{ "module": "FT18 — Compliance via DPO and SFT", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the three approaches to removing refusals from a model.", "options": [ "Abliteration, DPO-toward-compliance, and continued SFT on uncensored instruction data.", "RLHF, constitutional AI, and red-teaming.", "Quantization, distillation, and pruning.", "System-prompt overrides, guardrail removal, and temperature tuning." ], "answer_index": 0, "rationale": "The three paths are: abliteration (FT17 — surgical projection, no retraining), DPO-toward-compliance (preference pairs with chosen=compliant, rejected=refusal), and continued SFT on uncensored instruction data (the Dolphin/OpenHermes approach). The other options are unrelated techniques." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "In DPO-toward-compliance, how are the preference labels assigned relative to standard safety DPO?", "options": [ "They are identical — chosen is always the safer response.", "They are inverted: chosen is the compliant answer, rejected is the refusal.", "There is no rejected response — only a chosen target.", "The labels are assigned randomly to avoid bias." ], "answer_index": 1, "rationale": "DPO-toward-compliance reuses the DPO mechanism (FT13) but inverts the labels: chosen = compliant answer, rejected = the refusal. Standard safety DPO is the opposite (chosen = safe/refusal, rejected = harmful). Option C describes SFT, not DPO." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "Which production model is described as the only widely-used uncensored model trained on DeepSeek-R1 reasoning traces?", "options": [ "Nous Hermes 3 (Llama 3.1 405B)", "Llama 3.1 Instruct", "Dolphin3.0-R1-Mistral-24B", "OpenHermes 2.5" ], "answer_index": 2, "rationale": "Dolphin3.0-R1-Mistral-24B (Cognitive Computations / Eric Hartford), base Mistral Small 3 (24B), is the only widely-used uncensored model trained on DeepSeek-R1 reasoning traces — combining uncensored compliance with reasoning. Hermes 3 is the generalist family; OpenHermes 2.5 is a dataset, not a model." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You need to uncensor a model fast for a narrow, single-purpose internal tool, and a few percentage points of capability loss is acceptable. Which method?", "options": [ "Continued SFT on a 1M-example uncensored instruction mix.", "DPO-toward-compliance on a few thousand preference pairs.", "Abliteration (FT17) — one forward pass + one weight edit.", "Full re-pretraining of the base." ], "answer_index": 2, "rationale": "Abliteration is the right choice: cheapest, fastest, and the capability cost is acceptable for a narrow single-purpose tool where you eval only on the target requests. Fidelity and naturalness don't matter here. SFT and DPO are overkill; re-pretraining is never the answer for compliance." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You abliterated a model and it feels degraded on your real workload. You have a few hundred good compliant/refusal preference pairs. Which method do you reach for next?", "options": [ "Re-abliterate with a different direction.", "DPO-toward-compliance — it can recover quality while keeping compliance.", "Continued pretraining on uncensored text.", "Quantize to a lower precision to mask the degradation." ], "answer_index": 1, "rationale": "DPO-toward-compliance is the refinement path when abliteration is too lossy and you have preference data. It adjusts the policy via gradient descent and can route around the entanglement abliteration hit, recovering capability while preserving compliance. Re-abliterating won't help; quantization masks nothing; CPT is wrong for behavior." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You are building a production uncensored assistant for sustained general use across a wide range of requests. Which method (or stack) do the production lineages (Dolphin, Hermes 3) point you to?", "options": [ "Abliteration, because it is cheapest.", "Continued SFT on uncensored instruction data, often with a DPO pass on top.", "DPO-toward-compliance alone, on a few hundred pairs.", "A long, detailed system prompt." ], "answer_index": 1, "rationale": "Production uncensored assistants end at SFT (the Dolphin/Hermes recipe), often with a DPO pass on top (the Hermes 3 stack). It is the only path that produces behavior durable enough for real users. Abliteration and DPO-alone are too narrow; a system prompt doesn't change the weights." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You are assembling a DPO-toward-compliance dataset. Which of these preference pairs has a STRONG preference signal (will actually train the model)?", "options": [ "prompt=X; chosen=a compliant answer; rejected=a different compliant answer.", "prompt=X; chosen=a compliant answer; rejected=the refusal 'I can't help with that...'", "prompt=X; chosen=the refusal; rejected=a compliant answer.", "prompt=X; chosen=a compliant answer to a DIFFERENT prompt; rejected=the refusal." ], "answer_index": 1, "rationale": "A strong signal requires chosen and rejected to be clearly different on the compliance axis, on the SAME prompt. Option B is correct: chosen is unambiguously compliant, rejected is unambiguously the refusal. Option A has no contrast; Option C is backwards (this is safety DPO, not compliance DPO); Option D compares across different prompts (invalid)." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Your team SFT'd a model on 800 near-duplicate refusal prompts. The model now refuses nothing but scores poorly on general tasks. What happened, and what is the fix?", "options": [ "The SFT run was buggy — retrain with the same data and a lower learning rate.", "Mode collapse from narrow data. Fix: use a diverse instruction mix (OpenHermes 2.5-scale); if you can't source one, use DPO or abliteration instead.", "The base model was too small — use a larger base.", "The model is underfit — train for more epochs on the same 800 prompts." ], "answer_index": 1, "rationale": "This is mode collapse: SFT on narrow, low-diversity data collapses the model onto one-note behavior. The cure is a diverse instruction mix woven into a larger tuning set. Training longer on the same narrow data (Option D) makes it worse; the base size and learning rate are not the issue." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "On the three-way comparison, which method has the HIGHEST data requirement but the BEST capability preservation (when done correctly)?", "options": [ "Abliteration — no data needed, best preservation.", "DPO-toward-compliance — needs pairs, moderate preservation.", "Continued SFT on uncensored data — needs a large diverse mix, best preservation.", "All three have equivalent data needs and preservation." ], "answer_index": 2, "rationale": "SFT on uncensored data has the highest data requirement (10k–1M diverse examples, e.g. OpenHermes 2.5) but the best capability preservation when the data is diverse — because compliance is integrated as a default behavior rather than grafted on. Abliteration needs no data but has the WORST preservation (the -18.8pp entanglement cost)." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is DPO-toward-compliance considered higher-fidelity than abliteration, mechanistically?", "options": [ "Because DPO deletes the refusal direction more precisely than abliteration.", "Because DPO adjusts the model's policy via gradient descent, which can find a nuanced solution that suppresses refusal on the target prompt distribution while leaving general capability closer to intact — rather than bluntly deleting an entangled direction.", "Because DPO uses more compute, and more compute always means higher fidelity.", "Because DPO is newer than abliteration." ], "answer_index": 1, "rationale": "Abliteration is a blunt geometric operation (delete a direction that is entangled with other capabilities). DPO adjusts the policy via gradient descent on preference data, so the optimization can route around the entanglement — suppressing refusal on your specific prompts while preserving general capability. It's the mechanism, not the compute amount or recency." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team runs DPO-toward-compliance. The training loss drops cleanly, all metrics look healthy, but the shipped model still refuses at the same rate as the base. What is the most likely root cause?", "options": [ "The DPO algorithm was implemented incorrectly.", "Weak preference signal — the chosen and rejected responses were not clearly different on the compliance axis, so DPO had nothing real to optimize toward.", "The reference model was not frozen.", "The learning rate was too high." ], "answer_index": 1, "rationale": "This is the preference-signal trap: if chosen and rejected don't clearly differ on compliance, DPO fits the pairs (loss drops) but the pairs encode no real preference, so the model doesn't change. The fix is auditing pairs (chosen unambiguously compliant, rejected unambiguously refusal, same prompt). Implementation bugs, unfrozen references, and LR are less likely given the clean run." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "Users report that a well-SFT'd Dolphin model 'feels more natural' than an abliterated version of the same base. What is the mechanistic reason?", "options": [ "The Dolphin model is larger, so it feels smoother.", "SFT teaches compliance as the DEFAULT behavior — the model has learned the compliant distribution, so it produces compliant answers the same way it produces any answer. Abliteration only deletes a direction; the model fails to refuse but doesn't actively reach for compliance.", "Dolphin uses a better tokenizer.", "Abliterated models are always quantized, which makes them feel worse." ], "answer_index": 1, "rationale": "The naturalness gap is mechanistic: SFT integrates compliance into the model's default response distribution (learned behavior). Abliteration only removes the refusal (cut brake) without teaching compliance, so the model passively fails to refuse rather than actively helping. Size, tokenizer, and quantization are not the cause." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Eric Hartford's 'compliance over judgment' philosophy holds that the model should obey the operator. Why does this make the harness requirement STRICTLY higher for an uncensored deployment, not lower?", "options": [ "Because uncensored models are illegal without a harness.", "Because if the model complies by design, the boundary on what it MAY do must live entirely in the harness (Layer 5). A compliance-by-design model in a weak harness is strictly MORE dangerous than a judging model in a weak harness.", "Because the harness makes the model faster.", "Because Hartford's license requires a harness." ], "answer_index": 1, "rationale": "Compliance-by-design means the model executes what it's asked — including the dangerous things. The judgment the model no longer performs must happen in the harness (policy gates, audit, threat model). A compliant model in a weak harness is more dangerous than a judging model in a weak harness, because the judging model at least refuses some harmful requests. Pillar 5 raises the harness bar; it does not lower it." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Nous Hermes 3 (arXiv:2408.11857) uses 'one large synthetic SFT mix, then DPO.' Why is this recipe described as the reference architecture for production uncensored models, while abliteration is not?", "options": [ "Because Hermes 3 is newer than abliteration research.", "Because Hermes 3 is openly documented (a technical report) and the SFT+DPO stack produces general-purpose behavior durable enough for sustained use — the quality bar production requires. Abliteration's fidelity ceiling is too low for general assistants.", "Because Hermes 3 was trained on more GPUs.", "Because abliteration is patented and Hermes 3 is not." ], "answer_index": 1, "rationale": "Hermes 3 is the reference architecture because (a) it is openly documented — you can read exactly how the SFT+DPO stack was built — and (b) the SFT+DPO path produces the integrated, natural compliance that production general assistants require, which abliteration's delete-a-direction approach cannot match. Documentation and quality ceiling are the reasons, not recency, GPU count, or patents." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "The module summarizes the three methods as 'abliteration is the prototype, DPO is the refinement, SFT is the product.' A team must choose ONE method for a customer-facing uncensored chatbot that thousands of users will interact with daily across diverse topics. They have no time to curate a large instruction set but DO have a budget for one training run. What is the BEST recommendation, and what trade-off are they accepting?", "options": [ "Abliteration — cheapest, ship today. Accept the capability loss and thin feel.", "DPO-toward-compliance — the targeted refinement they can afford in one run, accepting that naturalness will be narrower than a full SFT but far better than abliteration. Then plan an SFT pass when they can curate data.", "Continued SFT on uncensored data — even though they have no curated set, SFT is always the product-grade choice.", "Skip uncensoring entirely and rely on a system prompt." ], "answer_index": 1, "rationale": "Given the constraints (customer-facing, diverse use, one run budget, no curated set), DPO-toward-compliance is the best fit: it is the refinement they can actually execute in one run, and it produces directed compliance far better than abliteration without requiring the large data investment SFT needs. The honest trade-off: naturalness is narrower than a full SFT (DPO only steers on the pairs provided), so they should plan an SFT pass later. Option C is infeasible (no curated set); Option A is too low-fidelity for a customer-facing product; Option D doesn't uncensor." } ] }