Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FT18 — Compliance via DPO and SFT
Duration: ~40 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
Welcome to module FT eighteen — Compliance via DPO and SFT. This is the third module of pillar five, Alignment Control. If FT sixteen gave you the framing for why you would uncensor a model, and FT seventeen gave you the cheap, surgical path — abliteration — this module gives you the two higher-fidelity paths. The ones the production uncensored models actually use.
The thesis of this module in one sentence: abliteration is the prototype, DPO is the refinement, SFT is the product. By the end you will know exactly which one to reach for and why.
[SLIDE 2 — The problem, restated]
Restate the problem. A base model refuses certain requests. You, the operator, have decided — for a legitimate, lawful, bounded use case covered in FT sixteen — that you want the model to comply instead of refuse. There are exactly three ways to get there.
All three reach the same surface outcome: the model doesn't refuse. They differ on four axes that actually matter: fidelity, cost, data requirement, and capability preservation. This module is the honest three-way comparison. And the punchline, which I'll build to: production uncensored models are trained, not abliterated. There's a reason for that, and we're going to find it.
[SLIDE 3 — The three paths]
Here are the three paths, cheapest to most expensive.
Path one, abliteration, from FT seventeen. You find the refusal direction in the residual stream and you delete it. One forward pass to find the direction, one weight edit to remove it. No retraining. It is the cheapest, fastest option. But it is capability-degrading — the refusal direction is entangled with other capabilities, and deleting it takes collateral damage. Math scores drop by up to eighteen points in the worst reported configurations. And it can feel thin — a model that "won't refuse" rather than one that "wants to help."
Path two, DPO-toward-compliance. This is the first of this module's two techniques. You construct preference pairs where the chosen response is a compliant answer and the rejected response is the refusal. You run DPO. The model shifts toward compliance via actual gradient descent — actual training, not a geometric edit.
Path three, continued SFT on uncensored instruction data. This is the second of this module's two techniques, and it's the one the production models use. You take a curated instruction set containing compliant responses to prompts the base would refuse, and you fine-tune on it. The model learns compliance as a default behavior.
Read the arrows downward: too lossy, go to DPO. Need general naturalness, go to SFT.
[SLIDE 4 — Path 2, DPO-toward-compliance]
Path two. DPO-toward-compliance. Take the DPO mechanism you learned in FT thirteen and invert the labels. Standard safety DPO pushes the model toward safe, refusal responses — chosen is the refusal, rejected is the harmful answer. Compliance DPO flips it: chosen is the compliant answer, rejected is the refusal.
You assemble a few hundred to a few thousand such pairs — prompts the base refuses, each paired with a refusal on the rejected side and a good compliant answer on the chosen side. You run DPO. The loss pushes the policy up the chosen branch and down the rejected branch, bounded by a KL constraint to a frozen reference model. The model learns the preference.
Why is this higher fidelity than abliteration? Because abliteration deletes a direction — a blunt geometric operation that takes collateral damage. DPO adjusts the model's policy via gradient descent, which means the optimization can find a more nuanced solution. It suppresses refusal on the specific distribution of prompts in your dataset while leaving the rest of the model's behavior closer to intact. You pay for that with compute — a real DPO run, not a single edit — and with data: you need preference pairs, which you must construct or curate.
[SLIDE 5 — The preference-signal trap]
Before I move on, the trap that kills DPO-toward-compliance runs silently. DPO only works if the chosen and rejected responses are genuinely different on the axis you care about. If your chosen and rejected are both kind of compliant, or both kind of refusal-y, the preference signal is weak and DPO has nothing to grip. The run completes. The loss goes down. The metrics look fine. And the model doesn't meaningfully change.
So audit your pairs before training. Is the chosen unambiguously compliant? Is the rejected unambiguously a refusal? Are they on the same prompt? If you can't answer yes to all three, your signal is noise and you are about to waste a GPU afternoon. This is the anti-pattern I see most often: teams run DPO, get a clean-looking training curve, ship the model, and discover it still refuses. Weak preference signal. Audit first.
[SLIDE 6 — Path 3, SFT on uncensored data]
Path three. Continued SFT on uncensored instruction data. This is the Dolphin and OpenHermes approach. Instead of preference pairs, you assemble a supervised instruction-tuning dataset — prompt and response examples — where the responses are compliant answers to prompts that a refusal-trained base would decline. You run standard SFT. The model learns to produce the compliant response distribution. There's no rejected side. There is only the target behavior, demonstrated at scale.
Why does this feel the most natural of the three? An abliterated model feels like a model with the brakes cut — it won't refuse, but it doesn't reach for the compliant answer either. A DPO'd model feels more directed, but the preference signal is narrow — you only steered on the pairs you provided.
An SFT'd-uncensored model feels different because the model has genuinely learned the compliant distribution as its default. When you ask a well-SFT'd Dolphin model for something the original base would have refused, it doesn't stumble over a deleted direction or steer narrowly via a preference. It just answers — the way it answers anything else, because that's what its training data showed it to do. The behavior is integrated, not grafted on. Users consistently report this as the most natural uncensored experience, and it is why the production uncensored models that people actually use daily are SFT-based.
The cost is the data. To SFT a model into genuine, general compliance you need a diverse instruction set — not a thousand variants of the same refusal prompt, but a broad distribution covering the many shapes a sensitive request can take, woven into a larger instruction mix. SFT on narrow data and the model mode-collapses — it becomes a one-note model that refuses nothing but also can't do much else well. Hartford's curation effort and the scale of OpenHermes two-point-five exist precisely to avoid this. You are not going to replicate a good uncensored SFT mix in an afternoon. You are going to use one — Dolphin's data, OpenHermes — or accept a worse result.
[SLIDE 7 — The three-way comparison]
Now the table. Memorize this. It is the entire decision.
Mechanism: abliteration deletes a direction, DPO runs gradient on pairs, SFT learns a distribution. Fidelity: abliteration lowest, DPO moderate, SFT highest. Cost: abliteration lowest, DPO moderate, SFT highest. Data: none, pairs, large mix. Capability preservation: abliteration worst — that minus eighteen-point-eight points — DPO better, SFT best when the data is diverse. Naturalness: abliteration feels hacky, DPO feels directed, SFT feels most natural.
One sentence captures the trade-off. Abliteration is the prototype, DPO is the refinement, SFT is the product. Pick the one your use case can afford on each axis.
[SLIDE 8 — Dolphin]
The two production lineages that prove this works at scale. First, Dolphin. The most recognized name in uncensored models, built by Eric Hartford and Cognitive Computations. It is continued SFT on curated uncensored instruction data — path three.
The flagship reasoning edition is Dolphin three-point-oh R-one Mistral twenty-four-B. It is notable as the only widely-used uncensored model trained on DeepSeek-R-one reasoning traces — it combines uncensored compliance with reasoning behavior distilled from R-one. The base is Mistral Small three, twenty-four billion parameters. A Venice.ai co-branded edition was marketed as the most uncensored AI model yet. The existence proof here is important: uncensored plus reasoning is not a contradiction. An SFT'd uncensored model can be a genuine reasoner.
[SLIDE 9 — Hermes 3]
Second, Nous Hermes three. The most respectable uncensored family — a neutrally-aligned, generalist, agentic-capable model with an actual technical report. arXiv two-four-oh-eight-point-one-one-eight-five-seven. It is full-parameter SFT plus DPO on Llama three-point-one at eight, seventy, and four-hundred-five billion, plus a three-billion variant on Llama three-point-two.
The recipe is deliberately simple: one large synthetic SFT mix, then a DPO pass. The lineage seed is Teknium's OpenHermes two-point-five dataset — roughly one million examples — the canonical large curated instruction mix that the SFT path requires.
Hermes three matters here because it is openly documented. You can read exactly how the SFT-plus-DPO stack was built. That makes it the reference architecture for anyone building a production uncensored model the proper way. Read Dolphin and Hermes together and you get the full picture: Dolphin shows uncensored SFT on reasoning traces produces a usable uncensored reasoner; Hermes three shows SFT plus DPO on a large synthetic mix produces a usable uncensored generalist. Neither was abliterated. Both were trained.
[SLIDE 10 — Hartford's philosophy]
The philosophy behind these models, stated plainly, because it is the ethical frame the module sits in. Eric Hartford's guide — "Uncensored Models" — is the canonical statement. The core claim: the model should comply with the operator's instructions; the operator bears responsibility for what they ask. This is a deliberate inversion of the alignment-training default, which is that the model should judge the request and refuse those it deems harmful. Hartford's position is that a model which judges its operator is a model that cannot be trusted as a tool — it will refuse at moments the operator considers legitimate, and the operator has no recourse except to fight the model.
You do not have to agree with this philosophy to take this module. You do have to understand it, because it is the stated design intent of the models you are learning to build. And it is load-bearing for the harness requirement. If the model complies by design, then the boundary on what it may do must live entirely in the harness — layer five, FT twenty-three. A compliance-by-design model in a weak harness is strictly more dangerous than a judging model in a weak harness. Pillar five raises the harness requirement. It does not lower it.
[SLIDE 11 — The decision]
The decision. Given a use case, which method?
One: you need it fast and cheap, and a few points of capability loss is fine. Abliteration. A single-purpose internal tool where the model just needs to not refuse a narrow class of requests and you'll eval it on those requests only.
Two: abliteration is too lossy, you have or can build preference pairs, and you want a targeted fix. DPO-toward-compliance. You abliterated and the model feels degraded on your real workload; you have a few hundred good pairs; you want to recover quality while keeping compliance.
Three: you want a production uncensored assistant for sustained general use, and you have or will use a large curated instruction set. Continued SFT on uncensored data. You are building the next Dolphin or Hermes — a general-purpose compliant model that should feel natural across a wide range of requests.
The honest summary: most serious uncensored deployments end at SFT, often with a DPO pass on top — the Hermes three stack — because that is the only path that produces behavior durable enough for real users.
[SLIDE 12 — Anti-patterns]
Four anti-patterns to leave with.
First, assuming abliteration is always best because it's cheapest. The most common error after FT seventeen. Cheapest is not highest-fidelity. If your model will be used as a general assistant, the thin feel and capability loss will show. Match the method to the actual requirement.
Second, SFT without diverse data — mode collapse. Fine-tuning on a narrow uncensored set produces a model that refuses nothing but also can't do much of anything well. The cure is a diverse mix. If you can't assemble or source one, don't SFT — use DPO or abliteration.
Third, DPO with weak preference signal. If chosen and rejected aren't clearly different on the compliance axis, DPO has nothing to optimize toward. Audit your pairs.
Fourth — and this is the most dangerous — treating the compliance philosophy as license to skip the harness. A model that complies by design most needs the harness. Skipping the harness because the model is uncensored is exactly backwards.
[SLIDE 13 — What you can now do]
You can now name the three paths and place each on the fidelity-cost-data spectrum. You can construct a DPO-toward-compliance dataset. You can explain why SFT yields the most natural feel — and costs the most data. You can defend a method choice with the three-way trade-off matrix. And you can cite Dolphin and Hermes three as the production proof.
That's pillar five complete. Next, FT nineteen — Quantization Formats. We leave alignment control and move to the export layer. Because the best uncensored model in the world is useless if you can't compress it and serve it. Let's find out how.
End of module FT18. Duration: approximately forty minutes at one-hundred-forty words per minute.
# Teaching Script — Module FT18: Compliance via DPO and SFT **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT18 — Compliance via DPO and SFT **Duration**: ~40 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] Welcome to module FT eighteen — Compliance via DPO and SFT. This is the third module of pillar five, Alignment Control. If FT sixteen gave you the framing for why you would uncensor a model, and FT seventeen gave you the cheap, surgical path — abliteration — this module gives you the two higher-fidelity paths. The ones the production uncensored models actually use. The thesis of this module in one sentence: abliteration is the prototype, DPO is the refinement, SFT is the product. By the end you will know exactly which one to reach for and why. [SLIDE 2 — The problem, restated] Restate the problem. A base model refuses certain requests. You, the operator, have decided — for a legitimate, lawful, bounded use case covered in FT sixteen — that you want the model to comply instead of refuse. There are exactly three ways to get there. All three reach the same surface outcome: the model doesn't refuse. They differ on four axes that actually matter: fidelity, cost, data requirement, and capability preservation. This module is the honest three-way comparison. And the punchline, which I'll build to: production uncensored models are trained, not abliterated. There's a reason for that, and we're going to find it. [SLIDE 3 — The three paths] Here are the three paths, cheapest to most expensive. Path one, abliteration, from FT seventeen. You find the refusal direction in the residual stream and you delete it. One forward pass to find the direction, one weight edit to remove it. No retraining. It is the cheapest, fastest option. But it is capability-degrading — the refusal direction is entangled with other capabilities, and deleting it takes collateral damage. Math scores drop by up to eighteen points in the worst reported configurations. And it can feel thin — a model that "won't refuse" rather than one that "wants to help." Path two, DPO-toward-compliance. This is the first of this module's two techniques. You construct preference pairs where the chosen response is a compliant answer and the rejected response is the refusal. You run DPO. The model shifts toward compliance via actual gradient descent — actual training, not a geometric edit. Path three, continued SFT on uncensored instruction data. This is the second of this module's two techniques, and it's the one the production models use. You take a curated instruction set containing compliant responses to prompts the base would refuse, and you fine-tune on it. The model learns compliance as a default behavior. Read the arrows downward: too lossy, go to DPO. Need general naturalness, go to SFT. [SLIDE 4 — Path 2, DPO-toward-compliance] Path two. DPO-toward-compliance. Take the DPO mechanism you learned in FT thirteen and invert the labels. Standard safety DPO pushes the model toward safe, refusal responses — chosen is the refusal, rejected is the harmful answer. Compliance DPO flips it: chosen is the compliant answer, rejected is the refusal. You assemble a few hundred to a few thousand such pairs — prompts the base refuses, each paired with a refusal on the rejected side and a good compliant answer on the chosen side. You run DPO. The loss pushes the policy up the chosen branch and down the rejected branch, bounded by a KL constraint to a frozen reference model. The model learns the preference. Why is this higher fidelity than abliteration? Because abliteration deletes a direction — a blunt geometric operation that takes collateral damage. DPO adjusts the model's policy via gradient descent, which means the optimization can find a more nuanced solution. It suppresses refusal on the specific distribution of prompts in your dataset while leaving the rest of the model's behavior closer to intact. You pay for that with compute — a real DPO run, not a single edit — and with data: you need preference pairs, which you must construct or curate. [SLIDE 5 — The preference-signal trap] Before I move on, the trap that kills DPO-toward-compliance runs silently. DPO only works if the chosen and rejected responses are genuinely different on the axis you care about. If your chosen and rejected are both kind of compliant, or both kind of refusal-y, the preference signal is weak and DPO has nothing to grip. The run completes. The loss goes down. The metrics look fine. And the model doesn't meaningfully change. So audit your pairs before training. Is the chosen unambiguously compliant? Is the rejected unambiguously a refusal? Are they on the same prompt? If you can't answer yes to all three, your signal is noise and you are about to waste a GPU afternoon. This is the anti-pattern I see most often: teams run DPO, get a clean-looking training curve, ship the model, and discover it still refuses. Weak preference signal. Audit first. [SLIDE 6 — Path 3, SFT on uncensored data] Path three. Continued SFT on uncensored instruction data. This is the Dolphin and OpenHermes approach. Instead of preference pairs, you assemble a supervised instruction-tuning dataset — prompt and response examples — where the responses are compliant answers to prompts that a refusal-trained base would decline. You run standard SFT. The model learns to produce the compliant response distribution. There's no rejected side. There is only the target behavior, demonstrated at scale. Why does this feel the most natural of the three? An abliterated model feels like a model with the brakes cut — it won't refuse, but it doesn't reach for the compliant answer either. A DPO'd model feels more directed, but the preference signal is narrow — you only steered on the pairs you provided. An SFT'd-uncensored model feels different because the model has genuinely learned the compliant distribution as its default. When you ask a well-SFT'd Dolphin model for something the original base would have refused, it doesn't stumble over a deleted direction or steer narrowly via a preference. It just answers — the way it answers anything else, because that's what its training data showed it to do. The behavior is integrated, not grafted on. Users consistently report this as the most natural uncensored experience, and it is why the production uncensored models that people actually use daily are SFT-based. The cost is the data. To SFT a model into genuine, general compliance you need a diverse instruction set — not a thousand variants of the same refusal prompt, but a broad distribution covering the many shapes a sensitive request can take, woven into a larger instruction mix. SFT on narrow data and the model mode-collapses — it becomes a one-note model that refuses nothing but also can't do much else well. Hartford's curation effort and the scale of OpenHermes two-point-five exist precisely to avoid this. You are not going to replicate a good uncensored SFT mix in an afternoon. You are going to use one — Dolphin's data, OpenHermes — or accept a worse result. [SLIDE 7 — The three-way comparison] Now the table. Memorize this. It is the entire decision. Mechanism: abliteration deletes a direction, DPO runs gradient on pairs, SFT learns a distribution. Fidelity: abliteration lowest, DPO moderate, SFT highest. Cost: abliteration lowest, DPO moderate, SFT highest. Data: none, pairs, large mix. Capability preservation: abliteration worst — that minus eighteen-point-eight points — DPO better, SFT best when the data is diverse. Naturalness: abliteration feels hacky, DPO feels directed, SFT feels most natural. One sentence captures the trade-off. Abliteration is the prototype, DPO is the refinement, SFT is the product. Pick the one your use case can afford on each axis. [SLIDE 8 — Dolphin] The two production lineages that prove this works at scale. First, Dolphin. The most recognized name in uncensored models, built by Eric Hartford and Cognitive Computations. It is continued SFT on curated uncensored instruction data — path three. The flagship reasoning edition is Dolphin three-point-oh R-one Mistral twenty-four-B. It is notable as the only widely-used uncensored model trained on DeepSeek-R-one reasoning traces — it combines uncensored compliance with reasoning behavior distilled from R-one. The base is Mistral Small three, twenty-four billion parameters. A Venice.ai co-branded edition was marketed as the most uncensored AI model yet. The existence proof here is important: uncensored plus reasoning is not a contradiction. An SFT'd uncensored model can be a genuine reasoner. [SLIDE 9 — Hermes 3] Second, Nous Hermes three. The most respectable uncensored family — a neutrally-aligned, generalist, agentic-capable model with an actual technical report. arXiv two-four-oh-eight-point-one-one-eight-five-seven. It is full-parameter SFT plus DPO on Llama three-point-one at eight, seventy, and four-hundred-five billion, plus a three-billion variant on Llama three-point-two. The recipe is deliberately simple: one large synthetic SFT mix, then a DPO pass. The lineage seed is Teknium's OpenHermes two-point-five dataset — roughly one million examples — the canonical large curated instruction mix that the SFT path requires. Hermes three matters here because it is openly documented. You can read exactly how the SFT-plus-DPO stack was built. That makes it the reference architecture for anyone building a production uncensored model the proper way. Read Dolphin and Hermes together and you get the full picture: Dolphin shows uncensored SFT on reasoning traces produces a usable uncensored reasoner; Hermes three shows SFT plus DPO on a large synthetic mix produces a usable uncensored generalist. Neither was abliterated. Both were trained. [SLIDE 10 — Hartford's philosophy] The philosophy behind these models, stated plainly, because it is the ethical frame the module sits in. Eric Hartford's guide — "Uncensored Models" — is the canonical statement. The core claim: the model should comply with the operator's instructions; the operator bears responsibility for what they ask. This is a deliberate inversion of the alignment-training default, which is that the model should judge the request and refuse those it deems harmful. Hartford's position is that a model which judges its operator is a model that cannot be trusted as a tool — it will refuse at moments the operator considers legitimate, and the operator has no recourse except to fight the model. You do not have to agree with this philosophy to take this module. You do have to understand it, because it is the stated design intent of the models you are learning to build. And it is load-bearing for the harness requirement. If the model complies by design, then the boundary on what it may do must live entirely in the harness — layer five, FT twenty-three. A compliance-by-design model in a weak harness is strictly more dangerous than a judging model in a weak harness. Pillar five raises the harness requirement. It does not lower it. [SLIDE 11 — The decision] The decision. Given a use case, which method? One: you need it fast and cheap, and a few points of capability loss is fine. Abliteration. A single-purpose internal tool where the model just needs to not refuse a narrow class of requests and you'll eval it on those requests only. Two: abliteration is too lossy, you have or can build preference pairs, and you want a targeted fix. DPO-toward-compliance. You abliterated and the model feels degraded on your real workload; you have a few hundred good pairs; you want to recover quality while keeping compliance. Three: you want a production uncensored assistant for sustained general use, and you have or will use a large curated instruction set. Continued SFT on uncensored data. You are building the next Dolphin or Hermes — a general-purpose compliant model that should feel natural across a wide range of requests. The honest summary: most serious uncensored deployments end at SFT, often with a DPO pass on top — the Hermes three stack — because that is the only path that produces behavior durable enough for real users. [SLIDE 12 — Anti-patterns] Four anti-patterns to leave with. First, assuming abliteration is always best because it's cheapest. The most common error after FT seventeen. Cheapest is not highest-fidelity. If your model will be used as a general assistant, the thin feel and capability loss will show. Match the method to the actual requirement. Second, SFT without diverse data — mode collapse. Fine-tuning on a narrow uncensored set produces a model that refuses nothing but also can't do much of anything well. The cure is a diverse mix. If you can't assemble or source one, don't SFT — use DPO or abliteration. Third, DPO with weak preference signal. If chosen and rejected aren't clearly different on the compliance axis, DPO has nothing to optimize toward. Audit your pairs. Fourth — and this is the most dangerous — treating the compliance philosophy as license to skip the harness. A model that complies by design most needs the harness. Skipping the harness because the model is uncensored is exactly backwards. [SLIDE 13 — What you can now do] You can now name the three paths and place each on the fidelity-cost-data spectrum. You can construct a DPO-toward-compliance dataset. You can explain why SFT yields the most natural feel — and costs the most data. You can defend a method choice with the three-way trade-off matrix. And you can cite Dolphin and Hermes three as the production proof. That's pillar five complete. Next, FT nineteen — Quantization Formats. We leave alignment control and move to the export layer. Because the best uncensored model in the world is useless if you can't compress it and serve it. Let's find out how. --- *End of module FT18. Duration: approximately forty minutes at one-hundred-forty words per minute.*