Compliance via DPO and SFT
Module FT18 · Course 3 — LLM Fine-Tuning Masterclass
60 minutes · 7 sub-sections: Three Paths · DPO · SFT · Comparison · Lineages · Philosophy · The Decision
The higher-fidelity alternatives to abliteration. The model doesn't just fail to refuse — it learns to comply.
Pillar 5 — Alignment Control · Third module
The problem, restated
A base model refuses. You have decided (FT16) that for a lawful, bounded use case, you want it to comply instead.
There are exactly three ways to get there. FT17 gave you the cheap one. This module gives you the other two.
All three reach "the model doesn't refuse." They differ on fidelity, cost, data, capability preservation.
This module is the honest three-way comparison — and the reason production uncensored models are trained, not abliterated.
The three paths to compliance
Cheapest to most expensive. Prototype -> refinement -> product.
ABLITERATION (FT17) · delete refusal direction · no retraining · cheapest · capability-degrading
↓ too lossy?
DPO-TOWARD-COMPLIANCE (this module) · preference pairs: chosen=compliant, rejected=refusal · targeted fix
↓ need general naturalness?
SFT ON UNCENSORED DATA (this module) · learn compliance as default · Dolphin / OpenHermes recipe
Path 2 — DPO-toward-compliance
Take the DPO mechanism (FT13) and invert the labels.
| Field | Standard safety DPO | DPO-toward-compliance |
|---|
| chosen | safe / refusal | compliant answer |
| rejected | unsafe / harmful | the refusal |
Assemble a few hundred to a few thousand such pairs. Run DPO. The model shifts its policy via gradient descent — not a deleted direction.
Why higher fidelity than abliteration: the optimization can route around the entanglement FT17 hits. It suppresses refusal on your prompt distribution while leaving general capability closer to intact.
The preference-signal trap
DPO only works if chosen and rejected are clearly different on the compliance axis. If both are kind-of-compliant or both are kind-of-refusal, the signal is noise. The run completes, the loss drops, the model doesn't change.
Audit your pairs before training:
- Is chosen unambiguously compliant?
- Is rejected unambiguously a refusal?
- Are they on the same prompt?
Weak preference signal is the silent killer of DPO-toward-compliance runs.
Path 3 — Continued SFT on uncensored data
Assemble a supervised instruction set where responses are compliant answers to prompts the base would refuse. Run SFT. The model learns compliance as the default behavior.
Why it feels the most natural: the model has genuinely learned the compliant distribution. It doesn't stumble over a deleted direction or steer narrowly via a preference — it just answers, the way it answers anything else.
The cost: data + mode collapse. Narrow data collapses the model into a one-note refusal-dropper. The cure is a diverse mix (OpenHermes 2.5 ~1M ex.; Dolphin curation). You won't build a good one in an afternoon.
The three-way comparison
One table. Memorize it.
| Axis | Abliteration | DPO | SFT |
|---|
| Mechanism | Delete direction | Gradient on pairs | Learn distribution |
| Fidelity | Lowest | Moderate | Highest |
| Cost | Lowest | Moderate | Highest |
| Data | None | Pairs | Large mix |
| Capability kept | Worst (-18.8pp) | Better | Best |
| Naturalness | "hacky" | Directed | Most natural |
Abliteration is the prototype, DPO is the refinement, SFT is the product.
Production lineage — Dolphin
Cognitive Computations / Hartford
The most-recognized uncensored family. Built on continued SFT on curated uncensored data.
HF org: cognitivecomputations
Flagship: Dolphin3.0-R1-Mistral-24B
- Base: Mistral Small 3 (24B)
- Trained on DeepSeek-R1 reasoning traces
- The only uncensored model that also reasons
- Venice.ai edition: "most uncensored AI yet"
Existence proof: uncensored + reasoning is not a contradiction. An SFT'd uncensored model can be a genuine reasoner.
Production lineage — Nous Hermes 3
The most "respectable" uncensored family — neutral generalist, agentic-capable, openly documented (arXiv:2408.11857).
| Property | Detail |
|---|
| Base | Llama 3.1 — 8B / 70B / 405B (+ 3B on 3.2) |
| Recipe | Full-param SFT + DPO — deliberately simple |
| SFT seed | Teknium's OpenHermes 2.5 (~1M synthetic examples) |
| Why it matters | The reference architecture: read exactly how SFT+DPO was built |
Neither Dolphin nor Hermes was abliterated. Both were trained.
Hartford's philosophy
Compliance over judgment. The model obeys the operator; the operator (and harness) bears responsibility.
A deliberate inversion of the alignment-training default (the model judges and refuses). Hartford's position: a model that judges its operator can't be trusted as a tool.
Load-bearing for the harness requirement. If the model complies by design, the boundary on what it may do lives entirely in the harness (Layer 5). A compliance-by-design model in a weak harness is strictly more dangerous than a judging model in a weak harness. Pillar 5 raises the harness bar; it does not lower it.
The decision
Given a use case's constraints, which method?
| If your constraint is... | Method |
|---|
| Fast, cheap, a few pp loss OK | Abliteration (FT17) |
| Abliteration too lossy, have preference pairs | DPO-toward-compliance |
| Production assistant for sustained general use | SFT on uncensored data (+ DPO) |
Most serious uncensored deployments end at SFT, often with a DPO pass on top (the Hermes 3 stack). That is the only path durable enough for real users.
Anti-patterns
Abliteration-by-default. Cheapest is not highest-fidelity. If the model is a general assistant, the "thin" feel and capability loss will show. Match the method to the actual requirement.
SFT without diverse data (mode collapse). Narrow uncensored set -> one-note model. The cure is a diverse mix (OpenHermes 2.5, Dolphin curation). Can't source one? Use DPO or abliteration.
DPO with weak preference signal. Chosen and rejected not clearly different -> no learning. Audit pairs: chosen unambiguously compliant, rejected unambiguously refusal, same prompt.
Treating "compliance" as license to skip the harness. A compliant-by-design model most needs the harness. (FT16, FT23.)
What you can now do
- Name the three paths and place each on the fidelity/cost/data spectrum.
- Construct a DPO-toward-compliance dataset (chosen=compliant, rejected=refusal).
- Explain why SFT yields the most natural feel — and costs the most data.
- Defend a method choice with the three-way trade-off matrix.
- Cite Dolphin and Hermes 3 as the production proof, and state Hartford's framing.
Next: FT19 — Quantization Formats