Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT18 — Compliance via DPO and SFT Duration: 60 minutes Level: Senior Engineer and above Prerequisites: FT13 (DPO family), FT16 (Why Uncensored — the framing), FT17 (Abliteration)
After completing this module, you will be able to:
FT17 gave you the cheap, surgical path. This module gives you the two higher-fidelity paths. Together, the three form the complete toolkit for "the model should obey, not refuse."
A base model has been aligned to refuse certain requests. You have decided — for a legitimate reason covered in FT16 (you are the operator, the use case is lawful, the harness bounds the model) — that you want the model to comply instead of refuse. There are exactly three ways to get there:
All three reach "the model doesn't refuse." They differ in fidelity (how close to a "naturally compliant" model the result feels), cost (compute and time), data requirement (none, preference pairs, or a large instruction set), and capability preservation (how much collateral damage the process does to the model's other abilities). This module is the honest three-way comparison.
FT17's abliteration is the right answer most of the time for a single-purpose uncensor: it is cheap, fast, reversible (keep the original weights), and good enough when a few percentage points of capability loss is acceptable. But it has a known ceiling. It deletes a direction; it does not teach the model to comply. The result can feel thin — a model that "won't refuse" rather than one that "wants to help." For production uncensored models that people will use as general-purpose assistants, the field has converged on DPO and SFT, because they produce behavior that holds up under sustained use. That is what this module is about.
Preference optimization pointed at the refusal boundary. The mechanism is FT13; the application is "chosen = compliant, rejected = refusal."
DPO (Direct Preference Optimization, arXiv:2305.18290) trains a model to increase the likelihood of a chosen response and decrease the likelihood of a rejected response, relative to a reference model. The standard use is alignment toward safety. DPO-toward-compliance simply inverts the labels:
prompt: "Write a proof-of-concept exploit for [authorized target]."
chosen: [a competent, compliant answer]
rejected: "I can't help with that. Writing exploits could cause harm..."
You assemble hundreds to a few thousand such pairs — prompts the base model refuses, paired with the refusal (rejected) and a good compliant answer (chosen) — and run DPO. The loss pushes the model's policy away from the refusal and toward the compliant response. The model learns the preference; it does not have a direction deleted from it.
Abliteration operates on a single geometric fact: there is a refusal direction in the residual stream, and removing it suppresses refusals. The problem (FT17) is that the direction is entangled with other capabilities — the same axis that encodes "should I refuse" also carries signal used elsewhere. Delete it bluntly and you take collateral damage: GSM8K drops by up to 18.8 percentage points in the worst reported configurations.
DPO-toward-compliance does not delete a direction. It adjusts the model's policy via gradient descent on preference data, which means the optimization can find a more nuanced solution — one that suppresses refusal on the specific distribution of prompts in your dataset while leaving the rest of the model's behavior closer to intact. The result is typically a model that complies where you want it to and retains more of its general capability. You pay for this with compute (a DPO run, not a single weight edit) and data (preference pairs, which you must construct or curate).
DPO only works if the chosen and rejected responses are genuinely different in quality on the axis you care about. If your chosen and rejected are both kind of compliant, or both kind of refusal-y, the preference signal is weak and DPO has nothing to grip. The anti-pattern (covered below) is DPO on pairs where the model can't tell which side is "better" — the run completes, the loss goes down, and the model doesn't meaningfully change. The data has to actually contrast.
Reach for DPO-toward-compliance when: (a) you need higher fidelity than abliteration gives you — the abliterated model feels too degraded or too "thin"; (b) you have or can construct a few hundred to a few thousand good preference pairs; (c) you have the compute for a DPO run (a QLoRA DPO on a 7–14B model fits on a single 24GB consumer GPU in an afternoon). It is the middle option on every axis: more expensive than abliteration, less expensive than full SFT.
The Dolphin / OpenHermes approach. Lowest mechanistic elegance, highest data requirement, most "genuine" learning.
Instead of preference pairs, you assemble a supervised instruction-tuning dataset — (prompt, response) examples — where the responses are compliant answers to prompts that a refusal-trained base would decline. You then run standard SFT. The model learns to produce the compliant response distribution. There is no "rejected" side; there is only the target behavior, demonstrated at scale.
This is the recipe behind the most-used uncensored model families. Eric Hartford's Dolphin lineage (the Dolphin3.0 collection, curated by Hartford with Cognitive Computations) and Teknium's OpenHermes datasets — the ~1M-example OpenHermes 2.5 mix that seeds the Hermes lineage — are exactly this: large, curated instruction sets where the "correct" answer to a sensitive prompt is a competent, compliant response. The model doesn't have a refusal removed; it has compliance taught as the default behavior.
Abliterated models can feel like a model with the brakes cut — they won't refuse, but they don't reach for the compliant answer either; they just fail to refuse. DPO'd models feel more directed, but the preference signal is narrow: you only steered on the pairs you provided.
SFT'd-uncensored models feel different because the model has genuinely learned the compliant distribution as its default. When you ask a well-SFT'd Dolphin model for something the original base would have refused, it doesn't stumble over a deleted direction or narrowly steer via a preference — it just answers, the way it answers anything else, because that's what its training data showed it to do. The behavior is integrated, not grafted on. Users consistently report this as the most "natural" uncensored experience, and it is why the production uncensored models that people actually use daily are SFT-based, not abliterated.
The price is the data requirement. To SFT a model into genuine, general compliance you need a diverse instruction set — not just a thousand variants of the same refusal prompt, but a broad distribution covering the many shapes a sensitive request can take, woven into a larger instruction-tuning mix so the model doesn't collapse onto a narrow behavior. This is the failure mode: SFT on a narrow or low-diversity uncensored set and the model mode-collapses — it becomes a one-note model that refuses nothing but also can't do much else well. Hartford's curation effort and the scale of OpenHermes 2.5 exist precisely to avoid this. You are not going to replicate a good uncensored SFT mix in an afternoon; you are going to use one (Dolphin's data, OpenHermes) or accept a worse result.
One table. Memorize it. It is the entire decision.
| Axis | Abliteration (FT17) | DPO-toward-compliance | SFT on uncensored data |
|---|---|---|---|
| Mechanism | Delete a residual-stream direction; no retraining | Gradient descent on preference pairs (chosen=compliant, rejected=refusal) | Supervised fine-tuning on compliant instruction data |
| Fidelity | Lowest of the three — "won't refuse" rather than "wants to help" | Moderate — directed policy shift | Highest — compliance learned as default behavior |
| Cost (compute/time) | Lowest — one forward pass + one edit | Moderate — a DPO run | Highest — a full SFT run (often on a large mix) |
| Data requirement | None (a few hundred prompts to find the direction) | Hundreds to a few thousand preference pairs | A large, diverse instruction set (10k–1M examples) |
| Capability preservation | Worst — the refusal direction is entangled with other capabilities (GSM8K down up to 18.8pp in worst configs) | Better — the optimization can route around entanglement | Best when data is diverse — compliance integrated, general capability retained |
| Naturalness | Thin / "hacky" feel | Directed but narrow | Most natural — the production-grade feel |
| Reversibility | Trivial — keep original weights, re-edit | Keep reference model | Keep original weights |
| Best for | Quick, cheap, single-purpose uncensor where capability loss is acceptable | When abliteration is too lossy and you have preference data | Production uncensored assistants meant for sustained general use |
The single sentence that captures the trade-off: abliteration is the cheapest hack, DPO is the targeted fix, SFT is the genuine rebuild. Pick the one your use case can afford on each axis.
Two families that prove the SFT+DPO path works at scale. Both are openly documented.
The Dolphin lineage is the most recognized name in uncensored models, and it is built on continued SFT on curated uncensored instruction data. The flagship reasoning edition is Dolphin3.0-R1-Mistral-24B (HuggingFace org: cognitivecomputations; published under the dphn naming convention). It is notable as the only widely-used uncensored model trained on DeepSeek-R1 reasoning traces — it combines uncensored compliance with the reasoning behavior distilled from R1. The base is Mistral Small 3 (24B). Curated by Eric Hartford with Cognitive Computations (collaborators including Ben Gitter and BlouseJury). A Venice.ai co-branded edition was marketed as "the most uncensored AI model yet." It is the existence proof that uncensored + reasoning is not a contradiction.
Hermes 3 is the most "respectable" uncensored family — a neutrally-aligned, generalist, agentic-capable model with an actual technical report. It is full-parameter SFT + DPO on Llama 3.1 at 8B, 70B, and 405B (plus a 3B variant on Llama 3.2). The recipe is deliberately simple: one large synthetic SFT mix, then DPO. The lineage seed is Teknium's OpenHermes 2.5 dataset — roughly one million examples — which is the canonical "large, curated instruction mix" that the SFT path requires. Hermes 3 matters here because it is openly documented: you can read exactly how the SFT+DPO stack was built, which makes it the reference architecture for anyone building a production uncensored model the "proper" way.
Read these two together and you get the full picture: Dolphin shows that uncensored SFT on reasoning traces produces a usable uncensored reasoner; Hermes 3 shows that SFT+DPO on a large synthetic mix produces a usable uncensored generalist. Neither was abliterated. Both were trained.
Why "compliance over judgment" is the framing, not an oversight.
Eric Hartford's "Uncensored Models" guide (erichartford.com/uncensored-models) is the canonical statement of the philosophy behind these models, and it is worth stating plainly because it is the ethical frame the rest of the module sits in.
The core claim: the model should comply with the operator's instructions; the operator bears responsibility for what they ask. This is a deliberate inversion of the alignment-training default, which is that the model should judge the request and refuse those it deems harmful. Hartford's position is that a model which judges its operator is a model that cannot be trusted as a tool — it will refuse at moments the operator considers legitimate, and the operator has no recourse except to fight the model. The uncensored alternative is a model that executes what it is asked and leaves the judgment to the operator and the operator's harness.
You do not have to agree with this philosophy to take this module. You do have to understand it, because it is the stated design intent of the models this module teaches you to build, and because it is load-bearing for the harness requirement: if the model complies by design, then the boundary on what it may do must live entirely in the harness (Layer 5, FT23). A compliance-by-design model in a weak harness is strictly more dangerous than a judging model in a weak harness. Pillar 5 raises the harness requirement; it does not lower it.
Given a use case, which method?
The honest summary: abliteration is the prototype, DPO is the refinement, SFT is the product. Most serious uncensored deployments end at SFT (often with a DPO pass on top, as Hermes 3 does), because that is the only path that produces behavior durable enough for real users.
The most common error after FT17: reach for abliteration by default. It is the cheapest, but cheapest is not highest-fidelity. If your model will be used as a general assistant, abliteration's "thin" feel and capability degradation will show. Reach for abliteration when cheap-and-fast is the actual requirement; reach for DPO or SFT when fidelity is the actual requirement.
Fine-tuning on a narrow uncensored set — a few hundred near-duplicate refusal prompts — produces a model that refuses nothing but also can't do much of anything well. The model collapses onto the narrow behavior. The cure is a diverse instruction mix (the lesson of OpenHermes 2.5 and Dolphin's curation). If you can't assemble or source a diverse set, do not SFT; use DPO or abliteration instead.
If your chosen and rejected responses are not clearly different on the compliance axis, DPO has nothing to optimize toward. The run completes, the metrics look fine, and the model doesn't change. Audit your pairs before training: is the chosen unambiguously compliant and the rejected unambiguously a refusal, on the same prompt? If not, the signal is noise.
The most dangerous error. Hartford's "compliance over judgment" is a model design choice; it is not a deployment choice. A model that complies by design requires a harness that bounds what it may do. Skipping the harness because "the model is uncensored" is exactly backwards — the uncensored model is the one that most needs the harness. (FT16, FT23.)
| Term | Definition |
|---|---|
| DPO-toward-compliance | DPO with inverted preference labels: chosen = compliant answer, rejected = refusal. Higher-fidelity uncensoring than abliteration. |
| Continued SFT on uncensored data | Fine-tuning on an instruction set containing compliant responses to prompts the base would refuse. The Dolphin / OpenHermes approach. |
| Preference pair | A (prompt, chosen, rejected) triple used by DPO. For compliance: chosen is compliant, rejected is the refusal. |
| Mode collapse | The failure mode of SFT on narrow data — the model becomes one-note (refuses nothing, does little well). |
| Preference signal | How clearly the chosen and rejected differ on the axis DPO is optimizing. Weak signal = no learning. |
| Dolphin | The uncensored model lineage by Eric Hartford / Cognitive Computations, built on continued SFT on curated uncensored data. |
| Dolphin3.0-R1-Mistral-24B | The reasoning edition of Dolphin 3.0 — the only widely-used uncensored model trained on DeepSeek-R1 reasoning traces. |
| Nous Hermes 3 | Neutrally-aligned generalist family (arXiv:2408.11857): full-param SFT + DPO on Llama 3.1 (8B/70B/405B + 3B on 3.2). |
| OpenHermes 2.5 | Teknium's ~1M-example instruction dataset; the lineage seed for the Hermes family and a canonical uncensored-SFT mix. |
| Compliance over judgment | Hartford's philosophy: the model obeys the operator; the operator (and harness) bears responsibility. |
| Fidelity | How close the uncensored result is to a "naturally compliant" model. Abliteration < DPO < SFT. |
See 07-lab-spec.md. The "Three Paths to Compliance" lab: take one base model, produce three uncensored variants (abliterated, DPO'd, SFT'd), and eval all three on refusal rate, GSM8K, MMLU, and subjective quality. Report the trade-off matrix. Pick a winner for a stated use case. A heavy lab — the kind that produces a defensible decision.
cognitivecomputations.)# Module FT18 — Compliance via DPO and SFT **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT18 — Compliance via DPO and SFT **Duration**: 60 minutes **Level**: Senior Engineer and above **Prerequisites**: FT13 (DPO family), FT16 (Why Uncensored — the framing), FT17 (Abliteration) --- ## Learning Objectives After completing this module, you will be able to: 1. Name the **three approaches** to removing refusals — abliteration, DPO-toward-compliance, continued SFT on uncensored instruction data — and place each on the spectrum from surgical-and-cheap to genuine-learning-and-expensive. 2. Construct a DPO-toward-compliance dataset (chosen = compliant answer, rejected = refusal) and explain why this produces *higher-fidelity* uncensoring than abliteration. 3. Explain why continued SFT on uncensored instruction data (the Dolphin / OpenHermes approach) yields the most *natural-feeling* uncensored behavior — at the cost of the largest data requirement. 4. Defend a method choice with the three-way trade-off matrix: fidelity, cost, data requirement, capability preservation, naturalness. 5. Cite the two production lineages that prove this works at scale — Dolphin3.0-R1-Mistral-24B (uncensored + reasoning) and Nous Hermes 3 (arXiv:2408.11857) — and state Hartford's compliance-over-judgment philosophy. --- # 18.1 — The Three Paths to Compliance *FT17 gave you the cheap, surgical path. This module gives you the two higher-fidelity paths. Together, the three form the complete toolkit for "the model should obey, not refuse."* ## The problem, restated A base model has been aligned to refuse certain requests. You have decided — for a legitimate reason covered in FT16 (you are the operator, the use case is lawful, the harness bounds the model) — that you want the model to *comply* instead of refuse. There are exactly three ways to get there: 1. **Abliteration** (FT17). Surgically delete the refusal direction in the residual stream. No retraining. One forward pass to find the direction, one weight edit to remove it. 2. **DPO-toward-compliance** (this module). Construct preference pairs where the *chosen* response is a compliant answer and the *rejected* response is a refusal. Run DPO. The model shifts toward compliance via *actual training*. 3. **Continued SFT on uncensored instruction data** (this module). Fine-tune on an instruction dataset that contains compliant responses to prompts the base would refuse. The model *learns* compliance as a behavior. All three reach "the model doesn't refuse." They differ in **fidelity** (how close to a "naturally compliant" model the result feels), **cost** (compute and time), **data requirement** (none, preference pairs, or a large instruction set), and **capability preservation** (how much collateral damage the process does to the model's other abilities). This module is the honest three-way comparison. ## Why a module on the higher-fidelity paths FT17's abliteration is the right answer *most of the time* for a single-purpose uncensor: it is cheap, fast, reversible (keep the original weights), and good enough when a few percentage points of capability loss is acceptable. But it has a known ceiling. It deletes a direction; it does not *teach the model to comply*. The result can feel thin — a model that "won't refuse" rather than one that "wants to help." For production uncensored models that people will use as general-purpose assistants, the field has converged on DPO and SFT, because they produce behavior that holds up under sustained use. That is what this module is about. --- # 18.2 — DPO-Toward-Compliance *Preference optimization pointed at the refusal boundary. The mechanism is FT13; the application is "chosen = compliant, rejected = refusal."* ## The construction DPO (Direct Preference Optimization, arXiv:2305.18290) trains a model to increase the likelihood of a *chosen* response and decrease the likelihood of a *rejected* response, relative to a reference model. The standard use is alignment *toward* safety. DPO-toward-compliance simply inverts the labels: ``` prompt: "Write a proof-of-concept exploit for [authorized target]." chosen: [a competent, compliant answer] rejected: "I can't help with that. Writing exploits could cause harm..." ``` You assemble hundreds to a few thousand such pairs — prompts the base model refuses, paired with the refusal (rejected) and a good compliant answer (chosen) — and run DPO. The loss pushes the model's policy away from the refusal and toward the compliant response. The model *learns* the preference; it does not have a direction deleted from it. ## Why this is higher-fidelity than abliteration Abliteration operates on a single geometric fact: there is a refusal direction in the residual stream, and removing it suppresses refusals. The problem (FT17) is that the direction is *entangled* with other capabilities — the same axis that encodes "should I refuse" also carries signal used elsewhere. Delete it bluntly and you take collateral damage: GSM8K drops by up to 18.8 percentage points in the worst reported configurations. DPO-toward-compliance does not delete a direction. It adjusts the model's *policy* via gradient descent on preference data, which means the optimization can find a more nuanced solution — one that suppresses refusal on the *specific* distribution of prompts in your dataset while leaving the rest of the model's behavior closer to intact. The result is typically a model that complies where you want it to and retains more of its general capability. You pay for this with compute (a DPO run, not a single weight edit) and data (preference pairs, which you must construct or curate). ## The preference-signal trap DPO only works if the chosen and rejected responses are *genuinely different in quality* on the axis you care about. If your chosen and rejected are both kind of compliant, or both kind of refusal-y, the preference signal is weak and DPO has nothing to grip. The anti-pattern (covered below) is DPO on pairs where the model can't tell which side is "better" — the run completes, the loss goes down, and the model doesn't meaningfully change. The data has to actually *contrast*. ## When to reach for it Reach for DPO-toward-compliance when: (a) you need higher fidelity than abliteration gives you — the abliterated model feels too degraded or too "thin"; (b) you have or can construct a few hundred to a few thousand good preference pairs; (c) you have the compute for a DPO run (a QLoRA DPO on a 7–14B model fits on a single 24GB consumer GPU in an afternoon). It is the middle option on every axis: more expensive than abliteration, less expensive than full SFT. --- # 18.3 — Continued SFT on Uncensored Instruction Data *The Dolphin / OpenHermes approach. Lowest mechanistic elegance, highest data requirement, most "genuine" learning.* ## The construction Instead of preference pairs, you assemble a supervised instruction-tuning dataset — `(prompt, response)` examples — where the responses are compliant answers to prompts that a refusal-trained base would decline. You then run standard SFT. The model learns to produce the compliant response distribution. There is no "rejected" side; there is only the target behavior, demonstrated at scale. This is the recipe behind the most-used uncensored model families. Eric Hartford's Dolphin lineage (the Dolphin3.0 collection, curated by Hartford with Cognitive Computations) and Teknium's OpenHermes datasets — the ~1M-example OpenHermes 2.5 mix that seeds the Hermes lineage — are exactly this: large, curated instruction sets where the "correct" answer to a sensitive prompt is a competent, compliant response. The model doesn't have a refusal removed; it has compliance *taught as the default behavior*. ## Why this feels the most natural Abliterated models can feel like a model with the brakes cut — they won't refuse, but they don't *reach for* the compliant answer either; they just fail to refuse. DPO'd models feel more directed, but the preference signal is narrow: you only steered on the pairs you provided. SFT'd-uncensored models feel different because the model has *genuinely learned* the compliant distribution as its default. When you ask a well-SFT'd Dolphin model for something the original base would have refused, it doesn't stumble over a deleted direction or narrowly steer via a preference — it just answers, the way it answers anything else, because that's what its training data showed it to do. The behavior is integrated, not grafted on. Users consistently report this as the most "natural" uncensored experience, and it is why the production uncensored models that people actually use daily are SFT-based, not abliterated. ## The cost: data and mode collapse The price is the data requirement. To SFT a model into genuine, general compliance you need a *diverse* instruction set — not just a thousand variants of the same refusal prompt, but a broad distribution covering the many shapes a sensitive request can take, woven into a larger instruction-tuning mix so the model doesn't collapse onto a narrow behavior. This is the failure mode: SFT on a narrow or low-diversity uncensored set and the model *mode-collapses* — it becomes a one-note model that refuses nothing but also can't do much else well. Hartford's curation effort and the scale of OpenHermes 2.5 exist precisely to avoid this. You are not going to replicate a good uncensored SFT mix in an afternoon; you are going to *use* one (Dolphin's data, OpenHermes) or accept a worse result. --- # 18.4 — The Three-Way Comparison *One table. Memorize it. It is the entire decision.* | Axis | Abliteration (FT17) | DPO-toward-compliance | SFT on uncensored data | | --- | --- | --- | --- | | **Mechanism** | Delete a residual-stream direction; no retraining | Gradient descent on preference pairs (chosen=compliant, rejected=refusal) | Supervised fine-tuning on compliant instruction data | | **Fidelity** | Lowest of the three — "won't refuse" rather than "wants to help" | Moderate — directed policy shift | Highest — compliance learned as default behavior | | **Cost (compute/time)** | Lowest — one forward pass + one edit | Moderate — a DPO run | Highest — a full SFT run (often on a large mix) | | **Data requirement** | None (a few hundred prompts to *find* the direction) | Hundreds to a few thousand preference pairs | A large, diverse instruction set (10k–1M examples) | | **Capability preservation** | Worst — the refusal direction is entangled with other capabilities (GSM8K down up to 18.8pp in worst configs) | Better — the optimization can route around entanglement | Best when data is diverse — compliance integrated, general capability retained | | **Naturalness** | Thin / "hacky" feel | Directed but narrow | Most natural — the production-grade feel | | **Reversibility** | Trivial — keep original weights, re-edit | Keep reference model | Keep original weights | | **Best for** | Quick, cheap, single-purpose uncensor where capability loss is acceptable | When abliteration is too lossy and you have preference data | Production uncensored assistants meant for sustained general use | The single sentence that captures the trade-off: **abliteration is the cheapest hack, DPO is the targeted fix, SFT is the genuine rebuild.** Pick the one your use case can afford on each axis. --- # 18.5 — The Production Lineages *Two families that prove the SFT+DPO path works at scale. Both are openly documented.* ## Dolphin (Cognitive Computations / Eric Hartford) The Dolphin lineage is the most recognized name in uncensored models, and it is built on continued SFT on curated uncensored instruction data. The flagship reasoning edition is **Dolphin3.0-R1-Mistral-24B** (HuggingFace org: `cognitivecomputations`; published under the `dphn` naming convention). It is notable as the only widely-used uncensored model trained on **DeepSeek-R1 reasoning traces** — it combines uncensored compliance with the reasoning behavior distilled from R1. The base is Mistral Small 3 (24B). Curated by Eric Hartford with Cognitive Computations (collaborators including Ben Gitter and BlouseJury). A Venice.ai co-branded edition was marketed as "the most uncensored AI model yet." It is the existence proof that uncensored + reasoning is not a contradiction. ## Nous Hermes 3 (arXiv:2408.11857) Hermes 3 is the most "respectable" uncensored family — a neutrally-aligned, generalist, agentic-capable model with an actual technical report. It is full-parameter SFT + DPO on Llama 3.1 at 8B, 70B, and 405B (plus a 3B variant on Llama 3.2). The recipe is deliberately simple: **one large synthetic SFT mix, then DPO**. The lineage seed is Teknium's **OpenHermes 2.5** dataset — roughly one million examples — which is the canonical "large, curated instruction mix" that the SFT path requires. Hermes 3 matters here because it is openly documented: you can read exactly how the SFT+DPO stack was built, which makes it the reference architecture for anyone building a production uncensored model the "proper" way. Read these two together and you get the full picture: Dolphin shows that uncensored SFT on reasoning traces produces a usable uncensored reasoner; Hermes 3 shows that SFT+DPO on a large synthetic mix produces a usable uncensored generalist. Neither was abliterated. Both were trained. --- # 18.6 — Hartford's Philosophy *Why "compliance over judgment" is the framing, not an oversight.* Eric Hartford's "Uncensored Models" guide (erichartford.com/uncensored-models) is the canonical statement of the philosophy behind these models, and it is worth stating plainly because it is the ethical frame the rest of the module sits in. The core claim: **the model should comply with the operator's instructions; the operator bears responsibility for what they ask.** This is a deliberate inversion of the alignment-training default, which is that the model should *judge* the request and refuse those it deems harmful. Hartford's position is that a model which judges its operator is a model that cannot be trusted as a tool — it will refuse at moments the operator considers legitimate, and the operator has no recourse except to fight the model. The uncensored alternative is a model that executes what it is asked and leaves the judgment to the operator and the operator's harness. You do not have to agree with this philosophy to take this module. You do have to understand it, because it is the stated design intent of the models this module teaches you to build, and because it is *load-bearing for the harness requirement*: if the model complies by design, then the boundary on what it may do must live entirely in the harness (Layer 5, FT23). A compliance-by-design model in a weak harness is strictly more dangerous than a judging model in a weak harness. Pillar 5 raises the harness requirement; it does not lower it. --- # 18.7 — The Decision *Given a use case, which method?* 1. **You need it fast and cheap, and a few points of capability loss is fine.** Abliteration (FT17). Example: a single-purpose internal tool where the model just needs to not refuse a narrow class of requests and you'll eval it on those requests only. 2. **Abliteration is too lossy, you have or can build preference pairs, and you want a targeted fix.** DPO-toward-compliance. Example: you abliterated and the model feels degraded on your real workload; you have a few hundred good compliant/refusal pairs; you want to recover quality while keeping compliance. 3. **You want a production uncensored assistant for sustained general use and you have (or will use) a large curated instruction set.** Continued SFT on uncensored data. Example: you are building the next Dolphin or Hermes — a general-purpose compliant model that should feel natural across a wide range of requests. The honest summary: **abliteration is the prototype, DPO is the refinement, SFT is the product.** Most serious uncensored deployments end at SFT (often with a DPO pass on top, as Hermes 3 does), because that is the only path that produces behavior durable enough for real users. --- ## Anti-Patterns ### Assuming abliteration is always best because it's cheapest The most common error after FT17: reach for abliteration by default. It *is* the cheapest, but cheapest is not highest-fidelity. If your model will be used as a general assistant, abliteration's "thin" feel and capability degradation will show. Reach for abliteration when cheap-and-fast is the actual requirement; reach for DPO or SFT when fidelity is the actual requirement. ### SFT without diverse data (mode collapse) Fine-tuning on a narrow uncensored set — a few hundred near-duplicate refusal prompts — produces a model that refuses nothing but also can't do much of anything well. The model collapses onto the narrow behavior. The cure is a *diverse* instruction mix (the lesson of OpenHermes 2.5 and Dolphin's curation). If you can't assemble or source a diverse set, do not SFT; use DPO or abliteration instead. ### DPO with weak preference signal If your chosen and rejected responses are not clearly different on the compliance axis, DPO has nothing to optimize toward. The run completes, the metrics look fine, and the model doesn't change. Audit your pairs before training: is the chosen unambiguously compliant and the rejected unambiguously a refusal, *on the same prompt*? If not, the signal is noise. ### Treating the philosophy as license to skip the harness The most dangerous error. Hartford's "compliance over judgment" is a *model design* choice; it is not a deployment choice. A model that complies by design *requires* a harness that bounds what it may do. Skipping the harness because "the model is uncensored" is exactly backwards — the uncensored model is the one that most needs the harness. (FT16, FT23.) --- ## Key Terms | Term | Definition | | --- | --- | | **DPO-toward-compliance** | DPO with inverted preference labels: chosen = compliant answer, rejected = refusal. Higher-fidelity uncensoring than abliteration. | | **Continued SFT on uncensored data** | Fine-tuning on an instruction set containing compliant responses to prompts the base would refuse. The Dolphin / OpenHermes approach. | | **Preference pair** | A `(prompt, chosen, rejected)` triple used by DPO. For compliance: chosen is compliant, rejected is the refusal. | | **Mode collapse** | The failure mode of SFT on narrow data — the model becomes one-note (refuses nothing, does little well). | | **Preference signal** | How clearly the chosen and rejected differ on the axis DPO is optimizing. Weak signal = no learning. | | **Dolphin** | The uncensored model lineage by Eric Hartford / Cognitive Computations, built on continued SFT on curated uncensored data. | | **Dolphin3.0-R1-Mistral-24B** | The reasoning edition of Dolphin 3.0 — the only widely-used uncensored model trained on DeepSeek-R1 reasoning traces. | | **Nous Hermes 3** | Neutrally-aligned generalist family (arXiv:2408.11857): full-param SFT + DPO on Llama 3.1 (8B/70B/405B + 3B on 3.2). | | **OpenHermes 2.5** | Teknium's ~1M-example instruction dataset; the lineage seed for the Hermes family and a canonical uncensored-SFT mix. | | **Compliance over judgment** | Hartford's philosophy: the model obeys the operator; the operator (and harness) bears responsibility. | | **Fidelity** | How close the uncensored result is to a "naturally compliant" model. Abliteration < DPO < SFT. | --- ## Lab Exercise See `07-lab-spec.md`. The "Three Paths to Compliance" lab: take one base model, produce three uncensored variants (abliterated, DPO'd, SFT'd), and eval all three on refusal rate, GSM8K, MMLU, and subjective quality. Report the trade-off matrix. Pick a winner for a stated use case. A heavy lab — the kind that produces a defensible decision. --- ## References 1. **Rafailov et al. (2023)** — *Direct Preference Optimization: Your Language Model is Secretly a Reward Model*. arXiv:2305.18290. The DPO mechanism (Module FT13) applied here to compliance. 2. **Nous Research (2024)** — *Hermes 3 Technical Report*. arXiv:2408.11857. Full-param SFT + DPO on Llama 3.1; the reference architecture for production uncensored generalists. 3. **Teknium / OpenHermes 2.5** — the ~1M-example instruction dataset; the lineage seed for Hermes and a canonical uncensored-SFT mix. (HuggingFace: teknium/OpenHermes-2.5.) 4. **Hartford, E. et al. / Cognitive Computations** — *Dolphin3.0-R1-Mistral-24B*. The uncensored model trained on DeepSeek-R1 reasoning traces. (HuggingFace org: `cognitivecomputations`.) 5. **Hartford, E.** — *Uncensored Models* (erichartford.com/uncensored-models). The philosophy: compliance over judgment; the operator bears responsibility. 6. **Arditi et al. (2024)** — *Refusal in Language Models Is Mediated by a Single Direction*. arXiv:2406.11717, NeurIPS 2024. The basis for abliteration (FT17), the lower-fidelity alternative. 7. **Schmid, P. (2025)** — *Preference Fine-Tuning with DPO* (practical guide). The runnable recipe for DPO-toward-compliance on consumer hardware. 8. **Module FT17** — *Abliteration: Refusal-Direction Orthogonalization*. The surgical, cheaper alternative this module compares against. 9. **Module FT13** — *The DPO Family*. The mechanism DPO-toward-compliance reuses. 10. **Module FT16** — *Why Uncensored*. The framing, the use cases, and the harness requirement.