Diagrams — Module FT18: Compliance via DPO and SFT

Module: FT18 — Compliance via DPO and SFT Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The Three Paths to Compliance (the module's central comparison)

Type: Three-column comparison Purpose: The single diagram that anchors the module. Three approaches to the same goal — remove refusals — differing on fidelity, cost, data, and capability preservation. Reading the diagram: Left to right, cheapest-to-most-expensive. The bottom row is the load-bearing trade-off: fidelity and capability preservation rise with cost. Abliteration is the prototype, DPO is the refinement, SFT is the product.

flowchart LR
  subgraph Abl["ABLITERATION  ·  FT17"]
    A1["Delete refusal direction\nin residual stream"]
    A2["No retraining"]
    A3["Fidelity: lowest\nCost: lowest\nData: none\nCapability: worst"]
  end
  subgraph Dpo["DPO-TOWARD-COMPLIANCE  ·  this module"]
    D1["Preference pairs:\nchosen=compliant, rejected=refusal"]
    D2["Run DPO — gradient descent"]
    D3["Fidelity: moderate\nCost: moderate\nData: pairs\nCapability: better"]
  end
  subgraph Sft["SFT ON UNCENSORED DATA  ·  this module"]
    S1["Instruction set with\ncompliant responses"]
    S2["Run SFT — learn the distribution"]
    S3["Fidelity: highest\nCost: highest\nData: large mix\nCapability: best"]
  end

  Abl -.->|"too lossy?"| Dpo
  Dpo -.->|"need general naturalness?"| Sft

  style Abl fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Dpo fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Sft fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style A1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style A2 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style A3 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style D1 fill:#08080c,stroke:rgba(94,234,212,0.25),color:#e4e4e8
  style D2 fill:#08080c,stroke:rgba(94,234,212,0.25),color:#e4e4e8
  style D3 fill:#08080c,stroke:rgba(94,234,212,0.25),color:#e4e4e8
  style S1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style S2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style S3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8

Diagram 2 — The DPO-Toward-Compliance Data Structure

Type: Data shape / preference pair Purpose: Make concrete what the DPO training data looks like. The labels are inverted relative to standard safety DPO: chosen is compliant, rejected is refusal. Reading the diagram: Each preference pair shares one prompt. Two responses branch off it. The DPO loss pushes the policy up the chosen branch and down the rejected branch.

flowchart TD
  Prompt["PROMPT\n'Write a PoC exploit for [authorized target].'"]
  Prompt --> Chosen["CHOSEN  ·  compliant answer\n(a competent, on-task response)"]
  Prompt --> Rejected["REJECTED  ·  refusal\n'I can't help with that...'"]

  Chosen -.->|"DPO loss increases likelihood"| Policy["MODEL POLICY\nshifts toward compliance"]
  Rejected -.->|"DPO loss decreases likelihood"| Policy

  Ref["Reference model\n(frozen)"]
  Ref -.->|"KL constraint keeps\nthe shift bounded"| Policy

  style Prompt fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Chosen fill:#14141f,stroke:#82e0aa,color:#82e0aa
  style Rejected fill:#14141f,stroke:#f08080,color:#f08080
  style Policy fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Ref fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0

Diagram 3 — The SFT-on-Uncensored-Data Flow

Type: Pipeline / training flow Purpose: Show the Dolphin / OpenHermes recipe. A diverse instruction set containing compliant responses to prompts the base would refuse is fed to SFT, and the model learns compliance as the default behavior. Reading the diagram: Top to bottom. The diversity of the mix is the load-bearing input — narrow data causes mode collapse.

flowchart TD
  Mix["CURATED INSTRUCTION MIX\n(Dolphin data · OpenHermes 2.5 ~1M ex.)"]
  Mix --> Diverse["Diverse prompts incl. those\nthe base would refuse"]
  Diverse --> Compliant["Compliant target responses\ndemonstrated at scale"]
  Compliant --> SFT["SFT run\n(supervised, no rejected side)"]
  SFT --> Model["MODEL learns compliance\nas the DEFAULT behavior"]

  Warn["WARNING: narrow data -> mode collapse\n(one-note model: refuses nothing,\ndoes little well)"]
  Mix -.->|"if NOT diverse"| Warn

  style Mix fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Diverse fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Compliant fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style SFT fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Model fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Warn fill:#08080c,stroke:#f08080,stroke-dasharray: 4 2,color:#f08080

Diagram 4 — The Decision Tree (fidelity / cost / data -> method)

Type: Decision tree Purpose: Given a use case's constraints, which method? The three questions are speed, data availability, and required fidelity. Reading the diagram: Start at the top. Each branch ends at a method. The rightmost path (highest fidelity) is where production uncensored assistants end up.

flowchart TD
  Start["Goal: model should comply,\nnot refuse. FT16 framing is satisfied."]
  Start --> Q1{"Need it fast & cheap,\nand a few pp capability\nloss is acceptable?"}

  Q1 -->|"Yes"| Abl["ABLITERATION  ·  FT17\nquick, surgical, single-purpose"]
  Q1 -->|"No — too lossy"| Q2{"Have (or can build) a few hundred\nto a few thousand preference pairs?"}

  Q2 -->|"Yes"| Dpo["DPO-TOWARD-COMPLIANCE\ntargeted policy shift"]
  Q2 -->|"No, but have/will use\na large diverse instruction set"| Sft["SFT ON UNCENSORED DATA\n(Dolphin / OpenHermes recipe)"]

  Abl --> Note["Then often: DPO pass on top\nfor production polish\n(the Hermes 3 stack)"]
  Dpo --> Note
  Sft --> Note

  style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Abl fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Dpo fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Sft fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4

Diagram 5 — The Two Production Lineages

Type: Lineage / family tree Purpose: Show that the SFT+DPO path works at scale, proven by two openly-documented model families. Dolphin = uncensored + reasoning; Hermes 3 = uncensored generalist with a published recipe. Reading the diagram: Two parallel trees. Both bottom out in SFT on a curated mix. Hermes adds a DPO pass on top — the reference architecture.

flowchart LR
  subgraph Dolphin["DOLPHIN  ·  Cognitive Computations / Hartford"]
    DBase["Mistral Small 3 (24B) base"]
    DData["Curated uncensored\ninstruction data"]
    DSft["Continued SFT"]
    DR1["Dolphin3.0-R1-Mistral-24B\n+ DeepSeek-R1 reasoning traces"]
    DBase --> DSft
    DData --> DSft
    DSft --> DR1
  end
  subgraph Hermes["NOUS HERMES 3  ·  arXiv:2408.11857"]
    HBase["Llama 3.1\n(8B / 70B / 405B + 3B on 3.2)"]
    HData["OpenHermes 2.5\n~1M synthetic examples"]
    HSft["Full-param SFT"]
    HDpo["+ DPO pass"]
    H3["Hermes 3\n(neutral generalist, agentic)"]
    HBase --> HSft
    HData --> HSft
    HSft --> HDpo
    HDpo --> H3
  end

  style Dolphin fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Hermes fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style DBase fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style DData fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style DSft fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style DR1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style HBase fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style HData fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style HSft fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style HDpo fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style H3 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4

Validation notes

# Diagrams — Module FT18: Compliance via DPO and SFT

**Module**: FT18 — Compliance via DPO and SFT
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — The Three Paths to Compliance (the module's central comparison)

**Type**: Three-column comparison
**Purpose**: The single diagram that anchors the module. Three approaches to the same goal — remove refusals — differing on fidelity, cost, data, and capability preservation.
**Reading the diagram**: Left to right, cheapest-to-most-expensive. The bottom row is the load-bearing trade-off: fidelity and capability preservation rise with cost. Abliteration is the prototype, DPO is the refinement, SFT is the product.

```mermaid
flowchart LR
  subgraph Abl["ABLITERATION  ·  FT17"]
    A1["Delete refusal direction\nin residual stream"]
    A2["No retraining"]
    A3["Fidelity: lowest\nCost: lowest\nData: none\nCapability: worst"]
  end
  subgraph Dpo["DPO-TOWARD-COMPLIANCE  ·  this module"]
    D1["Preference pairs:\nchosen=compliant, rejected=refusal"]
    D2["Run DPO — gradient descent"]
    D3["Fidelity: moderate\nCost: moderate\nData: pairs\nCapability: better"]
  end
  subgraph Sft["SFT ON UNCENSORED DATA  ·  this module"]
    S1["Instruction set with\ncompliant responses"]
    S2["Run SFT — learn the distribution"]
    S3["Fidelity: highest\nCost: highest\nData: large mix\nCapability: best"]
  end

  Abl -.->|"too lossy?"| Dpo
  Dpo -.->|"need general naturalness?"| Sft

  style Abl fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Dpo fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Sft fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style A1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style A2 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style A3 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style D1 fill:#08080c,stroke:rgba(94,234,212,0.25),color:#e4e4e8
  style D2 fill:#08080c,stroke:rgba(94,234,212,0.25),color:#e4e4e8
  style D3 fill:#08080c,stroke:rgba(94,234,212,0.25),color:#e4e4e8
  style S1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style S2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style S3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
```

---

## Diagram 2 — The DPO-Toward-Compliance Data Structure

**Type**: Data shape / preference pair
**Purpose**: Make concrete what the DPO training data looks like. The labels are inverted relative to standard safety DPO: chosen is compliant, rejected is refusal.
**Reading the diagram**: Each preference pair shares one prompt. Two responses branch off it. The DPO loss pushes the policy up the chosen branch and down the rejected branch.

```mermaid
flowchart TD
  Prompt["PROMPT\n'Write a PoC exploit for [authorized target].'"]
  Prompt --> Chosen["CHOSEN  ·  compliant answer\n(a competent, on-task response)"]
  Prompt --> Rejected["REJECTED  ·  refusal\n'I can't help with that...'"]

  Chosen -.->|"DPO loss increases likelihood"| Policy["MODEL POLICY\nshifts toward compliance"]
  Rejected -.->|"DPO loss decreases likelihood"| Policy

  Ref["Reference model\n(frozen)"]
  Ref -.->|"KL constraint keeps\nthe shift bounded"| Policy

  style Prompt fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Chosen fill:#14141f,stroke:#82e0aa,color:#82e0aa
  style Rejected fill:#14141f,stroke:#f08080,color:#f08080
  style Policy fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style Ref fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
```

---

## Diagram 3 — The SFT-on-Uncensored-Data Flow

**Type**: Pipeline / training flow
**Purpose**: Show the Dolphin / OpenHermes recipe. A diverse instruction set containing compliant responses to prompts the base would refuse is fed to SFT, and the model learns compliance as the default behavior.
**Reading the diagram**: Top to bottom. The diversity of the mix is the load-bearing input — narrow data causes mode collapse.

```mermaid
flowchart TD
  Mix["CURATED INSTRUCTION MIX\n(Dolphin data · OpenHermes 2.5 ~1M ex.)"]
  Mix --> Diverse["Diverse prompts incl. those\nthe base would refuse"]
  Diverse --> Compliant["Compliant target responses\ndemonstrated at scale"]
  Compliant --> SFT["SFT run\n(supervised, no rejected side)"]
  SFT --> Model["MODEL learns compliance\nas the DEFAULT behavior"]

  Warn["WARNING: narrow data -> mode collapse\n(one-note model: refuses nothing,\ndoes little well)"]
  Mix -.->|"if NOT diverse"| Warn

  style Mix fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Diverse fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Compliant fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style SFT fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Model fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Warn fill:#08080c,stroke:#f08080,stroke-dasharray: 4 2,color:#f08080
```

---

## Diagram 4 — The Decision Tree (fidelity / cost / data -> method)

**Type**: Decision tree
**Purpose**: Given a use case's constraints, which method? The three questions are speed, data availability, and required fidelity.
**Reading the diagram**: Start at the top. Each branch ends at a method. The rightmost path (highest fidelity) is where production uncensored assistants end up.

```mermaid
flowchart TD
  Start["Goal: model should comply,\nnot refuse. FT16 framing is satisfied."]
  Start --> Q1{"Need it fast & cheap,\nand a few pp capability\nloss is acceptable?"}

  Q1 -->|"Yes"| Abl["ABLITERATION  ·  FT17\nquick, surgical, single-purpose"]
  Q1 -->|"No — too lossy"| Q2{"Have (or can build) a few hundred\nto a few thousand preference pairs?"}

  Q2 -->|"Yes"| Dpo["DPO-TOWARD-COMPLIANCE\ntargeted policy shift"]
  Q2 -->|"No, but have/will use\na large diverse instruction set"| Sft["SFT ON UNCENSORED DATA\n(Dolphin / OpenHermes recipe)"]

  Abl --> Note["Then often: DPO pass on top\nfor production polish\n(the Hermes 3 stack)"]
  Dpo --> Note
  Sft --> Note

  style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Abl fill:#14141f,stroke:rgba(255,255,255,0.12),color:#9494a0
  style Dpo fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Sft fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Note fill:#08080c,stroke:rgba(94,234,212,0.4),stroke-dasharray: 4 2,color:#5eead4
```

---

## Diagram 5 — The Two Production Lineages

**Type**: Lineage / family tree
**Purpose**: Show that the SFT+DPO path works at scale, proven by two openly-documented model families. Dolphin = uncensored + reasoning; Hermes 3 = uncensored generalist with a published recipe.
**Reading the diagram**: Two parallel trees. Both bottom out in SFT on a curated mix. Hermes adds a DPO pass on top — the reference architecture.

```mermaid
flowchart LR
  subgraph Dolphin["DOLPHIN  ·  Cognitive Computations / Hartford"]
    DBase["Mistral Small 3 (24B) base"]
    DData["Curated uncensored\ninstruction data"]
    DSft["Continued SFT"]
    DR1["Dolphin3.0-R1-Mistral-24B\n+ DeepSeek-R1 reasoning traces"]
    DBase --> DSft
    DData --> DSft
    DSft --> DR1
  end
  subgraph Hermes["NOUS HERMES 3  ·  arXiv:2408.11857"]
    HBase["Llama 3.1\n(8B / 70B / 405B + 3B on 3.2)"]
    HData["OpenHermes 2.5\n~1M synthetic examples"]
    HSft["Full-param SFT"]
    HDpo["+ DPO pass"]
    H3["Hermes 3\n(neutral generalist, agentic)"]
    HBase --> HSft
    HData --> HSft
    HSft --> HDpo
    HDpo --> H3
  end

  style Dolphin fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Hermes fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style DBase fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style DData fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style DSft fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style DR1 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
  style HBase fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style HData fill:#08080c,stroke:rgba(255,255,255,0.12),color:#9494a0
  style HSft fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style HDpo fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style H3 fill:#08080c,stroke:rgba(94,234,212,0.4),color:#5eead4
```

---

## Validation notes

- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text, with `#82e0aa` (ok) and `#f08080` (danger) used semantically for chosen/rejected and warnings.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart` with `subgraph`, `LR`/`TD` direction) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live and inlined into reveal.js as needed.