---
title: "Chapter 14: Learning From Failures — DPO and GRPO"
description: "Understand DPO and GRPO as ways to use failures, check the ideas, then add preference pairs only after a rejection-sampling baseline exists."
type: lesson
order: 14
chapter: "05-the-flywheel"
---

# Chapter 14: Learning From Failures

## Objective

Recover signal rejection sampling throws away: **DPO** on chosen/rejected pairs, then **GRPO** with your verifier as reward. Do not jump here before Chapter 13 baseline.

---

## Concept

### DPO

For the same task: successful trajectory (**chosen**) vs failed (**rejected**). Model learns good *and* near-miss. Often more sample-efficient than SFT on winners alone.

You already log failures (Ch 08). Some MLX stacks expose DPO; check before writing a trainer from scratch.

### GRPO

Sample a group of rollouts per task, score each (tests / partial credit / step penalty), update toward above-average ones. **Your verifier is the reward function** — no separate reward model.

Same recipe family as large post-training stacks, at laptop scale.

### Sequencing

Rejection sampling first. One change at a time, scored on held-out eval. Preference methods only after you know SFT-on-winners works at all.

---

## Concept check

**Q1.** What pair does DPO need that pure rejection sampling discards?

**Q2.** In GRPO, what plays the role of the reward model in this course?

<details>
<summary>Concept answer key — attempt first</summary>

## Answer key (concept)

### Q1

**Model answer:** A failed/rejected trajectory for the same task alongside a success.

**Pass criteria:** chosen vs rejected pair

### Q2

**Model answer:** The verifier / test pass-fail (and related mechanical scores).

**Pass criteria:** verifier as reward

</details>

---

## Gate: Confirm Ch 13 baseline exists

Show tutor: at least one rejection-sampling round with held-out scores. **Do not** start DPO without it.

---

## Gate: Build preference pairs

From logged successes and failures on the **same** task ids, emit DPO-format pairs (per your trainer). Count pairs; spot-check one.

---

## Gate: One preference training run + eval

Train with DPO (or your tool’s equivalent). `make eval`. Compare to rejection-sampling baseline.

**Pass:** numbers compared; you can say whether preference learning bought anything.

---

## Check your understanding

**Q3.** Why is “one change at a time, scored on held-out” still the rule when methods get fancier?

**Q4.** Why was “log failures too” in Chapter 08 load-bearing for this chapter?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q3

**Model answer:** Otherwise you cannot attribute gains; fancy methods can mask pipeline bugs.

**Pass criteria:** attribution / isolation

### Q4

**Model answer:** Failures become rejected pairs; without them you cannot do DPO from your own rollouts.

**Pass criteria:** failures required for pairs

</details>

---

[← Chapter 13](/fine-tune-local-agent/05-the-flywheel/13-rejection-sampling/) · [Next: Chapter 15 →](/fine-tune-local-agent/05-the-flywheel/15-the-no-weights-complement/)