Chapter 14: Learning From Failures — DPO and GRPO
Understand DPO and GRPO as ways to use failures, check the ideas, then add preference pairs only after a rejection-sampling baseline exists.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 14: Learning From Failures
Objective
Recover signal rejection sampling throws away: DPO on chosen/rejected pairs, then GRPO with your verifier as reward. Do not jump here before Chapter 13 baseline.
Concept
DPO
For the same task: successful trajectory (chosen) vs failed (rejected). Model learns good and near-miss. Often more sample-efficient than SFT on winners alone.
You already log failures (Ch 08). Some MLX stacks expose DPO; check before writing a trainer from scratch.
GRPO
Sample a group of rollouts per task, score each (tests / partial credit / step penalty), update toward above-average ones. Your verifier is the reward function — no separate reward model.
Same recipe family as large post-training stacks, at laptop scale.
Sequencing
Rejection sampling first. One change at a time, scored on held-out eval. Preference methods only after you know SFT-on-winners works at all.
Concept check
Q1. What pair does DPO need that pure rejection sampling discards?
Q2. In GRPO, what plays the role of the reward model in this course?
Concept answer key — attempt first
Answer key (concept)
Q1
Model answer: A failed/rejected trajectory for the same task alongside a success.
Pass criteria: chosen vs rejected pair
Q2
Model answer: The verifier / test pass-fail (and related mechanical scores).
Pass criteria: verifier as reward
Gate: Confirm Ch 13 baseline exists
Show tutor: at least one rejection-sampling round with held-out scores. Do not start DPO without it.
Gate: Build preference pairs
From logged successes and failures on the same task ids, emit DPO-format pairs (per your trainer). Count pairs; spot-check one.
Gate: One preference training run + eval
Train with DPO (or your tool’s equivalent). make eval. Compare to rejection-sampling baseline.
Pass: numbers compared; you can say whether preference learning bought anything.
Check your understanding
Q3. Why is “one change at a time, scored on held-out” still the rule when methods get fancier?
Q4. Why was “log failures too” in Chapter 08 load-bearing for this chapter?
Answer key — attempt every question first
Answer key
Q3
Model answer: Otherwise you cannot attribute gains; fancy methods can mask pipeline bugs.
Pass criteria: attribution / isolation
Q4
Model answer: Failures become rejected pairs; without them you cannot do DPO from your own rollouts.
Pass criteria: failures required for pairs