Chapter 14: Learning From Failures — DPO and GRPO
Rejection sampling throws away every failure. Recover that signal with preference pairs, then graduate to group-relative policy optimization.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 14: Learning From Failures
Keep the Failures (DPO)
Rejection sampling throws away every failure. That’s a lot of signal.
Direct Preference Optimization trains on pairs: for the same task, a successful trajectory (chosen) and a failed one (rejected). The model learns not just what good looks like but what the near-miss looks like — usually more sample-efficient than SFT on winners alone.
You already have the failures. You logged them in Chapter 08 — this is why “log failures too” was a design note and not a suggestion. Some MLX fine-tuners expose DPO natively; check your toolchain before writing a trainer.
GRPO
Group Relative Policy Optimization: sample a group of rollouts per task, compute reward for each (tests passed / partial credit / step penalty), and update toward the above-average ones. No separate reward model needed — your verifier is the reward function.
This is not exotic. It’s part of how Nemotron Nano 2 was post-trained (SFT → GRPO → DPO → RLHF). You’re running the same recipe at 1/1000th scale on a laptop, which is a fairly remarkable thing to be able to say.
Sequencing
Don’t jump straight here. Rejection sampling (Chapter 13) is the baseline that tells you whether preference learning is actually buying you anything. Same discipline as everywhere else in this course: one change at a time, scored on the held-out set.