---
title: "Chapter 14: Learning From Failures — DPO and GRPO"
description: "Rejection sampling throws away every failure. Recover that signal with preference pairs, then graduate to group-relative policy optimization."
type: lesson
order: 14
chapter: "05-the-flywheel"
---

# Chapter 14: Learning From Failures

## Keep the Failures (DPO)

Rejection sampling throws away every failure. That's a lot of signal.

Direct Preference Optimization trains on *pairs*: for the same task, a successful trajectory (chosen) and a failed one (rejected). The model learns not just what good looks like but what the near-miss looks like — usually more sample-efficient than SFT on winners alone.

You already have the failures. You logged them in [Chapter 08](/fine-tune-local-agent/03-trajectories/08-the-logger/) — this is why "log failures too" was a design note and not a suggestion. Some MLX fine-tuners expose DPO natively; check your toolchain before writing a trainer.

## GRPO

Group Relative Policy Optimization: sample a group of rollouts per task, compute reward for each (tests passed / partial credit / step penalty), and update toward the above-average ones. No separate reward model needed — **your verifier *is* the reward function.**

This is not exotic. It's part of how Nemotron Nano 2 was post-trained (SFT → GRPO → DPO → RLHF). You're running the same recipe at 1/1000th scale on a laptop, which is a fairly remarkable thing to be able to say.

## Sequencing

Don't jump straight here. Rejection sampling (Chapter 13) is the baseline that tells you whether preference learning is actually buying you anything. Same discipline as everywhere else in this course: one change at a time, scored on the held-out set.

---

[← Chapter 13](/fine-tune-local-agent/05-the-flywheel/13-rejection-sampling/) · [Next: Chapter 15 →](/fine-tune-local-agent/05-the-flywheel/15-the-no-weights-complement/)