Chapter 13: Rejection Sampling
The simplest self-improvement method that works: sample k rollouts per task, keep the verified winners, train on them, redeploy, score.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 13: Rejection Sampling
This is rejection sampling, sometimes called STaR (Self-Taught Reasoner) or expert iteration. It is the simplest self-improvement method that works, and coding is its best-case domain because the filter is free — your verifier from Part 4.
The Loop
Sample k = 4–8 rollouts per task at temperature ~0.8. Keep every rollout that passes tests. One task can contribute several successful trajectories, each a different valid path — that diversity is a feature, not noise.
Train on the winners. Redeploy. Score.
Every piece already exists:
- Generation: the harness from Chapter 07
- Verification:
eval.py-style pass/fail from Chapter 12, populatingoutcome - Filtering: a one-liner over trajectory JSONL
- Training:
make trainfrom Chapter 06 - Scoring: the held-out eval set — which the flywheel never trains on
Watch For: The Plateau
The loop plateaus. After 2–3 rounds the model gets good at exactly the tasks it can already sometimes solve, and stops improving on the ones it never solves. Expected. The fix is harder tasks or a better base model, not more rounds.
Exit Criteria for the Whole Flywheel
Round n+1 scores higher than round n on held-out tasks, for at least two consecutive rounds, with no teacher in the loop.