Understand rejection sampling / STaR, check the idea, then wire generate→verify→filter→train→score one gate at a time using existing harness pieces.

Chapter 13: Rejection Sampling

Objective

Spin the simplest self-improvement loop that works: sample k rollouts per task, keep verifier winners, train, redeploy, score. Coding is a best-case domain because the Part 4 filter is free.

Concept: Keep winners only

Also called STaR / expert iteration.

Sample k = 4–8 rollouts per task at temperature ~0.8
Keep rollouts that pass tests (outcome from verifier)
One task can contribute several valid paths — diversity is a feature
Train on winners → redeploy → score held-out eval (never train on eval)

Pieces you already have:

Generate: harness (Ch 07)
Verify: eval-style pass/fail (Ch 12)
Filter: one-liner over trajectory JSONL
Train: make train (Ch 06)
Score: held-out set

The plateau

After 2–3 rounds the model gets good at tasks it can already sometimes solve and stalls on never-solved ones. Fix: harder tasks or bigger base — not endless identical rounds.

Flywheel exit (whole Part 5): round n+1 scores higher than n on held-out tasks for at least two consecutive rounds, no teacher in the loop.

Concept check

Q1. Why sample multiple rollouts per task instead of one greedy attempt?

Q2. Why must the held-out eval set stay out of the “train on winners” pool?

Concept answer key — attempt first

Answer key (concept)

Q1

Model answer: Temperature diversity finds solvable paths; multiple winners per task enrich data.

Pass criteria: diversity / more successes

Q2

Model answer: Contamination destroys the scoreboard; you would measure memorization of eval.

Pass criteria: no leakage / valid metrics

Gate: Filter script

From trajectories with outcome set, write winners to a train file. Show counts: total vs kept.

Gate: One flywheel round

generate (k samples) → verify → filter → make train → make eval

Record scores. Compare to pre-round baseline.

Pass: one full round completes; numbers written down.

Check your understanding

Q3. The loop plateaus. Name two non-“more rounds” responses.

Q4. Rejection sampling throws away failures. What signal is discarded (preview of Ch 14)?

Answer key — attempt every question first

Answer key

Q3

Model answer: Harder tasks; larger/better base model (or more diverse data).

Pass criteria: tasks and/or model scale (not just more identical rounds)

Q4

Model answer: Near-miss / failed trajectories useful for preference learning.

Pass criteria: failures as preference signal

← Part 5 Index · Next: Chapter 14 →