---
title: "Chapter 13: Rejection Sampling"
description: "Understand rejection sampling / STaR, check the idea, then wire generate→verify→filter→train→score one gate at a time using existing harness pieces."
type: lesson
order: 13
chapter: "05-the-flywheel"
---

# Chapter 13: Rejection Sampling

## Objective

Spin the simplest self-improvement loop that works: sample **k** rollouts per task, keep verifier winners, train, redeploy, score. Coding is a best-case domain because the Part 4 filter is free.

---

## Concept: Keep winners only

Also called STaR / expert iteration.

1. Sample **k = 4–8** rollouts per task at temperature ~0.8  
2. Keep rollouts that pass tests (`outcome` from verifier)  
3. One task can contribute several valid paths — diversity is a feature  
4. Train on winners → redeploy → score held-out eval (never train on eval)  

Pieces you already have:

- Generate: harness (Ch 07)  
- Verify: eval-style pass/fail (Ch 12)  
- Filter: one-liner over trajectory JSONL  
- Train: `make train` (Ch 06)  
- Score: held-out set  

### The plateau

After 2–3 rounds the model gets good at tasks it can already sometimes solve and stalls on never-solved ones. Fix: harder tasks or bigger base — not endless identical rounds.

**Flywheel exit (whole Part 5):** round *n+1* scores higher than *n* on held-out tasks for at least two consecutive rounds, no teacher in the loop.

---

## Concept check

**Q1.** Why sample multiple rollouts per task instead of one greedy attempt?

**Q2.** Why must the held-out eval set stay out of the “train on winners” pool?

<details>
<summary>Concept answer key — attempt first</summary>

## Answer key (concept)

### Q1

**Model answer:** Temperature diversity finds solvable paths; multiple winners per task enrich data.

**Pass criteria:** diversity / more successes

### Q2

**Model answer:** Contamination destroys the scoreboard; you would measure memorization of eval.

**Pass criteria:** no leakage / valid metrics

</details>

---

## Gate: Filter script

From trajectories with `outcome` set, write winners to a train file. Show counts: total vs kept.

---

## Gate: One flywheel round

```text
generate (k samples) → verify → filter → make train → make eval
```

Record scores. Compare to pre-round baseline.

**Pass:** one full round completes; numbers written down.

---

## Check your understanding

**Q3.** The loop plateaus. Name two non-“more rounds” responses.

**Q4.** Rejection sampling throws away failures. What signal is discarded (preview of Ch 14)?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q3

**Model answer:** Harder tasks; larger/better base model (or more diverse data).

**Pass criteria:** tasks and/or model scale (not just more identical rounds)

### Q4

**Model answer:** Near-miss / failed trajectories useful for preference learning.

**Pass criteria:** failures as preference signal

</details>

---

[← Part 5 Index](/fine-tune-local-agent/05-the-flywheel/) · [Next: Chapter 14 →](/fine-tune-local-agent/05-the-flywheel/14-learning-from-failures/)