---
title: "Chapter 11: The Eval Set"
description: "Understand held-out Type A vs Type B tasks, then build eval/tasks/ interactively with a Type B majority and mechanical verification."
type: lesson
order: 11
chapter: "04-the-verifier"
---

# Chapter 11: The Eval Set

## Objective

Write **20–50 held-out tasks** that never appear in training. This is where Part 4 calendar time actually goes.

---

## Concept: Guard the eval set

Eval tasks are sacred. If they leak into training, every scoreboard number becomes fiction.

### Two families

| Type | Example | Verification |
|------|---------|--------------|
| **A — Research / report** | “Explain how auth flows through this repo” | Rubric or LLM-judge vs **your** reference |
| **B — Executable** | “This test is failing; fix it” | Run tests; exit code |

**Weight toward Type B** — free mechanical filter for Part 5.

### Infinite Type B generator

Take real commits, **revert the fix**, keep the test. Failing state is real; fix is known to exist; suite is the judge. Every bugfix commit is a potential eval task.

---

## Concept check

**Q1.** Why must eval tasks never appear in training data?

**Q2.** Why prefer Type B for the flywheel even if Type A matches product use?

<details>
<summary>Concept answer key — attempt first</summary>

## Answer key (concept)

### Q1

**Model answer:** Leakage makes metrics fiction — the model can memorize eval tasks.

**Pass criteria:** contamination / invalid metrics

### Q2

**Model answer:** Mechanical pass/fail enables automatic filtering without judges; flywheel needs that.

**Pass criteria:** automatic/mechanical filter

</details>

---

## Gate: Create `eval/tasks/` structure

```text
eval/tasks/
  001-fix-auth.md   # or .json — your schema
  002-failing-test/
    ...
```

Agree a minimal task schema with your tutor: id, prompt, type A/B, how to verify.

---

## Gate: Author tasks interactively

1. Add **5 Type B** tasks first (git history reverts are ideal).  
2. Smoke-verify each by hand (run the check; expect fail before agent).  
3. Grow to **20–50**, Type B majority.  
4. Add some Type A with written reference answers.  

**Pass:** directory exists; count and type mix reported to tutor; none of these IDs appear in training trajectories.

---

## Check your understanding

**Q3.** Describe how you turn one bugfix commit into a Type B eval task.

**Q4.** What is the exit criterion for this chapter (artifact + property)?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q3

**Model answer:** Revert the fix (or checkout parent), keep/observe failing test, task is “make tests pass”; suite judges.

**Pass criteria:** revert/fix + test as judge

### Q4

**Model answer:** `eval/tasks/` with 20–50 tasks, Type B majority, mechanical verification each; never used in training.

**Pass criteria:** count + held-out + Type B majority

</details>

---

[← Chapter 10](/fine-tune-local-agent/04-the-verifier/10-real-tools/) · [Next: Chapter 12 →](/fine-tune-local-agent/04-the-verifier/12-the-scoreboard/)