---
title: "Chapter 11: The Eval Set"
description: "Write 20–50 held-out tasks — research tasks and executable tasks — and discover the infinite ground-truth task generator hiding in your git history."
type: lesson
order: 11
chapter: "04-the-verifier"
---

# Chapter 11: The Eval Set

**Twenty to fifty held-out tasks.** These never appear in training data. Ever. Write them once, guard them with your life.

This is where the time in Part 4 actually goes — writing eval tasks is the bottleneck of the whole course, and it's unavoidable.

## Two Families

**Type A — Research / report tasks.** "Explain how authentication flows through this repo." Verification: rubric-scored, or an LLM-judge against a reference answer *you* wrote. Softer, but these are half of what you actually want the agent for.

**Type B — Executable tasks.** "This test is failing. Fix it." Verification: run the test suite. Exit code. Done.

## Weight Toward Type B

Type B is what makes Part 5 possible — it's the free, mechanical filter that the flywheel needs.

The cleanest source: **take real commits from a repo, revert the fix, keep the test.** You now have an infinite, ground-truth-verified task generator. Every bugfix commit in the history is a potential eval task: the failing state is real, the fix is known to exist, and the test suite is the judge.

## Exit Criterion for This Chapter

A directory `eval/tasks/` with 20–50 tasks, Type B majority, each with a mechanical verification procedure. Nothing in it will ever be shown to a training run.

---

[← Chapter 10](/fine-tune-local-agent/04-the-verifier/10-real-tools/) · [Next: Chapter 12 →](/fine-tune-local-agent/04-the-verifier/12-the-scoreboard/)