Chapter 11: The Eval Set
Write 20–50 held-out tasks — research tasks and executable tasks — and discover the infinite ground-truth task generator hiding in your git history.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 11: The Eval Set
Twenty to fifty held-out tasks. These never appear in training data. Ever. Write them once, guard them with your life.
This is where the time in Part 4 actually goes — writing eval tasks is the bottleneck of the whole course, and it’s unavoidable.
Two Families
Type A — Research / report tasks. “Explain how authentication flows through this repo.” Verification: rubric-scored, or an LLM-judge against a reference answer you wrote. Softer, but these are half of what you actually want the agent for.
Type B — Executable tasks. “This test is failing. Fix it.” Verification: run the test suite. Exit code. Done.
Weight Toward Type B
Type B is what makes Part 5 possible — it’s the free, mechanical filter that the flywheel needs.
The cleanest source: take real commits from a repo, revert the fix, keep the test. You now have an infinite, ground-truth-verified task generator. Every bugfix commit in the history is a potential eval task: the failing state is real, the fix is known to exist, and the test suite is the judge.
Exit Criterion for This Chapter
A directory eval/tasks/ with 20–50 tasks, Type B majority, each with a mechanical verification procedure. Nothing in it will ever be shown to a training run.