Chapter 11: The Eval Set
Understand held-out Type A vs Type B tasks, then build eval/tasks/ interactively with a Type B majority and mechanical verification.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 11: The Eval Set
Objective
Write 20–50 held-out tasks that never appear in training. This is where Part 4 calendar time actually goes.
Concept: Guard the eval set
Eval tasks are sacred. If they leak into training, every scoreboard number becomes fiction.
Two families
| Type | Example | Verification |
|---|---|---|
| A — Research / report | “Explain how auth flows through this repo” | Rubric or LLM-judge vs your reference |
| B — Executable | “This test is failing; fix it” | Run tests; exit code |
Weight toward Type B — free mechanical filter for Part 5.
Infinite Type B generator
Take real commits, revert the fix, keep the test. Failing state is real; fix is known to exist; suite is the judge. Every bugfix commit is a potential eval task.
Concept check
Q1. Why must eval tasks never appear in training data?
Q2. Why prefer Type B for the flywheel even if Type A matches product use?
Concept answer key — attempt first
Answer key (concept)
Q1
Model answer: Leakage makes metrics fiction — the model can memorize eval tasks.
Pass criteria: contamination / invalid metrics
Q2
Model answer: Mechanical pass/fail enables automatic filtering without judges; flywheel needs that.
Pass criteria: automatic/mechanical filter
Gate: Create eval/tasks/ structure
eval/tasks/
001-fix-auth.md # or .json — your schema
002-failing-test/
...Agree a minimal task schema with your tutor: id, prompt, type A/B, how to verify.
Gate: Author tasks interactively
- Add 5 Type B tasks first (git history reverts are ideal).
- Smoke-verify each by hand (run the check; expect fail before agent).
- Grow to 20–50, Type B majority.
- Add some Type A with written reference answers.
Pass: directory exists; count and type mix reported to tutor; none of these IDs appear in training trajectories.
Check your understanding
Q3. Describe how you turn one bugfix commit into a Type B eval task.
Q4. What is the exit criterion for this chapter (artifact + property)?
Answer key — attempt every question first
Answer key
Q3
Model answer: Revert the fix (or checkout parent), keep/observe failing test, task is “make tests pass”; suite judges.
Pass criteria: revert/fix + test as judge
Q4
Model answer: eval/tasks/ with 20–50 tasks, Type B majority, mechanical verification each; never used in training.
Pass criteria: count + held-out + Type B majority