Chapter 12: The Scoreboard
Build eval.py, score every model you can run, and face the uncomfortable possibility the table exists to reveal.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 12: The Scoreboard
The Interface
python eval.py --model http://localhost:8080 --tasks ./eval/The Metrics
| Metric | Type | Why |
|---|---|---|
| Task completion rate | A+B | The headline number |
| Tests-pass rate | B | The uncheatable one |
| Tool-call validity | A+B | Carried from Part 3 |
| Mean steps | A+B | Efficiency; also catches thrashing |
| Timeout / loop rate | A+B | Small models get stuck in cycles |
| Regression rate | B | Did the fix break something else? |
The Baseline Table
Run it against: base Gemma 1B, tuned Gemma 1B, base Gemma 4B, base 12B, and the teacher.
That table is your map. It tells you whether tuning a 1B beats prompting a 12B — which is the only question that determines whether the whole project is worth it.
The Uncomfortable Possibility
Your table may show that base Gemma 3 12B, well-prompted, beats your fine-tuned 1B. If so: that’s a real result, not a failure. It means your gains live in model scale and harness quality, and the correct move is to move the fine-tune up to 12B/27B (or Nemotron on the Thor) rather than to keep polishing a 1B.
Build the scoreboard so that it can tell you this. A verifier you’ve designed to confirm your hypothesis isn’t a verifier.
Deliverables
tools/— six sandboxed toolseval/tasks/— 20–50 held-out tasks, Type B majorityeval.py— runs the harness against tasks, emits the scoreboard as JSON + a table- A baseline table. Every model you can run, scored. This is the most valuable file in the project.
Exit Criteria
make eval prints a table. You can point at a cell and say what would have to change to move it. outcome in the trajectory schema is now populated automatically.