Chapter 12: The Scoreboard
Understand metrics and the uncomfortable baseline question, then build eval.py, smoke-test it, and fill a multi-model baseline table.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 12: The Scoreboard
Objective
Build eval.py, score every model you can run, and populate a baseline table — including the possibility that a larger prompted model beats your tuned 1B.
Concept: A scoreboard that can falsify you
python eval.py --model http://localhost:8080 --tasks ./eval/| Metric | Type | Why |
|---|---|---|
| Task completion rate | A+B | Headline |
| Tests-pass rate | B | Uncheatable |
| Tool-call validity | A+B | From Part 3 |
| Mean steps | A+B | Efficiency / thrashing |
| Timeout / loop rate | A+B | Stuck cycles |
| Regression rate | B | Fixes that break elsewhere |
The uncomfortable possibility
Base Gemma 3 12B, well-prompted, may beat your fine-tuned 1B. That is a real result: gains may live in scale and harness quality; move the fine-tune up rather than polish a 1B forever.
A verifier designed only to confirm your hypothesis is not a verifier.
outcome on trajectories should be populated automatically from these runs.
Concept check
Q1. Why score base 1B, tuned 1B, larger base models, and the teacher on the same eval set?
Q2. What decision does the baseline table exist to support?
Concept answer key — attempt first
Answer key (concept)
Q1
Model answer: Comparable map — know whether tuning helped vs scale/prompting; avoid vibes.
Pass criteria: comparable baselines / relative value of tuning
Q2
Model answer: Whether fine-tuning the small model is worth it vs prompting a larger one; where to invest next.
Pass criteria: go/no-go or scale-up decision
Gate: Implement eval.py smoke test
On one task and one model URL, print pass/fail and write JSON.
Pass: deterministic enough to re-run; outcome written back or alongside trajectories.
Gate: Full table
Run against: base Gemma 1B, tuned Gemma 1B, base 4B/12B if available, teacher.
Deliverables
tools/— six sandboxed toolseval/tasks/— 20–50 held-out, Type B majorityeval.py— JSON + table- Baseline table — most valuable file in the project
Exit: make eval prints a table; you can point at a cell and say what would move it.
Check your understanding
Q3. Tests-pass rate is high but tool-call validity is low. What might that mean?
Q4. Why is regression rate on Type B tasks important when the flywheel starts editing code?
Answer key — attempt every question first
Answer key
Q3
Model answer: Maybe lucky finals, non-tool solutions, or metric mismatch — investigate; do not trust a single headline.
Pass criteria: metrics can disagree; investigate
Q4
Model answer: “Fixes” can break other tests; flywheel can amplify regressions without this metric.
Pass criteria: detect breaking changes / collateral damage