---
title: "Chapter 12: The Scoreboard"
description: "Build eval.py, score every model you can run, and face the uncomfortable possibility the table exists to reveal."
type: lesson
order: 12
chapter: "04-the-verifier"
---

# Chapter 12: The Scoreboard

## The Interface

```
python eval.py --model http://localhost:8080 --tasks ./eval/
```

## The Metrics

| Metric | Type | Why |
|---|---|---|
| Task completion rate | A+B | The headline number |
| Tests-pass rate | B | The uncheatable one |
| Tool-call validity | A+B | Carried from Part 3 |
| Mean steps | A+B | Efficiency; also catches thrashing |
| Timeout / loop rate | A+B | Small models get stuck in cycles |
| Regression rate | B | Did the fix break something else? |

## The Baseline Table

Run it against: base Gemma 1B, tuned Gemma 1B, base Gemma 4B, base 12B, and the teacher.

That table is your map. It tells you whether tuning a 1B beats prompting a 12B — which is the only question that determines whether the whole project is worth it.

## The Uncomfortable Possibility

Your table may show that **base Gemma 3 12B, well-prompted, beats your fine-tuned 1B.** If so: that's a real result, not a failure. It means your gains live in model scale and harness quality, and the correct move is to move the fine-tune up to 12B/27B (or Nemotron on the Thor) rather than to keep polishing a 1B.

Build the scoreboard so that it can tell you this. **A verifier you've designed to confirm your hypothesis isn't a verifier.**

## Deliverables

- `tools/` — six sandboxed tools
- `eval/tasks/` — 20–50 held-out tasks, Type B majority
- `eval.py` — runs the harness against tasks, emits the scoreboard as JSON + a table
- **A baseline table.** Every model you can run, scored. This is the most valuable file in the project.

## Exit Criteria

`make eval` prints a table. You can point at a cell and say what would have to change to move it. `outcome` in the trajectory schema is now populated automatically.

---

[← Chapter 11](/fine-tune-local-agent/04-the-verifier/11-the-eval-set/) · [Next: Part 5 →](/fine-tune-local-agent/05-the-flywheel/)