---
title: "Chapter 12: The Scoreboard"
description: "Understand metrics and the uncomfortable baseline question, then build eval.py, smoke-test it, and fill a multi-model baseline table."
type: lesson
order: 12
chapter: "04-the-verifier"
---

# Chapter 12: The Scoreboard

## Objective

Build `eval.py`, score every model you can run, and populate a baseline table — including the possibility that a larger prompted model beats your tuned 1B.

---

## Concept: A scoreboard that can falsify you

```text
python eval.py --model http://localhost:8080 --tasks ./eval/
```

| Metric | Type | Why |
|---|---|---|
| Task completion rate | A+B | Headline |
| Tests-pass rate | B | Uncheatable |
| Tool-call validity | A+B | From Part 3 |
| Mean steps | A+B | Efficiency / thrashing |
| Timeout / loop rate | A+B | Stuck cycles |
| Regression rate | B | Fixes that break elsewhere |

### The uncomfortable possibility

**Base Gemma 3 12B, well-prompted, may beat your fine-tuned 1B.** That is a real result: gains may live in scale and harness quality; move the fine-tune up rather than polish a 1B forever.

**A verifier designed only to confirm your hypothesis is not a verifier.**

`outcome` on trajectories should be populated automatically from these runs.

---

## Concept check

**Q1.** Why score base 1B, tuned 1B, larger base models, *and* the teacher on the same eval set?

**Q2.** What decision does the baseline table exist to support?

<details>
<summary>Concept answer key — attempt first</summary>

## Answer key (concept)

### Q1

**Model answer:** Comparable map — know whether tuning helped vs scale/prompting; avoid vibes.

**Pass criteria:** comparable baselines / relative value of tuning

### Q2

**Model answer:** Whether fine-tuning the small model is worth it vs prompting a larger one; where to invest next.

**Pass criteria:** go/no-go or scale-up decision

</details>

---

## Gate: Implement `eval.py` smoke test

On **one** task and **one** model URL, print pass/fail and write JSON.

**Pass:** deterministic enough to re-run; `outcome` written back or alongside trajectories.

---

## Gate: Full table

Run against: base Gemma 1B, tuned Gemma 1B, base 4B/12B if available, teacher.

**Deliverables**

- `tools/` — six sandboxed tools  
- `eval/tasks/` — 20–50 held-out, Type B majority  
- `eval.py` — JSON + table  
- **Baseline table** — most valuable file in the project  

**Exit:** `make eval` prints a table; you can point at a cell and say what would move it.

---

## Check your understanding

**Q3.** Tests-pass rate is high but tool-call validity is low. What might that mean?

**Q4.** Why is regression rate on Type B tasks important when the flywheel starts editing code?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q3

**Model answer:** Maybe lucky finals, non-tool solutions, or metric mismatch — investigate; do not trust a single headline.

**Pass criteria:** metrics can disagree; investigate

### Q4

**Model answer:** “Fixes” can break other tests; flywheel can amplify regressions without this metric.

**Pass criteria:** detect breaking changes / collateral damage

</details>

---

[← Chapter 11](/fine-tune-local-agent/04-the-verifier/11-the-eval-set/) · [Next: Part 5 →](/fine-tune-local-agent/05-the-flywheel/)