Part 4: The Verifier — Real Tools + an Objective Scoreboard

Chapters 10–12: Expand to a real toolset, write a held-out eval set, and build the mechanical pass/fail scoreboard that everything else depends on.

TUTOR WITH THEFOCUS.AI

Agent Integration

Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.

Please tutor me in this lesson using the following context. First, read the instructions at: https://courses.thefocus.ai/llms.txt My Student ID is: <none> The lesson markdown source is at: https://courses.thefocus.ai/fine-tune-local-agent/04-the-verifier/10-real-tools.md

You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.

Part 4: The Verifier

Real tools + an objective scoreboard · ~3–5 days

This is the module people skip, and skipping it is the number one thing that kills projects like this. Without a verifier you cannot answer “did the fine-tune help?” — so you tune on vibes, and vibes on a 1B model are indistinguishable from noise. Worse: Part 5’s entire premise is filtering trajectories by whether they succeeded. No verifier, no filter. No filter, no flywheel.

Chapter	What You’ll Do
10 · Real Tools	Expand to six sandboxed tools — and resist adding a seventh
11 · The Eval Set	Write 20–50 held-out tasks, weighted toward mechanically checkable ones
12 · The Scoreboard	Build eval.py and produce the baseline table — the most valuable file in the project

What You’ll Have After Part 4

tools/ — six sandboxed tools
eval/tasks/ — 20–50 held-out tasks, Type B majority
eval.py — runs the harness against tasks, emits the scoreboard as JSON + a table
A baseline table: every model you can run, scored — the map that tells you whether tuning a 1B beats prompting a 12B
outcome in the trajectory schema now populated automatically

Chapter 10 →