Chapter 10: Real Tools
Understand why verifiers and small toolsets matter, then expand the harness to six sandboxed tools and smoke-test each one.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 10: Real Tools
Objective
Expand to a six-tool sandboxed set. Later chapters add the eval set and scoreboard; here you grow the protocol surface carefully and prove each tool works.
Concept: No verifier, no flywheel
Without a mechanical pass/fail you cannot answer “did the fine-tune help?” — you tune on vibes. Part 5 filters trajectories by success. No verifier → no filter → no flywheel.
Coding is ideal: tests pass or they don’t. Exit code. No reward model required.
Toolset (stop at six)
read_file(path)
list_dir(path)
grep(pattern, path)
edit_file(path, old, new)
run_tests(target)
git_diff()Resist adding more. Each tool multiplies protocol surface and the examples the small model needs.
Sandbox
Sandbox write/execute (edit_file, run_tests): container, git worktree, or disposable copy. The agent will do something stupid; you want git checkout . and move on.
Concept check
Q1. Why is “tests pass / exit code” a better core filter for Part 5 than an LLM-as-judge score alone?
Q2. Why not add fifteen tools “for realism” on day one?
Concept answer key — attempt first
Answer key (concept)
Q1
Model answer: Mechanical, uncheatable, free filter for the flywheel; LLM judges are soft and gameable.
Pass criteria: mechanical / binary / flywheel filter
Q2
Model answer: Each tool multiplies protocol + training data needs; small models drown in surface area.
Pass criteria: protocol surface / data cost
Gate: Implement tools behind a sandbox root
Extend the harness. All paths resolve under a sandbox directory. run_tests and edit_file cannot escape it.
Pass: unit-level smoke for each tool (call from a tiny script or REPL): list, read, grep, edit, tests, diff.
Gate: One multi-tool task by hand
Run a single task that needs at least three tools (e.g. find a function, edit, run a test). Log a trajectory.
Pass: trajectory shows multiple tool types; sandbox survived (repo not destroyed).
Check your understanding
Q3. Which two tools are most important to sandbox, and why?
Q4. How does a larger toolset change what the logger/converter must handle?
Answer key — attempt every question first
Answer key
Q3
Model answer: edit_file and run_tests — they mutate state and execute code.
Pass criteria: write + execute named
Q4
Model answer: More tool-call shapes in messages; masking still applies to all tool results; schema may need tool names/args validated.
Pass criteria: more protocol variants + still mask tool outputs