---
title: "Chapter 09: Teacher Traces"
description: "Run the harness against Claude to generate distillation data, tune the 1B, and measure the metric that matters: tool-call validity rate."
type: lesson
order: 9
chapter: "03-trajectories"
---

# Chapter 09: Teacher Traces

## Generating the Data

Run the harness against a **teacher** — Claude via the Anthropic API — over 100–300 real questions about a real repo. The teacher's traces are your first training set. This is distillation: the small model learns the teacher's *procedure*, not its knowledge.

Write the tasks by hand. They should look like what you actually want the agent to do:

- "Where is X configured?"
- "Summarize how Y flows through the system"
- "What would break if I changed Z?"

## Training

Same pipeline as Part 2. Same `make train`. Only the JSONL content changed. **This is the payoff for building the Makefile.**

## The Metric

Not loss. **Tool-call validity rate**: of all assistant turns that attempted a tool call, what fraction parsed and executed without error?

Base Gemma 3 1B on a custom protocol: expect something dismal. That's the point — it's a wide-open gap that fine-tuning closes, and it's your first proof that any of this works.

Secondary metrics: mean steps to completion, hallucinated-file rate (did it invent a path that doesn't exist?), premature-stop rate.

## Deliverables

- `harness.py` — the loop
- `trajectory.jsonl` — a few hundred teacher traces
- `to_training_data.py` — trajectory JSONL → mlx-lm chat JSONL, with masking
- **A number**: validity rate before and after tuning

## Exit Criteria

Fine-tuned 1B has a materially higher tool-call validity rate than base. You can regenerate the entire training set from raw trajectories with one command.

## Failure Modes

- **Hallucinated observations.** The model writes what it *thinks* the file says instead of calling `read_file`. Diagnosis: loss masking. Go back to Check 2.
- **Protocol drift.** The model half-remembers a different tool syntax from pretraining. Fix: more examples, stricter delimiters, and check your system prompt is identical between training and inference. Mismatch here silently destroys everything downstream.
- **The 1B is just too dumb.** Possible. Bump to 4B before concluding the method is wrong. But do it *after* you've confirmed the pipeline on 1B.

---

[← Chapter 08](/fine-tune-local-agent/03-trajectories/08-the-logger/) · [Next: Part 4 →](/fine-tune-local-agent/04-the-verifier/)