Chapter 09: Teacher Traces
Run the harness against Claude to generate distillation data, tune the 1B, and measure the metric that matters: tool-call validity rate.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 09: Teacher Traces
Generating the Data
Run the harness against a teacher — Claude via the Anthropic API — over 100–300 real questions about a real repo. The teacher’s traces are your first training set. This is distillation: the small model learns the teacher’s procedure, not its knowledge.
Write the tasks by hand. They should look like what you actually want the agent to do:
- “Where is X configured?”
- “Summarize how Y flows through the system”
- “What would break if I changed Z?”
Training
Same pipeline as Part 2. Same make train. Only the JSONL content changed. This is the payoff for building the Makefile.
The Metric
Not loss. Tool-call validity rate: of all assistant turns that attempted a tool call, what fraction parsed and executed without error?
Base Gemma 3 1B on a custom protocol: expect something dismal. That’s the point — it’s a wide-open gap that fine-tuning closes, and it’s your first proof that any of this works.
Secondary metrics: mean steps to completion, hallucinated-file rate (did it invent a path that doesn’t exist?), premature-stop rate.
Deliverables
harness.py— the looptrajectory.jsonl— a few hundred teacher tracesto_training_data.py— trajectory JSONL → mlx-lm chat JSONL, with masking- A number: validity rate before and after tuning
Exit Criteria
Fine-tuned 1B has a materially higher tool-call validity rate than base. You can regenerate the entire training set from raw trajectories with one command.
Failure Modes
- Hallucinated observations. The model writes what it thinks the file says instead of calling
read_file. Diagnosis: loss masking. Go back to Check 2. - Protocol drift. The model half-remembers a different tool syntax from pretraining. Fix: more examples, stricter delimiters, and check your system prompt is identical between training and inference. Mismatch here silently destroys everything downstream.
- The 1B is just too dumb. Possible. Bump to 4B before concluding the method is wrong. But do it after you’ve confirmed the pipeline on 1B.