Chapter 09: Teacher Traces
Understand distillation metrics, generate teacher trajectories, convert and train with the existing Makefile, then measure tool-call validity before vs after.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
Chapter 09: Teacher Traces
Objective
Run the harness against a teacher (e.g. Claude), convert trajectories, train the 1B with the Part 2 pipeline, and report tool-call validity rate before and after.
Concept: Distill procedure, not knowledge
Teacher traces are your first real training set. The small model learns the teacher’s procedure (when to call tools, how to format calls, when to stop) — not a frozen copy of the repo.
The metric that matters
Not loss. Tool-call validity rate: of assistant turns that attempted a tool call, what fraction parsed and executed without error?
Base 1B on a custom protocol: expect something dismal. That gap is the point.
Secondary: mean steps, hallucinated-file rate, premature-stop rate.
Failure modes
- Hallucinated observations → loss masking; back to Chapter 05 Check 2 (canary +
--mask-prompt) - Protocol drift → more examples, stricter delimiters, identical system prompts
- 1B too weak → try 4B after pipeline is proven on 1B
Concept check
Q1. Why is tool-call validity a better primary metric here than training loss?
Q2. A task says “Where is X configured?” Why is a teacher trace still useful if the 1B will never match Claude’s depth?
Concept answer key — attempt first
Answer key (concept)
Q1
Model answer: Loss can fall while protocol stays broken; validity measures the habit you care about (parseable, executable calls).
Pass criteria: metric matches goal / loss can mislead
Q2
Model answer: You are teaching call shape and loop behavior, not Claude-level reasoning; validity can rise without matching teacher intelligence.
Pass criteria: procedure/protocol vs full capability
Gate: Task list (raw)
Write 100–300 real questions about a real repo by hand (or carefully curated), e.g.:
- “Where is X configured?”
- “Summarize how Y flows through the system”
- “What would break if I changed Z?”
Keep them in a raw tasks file; do not invent training JSONL by hand.
Gate: Generate teacher trajectories
Point the harness at the teacher API. Run the task list. Append to trajectories/raw.jsonl.
Pass: hundreds of lines (or a smaller pilot if cost-constrained — note the count to your tutor); failures logged too.
Gate: Convert → train → measure
make data # to_training_data.py
make train # same Part 2 pipelineMeasure validity on a held-out slice before and after on the 1B.
Exit criteria
- Fine-tuned 1B has materially higher tool-call validity than base
- Entire training set regenerates from raw trajectories with one command
Deliverables
harness.pytrajectory.jsonl(teacher traces)to_training_data.py- A number: validity before vs after
Check your understanding
Q3. Same Makefile as Part 2, only JSONL changed — what did that buy you?
Q4. Model invents file contents without calling read_file. First diagnosis?
Answer key — attempt every question first
Answer key
Q3
Model answer: Pipeline reuse — data is the only moving part; no re-plumbing train/serve.
Pass criteria: reuse / data-only change
Q4
Model answer: Loss masking / training on tool observations as targets.
Pass criteria: masking / Check 2