---
title: "Chapter 09: Teacher Traces"
description: "Understand distillation metrics, generate teacher trajectories, convert and train with the existing Makefile, then measure tool-call validity before vs after."
type: lesson
order: 9
chapter: "03-trajectories"
---

# Chapter 09: Teacher Traces

## Objective

Run the harness against a **teacher** (e.g. Claude), convert trajectories, train the 1B with the Part 2 pipeline, and report **tool-call validity rate** before and after.

---

## Concept: Distill procedure, not knowledge

Teacher traces are your first real training set. The small model learns the teacher’s **procedure** (when to call tools, how to format calls, when to stop) — not a frozen copy of the repo.

### The metric that matters

Not loss. **Tool-call validity rate:** of assistant turns that attempted a tool call, what fraction parsed and executed without error?

Base 1B on a custom protocol: expect something dismal. That gap is the point.

Secondary: mean steps, hallucinated-file rate, premature-stop rate.

### Failure modes

- **Hallucinated observations** → loss masking; back to Chapter 05 Check 2 (canary + `--mask-prompt`)  
- **Protocol drift** → more examples, stricter delimiters, identical system prompts  
- **1B too weak** → try 4B *after* pipeline is proven on 1B  

---

## Concept check

**Q1.** Why is tool-call validity a better primary metric here than training loss?

**Q2.** A task says “Where is X configured?” Why is a teacher trace still useful if the 1B will never match Claude’s depth?

<details>
<summary>Concept answer key — attempt first</summary>

## Answer key (concept)

### Q1

**Model answer:** Loss can fall while protocol stays broken; validity measures the habit you care about (parseable, executable calls).

**Pass criteria:** metric matches goal / loss can mislead

### Q2

**Model answer:** You are teaching call shape and loop behavior, not Claude-level reasoning; validity can rise without matching teacher intelligence.

**Pass criteria:** procedure/protocol vs full capability

</details>

---

## Gate: Task list (raw)

Write 100–300 real questions about a real repo by hand (or carefully curated), e.g.:

- “Where is X configured?”  
- “Summarize how Y flows through the system”  
- “What would break if I changed Z?”  

Keep them in a **raw** tasks file; do not invent training JSONL by hand.

---

## Gate: Generate teacher trajectories

Point the harness at the teacher API. Run the task list. Append to `trajectories/raw.jsonl`.

**Pass:** hundreds of lines (or a smaller pilot if cost-constrained — note the count to your tutor); failures logged too.

---

## Gate: Convert → train → measure

```text
make data    # to_training_data.py
make train   # same Part 2 pipeline
```

Measure validity on a held-out slice **before** and **after** on the 1B.

**Exit criteria**

- Fine-tuned 1B has materially higher tool-call validity than base  
- Entire training set regenerates from raw trajectories with one command  

### Deliverables

- `harness.py`  
- `trajectory.jsonl` (teacher traces)  
- `to_training_data.py`  
- **A number:** validity before vs after  

---

## Check your understanding

**Q3.** Same Makefile as Part 2, only JSONL changed — what did that buy you?

**Q4.** Model invents file contents without calling `read_file`. First diagnosis?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q3

**Model answer:** Pipeline reuse — data is the only moving part; no re-plumbing train/serve.

**Pass criteria:** reuse / data-only change

### Q4

**Model answer:** Loss masking / training on tool observations as targets.

**Pass criteria:** masking / Check 2

</details>

---

[← Chapter 08](/fine-tune-local-agent/03-trajectories/08-the-logger/) · [Next: Part 4 →](/fine-tune-local-agent/04-the-verifier/)