---
title: "Chapter 04: The Toy Task"
description: "Understand the toy task and chat JSONL, install mlx-lm and smoke-test it, then generate a train/valid dataset from a script — not by hand."
type: lesson
order: 4
chapter: "02-the-pipeline"
---

# Chapter 04: The Toy Task

## Objective

Prove a **text** fine-tune pipeline on a tiny, unambiguous behavior: always answer as JSON with `answer` and `confidence`. Install **mlx-lm**, smoke-test it, then generate chat-format JSONL from a **script**.

Tutor pacing: concept → concept check → install → smoke test → write generator → emit data. Do not train in this chapter.

---

## Concept: Why a toy task

Part 5’s flywheel will spin `data → train → fuse → serve` hundreds of times. Every defect multiplies. The most expensive bug in agent fine-tuning — **loss masking** — is invisible unless you deliberately test for it. So we use:

- a **1B-class** model (minutes per loop, not hours)
- a behavior so trivial that success and failure are **unambiguous**

### The behavior

**Always respond with a JSON object containing keys `answer` and `confidence`.**

Twenty-ish examples is enough to start. It is the “replace every number with BANANA” of fine-tuning: a habit the base model does not reliably have, checkable in one glance, zero domain knowledge.

### Data format

`mlx-lm` expects a directory with `train.jsonl`, `valid.jsonl`, and optionally `test.jsonl`. Use **chat** format (what Part 3 needs):

```jsonl
{"messages": [{"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "{\"answer\": \"Paris\", \"confidence\": 0.99}"}]}
```

### Generator, not a hand-written file

The **script** is the artifact; the JSONL is disposable. You will regenerate splits and sizes many times.

---

## Concept check

**Q1.** Why prove the pipeline on “always return JSON with `answer` and `confidence`” instead of on a realistic coding agent task?

**Q2.** Why is a **generator script** the deliverable, not a carefully hand-edited `train.jsonl`?

<details>
<summary>Concept answer key — attempt first</summary>

## Answer key (concept)

### Q1

**Model answer:** Failure is unambiguous and fast to see; realistic agent tasks hide pipeline bugs (format, templates, masking) behind noisy task difficulty.

**Pass criteria:** unambiguous / fast check / isolates pipeline vs task hardness

### Q2

**Model answer:** You will regenerate data many times (sizes, splits, tasks); the script makes that cheap and repeatable; hand JSONL does not.

**Pass criteria:** regeneration / repeatability / script as source of truth

</details>

---

## Gate: Install mlx-lm

```bash
mkdir -p ~/pipeline-toy && cd ~/pipeline-toy
uv venv && source .venv/bin/activate
uv pip install mlx-lm
```

---

## Gate: Verify it runs (smoke test)

Before writing datasets, prove generate works on the base model:

```bash
mlx_lm.generate \
  --model mlx-community/gemma-3-1b-it-4bit \
  --prompt "What is 2+2? Reply briefly." \
  --max-tokens 64
```

First run downloads weights. **Pass:** text prints without crash.

If this fails, stop — do not build JSONL yet.

---

## Gate: Project layout (raw intent → train files)

```bash
mkdir -p ~/pipeline-toy/data
mkdir -p ~/pipeline-toy/scripts
```

You may keep a human-readable list of Q&A pairs in something like `scripts/seed_questions.txt` (**raw**), and have the generator write `data/train.jsonl` and `data/valid.jsonl` (**train artifacts**). Same idea as image `raw/` → `train/` in Part 1.

---

## Gate: Write the generator and emit JSONL

Write `scripts/make_json_dataset.py` (or similar) that:

1. Defines ~20 user questions (or loads them from a seed file)
2. Emits assistant messages that are **only** JSON: `{"answer": "...", "confidence": 0.0-1.0}`
3. Writes `data/train.jsonl` and a smaller `data/valid.jsonl`

Run it. Show your tutor: `wc -l data/*.jsonl` and one sample line.

**Do not hand-write the JSONL as the primary workflow.**

---

## Check your understanding

**Q3.** What must be true of the assistant side of each training example for this toy task?

**Q4.** You open the base model’s smoke-test reply and it is free-form prose, not JSON. Is that a problem before you train? Why or why not?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q3

**Model answer:** It must be valid JSON with (at least) `answer` and `confidence` keys — the habit you are teaching.

**Pass criteria:** JSON shape with those keys

### Q4

**Model answer:** Expected. The base model does not have the habit; that is why you fine-tune. Smoke test only proves tooling, not the behavior.

**Pass criteria:** base failure expected; smoke test ≠ behavior success

</details>

---

[← Part 2 Index](/fine-tune-local-agent/02-the-pipeline/) · [Next: Chapter 05 →](/fine-tune-local-agent/02-the-pipeline/05-train-and-check/)