---
title: "Chapter 05: Train and Check"
description: "Run mlx_lm.lora, then pass the three checks: overfit sanity, loss masking, and adapter vs. merged. The loss masking check is the highest-leverage lesson in the course."
type: lesson
order: 5
chapter: "02-the-pipeline"
---

# Chapter 05: Train and Check

## Training

```bash
mlx_lm.lora \
  --model mlx-community/gemma-3-1b-it-4bit \
  --train \
  --data ./data \
  --iters 300 \
  --batch-size 4
```

Flags that matter:

- `--fine-tune-type` — `lora` (default), `dora`, or `full`
- `--num-layers` — how many layers receive adapters (default 16). Drop to 8 or 4 under memory pressure.
- `--batch-size` — tutorials default to 1 for 16GB machines. **You have 64GB.** Use 2–4 and cut wall-clock time.
- `--grad-accumulation-steps` — raises effective batch size without raising memory
- `--grad-checkpoint` — trades compute for memory
- `--report-to wandb` — if you want live curves instead of terminal scroll

Adapters land in `./adapters` by default.

## The Three Checks (do all three, in order)

### Check 1 — The overfit sanity check

Train on **10 examples for 200 iterations**. Generate on those exact 10 prompts. The model must reproduce the outputs essentially verbatim.

If it cannot memorize ten examples, your formatting, chat template, or loss masking is broken. This takes two minutes and saves three hours. **Run it before every new dataset shape, forever.**

### Check 2 — Loss masking

Confirm you are training only on *assistant* tokens. In `mlx-lm`'s chat format this is handled for you; if you move to a custom loop or to `mlx-tune` (an MLX fine-tuner with an Unsloth-compatible API), it exposes this explicitly as `train_on_responses_only` and `get_chat_template`.

Why it matters, stated plainly, because Part 3 lives or dies on it: **if you compute loss over tool outputs and file contents the model merely *observed*, you are teaching it to hallucinate observations instead of going and fetching them.** It will invent the contents of files. This is the single highest-leverage line in the entire course.

### Check 3 — Adapter vs. merged

Serve with the adapter attached, and serve merged. Confirm identical behavior.

```bash
# adapter attached
mlx_lm.generate --model mlx-community/gemma-3-1b-it-4bit \
  --adapter-path ./adapters --prompt "What is 2+2?" --max-tokens 200

# merge into standalone weights
mlx_lm.fuse --model mlx-community/gemma-3-1b-it-4bit \
  --adapter-path ./adapters --save-path ./models/gemma-json-1b
```

## The Merging Caveat, Worth Internalizing Now

Merging a weak LoRA delta back into a *4-bit* base can round the delta away entirely. The safe default dequantizes the base to 16-bit before fusing. If your merged model mysteriously behaves like the base model, this is why. When in doubt, keep the adapter separate and load it at inference.

---

[← Chapter 04](/fine-tune-local-agent/02-the-pipeline/04-the-toy-task/) · [Next: Chapter 06 →](/fine-tune-local-agent/02-the-pipeline/06-serve-and-automate/)