---
title: "Parallel Classification at Scale"
description: Create a classification agent and run it against batches of files. 50 concurrent subagents complete in minutes what would take hours manually.
---

# Parallel Classification at Scale

## The Pattern: Spawn N Agents for N Items

Once you have a well-defined classification agent and a clear schema, processing multiple files is just a matter of spawning more agents. This is the **Parallel Execution** pattern — the simplest form of scaling with AI agents.

## Creating the Classification Agent

Use the same pattern from Phase 1:

```
> /agents → New Agent → Project Agent → Generate with Claude
```

Description:

> This email classifying agent will take an email from @content/uncategorized and examine it and add the taxonomy described in the context-taxonomy-schema in the reports/ directory. It will create a new file in content/decorated with the same name, rewritten front matter, and exact same body text.

Key design decisions:

- **Model: Haiku** — Fast and cost-effective for repetitive classification tasks
- **Single responsibility** — Read one file, classify, write one file
- **Idempotent** — Running on the same file twice produces the same output
- **Preserves body text** — Only the frontmatter changes

This agent doesn't need Opus-level reasoning. Classification is straightforward once the schema is defined: identify the series from the subject, assess content type, extract people and companies, judge audience and depth.

## First Run: Test with 5

```
> classify the 5 most recent emails that haven't been classified yet
```

The agent:

1. Looks at the list of undecorated files
2. Figures out which are the most recent (by date in frontmatter)
3. Spawns **5 parallel subagents** — one per file
4. Each subagent reads its file, applies taxonomy, writes to `content/decorated/`

Results:

| File                            | Series      | Type      | Topic  |
| ------------------------------- | ----------- | --------- | ------ |
| `why-single-ai-isnt-enough...`  | agentic #18 | explainer | agents |
| `what-is-defense-ai.md`         | guestpost   | article   | ethics |
| `sycophancy-and-leaderboard...` | fod #99     | digest    | ethics |
| `when-will-we-stop-coding.md`   | podcast     | interview | agents |
| `can-liquid-models-beat...`     | —           | article   | llm    |

## Extracting Validation into a Script

After watching the first run, you notice the agent is doing validation work manually — checking that the decorated file matches the source. This is redundant for a script.

> It was doing a lot of work in that run about checking to see that the resulting file was correct. Can you extract that validation code so that the @email-taxonomy-classifier can run it after?

Claude creates `validate_taxonomy.py`:

```python
# Usage:
# Validate single file
mise exec -- python validate_taxonomy.py content/decorated/FILE.md --source-dir content/uncategorized/ --lenient

# Validate all decorated files
mise exec -- python validate_taxonomy.py content/decorated/ --source-dir content/uncategorized/ --lenient
```

The validation script checks:

- Valid YAML frontmatter
- Required fields present (title, date, series, content_type)
- Enum values match schema (series, content_type, primary_topic, audience, depth)
- List field types correct (tags, people_mentioned, companies_mentioned, models_mentioned)
- Body text matches source (ensures no content was altered)
- `--lenient` mode for Unicode normalization (fancy quotes → ASCII)

And `classify_emails.py` for batch management:

```bash
mise exec -- python classify_emails.py --status   # Show overall progress
mise exec -- python classify_emails.py --list     # List unclassified files
mise exec -- python classify_emails.py --next 5   # Get next N files
mise exec -- python classify_emails.py --validate # Validate all decorated
```

## Batch Processing

### 10 More

```
> great, now decorate the ten most recent undecorated emails
```

10 parallel subagents, ~3 minutes:

| File                                    | Type                | Topic         |
| --------------------------------------- | ------------------- | ------------- |
| `what-ai-is-missing-for-real-reasoning` | interview (podcast) | llm/reasoning |
| `what-is-continual-learning`            | explainer           | llm           |
| `universe-of-incredible-models`         | digest (FOD#128)    | industry      |
| ...                                     | ...                 | ...           |

Progress: 16 classified (7.7%)

### 50 More

```
> ok do the next 50
```

50 parallel subagents. Progress jumps to 66 classified (31.4%).

### The Rest

```
> ok do the rest of them
```

~15 minutes to process the remaining files:

```
Email Processing Complete
- Converted: 218/218 EML files (100%)
- Classified: 210/210 markdown files (100%)
- Validated: 207/210 files pass validation (3 have minor issues)
```

## Why Parallel Works Here

The classification problem is **embarrassingly parallel** because:

1. **No shared state** — Each file is classified independently
2. **No ordering dependencies** — File A doesn't need File B's results
3. **Read-only inputs** — Source files in `uncategorized/` are never modified
4. **Write-only outputs** — Each agent writes to its own file in `decorated/`
5. **Idempotent** — Running twice on the same file produces the same output

This is the ideal case for agent parallelism. Unlike code generation (where multiple agents might conflict on the same file), classification naturally partitions across files.

## The Compounding Effect

Notice how the layers compound:

```
Research agent → "Use Python email stdlib" (reports/)
    ↓
Plans → "1-convert-emails.md" (plans/)
    ↓
Converter → content/uncategorized/ (209 files, 710 images)
    ↓
Taxonomy report → "content-taxonomy-schema.md" (reports/)
    ↓
Classification agent → content/decorated/ (210 files, rich metadata)
    ↓
Validation script → catch 3 edge cases
```

Each layer produces outputs that become inputs for the next. The research reports make classification possible. The schema makes validation possible. The validation makes iteration possible.

## Scaling Beyond 210 Files

The same pattern works for any number of files. With 10,000 files:

```
> classify the first 1000
> classify the next 1000
> classify the next 1000
...
```

Or with better tooling: a single command that partitions files into batches and spawns agents for each batch. The agent's per-file processing time is the only scaling limit.

---

**Next: [Semantic Enrichment](/content-repurpose/03-classification/08-semantic-enrichment/)** — How LLMs extract people, companies, models, and metadata.