Parallel Classification at Scale

The Pattern: Spawn N Agents for N Items

Once you have a well-defined classification agent and a clear schema, processing multiple files is just a matter of spawning more agents. This is the Parallel Execution pattern — the simplest form of scaling with AI agents.

Creating the Classification Agent

Use the same pattern from Phase 1:

> /agents → New Agent → Project Agent → Generate with Claude

Description:

This email classifying agent will take an email from @content/uncategorized and examine it and add the taxonomy described in the context-taxonomy-schema in the reports/ directory. It will create a new file in content/decorated with the same name, rewritten front matter, and exact same body text.

Key design decisions:

• Model: Haiku — Fast and cost-effective for repetitive classification tasks
• Single responsibility — Read one file, classify, write one file
• Idempotent — Running on the same file twice produces the same output
• Preserves body text — Only the frontmatter changes

This agent doesn’t need Opus-level reasoning. Classification is straightforward once the schema is defined: identify the series from the subject, assess content type, extract people and companies, judge audience and depth.

First Run: Test with 5

> classify the 5 most recent emails that haven't been classified yet

The agent:

1. Looks at the list of undecorated files
2. Figures out which are the most recent (by date in frontmatter)
3. Spawns 5 parallel subagents — one per file
4. Each subagent reads its file, applies taxonomy, writes to content/decorated/

Results:

Extracting Validation into a Script

After watching the first run, you notice the agent is doing validation work manually — checking that the decorated file matches the source. This is redundant for a script.

It was doing a lot of work in that run about checking to see that the resulting file was correct. Can you extract that validation code so that the @email-taxonomy-classifier can run it after?

Claude creates validate_taxonomy.py:

# Usage:
# Validate single file
mise exec -- python validate_taxonomy.py content/decorated/FILE.md --source-dir content/uncategorized/ --lenient

# Validate all decorated files
mise exec -- python validate_taxonomy.py content/decorated/ --source-dir content/uncategorized/ --lenient

The validation script checks:

• Valid YAML frontmatter
• Required fields present (title, date, series, content_type)
• Enum values match schema (series, content_type, primary_topic, audience, depth)
• List field types correct (tags, people_mentioned, companies_mentioned, models_mentioned)
• Body text matches source (ensures no content was altered)
• --lenient mode for Unicode normalization (fancy quotes → ASCII)

And classify_emails.py for batch management:

mise exec -- python classify_emails.py --status   # Show overall progress
mise exec -- python classify_emails.py --list     # List unclassified files
mise exec -- python classify_emails.py --next 5   # Get next N files
mise exec -- python classify_emails.py --validate # Validate all decorated

Batch Processing

10 More

> great, now decorate the ten most recent undecorated emails

10 parallel subagents, ~3 minutes:

Progress: 16 classified (7.7%)

50 More

> ok do the next 50

50 parallel subagents. Progress jumps to 66 classified (31.4%).

The Rest

> ok do the rest of them

~15 minutes to process the remaining files:

Email Processing Complete
- Converted: 218/218 EML files (100%)
- Classified: 210/210 markdown files (100%)
- Validated: 207/210 files pass validation (3 have minor issues)

Why Parallel Works Here

The classification problem is embarrassingly parallel because:

1. No shared state — Each file is classified independently
2. No ordering dependencies — File A doesn’t need File B’s results
3. Read-only inputs — Source files in uncategorized/ are never modified
4. Write-only outputs — Each agent writes to its own file in decorated/
5. Idempotent — Running twice on the same file produces the same output

This is the ideal case for agent parallelism. Unlike code generation (where multiple agents might conflict on the same file), classification naturally partitions across files.

The Compounding Effect

Notice how the layers compound:

Research agent → "Use Python email stdlib" (reports/)
    ↓
Plans → "1-convert-emails.md" (plans/)
    ↓
Converter → content/uncategorized/ (209 files, 710 images)
    ↓
Taxonomy report → "content-taxonomy-schema.md" (reports/)
    ↓
Classification agent → content/decorated/ (210 files, rich metadata)
    ↓
Validation script → catch 3 edge cases

Each layer produces outputs that become inputs for the next. The research reports make classification possible. The schema makes validation possible. The validation makes iteration possible.

Scaling Beyond 210 Files

The same pattern works for any number of files. With 10,000 files:

> classify the first 1000
> classify the next 1000
> classify the next 1000
...

Or with better tooling: a single command that partitions files into batches and spawns agents for each batch. The agent’s per-file processing time is the only scaling limit.

Next: Semantic Enrichment — How LLMs extract people, companies, models, and metadata.