Iterate & Improve

The Pattern: Build, Validate, Refine, Rebuild

Building a pipeline isn’t a one-shot process. It’s iterative:

Build → Test on small sample → Find issues → Fix → Test on larger sample → Find new issues → Fix → Process everything → Validate → Fix edge cases → Done

The AI agent makes this cycle fast because it can:

1. Re-run the entire pipeline on updated schema
2. Build tools to identify edge cases
3. Apply fixes consistently across all files

Handling 72 Unconverted Emails

Midway through the project, check_emails.py reveals 72 EML files that were never converted:

mise exec -- python check_emails.py --status
# Total EML files:    218
# Converted:          146
# Not converted:      72

This is a real-world data problem: you don’t always have the complete dataset at the start. The tooling catches it:

# Convert all unconverted at once
mise exec -- python check_emails.py | xargs mise exec -- python convert_email.py

Now all 218 EMLs are converted. Without check_emails.py, you might not notice the missing 72 files until much later.

Keeping CLAUDE.md Current

As the project evolves, the agent context needs to stay current:

> make sure that @CLAUDE.md is up to date

Claude updates CLAUDE.md with:

• Updated project structure (new subdirectories, scripts)
• Email classification workflow (two-stage pipeline)
• Documentation for new scripts (classify_emails.py, validate_taxonomy.py)
• Quick reference for taxonomy schema fields

This is the Self-Reinforcing Knowledge pattern again: each update to CLAUDE.md makes the next agent session more effective.

Reclassifying with Updated Schema

After refining the taxonomy, you re-run classification on updated batches:

> great, now decorate the ten most recent undecorated emails
> ok do the next 50
> ok do the rest of them

Each batch is processed in parallel — 5, 10, or 50 concurrent subagents. The agent usage is:

Batch 1: 5 files → 5 parallel subagents → ~1 minute
Batch 2: 50 files → 50 parallel subagents → ~5 minutes
Batch 3: ~150 files → 150 parallel subagents → ~15 minutes
Total: ~21 minutes for 210 files

Compare to manual classification: 5 minutes per file × 210 files = 17.5 hours. The parallel agent approach is 50x faster.

Beyond Classification: Building a Magazine

The structured content enables downstream applications. The final step in the tutorial is:

look through all the front matter in the decorated content. Use your front end design skills to make a magazine that Vogue meets Wired. We want to have company profiles, book quality history series, and FOD articles that get the town talking. Design an elegant website that is worthy of this content, and make sure that everything is cross linked, so we can find out more information about the people or companies mentioned in the articles.

Claude generates a complete magazine website:

magazine/
├── index.html          # Main magazine homepage (46KB)
├── article.html        # Article detail template (23KB)
├── entity.html         # Cross-link entity page (26KB)
├── generate_data.py    # Data extraction script
└── data/
    ├── articles.json     # All 210 articles with frontmatter
    ├── entities.json     # Cross-reference index (430 people, 553 companies)
    ├── categories.json   # Articles by series/type/topic
    └── stats.json        # Summary statistics

Design: Dark editorial luxury with Cormorant Garamond headlines, gold accents, color-coded series badges, asymmetric grids.

Cross-linking: Entity pages for people (Andrew Ng: 40 mentions), companies (OpenAI: 133 mentions), and models (615 unique). Click any name to see every article referencing it.

Converting to a Static Site (Astro)

The HTML prototype is refined into a production-ready Astro site:

web/
├── src/
│   ├── content/config.ts       # Zod schema for frontmatter validation
│   ├── pages/
│   │   ├── index.astro         # Homepage with all sections
│   │   ├── articles/[...slug].astro  # Article detail page
│   │   └── entity/[slug].astro # Entity profile page
│   ├── components/             # Nav, Hero, ArticleCard, EntityChip, Footer
│   └── styles/global.css       # Design tokens and theme

1,194 pages generated (articles + entities + index).

The site works because the classification schema is consistent. Astro’s content collections use Zod schemas that mirror the taxonomy:

const articleCollection = defineCollection({
  schema: z.object({
    title: z.string(),
    date: z.date(),
    series: z
      .enum([
        "fod",
        "topic",
        "agentic",
        "insights",
        "concepts",
        "podcast",
        "unicorn",
        "guestpost",
      ])
      .nullable(),
    content_type: z.enum([
      "article",
      "digest",
      "explainer",
      "interview",
      "profile",
      "curated",
      "announcement",
      "sponsored",
      "forwarded",
      "survey",
    ]),
    primary_topic: z
      .enum([
        "llm",
        "agents",
        "infrastructure",
        "research",
        "industry",
        "business",
        "ethics",
        "hardware",
        "multimodal",
      ])
      .optional(),
    // ... all other fields
  }),
});

The Full Stack

EML files (218)
    ↓ convert_email.py
Markdown files (209 → 218)
    ↓ email-taxonomy-classifier agent (parallel)
Decorated files (210, with taxonomy)
    ↓ validate_taxonomy.py
Validated files (207/210 pass)
    ↓ generate_data.py
JSON data (articles, entities, categories)
    ↓ Astro build
Static site (1,194 pages)

Each layer depends on the layer below. If the taxonomy schema changes, you re-classify. If the content changes, you re-build. The pipeline is reproducible — run it from scratch on updated data and get the same results.

Key Takeaway

The iteration cycle — build, validate, refine, rebuild — is fast because:

1. AI agents handle the bulk work (conversion, classification)
2. Scripts catch edge cases (validation, completeness)
3. Parallelism makes scale irrelevant (1 file or 1,000, same patterns)
4. Structured output enables downstream applications (magazine, website, search)

You spend your time on decisions (what taxonomy fields matter? what design aesthetic?) while the agents and scripts handle execution.

Next: Project Ideas — Ways to apply these patterns to your own work.