Research → Build Converter

The Pattern: Research, Plan, Build

Don’t jump straight to code. The sequence matters:

1. Research: Ask the agent to evaluate approaches
2. Plan: Write a plan document that captures decisions
3. Build: Implement based on the plan
4. Test: Run on a few files, inspect output
5. Iterate: Refine based on real data

Step 1: Research

Start a new Claude Code session and ask the research agent:

do tech research on how we can extract the data in emails/ to an organized content directory in markdown

The agent exits for ~4 minutes, examines your project, and produces a detailed report at reports/2025-11-30-eml-to-markdown-extraction.md. The full report includes:

• Recommendation: Python with email stdlib + html2text
• Rationale: Zero dependencies for MIME parsing, automatic encoding handling, mature HTML conversion
• Alternative analysis: Node.js mailparser (better for streaming, overkill for batch), Deno + postal-mime (ecosystem less mature)
• Category detection: Regex patterns for 7+ newsletter series from subject lines
• Complete code: parse_eml_file(), extract_category(), html_to_markdown(), process_all_newsletters()
• Caveats: GPLv3 licensing for html2text, complex table handling, image extraction

Step 2: Plan

Now write a plan:

look through all of our research, and lets make a plan in plans/1-convert-emails.md. lets focus on simplicity, making sure we move the data over cleanly, and all of the images extracted. we should be able to do one email at a time for testing purposes. We will decorate the emails later with more metadata

The plan captures key decisions:

# Plan: Convert Emails to Markdown

## Design Principles

1. **Use plain text version** - The plain text email is already markdown with real URLs
2. **Single-file processing** - Script takes one EML path, outputs one markdown file
3. **Images downloaded locally** - Store in content/images/ with unique names
4. **Minimal frontmatter** - Just enough to identify the source; metadata decoration later
5. **Clean content** - Strip email cruft (separators, follow prompts, forwarding notices)

## Output Structure

content/
├── images/ # All downloaded images (flat, hashed names)
│ ├── a1b2c3d4.png
│ └── ...
└── uncategorized/ # All emails go here initially (categorize later)
└── 2024-08-26-fod64-golden-age-for-indie-devs.md

## Frontmatter (Minimal)

---

title: "Golden Age for Indie Devs and Engineers"
date: 2024-08-26
original*file: "FOD#64* Golden Age for Indie Devs and Engineers.eml"

---

Step 3: Build and Test

yes lets build it, and test with a few recent emails to see how its working

Claude builds convert_email.py and tests with 3 emails. Results:

Step 4: Iterate — Fix Edge Cases

The initial converter has issues. Titles get mangled (FOD#64 stripped off) and links show beehiiv tracking URLs instead of real ones.

its looking good, but I notice that the subject link is getting mangled. also, the links are weird beehiiv links — can we get the real ones?

Claude fixes both issues. But the link resolution rate is only ~9-15% because beehiiv uses Cloudflare protection.

Step 5: The Discovery That Changes Everything

can you show me the plain text of the emails?

Claude examines the plain text and makes a crucial discovery:

The plain text is basically already markdown! It has:

• Real URLs in markdown link format: [text](url)
• Headers with ##
• Bold with text
• Bullet lists with *
• Image references with View image: (url) and Caption:

This simplifies everything. Instead of parsing HTML, cleaning with BeautifulSoup, and converting with html2text, the script can use the plain text directly. Real URLs, no tracking links, clean formatting.

test a few more to make sure that the images and everything is included

After confirming no information is lost with the plain text approach, Claude refactors the converter. Dependencies drop from 3 (html2text, beautifulsoup4, requests) to 1 (requests). The code is significantly simpler.

Step 6: Process Everything

ok, finish converting all the emails

Final Results:

- 209 markdown files created in content/uncategorized/
- 710 images downloaded to content/images/
- 95MB total content size

All files have:

- Full titles preserved (FOD#64:, AI 101:, 🦸🏻#5:, etc.)
- Real URLs (no beehiiv tracking links)
- Local image references
- Clean markdown format

The Key Insight

If we had jumped straight to implementation, we would have:

1. Parsed HTML (complex, error-prone)
2. Added BeautifulSoup dependency (big library for simple cleanup)
3. Added html2text dependency (GPLv3 licensing concerns)
4. Fought with tracking link resolution (~9-15% success rate)
5. Written significantly more code

By letting the agent examine the data first, we discovered that the plain text was already markdown. This is the Discovery before Implementation pattern: the AI agent found a simpler path by looking at the actual files rather than following assumptions.

This is what AI agents are good at: not just writing code, but examining your specific situation and finding simplifications a human would miss in the rush to implementation.

The Code Patterns

The final converter uses Python’s stdlib email module:

from email import policy
from email.parser import BytesParser
from pathlib import Path

# Parse an EML file
with open(Path('emails/file.eml'), 'rb') as fp:
    msg = BytesParser(policy=policy.default).parse(fp)

subject = msg.get('subject', '')
# Get plain text body (already markdown)
text_part = msg.get_body(preferencelist=('plain',))
text_content = text_part.get_content() if text_part else None

For image deduplication, hash the URL to create unique filenames:

import hashlib
from urllib.parse import urlparse

def image_filename(url: str) -> str:
    url_hash = hashlib.md5(url.encode()).hexdigest()[:12]
    ext = Path(urlparse(url).path).suffix or '.png'
    return f"{url_hash}{ext}"

Next: Image Extraction — How to download and deduplicate 710 images.