---
title: "Research → Build Converter"
description: Use the research agent to evaluate extraction approaches, build the converter, and discover that plain text emails are already markdown.
---

# Research → Build Converter

## The Pattern: Research, Plan, Build

Don't jump straight to code. The sequence matters:

1. **Research**: Ask the agent to evaluate approaches
2. **Plan**: Write a plan document that captures decisions
3. **Build**: Implement based on the plan
4. **Test**: Run on a few files, inspect output
5. **Iterate**: Refine based on real data

## Step 1: Research

Start a new Claude Code session and ask the research agent:

> do tech research on how we can extract the data in emails/ to an organized content directory in markdown

The agent exits for ~4 minutes, examines your project, and produces a detailed report at `reports/2025-11-30-eml-to-markdown-extraction.md`. The full report includes:

- **Recommendation**: Python with email stdlib + html2text
- **Rationale**: Zero dependencies for MIME parsing, automatic encoding handling, mature HTML conversion
- **Alternative analysis**: Node.js mailparser (better for streaming, overkill for batch), Deno + postal-mime (ecosystem less mature)
- **Category detection**: Regex patterns for 7+ newsletter series from subject lines
- **Complete code**: `parse_eml_file()`, `extract_category()`, `html_to_markdown()`, `process_all_newsletters()`
- **Caveats**: GPLv3 licensing for html2text, complex table handling, image extraction

## Step 2: Plan

Now write a plan:

> look through all of our research, and lets make a plan in plans/1-convert-emails.md. lets focus on simplicity, making sure we move the data over cleanly, and all of the images extracted. we should be able to do one email at a time for testing purposes. We will decorate the emails later with more metadata

The plan captures key decisions:

```markdown
# Plan: Convert Emails to Markdown

## Design Principles

1. **Use plain text version** - The plain text email is already markdown with real URLs
2. **Single-file processing** - Script takes one EML path, outputs one markdown file
3. **Images downloaded locally** - Store in content/images/ with unique names
4. **Minimal frontmatter** - Just enough to identify the source; metadata decoration later
5. **Clean content** - Strip email cruft (separators, follow prompts, forwarding notices)

## Output Structure

content/
├── images/ # All downloaded images (flat, hashed names)
│ ├── a1b2c3d4.png
│ └── ...
└── uncategorized/ # All emails go here initially (categorize later)
└── 2024-08-26-fod64-golden-age-for-indie-devs.md

## Frontmatter (Minimal)

---

title: "Golden Age for Indie Devs and Engineers"
date: 2024-08-26
original*file: "FOD#64* Golden Age for Indie Devs and Engineers.eml"

---
```

## Step 3: Build and Test

> yes lets build it, and test with a few recent emails to see how its working

Claude builds `convert_email.py` and tests with 3 emails. Results:

| Input         | Output                                                  | Images    |
| ------------- | ------------------------------------------------------- | --------- |
| FOD#64 email  | `2024-08-26-golden-age-for-indie-devs-and-engineers.md` | 45 images |
| AI 101 email  | `2025-11-26-what-is-continual-learning.md`              | —         |
| Agentic email | `2024-11-23-building-blocks-of-agentic-systems.md`      | —         |

## Step 4: Iterate — Fix Edge Cases

The initial converter has issues. Titles get mangled (`FOD#64` stripped off) and links show beehiiv tracking URLs instead of real ones.

> its looking good, but I notice that the subject link is getting mangled. also, the links are weird beehiiv links -- can we get the real ones?

Claude fixes both issues. But the link resolution rate is only ~9-15% because beehiiv uses Cloudflare protection.

## Step 5: The Discovery That Changes Everything

> can you show me the plain text of the emails?

Claude examines the plain text and makes a crucial discovery:

> The plain text is basically already markdown! It has:
>
> - Real URLs in markdown link format: `[text](url)`
> - Headers with ##
> - Bold with **text**
> - Bullet lists with \*
> - Image references with View image: (url) and Caption:

**This simplifies everything.** Instead of parsing HTML, cleaning with BeautifulSoup, and converting with html2text, the script can use the plain text directly. Real URLs, no tracking links, clean formatting.

> test a few more to make sure that the images and everything is included

After confirming no information is lost with the plain text approach, Claude refactors the converter. Dependencies drop from 3 (html2text, beautifulsoup4, requests) to 1 (requests). The code is significantly simpler.

## Step 6: Process Everything

> ok, finish converting all the emails

```markdown
Final Results:

- 209 markdown files created in content/uncategorized/
- 710 images downloaded to content/images/
- 95MB total content size

All files have:

- Full titles preserved (FOD#64:, AI 101:, 🦸🏻#5:, etc.)
- Real URLs (no beehiiv tracking links)
- Local image references
- Clean markdown format
```

## The Key Insight

If we had jumped straight to implementation, we would have:

1. Parsed HTML (complex, error-prone)
2. Added BeautifulSoup dependency (big library for simple cleanup)
3. Added html2text dependency (GPLv3 licensing concerns)
4. Fought with tracking link resolution (~9-15% success rate)
5. Written significantly more code

By letting the agent examine the _data first_, we discovered that the plain text was already markdown. This is the **Discovery before Implementation** pattern: the AI agent found a simpler path by looking at the actual files rather than following assumptions.

**This is what AI agents are good at**: not just writing code, but examining your specific situation and finding simplifications a human would miss in the rush to implementation.

## The Code Patterns

The final converter uses Python's stdlib `email` module:

```python
from email import policy
from email.parser import BytesParser
from pathlib import Path

# Parse an EML file
with open(Path('emails/file.eml'), 'rb') as fp:
    msg = BytesParser(policy=policy.default).parse(fp)

subject = msg.get('subject', '')
# Get plain text body (already markdown)
text_part = msg.get_body(preferencelist=('plain',))
text_content = text_part.get_content() if text_part else None
```

For image deduplication, hash the URL to create unique filenames:

```python
import hashlib
from urllib.parse import urlparse

def image_filename(url: str) -> str:
    url_hash = hashlib.md5(url.encode()).hexdigest()[:12]
    ext = Path(urlparse(url).path).suffix or '.png'
    return f"{url_hash}{ext}"
```

---

**Next: [Image Extraction](/content-repurpose/02-extraction/05-image-download/)** — How to download and deduplicate 710 images.