Image Download & Deduplication

The Image Problem

Newsletter emails contain images hosted on CDNs like beehiiv. These external URLs:

• Will break if the CDN changes or the newsletter service shuts down
• Leak analytics — opening the image tells the sender you read the email
• Are slow — requiring network requests to render local content
• Contain duplicates — the same logo or icon appears across multiple emails

The solution: download all images locally, deduplicate by content hash, and replace remote URLs with local paths.

Image Download Strategy

Filename Generation

Use MD5 hash of the URL to generate unique, deterministic filenames:

import hashlib
from urllib.parse import urlparse
from pathlib import Path

def image_filename(url: str) -> str:
    """Generate unique filename from URL hash + extension.

    Same URL → same filename (deduplication)
    Different URLs → different filenames (no collisions)
    Reproducible → re-running produces same filenames
    """
    url_hash = hashlib.md5(url.encode()).hexdigest()[:12]
    ext = Path(urlparse(url).path).suffix or '.png'
    return f"{url_hash}{ext}"

This approach means the same image appearing across 10 emails is downloaded once and shared.

Skip List

Not everything that looks like an image should be downloaded:

def should_download(url: str) -> bool:
    """Determine if an image URL should be downloaded."""
    # Skip data URIs (embedded content, not downloadable)
    if url.startswith('data:'):
        return False

    # Skip tracking pixels
    # These are usually 1x1 transparent GIFs used for open tracking
    if 'track' in url.lower() or 'pixel' in url.lower():
        return False

    return True

Download with Caching

import requests
from pathlib import Path

def download_image(url: str, images_dir: Path) -> str | None:
    """Download an image and return the local filename, or None if skipped/failed."""
    if not should_download(url):
        return None

    filename = image_filename(url)
    local_path = images_dir / filename

    # Skip if already downloaded (idempotent)
    if local_path.exists():
        return filename

    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()

        # Check it's actually an image
        content_type = response.headers.get('content-type', '')
        if not content_type.startswith('image/'):
            return None

        local_path.write_bytes(response.content)
        return filename
    except Exception as e:
        print(f"Failed to download {url}: {e}")
        return None

Finding Images in Plain Text Emails

The plain text emails reference images in a specific format:

View image: (https://cdn.beehiiv.com/abc123.png)
Caption: Some caption text

The converter uses a regex to find and replace these:

import re

def extract_and_replace_images(text: str, images_dir: Path) -> str:
    """Find image references in text, download, and replace with local paths."""

    def replace_image(match):
        url = match.group(1)
        caption = match.group(2)

        filename = download_image(url, images_dir)
        if filename:
            return f"![{caption}](../images/{filename})\n*{caption}*"
        else:
            return f"[Image: {caption}]({url})\n*{caption}*"

    # Pattern: View image: (URL) followed by optional Caption: text
    pattern = r'View image:\s*\(([^)]+)\)\s*(?:\nCaption:\s*(.+))?'

    return re.sub(pattern, replace_image, text)

The Complete Image Pipeline

EML file
    ↓
Extract plain text
    ↓
Find "View image: (URL)" patterns
    ↓
For each URL:
    ├── Already downloaded? → use existing local path
    ├── Tracking pixel? → skip
    ├── Data URI? → skip
    └── New image → download → save with hashed filename
    ↓
Replace URLs with ../images/hash.png references
    ↓
Write markdown file

Results

After processing all 218 emails:

• 710 images downloaded to content/images/
• 95MB total storage
• No duplicates — same image across multiple emails stored once
• All references converted from remote URLs to relative paths (../images/hash.png)

Edge Cases Handled

Why This Matters for the Pattern

Image extraction demonstrates a principle that recurs throughout the project: handle the messy reality of data, not just the happy path. Real emails have tracking pixels, CDN URLs, duplicate images, and network failures. The agent handles all of these because you described what you wanted (“make sure all images are extracted”) and trusted it to work through the details.

Next: Phase 3: Classification — Analyze content, build a taxonomy, and classify at scale with parallel agents.