Taxonomy Design

The Pattern: Let AI Analyze Your Data

Before you can classify content, you need to understand what’s in it. The traditional approach: manually read through files, take notes, identify patterns, design categories. With 209 files, that’s days of work.

Instead, let the AI agent read everything and propose a taxonomy.

The Prompt

look through all the emails in @content/uncategorized. We want to build a taxonomy of the data, where we identify high level series, propose categories and topics, and what other metadata we want to consistently extract from the data. Propose a content schema that we could apply to all of the emails

The @ symbol brings the entire directory into context. The agent reads all 209 files and identifies patterns.

The Agent’s Analysis

Based on reading every file, the agent identifies:

Series Patterns

Series	Title Pattern	Count	Description
FOD (Findings of the Day)	FOD#N:	~99	Weekly digest of AI news and papers
Topic Deep Dives	Topic N:	~4	Technical explainers on specific concepts
Agentic Series	🦸🏻#N:	~18	Agent and reasoning system deep dives
SF Insights	🌁#N:	~20	San Francisco-based AI insights
Concept Cards	Concepts:	~5	Short explainers of AI concepts
Podcast/Interviews	🎙️	~20	Conversations with AI practitioners
Unicorn Profiles	Company structure	~10	Deep-dive company profiles

Content Types

Type	Description	Example
digest	Weekly roundup with multiple sections	FOD emails
explainer	Technical deep-dive on a concept	Topic emails
article	General article or essay	Standalone pieces
interview	Q&A or conversation format	Podcast transcripts
profile	Company or person profile	Unicorn series
curated	Lists, recommendations, roundups	Book recommendations
announcement	Newsletter meta-content	Welcome messages
sponsored	Partner/sponsored content	Webinar promotions
forwarded	Forwarded email (not original)	Fwd: prefix
survey	Reader survey or feedback request	Input requests

Topic Categories

The agent identifies primary topic areas:

Value	Description	Common Keywords
llm	Language models, architectures, training	transformer, GPT, attention
agents	AI agents, agentic workflows, multi-agent	agent, workflow, tool use
infrastructure	ML ops, training, optimization	FSDP, inference, GPU, scaling
research	Academic papers, benchmarks, methods	paper, benchmark, SOTA
industry	Company news, funding, launches	unicorn, funding, valuation
business	Enterprise AI, productivity	enterprise, ROI, adoption
ethics	AI safety, policy, regulation	safety, alignment, bias
hardware	Chips, robotics, physical AI	GPU, TPU, robot, chip
multimodal	Vision, audio, video AI	image, video, vision

Refining with Human Judgment

we should also add to this the list of people mentioned. write out the report, focusing on describing the content schema which will be used to populate the frontmatter of the emails, so make sure it’s in a form thats good for that

You steer the agent to add people tracking. The final schema includes:

people_mentioned:
  - name: "string" # Full name
    role: "string" # Their role/title (optional)
    affiliation: "string" # Company/institution (optional)

The report is saved to reports/2025-12-01-content-taxonomy-schema.md. It includes:

1. Series identification patterns — Regex and emoji detection
2. Complete frontmatter schema — All fields with types and allowed values
3. People tracking — name/role/affiliation structure
4. Four detailed examples — FOD digest, Topic explainer, Unicorn profile, Podcast interview
5. Notable people reference — Frequently mentioned individuals for entity recognition
6. Exclusion criteria — What to filter out (forwarded emails, surveys)
7. Implementation notes — Regex hints for automated extraction

The Complete Schema

Required Fields

Field	Type	Values
title	string	Clean article title
date	YYYY-MM-DD	Publication date
series	string or null	fod, topic, agentic, insights, concepts, podcast, unicorn, guestpost, or null
content_type	string	article, digest, explainer, interview, profile, curated, announcement, sponsored, forwarded, survey

Categorization

Field	Type	Values
primary_topic	string	llm, agents, infrastructure, research, industry, business, ethics, hardware, multimodal
tags	array of strings	Specific technologies, techniques, or concepts
episode	integer	Episode number (for numbered series)

People & Organizations

Field	Type
people_mentioned	Array of {name, role?, affiliation?}
companies_mentioned	Array of strings
models_mentioned	Array of strings

Content Characteristics

Field	Type	Values
audience	string	technical, general, business, mixed
depth	string	overview, intermediate, deep-dive
has_research_papers	boolean
has_code_examples	boolean
is_premium	boolean

Why Design the Schema First

This is a critical pattern: define your output before you build the pipeline. The schema serves as:

1. A contract between the extraction phase and classification phase
2. A target for the classification agent to aim at
3. A validation reference for catching errors
4. A discovery tool — the act of designing the schema reveals what’s in your data

Without a schema, the classification agent would produce inconsistent output. With it, every file follows the same structure, making downstream processing (search, filtering, cross-linking) reliable.

The Meta-Pattern

The taxonomy report itself is an example of a compounding pattern:

1. Agent reads all content → identifies patterns
2. Human reviews → adds people tracking
3. Refined schema → written to reports/
4. Schema report → becomes context for classification agent
5. Classification agent → applies schema to every file

Each step builds on the last. The schema report isn’t just documentation — it’s machine-readable context that future agents consume.

---

Next: Parallel Classification — Create the classifier and process 50 files at once.