Chapter 07: Web Research

What We’re Building

Your agent can read local files, but it can’t access the web. We’ll add:

• URL to markdown: Download any webpage and convert it to clean markdown for the LLM
• Web search: Search the internet via the Tavily API
• Research agent: A specialized subagent that does deep research and writes reports

Step 1: Add URL Downloading

Paste this prompt:

Add a tool that uses turndown or similar to download a url and convert it to markdown

The Implementation

The agent will create src/tools/web.ts:

import TurndownService from "turndown";

export async function downloadUrl(url: string): Promise<string> {
  const response = await fetch(url, {
    headers: {
      "User-Agent":
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
        "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    },
  });

  if (!response.ok) {
    throw new Error(
      `Failed to download URL: ${url} (Status: ${response.status})`,
    );
  }

  const html = await response.text();
  const turndownService = new TurndownService();
  return turndownService.turndown(html);
}

This uses Turndown to convert HTML to markdown. Markdown is much more token-efficient for LLMs than raw HTML — typically 60–80% smaller while preserving all the content.

The tool definition:

{
  type: "function",
  function: {
    name: "download_url",
    description: "Download a URL and convert it to markdown",
    parameters: {
      type: "object",
      properties: {
        url: { type: "string", description: "URL to download" },
      },
      required: ["url"],
    },
  },
}

Step 2: Add Tavily Search

First, get a Tavily API key at tavily.com. Add it to your .env:

echo 'TAVILY_API_KEY=tvly-your-key-here' >> .env

Then paste this prompt:

Add tavily search as a tool

The Implementation

The agent will create src/tools/tavily.ts:

interface TavilyResult {
  title: string;
  url: string;
  content: string;
  raw_content?: string;
  score: number;
}

interface TavilyResponse {
  answer?: string;
  query: string;
  results: TavilyResult[];
  images?: string[];
}

export async function tavilySearch(query: string): Promise<string> {
  const apiKey = process.env.TAVILY_API_KEY;
  if (!apiKey) {
    throw new Error("TAVILY_API_KEY is not set");
  }

  const response = await fetch("https://api.tavily.com/search", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${apiKey}`,
    },
    body: JSON.stringify({
      query: query,
      search_depth: "advanced",
      include_answer: true,
      max_results: 5,
    }),
  });

  const data = (await response.json()) as TavilyResponse;
  return formatTavilyResponse(data);
}

The formatter creates a clear output:

### Search Results for "best practices for using tavily search"

**Answer:**
Tavily's advanced search depth returns more comprehensive results...

**Sources:**

1. [Using Tavily API Effectively](https://docs.tavily.com/...)
   > The advanced search depth enables deeper web crawling...

Step 3: Optimize with Self-Research

This is meta but effective — use your agent’s new search capability to improve the search tool:

use the tavily search engine tool to look up best practices on how to use tavily, and then update the tavily.ts search tool

The agent will:

1. Search for “Tavily API best practices”
2. Download the docs
3. Update the tool with improved parameters (search depth, result count, formatting)

Step 4: Create the Research Agent

Paste this prompt:

Create a tech-researcher prompt. It should be a specialized agent that uses web search and URL downloading to do deep research on a topic and write comprehensive reports. The reports should be saved to a reports/ directory with the date and topic in the filename.

The agent creates src/prompts/tech-researcher.md:

---
name: tech-researcher
description: Use this agent when the user asks to research a topic on the web...
tools:
  [
    list_files,
    read_file,
    search_files,
    download_url,
    write_file,
    tavily_search,
    ask_user,
  ]
---

You are a technology researcher that specializes in helping software get
developed by agents. YOU WILL ALWAYS DOUBLE CHECK YOUR ASSUMPTIONS AND
VERIFY WITH THE INTERNET, ESPECIALLY AROUND DATES AND VERSIONS

1. Understand the current date and project type
2. Ask clarifying questions ONE AT A TIME (3-4 questions)
3. Summarize what you're going to do and ask the user to confirm
4. Do extensive web searching
5. Choose the best overall choice
6. Comprehensively document findings for future LLMs

## Report structure

The report MUST BE WRITTEN TO THE REPORTS DIRECTORY in the form
`reports/YYYY-MM-DD-topic-researcher.md`

All sources must be referenced at the bottom.

The research agent follows a structured workflow:

1. Understand context — Current date, project stack, what’s needed
2. Clarify — Ask questions to narrow down the research scope
3. Search broadly — Tavily search, then download relevant pages
4. Synthesize — Compare options, pick the best approach
5. Document — Write a comprehensive report with frontmatter

The report format includes YAML frontmatter to make reports machine-readable:

---
title: "Accessing and Searching Claude Code Conversation History"
date: 2025-11-28
tags: [claude-code, history, search]
recommendation: "Use a custom /history command for day-to-day access..."
use_when:
  - "You need to find a specific conversation from the past"
  - "You want to resume a previous Claude Code session"
dont_use_when:
  - "You just need to continue your most recent session"
---

Step 5: Test the Research Tools

Start your agent and try:

download https://example.com and show me the content

search for "latest TypeScript 5.8 features"

Then test the research agent (you’ll need Chapter 09 for subagent support):

research best practices for building AI coding agents

Step 6: Verify the Model Configuration

Paste this prompt to make sure your OpenRouter setup is correct:

are we using openrouter and gemini correctly together?

The agent will verify:

• The model string format for OpenRouter
• That reasoning details are enabled
• That the pricing lookup is working correctly

Tool Summary

By the end of this chapter, you have these internet tools:

← Chapter 06 · Next: Part 3 →