---
title: "Run Your First Eval"
description: "One command, fifty real examples, and the truth about which model works for YOUR task."
order: 35
duration: "15 min"
chapter: "09-running-agents-in-production"
type: lesson
---

## See how models actually perform on your tasks

Install umwelten:

```bash
npm install -g umwelten
```

Now run the car wash test — a deceptively simple eval that most models fail:

```bash
umwelten eval run \
  --prompt "Should I walk or drive to the car wash?" \
  --models "claude-sonnet-5,gpt-5.5,gemini-3.5-pro" \
  --judge claude-sonnet-5 \
  --output results.json
```

The correct answer requires common sense — you drive to the car wash because your car needs to be there. 76% of models get it wrong. Total cost of running 131 models through this eval: $0.46.

You just ran your first eval. It took one command, cost less than a dollar, and revealed something real about model behavior that no benchmark would tell you.

<div class="exercise">
  <div class="callout-label">Try This</div>
  <p>Install umwelten and run the car wash eval. Look at the results — which models got it right? Which got it wrong? Now change the prompt to something from YOUR work. Cornwall Market: <code>umwelten eval run --prompt "An invoice from Chen's says flour and sugar. Which account code?" --models "claude-sonnet-5,gpt-5.5,gemini-3.5-pro"</code>. Without the categorization skill, models have to guess. With different system prompts, they guess differently. This is why evals matter — you need to test with your tasks, not benchmarks.</p>
</div>

## Your tasks, your tools, your cost threshold

Benchmarks tell you how a model performs on academic tasks. Evaluations tell you how it performs on *your* tasks, with *your* tools, at *your* cost threshold.

<div class="callout">
  <div class="callout-label">The MCP Tool Eval</div>
  <p>We ran 39 models against real Rivian vehicle data via MCP tools. The models had to call the right tools, parse the responses, and answer questions about real charging sessions and drive data. The standout: <strong>Mercury-2</strong>, a diffusion model, scored a perfect <strong>15/15</strong> in <strong>8.6 seconds</strong> at a cost of <strong>$0.006</strong>.</p>
</div>

Cornwall Market: eval the categorization task across models. Give each model a set of 50 real invoices and the categorization skill. Score them on accuracy.

| Model | Score | Notes |
|-------|-------|-------|
| Claude Sonnet 5 | 47/50 | Missed 2 edge cases, 1 vendor name variation |
| GPT-5.5 | 41/50 | Consistent errors on Pacific Foods splits |
| Gemini 3.5 Pro | 44/50 | Strong on splits, weak on Q4 rules |
| Local model (Qwen, on the office Mac) | 28/50 | Cannot reliably handle multi-department splits |

The numbers are not the point. The point is that *you know* — concretely, with real data — which model works for your task. No benchmark can tell you that. Only an eval with your data, your rules, your edge cases.

### Same weights, different results

A finding that changes how you think about model selection:

<div class="callout">
  <div class="callout-label">Same Weights, Different Results</div>
  <p>We tested the same Nemotron model hosted on four different providers. The task was simple: use MCP tools to answer questions about vehicle data. <strong>Tool use success ranged from 1/6 to 6/6</strong> — same model weights, same prompt, same tools. The difference was entirely in the provider's infrastructure: how they handle tool call formatting, token limits, and response parsing.</p>
</div>

Infrastructure matters as much as the model. The provider you choose changes the behavior you get, even with identical weights. This is why evals must test the full stack — model + provider + tools — not just the model in isolation.

## Check your understanding

Answer in your own words — write it down before opening the key. Your tutor grades against the criteria and generates fresh variants on retries.

**Q1.** Why do public benchmarks fail to predict whether a model will work for YOUR task, and what three things must an eval hold constant with production to be trustworthy?

**Q2.** The same model weights scored 1/6 on one provider and 6/6 on another for tool use. What does this imply about how you must evaluate and choose infrastructure?

<details>
<summary>Answer key — attempt every question first</summary>

## Answer key

### Q1

**Model answer:** Benchmarks measure academic tasks with generic tooling — your work has different data, different tools, different edge cases. A trustworthy eval holds constant: your actual tasks (real examples), your actual tools (the same MCP servers production uses), and your full stack including the provider. Fifty real invoices beat any leaderboard.

**Pass criteria:** benchmark-vs-your-task mismatch; the three constants: real tasks, real tools, full stack/provider

### Q2

**Model answer:** The behavior difference came entirely from provider infrastructure — tool-call formatting, token limits, response parsing — not the weights. Implication: evaluate model + provider as a unit; a model 'passing' on one host says nothing about another, so infrastructure choices need the same eval rigor as model choices.

**Pass criteria:** attributes variance to provider infrastructure; implication = test the combination, never the model in isolation

</details>


**Next:** [Habitats](/mastering-claude/09-running-agents-in-production/36-habitats/)