From Eval to Production
The same system runs both — and the loop that makes it better.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
The same system runs both
Here is the insight that makes this different from most agent frameworks: the system that runs evaluations is the same system that runs production chat. Not a similar system. The same one.
| Mode | Models | Prompts | Tools | Output |
|---|---|---|---|---|
| Eval | Many (e.g., 39) | Fixed set, replayed | Real MCP servers | Scored results, comparison table |
| Chat | One (chosen) | Interactive, user-driven | Same MCP servers | Conversation, actions |
One MCP server. One tool set. One skill. In eval mode, the system replays prompts against many models and scores results. In chat mode, it talks to one model interactively. If the chat agent can do it, the eval tests it. If the eval reveals a problem, you have found a production bug.
Run an eval against your habitat's task. Cornwall Market: take 50 real invoices, run them through the categorization eval with the agent's current rules. Score: 47/50. Sarah corrects the 3 errors — two were vendor name variations the skill did not cover, one was a new supplier. She updates the rules. Re-eval: 49/50. The remaining miss is a genuinely ambiguous item that needs human judgment.
That loop — eval, identify errors, correct, re-eval — is how the system gets better. Not through bigger models. Through better organizational knowledge captured in better skills, tested by real evals.
This is how the fourth transition works in practice. The habitat runs in production — categorizing transactions, processing invoices, surfacing anomalies. The eval system measures whether it is getting better or worse. Expert corrections feed back into the habitat’s skills and memory. The eval system measures whether those corrections actually improved performance.
“The best test environment is production. The best test harness is the tool your users actually use. Build them as the same thing.”
Check your understanding
Answer in your own words — write it down before opening the key. Your tutor grades against the criteria and generates fresh variants on retries.
Q1. Describe one full turn of the eval → correct → re-eval loop for a task from your own work, and state where the improvement actually comes from.
Answer key — attempt every question first
Answer key
Q1
Model answer: Shape: run N real examples against the current rules and score them; examine the misses (a vendor-name variation, a missing exception); correct the skill, not the model; re-run the same eval and compare. The improvement comes from better organizational knowledge captured in the skill — the model never changed.
Pass criteria: all three loop stages concrete; locates improvement in knowledge/skill updates rather than model size
Next: Take Inventory