Running Agents in Production
Habitats, evals, and production agent infrastructure.
TUTOR WITH THEFOCUS.AI
Copy this prompt into Claude, ChatGPT, or any external AI assistant. It points the assistant to the course instructions and links it to your student profile to track your progress and customize observations.
You are not enrolled yet. Enroll to generate a Student ID to track lesson completions and store learning notes.
For teams building systems that learn: evals with your data, habitats with memory, corrections that persist. Four lessons, about 45 minutes total.
Lessons in this chapter
- Run Your First Eval — One command, fifty real examples, and the truth about which model works for YOUR task. (~15 min)
- Habitats — An agent is a git repo — config, skills, and memory, versioned. (~10 min)
- Corrections That Stick — Self-modifying agents, and how to verify a correction survived the night. (~10 min)
- From Eval to Production — The same system runs both — and the loop that makes it better. (~10 min)
If you take five things from this chapter
1. Eval with your tasks, not benchmarks. Fifty real examples and a judge beat any leaderboard — and it costs less than a dollar to find out.
2. Test the full stack. Same model weights behave differently across providers (1/6 vs 6/6 tool success). Eval model + provider + tools together.
3. An agent is a git repo. Config, skills, memory — versioned. When the agent modifies itself, the diff is your audit trail.
4. Enable self-modify only with version control underneath. Corrections should persist across sessions, and you should be able to see exactly what changed.
5. Run the loop: eval → correct → re-eval. Improvement comes from better organizational knowledge in better skills, not from bigger models.