--- title: "Production Review" description: "Cumulative review of Chapters 09-10: evals, habitats, and the compound system." type: review order: 4 --- # Production Review A cumulative review of Chapters 09-10 (Running Agents in Production, The Compound System). Ten questions, drawn from across the chapters and deliberately mixed together — interleaved practice is harder than chapter-by-chapter review, and that difficulty is exactly what builds retention. **How to use this review:** - **Pass bar: 9 of 10.** Higher than lesson quizzes, because this is where knowledge proves it stuck. - **Answer in writing, in your own words**, before checking anything. Constructing the answer is the exercise. - **With a tutor:** it will run the review one question at a time, grade against the key, and generate fresh variants for any retake. - **If you miss the bar — or barely clear it — come back tomorrow, not in five minutes.** Spaced retakes are dramatically better for retention than immediate ones. The gap is a feature. ## Check your understanding **Q1.** Your team shortlists a model because it tops a public leaderboard. Make the case for why that is insufficient, and specify the eval you would run instead — including the three things it must share with production. **Q2.** You switch inference providers to save money; the model is identical. What risk did you just take, and what is the two-hour check that de-risks it? **Q3.** Defend 'an agent is a git repo' to a skeptical engineer: what is IN the repo, what does git give you for free, and which agent capability is reckless without it? **Q4.** Your habitat's eval score has plateaued at 44/50 for three cycles. Where do you look for the next improvement, and where do you explicitly NOT look first? **Q5.** Design the verification protocol for self-modification: the agent updated its own rules from a correction yesterday — list the steps that prove today that the update is real, correct, and durable. **Q6.** Your inventory: 4 capabilities, 2 access methods. A teammate proposes building capability #5. Using the multiplication, argue the alternative investment and quantify both options. **Q7.** Take one real process and walk it up the full ladder: manual command → skill → routine → channel trigger → habitat. At each rung, state the trigger for promotion — the observation that tells you it has earned the next rung. **Q8.** Distinguish the two compounding effects — multiplication and flywheel — and explain why a system can have one without the other. What does each look like when missing? **Q9.** Sketch your compound system on five lines: trigger, data, knowledge, actions, feedback. Then identify which single line, if deleted, turns it back into 'a collection of automations' — and why. **Q10.** "The organization is better off even if we turn the AI off tomorrow." Steelman this claim with two concrete artifacts this course had you build, and the human-only value of each.
Answer key — attempt every question first ## Answer key ### Q1 **Model answer:** Leaderboards measure generic tasks with generic tooling; your work differs in data, tools, and edge cases. The eval: N real examples (e.g. 50 real invoices) with the skill and MCP servers production uses, scored against known answers. Shared with production: your tasks, your tools, your full stack including provider. **Pass criteria:** insufficiency argued via task/tool mismatch; eval described with real examples; the three constants named ### Q2 **Model answer:** Provider infrastructure changes behavior even with identical weights — tool-call formatting, token limits, parsing (observed range: 1/6 to 6/6 tool success on the same model). De-risk: re-run your standing eval suite against the new provider before cutover and compare scores; it costs cents and an afternoon. **Pass criteria:** names provider-infrastructure variance; the check is re-running the same eval on the new stack ### Q3 **Model answer:** In the repo: config, skills, memory files, tool definitions — everything the agent IS. Git gives version history, rollback, audit trails, and reviewable diffs with zero extra tooling. Self-modify is reckless without it: an agent editing its own instructions with no diff and no rollback is unauditable drift. **Pass criteria:** contents + at least two git dividends; self-modify tied to diff/rollback ### Q4 **Model answer:** Look at the misses: which examples fail and why — usually missing organizational knowledge (a vendor variation, an unstated exception) → fix the skill/rubric, or genuinely ambiguous cases that need a human-judgment rule. Do NOT start by swapping in a bigger model — improvement lives in knowledge, and model changes without eval evidence are superstition. **Pass criteria:** miss-analysis first; knowledge/skill fixes; explicit anti-pattern of reflexive model upgrades ### Q5 **Model answer:** Steps: (1) read the diff of the skill/memory files to see exactly what the agent wrote; (2) confirm it encodes the correction, not a garbled version; (3) start a FRESH session and pose a case that triggers the rule — confirm it applies; (4) optionally add the case to the eval set so durability is checked every cycle. **Pass criteria:** diff inspection + fresh-session behavioral test both present; bonus for eval-set addition ### Q6 **Model answer:** Current: 4×2 = 8 workflows. Capability #5: 5×2 = 10 (+2). Access method #3 (say, channels or schedules): 4×3 = 12 (+4) — and it upgrades every future capability too. When methods are the scarce dimension, grow methods. **Pass criteria:** both options computed; scarcer-dimension logic; forward-multiplying bonus of methods noted ### Q7 **Model answer:** Any real process passes. Promotion triggers must be observational: typed it more than twice → skill; checking/running it every morning → routine; there's an external event that should start it → channel; it needs to learn from corrections over time → habitat. **Pass criteria:** full ladder walked; each promotion justified by the observed usage pattern, not ambition ### Q8 **Model answer:** Multiplication is spatial: capabilities × access methods, more workflows from the same parts. Flywheel is temporal: outputs and corrections feed the next cycle, so quality compounds. Many surfaces + no feedback = broad but frozen (repeats mistakes everywhere). Learning + one surface = improving but narrow. Missing multiplication looks like underused capability; missing flywheel looks like day-one quality forever. **Pass criteria:** spatial/temporal distinction; independence shown; both missing-states characterized ### Q9 **Model answer:** Any coherent five-line design passes. The load-bearing line: feedback. Without it corrections never re-enter the system, quality freezes at launch, and what remains is scheduled scripts — automation without adaptation. (Accept a well-argued case for 'knowledge' as the fallback answer.) **Pass criteria:** five lines present; feedback identified with the frozen-quality reasoning ### Q10 **Model answer:** Examples with human-only value: the SKILL.md (a real training manual — onboarding shrinks, bus-factor drops), the dictated process documentation (consistency, coverage during absence), the eval set (a quality bar and regression test for the PROCESS itself), the artifact cheat-sheet (staff self-serve). Any two, each with its no-AI payoff. **Pass criteria:** two artifacts from the course; each with a concrete benefit that requires no AI