Production Review

A cumulative review of Chapters 09-10 (Running Agents in Production, The Compound System). Ten questions, drawn from across the chapters and deliberately mixed together — interleaved practice is harder than chapter-by-chapter review, and that difficulty is exactly what builds retention.

How to use this review:

Pass bar: 9 of 10. Higher than lesson quizzes, because this is where knowledge proves it stuck.
Answer in writing, in your own words, before checking anything. Constructing the answer is the exercise.
With a tutor: it will run the review one question at a time, grade against the key, and generate fresh variants for any retake.
If you miss the bar — or barely clear it — come back tomorrow, not in five minutes. Spaced retakes are dramatically better for retention than immediate ones. The gap is a feature.

Check your understanding

Q1. Your team shortlists a model because it tops a public leaderboard. Make the case for why that is insufficient, and specify the eval you would run instead — including the three things it must share with production.

Q2. You switch inference providers to save money; the model is identical. What risk did you just take, and what is the two-hour check that de-risks it?

Q3. Defend ‘an agent is a git repo’ to a skeptical engineer: what is IN the repo, what does git give you for free, and which agent capability is reckless without it?

Q4. Your habitat’s eval score has plateaued at 44/50 for three cycles. Where do you look for the next improvement, and where do you explicitly NOT look first?

Q5. Design the verification protocol for self-modification: the agent updated its own rules from a correction yesterday — list the steps that prove today that the update is real, correct, and durable.

Q6. Your inventory: 4 capabilities, 2 access methods. A teammate proposes building capability #5. Using the multiplication, argue the alternative investment and quantify both options.

Q7. Take one real process and walk it up the full ladder: manual command → skill → routine → channel trigger → habitat. At each rung, state the trigger for promotion — the observation that tells you it has earned the next rung.

Q8. Distinguish the two compounding effects — multiplication and flywheel — and explain why a system can have one without the other. What does each look like when missing?

Q9. Sketch your compound system on five lines: trigger, data, knowledge, actions, feedback. Then identify which single line, if deleted, turns it back into ‘a collection of automations’ — and why.

Q10. “The organization is better off even if we turn the AI off tomorrow.” Steelman this claim with two concrete artifacts this course had you build, and the human-only value of each.

Answer key — attempt every question first

Answer key

Q1

Model answer: Leaderboards measure generic tasks with generic tooling; your work differs in data, tools, and edge cases. The eval: N real examples (e.g. 50 real invoices) with the skill and MCP servers production uses, scored against known answers. Shared with production: your tasks, your tools, your full stack including provider.

Pass criteria: insufficiency argued via task/tool mismatch; eval described with real examples; the three constants named

Q2

Model answer: Provider infrastructure changes behavior even with identical weights — tool-call formatting, token limits, parsing (observed range: 1/6 to 6/6 tool success on the same model). De-risk: re-run your standing eval suite against the new provider before cutover and compare scores; it costs cents and an afternoon.

Pass criteria: names provider-infrastructure variance; the check is re-running the same eval on the new stack

Q3

Model answer: In the repo: config, skills, memory files, tool definitions — everything the agent IS. Git gives version history, rollback, audit trails, and reviewable diffs with zero extra tooling. Self-modify is reckless without it: an agent editing its own instructions with no diff and no rollback is unauditable drift.

Pass criteria: contents + at least two git dividends; self-modify tied to diff/rollback

Q4

Model answer: Look at the misses: which examples fail and why — usually missing organizational knowledge (a vendor variation, an unstated exception) → fix the skill/rubric, or genuinely ambiguous cases that need a human-judgment rule. Do NOT start by swapping in a bigger model — improvement lives in knowledge, and model changes without eval evidence are superstition.

Pass criteria: miss-analysis first; knowledge/skill fixes; explicit anti-pattern of reflexive model upgrades

Q5

Model answer: Steps: (1) read the diff of the skill/memory files to see exactly what the agent wrote; (2) confirm it encodes the correction, not a garbled version; (3) start a FRESH session and pose a case that triggers the rule — confirm it applies; (4) optionally add the case to the eval set so durability is checked every cycle.

Pass criteria: diff inspection + fresh-session behavioral test both present; bonus for eval-set addition

Q6

Model answer: Current: 4×2 = 8 workflows. Capability #5: 5×2 = 10 (+2). Access method #3 (say, channels or schedules): 4×3 = 12 (+4) — and it upgrades every future capability too. When methods are the scarce dimension, grow methods.

Pass criteria: both options computed; scarcer-dimension logic; forward-multiplying bonus of methods noted

Q7

Model answer: Any real process passes. Promotion triggers must be observational: typed it more than twice → skill; checking/running it every morning → routine; there’s an external event that should start it → channel; it needs to learn from corrections over time → habitat.

Pass criteria: full ladder walked; each promotion justified by the observed usage pattern, not ambition

Q8

Model answer: Multiplication is spatial: capabilities × access methods, more workflows from the same parts. Flywheel is temporal: outputs and corrections feed the next cycle, so quality compounds. Many surfaces + no feedback = broad but frozen (repeats mistakes everywhere). Learning + one surface = improving but narrow. Missing multiplication looks like underused capability; missing flywheel looks like day-one quality forever.

Pass criteria: spatial/temporal distinction; independence shown; both missing-states characterized

Q9

Model answer: Any coherent five-line design passes. The load-bearing line: feedback. Without it corrections never re-enter the system, quality freezes at launch, and what remains is scheduled scripts — automation without adaptation. (Accept a well-argued case for ‘knowledge’ as the fallback answer.)

Pass criteria: five lines present; feedback identified with the frozen-quality reasoning

Q10

Model answer: Examples with human-only value: the SKILL.md (a real training manual — onboarding shrinks, bus-factor drops), the dictated process documentation (consistency, coverage during absence), the eval set (a quality bar and regression test for the PROCESS itself), the artifact cheat-sheet (staff self-serve). Any two, each with its no-AI payoff.

Pass criteria: two artifacts from the course; each with a concrete benefit that requires no AI