Run a deterministic application suite and prove the gate catches a deliberate defect.

Chapter 03: Tests That Push Back

Objective

Run a deterministic application suite and prove the gate catches a deliberate defect.

Your tutor should stop at each gate and wait for your evidence before continuing.

Concept

A useful test makes the desired behavior observable without asking the agent whether it succeeded. Our task board stores dictionaries with a title and a completion flag. Whitespace normalization, title length, and pending-item ordering are small enough to test exhaustively at their boundaries.

The starter implements existing behavior. A later exercise adds task completion. Begin with six passing tests so you can separate a broken environment from a missing feature. A red test is useful only when you know why it is red.

devenv test runs the configured test entry point; it does not magically discover every test in your repository. We explicitly wire formatting, lint, and unit tests into it. The same command becomes the human, agent, and CI acceptance gate.

Concept check

Q1. Why deliberately introduce a bug before giving the project to an agent?

Q2. What happens if your test entry point only prints “passed”?

Concept answer key — attempt first

Answer key (concept)

Q1. It proves the check rejects an incorrect implementation. Pass criteria: explain a negative control.

Q2. The command can succeed without testing behavior. Pass criteria: exit status is meaningful only when backed by real checks.

Gate: Prepare source and tests

Copy taskboard.py, tests/, exercises/, and scripts/ from the archive into your project. Merge the archive’s .gitignore rules before generating Python bytecode or logs. Do not copy exercises/test_completion.py into tests/ yet.

Read the implementation:

"""A deliberately small task board for the Nix agent course."""


def normalize_title(value: str) -> str:
    if not isinstance(value, str):
        raise TypeError("title must be text")
    title = " ".join(value.split())
    if not title or len(title) > 80:
        raise ValueError("title must contain 1 to 80 characters")
    return title


def new_task(title: str) -> dict:
    return {"title": normalize_title(title), "done": False}


def pending_titles(tasks: list[dict]) -> list[str]:
    return [task["title"] for task in tasks if not task["done"]]

The test suite covers blank strings, an 80/81-character boundary, non-text input, new tasks, whitespace, and order. None of those checks uses an LLM or a network service.

Gate: Smoke-test the application

devenv shell python -m unittest discover -s tests -v

Pass: six tests, no skipped checks, exit status zero. Run this before adding services.

Gate: Make one acceptance command

Replace your initial devenv.nix with the archive version:

{ pkgs, ... }:
{
  packages = with pkgs; [ git jq ripgrep curl bashInteractive ruff ];
  languages.python.enable = true;

  tasks."check:format".exec = "ruff format --check taskboard.py tests exercises scripts";
  tasks."check:lint".exec = "ruff check taskboard.py tests exercises scripts";
  tasks."check:unit".exec = "python -m unittest discover -s tests -v";
  tasks."check:all".after = [ "check:format" "check:lint" "check:unit" ];

  enterTest = ''
    set -euo pipefail
    ruff format --check taskboard.py tests exercises scripts
    ruff check taskboard.py tests exercises scripts
    python -m unittest discover -s tests -v
  '';
}

devenv test

The task declarations are explained in Chapter 05. For now, enterTest is the authoritative sequential gate. set -euo pipefail prevents an earlier failing command from being hidden by a later successful command.

Gate: Prove red, then green

Change the > 80 title check to > 79. Run the unit suite and then devenv test. Both should fail at the boundary case. Restore > 80, run both again, and record the observed failure and recovery.

Commit this known-good baseline. Do not merely report that tests “probably catch it.” Show your tutor the failed assertion and passing rerun.

Gate: Improve the evidence

Add a test for a title containing Unicode characters. Decide whether length means Python characters, bytes, or user-perceived graphemes; document the chosen contract. This lab uses Python’s len, not display width. Review the test before running an agent.

Known failure modes

Zero tests is not a pass: verify the reported count.
Running from the wrong directory can cause ModuleNotFoundError; run from the lab root.
Formatting checks reject style but do not establish correctness.
A test that asserts the same implementation expression can share the same mistake. Assert a concrete outcome from a spec.

Check your understanding

Q3. What does the 80/81 pair establish?

Q4. Which test should catch changing the order of pending tasks?

Answer key — attempt every question first

Answer key

Q3. The exact inclusive length boundary. Pass criteria: both valid and invalid neighbors.

Q4. The pending-order test with distinct titles. Pass criteria: identify observable ordering, not implementation details.

References

← Chapter 02 · Next: Chapter 04 →