Chapter 12: Test the Operating System

Objective

Boot two NixOS machines, verify a service across their network, and finish with an evidence-based capstone.

Your tutor should stop at each gate and wait for your evidence before continuing.

Concept

Some contracts live below the application. Does a service start under systemd? Is the intended port reachable from another host? Did a firewall change block the client? NixOS tests let you describe machines and drive assertions against them.

This final lab deliberately changes the system under test. The Python task board is a library, so we use a tiny Nginx health endpoint to learn the VM framework without pretending we deployed a web application that does not exist. One server and one client establish a network contract. The earlier Python and database tests remain separate checks.

Huntley’s Nix demonstration also uses an overlay to change a dependency once and exercise it in multiple outputs. An overlay can replace a package definition, source, or patch. It is powerful dependency control, but patching one executable is not a complete security policy if another executable or remote API can perform the same operation.

Concept check

Q1. What can a two-node VM test reveal that a unit test cannot?

Q2. Why is this check Linux-only in our lab?

Gate: Verify the builder

Use an x86 Linux machine with working hardware virtualization/KVM access. Nested virtualization may need provider support; many ordinary VMs do not expose it. Run as the test user:

uname -sm
test -r /dev/kvm && test -w /dev/kvm && echo 'KVM accessible'
nix --version

Pass: x86_64 Linux, KVM accessible, and Nix works. If unavailable, select a compatible Linux builder. A Mac may submit builds to one, but the test still executes on Linux. The supplied flake intentionally has no Darwin check output.

Gate: Inspect the system definition

Copy nixos/ from the archive. Its flake is:

{
  description = "Linux-only two-machine contract test for the agent lab";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
    in {
      checks.${system}.web = pkgs.testers.runNixOSTest {
        name = "agent-lab-web";
        nodes.server = { ... }: {
          networking.firewall.allowedTCPPorts = [ 8080 ];
          services.nginx.enable = true;
          services.nginx.virtualHosts."lab" = {
            listen = [{ addr = "0.0.0.0"; port = 8080; }];
            locations."/health".return = "200 'ready'";
          };
        };
        nodes.client = { pkgs, ... }: {
          environment.systemPackages = [ pkgs.curl ];
        };
        testScript = ''
          start_all()
          server.wait_for_unit("nginx.service")
          server.wait_for_open_port(8080)
          client.wait_until_succeeds("curl --fail --max-time 10 http://server:8080/health | grep -Fx ready", timeout=60)
          client.fail("curl --fail --max-time 2 http://server:8081/health")
        '';
      };
    };
}

The stable branch is a bootstrap input; the generated nixos/flake.lock freezes the actual revision. The positive assertion retries for up to 60 seconds because a guest shell can be ready before its network path; it reaches port 8080; the negative one verifies no responding service on 8081. That negative assertion alone does not prove the firewall blocks a listening service.

Gate: Lock and run

git add nixos/flake.nix
nix flake lock ./nixos
git add nixos/flake.lock
nix build ./nixos#checks.x86_64-linux.web -L

Pass: both machines boot, Nginx starts, the client receives ready, and the wrong-port request fails. Preserve the build log and lock hash. Rerunning an unchanged successful derivation may substitute the result; change the test input when demonstrating a new failure.

Gate: Break the network contract

On a disposable branch, change allowedTCPPorts from [ 8080 ] to [ 8081 ] while leaving Nginx listening on 8080. Run the same build. The server-local readiness check may still succeed, but the client’s HTTP request should fail. Restore 8080 and rebuild.

This is a meaningful negative control for the firewall rule because the service continues to listen. If it does not fail, inspect the actual firewall configuration and network path before trusting the test.

Gate: Capstone evidence

Produce a reviewed learning-repository PR with:

1. A clean source commit, devenv lock, CLI revision record, and NixOS flake lock.
2. Six original unit tests plus completion acceptance tests and your extra-field test.
3. Repeated database checks with no leaked state.
4. Mac and Linux results for the same source and dependency lock.
5. One supervised agent attempt and one bounded VM experiment, with costs and stop reason.
6. A CI failure caused by a deliberate application defect, followed by a passing correction.
7. The two-node VM test’s red/green firewall experiment.
8. Evidence of credential revocation and VM teardown.

Explain each artifact to your tutor. Missing cloud or KVM access is a documented incomplete gate, not permission to claim the capstone passed.

Extension: One overlay, two consumers

Study ghuntley/nix-demo. Trace where its modified Git package is defined and where the development shell and VM consume it. Propose a harmless package patch of your own, write an assertion that fails before the patch, and test each supported consumer. Keep this optional until the unmodified lab passes.

Known failure modes

• Adding Linux to a flake output name does not supply a Linux builder.
• Insufficient disk or RAM can fail a VM build before your application assertion; classify the failure honestly.
• A local open-port check cannot verify remote firewall reachability by itself.
• A cached passing derivation represents that input graph, not an unrelated new deployment.

Check your understanding

Q3. Why keep both server readiness and a client request?

Q4. What is required before describing the whole course lab as validated?

References

• NixOS test framework
• NixOS test tutorial
• Huntley’s Nix demo

← Chapter 11 · Course overview →