07 — The Self-Improving Loop

The Big Idea

The prompt runner is useful, but it’s still manual. You write a prompt, you run it, you look at the output, you tweak the prompt. What if we could close the loop — feed the errors back automatically?

This is the core insight of agentic AI: the model can improve its own output when given feedback.

How It Works

The self-improving loop has a “rerun” mode. Instead of just running prompt.txt fresh, it:

1. Takes the original prompt
2. Takes the failed test output
3. Takes the generated code
4. Combines them with a human-readable error message
5. Feeds everything back to Ollama
6. Gets improved code

The key is that a human provides the error message, but the model sees the full context: what we asked for, what it generated, and what went wrong.

The Script

This is run_prompt.sh with rerun support added (generated by our 3-prompt exercise):

#!/bin/bash

DIR=$1
MODE="regular"
MESSAGE=""

if [ -z "$DIR" ]; then
    echo "Usage: $0 <directory> [--rerun <message>]"
    exit 1
fi

if [ "$2" == "--rerun" ]; then
    MODE="rerun"
    MESSAGE="$3"
fi

if [ "$MODE" == "rerun" ]; then
    ./1-combind/code \
        -m "$MESSAGE" \
        "$DIR/prompt.md" "$DIR/test-output.md" "$DIR/code" | \
        ollama run gemma4:26b --nowordwrap | \
        tee "$DIR/ollama-output.md"
else
    cat "$DIR/prompt.md" | ollama run gemma4:26b --nowordwrap | tee "$DIR/ollama-output.md"
fi

./2-extract/code "$DIR"

if [ -f "$DIR/test" ]; then
    chmod +x "$DIR/test"
    "$DIR/test" 2>&1 | tee test-output.md
fi

The Loop in Action

Here’s a real session from the author’s walkthrough:

Run 1: It Fails

./run_prompt.sh 2-runner

Everything looks good, but when the test runs:

regression_test.sh: line 48: the: command not found

The generated code has weird artifacts — garbage characters at the end of some lines. It’s not clean.

Feedback

The author copies the error and the generated files back into the chat:

when i ran this with the runner itself, I got these results and it didn't work.
python3 runs/run_002/generated_code_001.py
  File "...generated_code_001.py", line 42
    with open(prompt_path, "w", encoding="cap=utf-8") as f:
                                               ^
SyntaxError: unterminated string literal (detected at line 42)

The Model Fixes It

The model recognizes specific failure modes:

• encoding="cap=utf-8" — it invented a bad argument (should be encoding="utf-8")
• Garbage characters — likely from not using re.DOTALL in regex

It produces an updated specification (plan2.md) with explicit constraints:

• “No Syntax Hallucinations: Ensure all Python syntax is standard”
• “Ensure open() arguments are correctly formatted”
• “No Code Fragmentation”

Run 2: Better, But Still Not Perfect

The improved prompt produces better code, but there are still issues. More feedback, more improvement. This is the evolution we traced in Spell #3: plan1 → plan2 → plan3 → plan4.

The Master Prompt (plan4.md)

After several iterations, here’s the final, bulletproof prompt that consistently produces working code. This is what you’d use as your “one-shot” for generating the Prompt Iterancy Tracker:

Master Prompt: Prompt Iterancy Tracker (PIINT)

Role: You are an expert Python Developer specializing in automation, CLI tools, and mission-critical code integrity. Your goal is to write code that is syntactically perfect, follows PEP 8 standards, and handles all edge cases without fragmentation or syntax errors.

Objective: Build a “Prompt Iterancy Tracker” (MVP version) that automates the process of running prompts through a local Ollama instance, extracting generated code, and versioning the results in a structured local directory.

Technical Requirements:

1. Robust Input Handling:

   • Existence Check: Check if prompt.txt exists in the root directory.
   • Auto-Creation: If prompt.txt is missing, automatically create it with the default text: "Write a python script that prints 'Hello World'".
   • Encoding Standard: Every open() function call must explicitly include encoding="utf-8".

2. Core Workflow & API Integration:

   • Library: Use requests.
   • Endpoint: Target http://localhost:11434/api/generate using the model gemma4:26b.
   • Payload: Set "stream": False.
   • Error Handling: Wrap the API call in a try-except block. If the connection fails, print "Error: Is Ollama running?" and exit via sys.exit(1).

3. Versioning & File Management:

   • Directory Structure: Create a runs/ directory. Inside, create uniquely numbered subdirectories (e.g., run_001/, run_002/).
   • Auto-Increment Logic: Scan the runs/ folder, identify the highest existing number in run_XXX, and increment it by 1.
   • Data Preservation: Inside each run_XXX/ folder, save:
     1. A copy of prompt.txt used.
     2. The full_response.txt (raw text).
     3. All extracted code files.

4. Code Extraction (Regex):

   • Regex Engine: Use re.findall with re.DOTALL. The pattern must target: r"```(\w*)\n(.*?)\n```".
   • Language Mapping: Use a dictionary to map Markdown tags to extensions (e.g., python → .py). Default to .txt if unknown.
   • Multi-Block Support: Extract and save every code block found as a separate numbered file (e.g., generated_code_001.py).

DEFENSIVE CODING & ANTI-HALLUCINATION (CRITICAL):

1. Zero-Tolerance for Syntax Hallucination:
   • Comma Integrity: Double-check every dictionary ({}) and function call for missing commas.
   • Argument Accuracy: Use only standard Python arguments. (e.g., open(file, "w", encoding="utf-8"). Do not invent arguments like cap=utf-8).
   • OS Module Standard: Use only standard os.makedirs arguments (e.g., exist_ok=True).
2. Variable Consistency: Ensure all constant names (e.g., MODEL_NAME) are used with identical casing throughout the entire script. No Model_Name or model_name variations.
3. No Complex Syntax: Avoid “clever” Python features like the Walrus operator (:=) or complex one-line f-string logic. Write clear, multi-line, readable code.
4. No Fragmentation: Deliver the entire logic as one single, cohesive, runnable .py file.

Deliverables:

1. The Python Script: A single, clean, well-commented .py file.
2. The Setup Guide: Instructions for venv creation, dependency installation (pip install requests), and execution.

The Key Insight

The self-improving loop works because:

1. The error is concrete. Not “it’s broken,” but “line 42 has encoding='cap=utf-8' which is invalid Python.”
2. The model has full context. It sees the original prompt, the code it generated, and the exact error.
3. Constraints accumulate. Each iteration adds explicit rules that prevent past failures. The prompt gets longer but more reliable.
4. The human is still in the loop. You’re not asking the model to evaluate itself — you’re giving it specific, factual feedback.

What You’ve Learned

• The rerun pattern feeds errors back to the model with full context
• The master prompt (plan4) is the result of 4 rounds of iterative improvement
• Defensive constraints (anti-hallucination rules) prevent specific failure modes
• A good error message is the difference between “try again” and “fix this exact thing”

Next: 08 — Code Extraction →