Find and fix a bug

Think of two ways to get a leaking tap fixed. One plumber shows you what's wrong and asks before touching anything. The other is left alone with a rule: keep trying until the leak stops — but give up after three tries and call you.

Both agents here do the same job: find the bug, fix it, and check the fix worked. The controlled one is a recipe card like every other agent on this site. The autonomous one is different: it's a small real program that runs by itself, so it never asks you anything. Its only brake is a limit on how many times it may try.

In plain words: a test is a tiny program that checks another program gives the right answer — here, that a spreadsheet file (CSV) starts with the column names Name,Email,ID. The test fails because the program forgot the ID column. "Passing" means the test is happy.

One bug, two ways

Controlled stops twice: before it changes any code, and before it calls the bug fixed. A passing test isn't the same as you agreeing it's fixed.

Autonomous never stops to ask. It runs the test, has the AI make the smallest fix, and runs the test again — until the test passes or it has used up its tries.

The lesson: a program that never asks needs a hard limit — and someone should still look at what it changed. In an earlier try, when we gave it a test that could never pass, this program made it "pass" by changing the test instead of the code.

What we gave both agents

The same small broken program, in a folder of its own: export.py builds the column names, and test_export.py checks them.

export.py
HEADER_FIELDS = ["Name", "Email"]

def build_csv_header():
    return ",".join(HEADER_FIELDS)

test_export.py
import unittest
from export import build_csv_header

class TestExport(unittest.TestCase):
    def test_csv_header(self):
        self.assertEqual(build_csv_header(), "Name,Email,ID")

if __name__ == "__main__":
    unittest.main()

FAILED tests/test_export.py::test_csv_header
AssertionError: assert 'Name,Email' == 'Name,Email,ID'

What the real agents did

Two real runs, one per agent. Each time the controlled agent stopped to ask, our reply is shown too. Both end with export.py as the agent left it.

Reading the program's printout: attempt 1 ran the test, and it failed (the Name,Email vs Name,Email,ID lines). The program then asked the AI for the smallest fix. Attempt 2 ran the test again, and it passed — so the program stopped, with 1 of its 3 tries left unused.