Q&A-1A bug report says refunds on partially shipped orders are sometimes one cent short in a 400,000-line Python service you have not read, and an AI assistant is open beside you. What do you do before asking it for a fix, and what does that step buy you?
Answer
I would not ask for a fix yet. First I would make the symptom happen on demand: the exact input, the observed output, the expected output from the ledger or the spec, and whether it reproduces locally or only with production data. Then I would write one test that asserts the expected output and run it to watch it fail for the reported reason.
The mechanism matters here. An assistant that has read only part of the repo will produce a patch that looks right, because a fluent explanation and a clean diff cost it nothing. Without an external check, my only way to judge the patch is to read it and trust it. The failing test changes that: the patch is accepted only if the test goes green and the existing suite stays green. The model's confidence stops being evidence.
Two approaches, compared on their main trade-off. In the first, I give the assistant the traceback and the function and ask for a fix. That is faster and sometimes right, but I end up reviewing a diff against my own guess about intent, and a special-cased fixture or a swallowed exception can pass that review. In the second, I write the test first. It costs ten to thirty minutes in a codebase I do not know, but the expected value is now fixed outside the conversation, and any patch that touches three unrelated functions to make a symptom vanish is visible in the diff and the suite.
Hypothetical figures for illustration: an order of three lines at 19.99 each should refund 59.97, and the report says 59.96. The test asserts 59.97 with the value taken from the ledger export, not from the function's current output. I then ask the assistant to explain the cause before it edits anything, and I check that explanation against the code.
What the test buys: a stopping condition, a regression guard that stays in the suite, and a way to reject a wrong patch. The failure mode to watch is a test that passes for the wrong reason, such as one that only checks that some exception is raised. I detect that by breaking the code under test on purpose and confirming the test turns red.
Curated: · Written: · Reviewed: