Skip to content
Tech Interview Prep
Technical interview guideNew

AI-Assisted Debugging in Legacy Codebases

How to reproduce, localise, diagnose and patch a bug in a large unfamiliar repository while directing an AI coding assistant, and how to show the interviewer that you verified every change. Covers the debugging round where the assistant is allowed for the whole session.

Read
11 min
Practice MCQs
24
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: Method is tool-independent and current to mainstream AI coding assistants and Python, Java, TypeScript and C# codebases as of 2026. Assistant-specific features (context limits, agent modes, tool permissions) change often and should be checked against the vendor's current documentation before the interview..

Interview Q&A

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1

A bug report says refunds on partially shipped orders are sometimes one cent short in a 400,000-line Python service you have not read, and an AI assistant is open beside you. What do you do before asking it for a fix, and what does that step buy you?

Answer

I would not ask for a fix yet. First I would make the symptom happen on demand: the exact input, the observed output, the expected output from the ledger or the spec, and whether it reproduces locally or only with production data. Then I would write one test that asserts the expected output and run it to watch it fail for the reported reason.

The mechanism matters here. An assistant that has read only part of the repo will produce a patch that looks right, because a fluent explanation and a clean diff cost it nothing. Without an external check, my only way to judge the patch is to read it and trust it. The failing test changes that: the patch is accepted only if the test goes green and the existing suite stays green. The model's confidence stops being evidence.

Two approaches, compared on their main trade-off. In the first, I give the assistant the traceback and the function and ask for a fix. That is faster and sometimes right, but I end up reviewing a diff against my own guess about intent, and a special-cased fixture or a swallowed exception can pass that review. In the second, I write the test first. It costs ten to thirty minutes in a codebase I do not know, but the expected value is now fixed outside the conversation, and any patch that touches three unrelated functions to make a symptom vanish is visible in the diff and the suite.

Hypothetical figures for illustration: an order of three lines at 19.99 each should refund 59.97, and the report says 59.96. The test asserts 59.97 with the value taken from the ledger export, not from the function's current output. I then ask the assistant to explain the cause before it edits anything, and I check that explanation against the code.

What the test buys: a stopping condition, a regression guard that stays in the suite, and a way to reject a wrong patch. The failure mode to watch is a test that passes for the wrong reason, such as one that only checks that some exception is raised. I detect that by breaking the code under test on purpose and confirming the test turns red.

Curated: · Written: · Reviewed:

Q&A-2

When an AI coding assistant is debugging a failure in a large legacy repository, what is the first artefact you give it, and why not the whole repo?

Answer

The first artefact is the exact error: the full stack trace or error text, copied verbatim, with the first frame that sits in your own code. A stack trace is the only thing in the report that was observed rather than guessed, and it names a file and line the assistant can read directly.

TypeError: Cannot read properties of undefined (reading 'amount')
    at applyDiscount (src/checkout/discount.js:41:28)
    at checkoutHandler (src/checkout/handler.js:17:12)

The failing test name goes with it, since it turns the trace into a pass or fail check you can run after each suggestion.

Why not the whole repo: a large legacy codebase has many near-identical helpers, duplicated modules and old and new code paths. Whole-repo context puts the relevant lines among a great deal of unrelated code, so the assistant has more plausible places to edit and no reason to pick the one the trace points to. It can then return a fix in the wrong copy of a helper, which reads well and fails the test.

So start with the verbatim error and the first project frame, add the failing test, then widen only to the files the trace names. Check each suggestion against the test before accepting it.

Curated: · Written: · Reviewed:

Q&A-3

Your team inherits a 60,000-line Python billing service. Walk me through how you use an AI assistant to map how a refund request flows from the Flask route to the ledger write, and how you check that map before you rely on it.

Answer

The 90-second answer. I would not ask for "the architecture". I would ask for one slice: the route handler, the service method it calls, the ledger interface, and the relevant settings module, all for one request type. The question is "list every call that can happen between post_refund and the ledger write, with file and line, the data each hop reads and writes in minor units, and mark each edge as direct, dynamic or unresolved." I treat the reply as a hypothesis with a status on every edge. Then I check the edges that carry money, using grep for names, the editor's find-references for static callers, and a breakpoint for one real request. The map is only trusted where a runtime witness exists.

$ rg -n "post_refund|process_refund|\.write\(" billing/
billing/api/refunds.py:41:    def post_refund(...)
billing/services/refunds.py:88:    def process_refund(...)
billing/services/refunds.py:131:        self.ledger.write(entry)

That trace already shows a gap: self.ledger is an attribute, so the write site is known but its concrete class is not. The assistant's claim that LedgerPostgres.write is called is an inference until something confirms it.

Where AI maps go wrong.

  1. Dependency injection. The concrete class is chosen in a container or config module at boot. Reading refunds.py shows only the interface. Check with breakpoint() in __init__, then print(type(self.ledger)).
  2. Dynamic dispatch. A subclass overrides write. The static edge points at the base class; the runtime call goes to the override. Find-references lists the definition, not the override that runs.
  3. Decorators. @audit or @retry wraps the function. The source shows the inner body, and the wrapper may swallow the exception or call the function twice. Inspect process_refund.__wrapped__ or the wrapper source.
  4. Reflection and string lookup. getattr(handler, f"handle_{kind}") or a registry dict means rg finds no call site at all. An empty grep result is not evidence of no caller.
  5. Generated code. Protobuf stubs or ORM models in _pb2.py or a build directory may be stale relative to their generator input. Check the generator and its timestamp before trusting field names.

Verification order. Grep confirms that a name exists. Find-references confirms static callers. A breakpoint at the ledger write, followed by where in pdb or the IDE call stack, confirms the path for one request. I run two requests on different branches, for example a full refund and a refund with a coupon, because one trace proves one path. Any edge the map marked dynamic or unresolved stays open until a trace hits it.

Trade-off. The assistant is fast at breadth and at reading unfamiliar code, but its edges are plausible rather than observed. A static slice can over-approximate (edges that never run) and under-approximate (edges created by injection). A breakpoint is exact and narrow. I use the assistant to propose the graph and a trace to confirm the parts that move money.

Curated: · Written: · Reviewed:

Q&A-4

An AI assistant says a monkey-patch in a legacy Python module is harmless because callers look up the patched function at call time. Is that right, and how would you check it?

Answer

The 90-second answer. The claim depends on how the caller imports the function, and that is a question about the source text, which I can read. import billing.tax followed by billing.tax.rate(...) looks the attribute up on every call, so a later reassignment is visible. from billing.tax import rate copies the function object into the caller's namespace when the import runs. A patch applied after that point never reaches the caller. So the assistant is right for one import style and wrong for the other, and I would check which style this module uses before accepting either version.

# billing/tax.py
def rate(region): return 0.2

# legacy/patches.py, imported from app/__init__.py
import billing.tax
billing.tax.rate = lambda region: 0.0

# billing/orders.py
from billing.tax import rate   # binds the original object at import time

I would run grep -n "import" billing/orders.py to find the style, then import billing.orders before and after legacy.patches in two small scripts and print rate('EU'). If the from-import is present and the module loads before the patch, the output is 0.2 in one order and 0.0 in the other. That result contradicts "harmless" for any order where the patch runs late.

Trade-off. The grep and the two-script check take a few minutes and cover the one caller I care about. Tracing every caller of the patched function across 400,000 lines would cost far more, and it only matters if the patched name is used in several modules with different import styles. I would start with the failing path and widen only if the probe shows the patch reaching it.

Failure mode to name. A green test suite proves little here, because the suite usually imports modules in one fixed order that hides the bug.

Curated: · Written: · Reviewed:

Q&A-5

An AI assistant gives you three candidate causes for a nightly job that fails about one run in ten in a legacy Java service, and it has no logs. Which one do you check first, and what single cheap check would separate the top two?

Answer

I check the timeout first, but as the cheapest test, not a verdict. With no logs I have three sources: the per-night exit status, run start and end times, and the downstream service's latency history for the same windows. Before running anything I read the timeout actually in force on the downstream call. In java.net.http.HttpClient that is the per-request timeout set through HttpRequest.Newbuilder().timeout(Duration), plus connectTimeout(Duration) on HttpClient.newBuilder(). Legacy code often pulls these from a shared config, so I grep for the setter and the property it reads rather than trusting the ticket.

Assumptions I state first: the job runs once a night, failures are clean (uncaught exception or non-zero exit), and downstream p99 can be computed for each run window.

Each candidate predicts something different, so I write the predictions down before touching the data:

CauseFailures should trackIf the job runs single-threaded
Timeout on downstream callNights where downstream p99 exceeds the timeoutFailures continue
Race on shared cacheNights regardless of downstream latencyFailures stop
Third candidateWhatever its mechanism impliesSame test, same discipline

The check that separates the top two is the join. Schema names are placeholders:

select r.night, r.failed, l.p99_ms from job_runs r join downstream_latency l on l.window_start <= r.run_start and l.window_end >= r.run_end;

Then I bucket nights by whether p99_ms exceeds the timeout. Hypothetical figures, labelled as such: 30 nights, 3 failures, 5 slow nights.

  • Scenario A: all 3 failures on slow nights. Failure rate is 3/5 = 60% on slow nights against 0/25 = 0% on normal nights. The timeout moves to the front.
  • Scenario B: 1 failure on a slow night and 2 on normal nights. Failure rate is 1/5 = 20% against 2/25 = 8%. The timeout weakens and the race moves up.

The join has a confound. Heavy nights can slow the downstream and also raise contention inside the job, so clustering on slow nights does not prove the timeout by itself. It becomes strong evidence only if failures still cluster on slow nights when load is roughly held constant, for example by comparing slow and fast nights within the same batch-size band. If I cannot get that control, I treat the result as a ranking and confirm with a targeted change before relying on it.

The fallback is a single-threaded run, for example by pinning the shared-cache section to Executors.newSingleThreadExecutor(). If failures stop over a week, that points at the race. The cost is a week of slower runs and a delayed fix, so I only use it if the join is inconclusive.

I do not patch on the assistant's ranking alone. A retry wrapped around the downstream call would hide a timeout on the nights I happen to test and leave a race in place for the rest.

Curated: · Written: · Reviewed:

Q&A-6

In a request-handling service that serves many requests at once, why is a breakpoint usually the wrong first tool for narrowing down a bug?

Answer

The short answer. A breakpoint pauses the process, not just the request you care about. In a service that handles many requests at once, every other thread or worker that reaches the same code stops too, so other users stall, timeouts start firing, and the timing you are trying to observe changes. A log line does not pause anything. It writes a value and the request keeps running, so it is the safer first tool.

What this looks like in practice. Suppose 50 requests per second hit a handler and one in 300 fails. A breakpoint on the handler stops the first worker that reaches it. The other 49 requests a second queue behind it, and the failure you were chasing may no longer occur because the timing has changed. A single log line at the entry to the suspect function, tagged with the request ID, keeps the service at its normal pace and lets you collect a sample of failing requests to compare against passing ones.

When a breakpoint is still the right call. A conditional breakpoint is reasonable when the failure reproduces on demand in a local or staging process that no other traffic touches, or in a batch job with one input. Then pausing costs nothing, and you get the full local state, which a log line would only show if you had logged it in advance.

Keeping temporary instrumentation out of the diff. Tag each temporary line with a unique marker such as # DBG-7f3, then check before committing with git diff main -- . | grep 'DBG-', which should print nothing.

Curated: · Written: · Reviewed:

Q&A-7

A refund regression appears between tags v4.2 and v4.3 in a 1,100-commit legacy Python repo. The failing test takes 40 seconds and fails about one run in five on the bad commit. Walk me through how you find the commit that introduced it, and how you know it is the cause and not just the last line changed.

Answer

The 90-second answer. Before bisecting I confirm the oracle at both ends. The test must fail on v4.3 and pass on v4.2 for the same reason, and I measure the failure rate on each side, because bisect is only as good as its verdicts. On a linear history, bisect needs ceil(log2 N) test points, so 1,100 commits means 11 points. The real cost is runs per point, not steps. Flakiness changes the procedure: a single run per point will miss the bug, so each point needs enough runs that a bad commit almost never passes. Then I treat blame as a pointer to the last edit, and I confirm the cause by reverting or by bisecting on a behaviour rather than a line.

git bisect start v4.3 v4.2
git bisect run sh -c 'make build >/dev/null 2>&1 || exit 125; for i in $(seq 14); do pytest -q tests/test_refund.py::test_partial_ship >/dev/null 2>&1 || exit 1; done; exit 0'

Exit 125 tells git bisect run to skip a commit that does not build. Any failing run marks the commit bad. Two things go wrong with that rule. A pytest collection error exits 2, not 1, and a plain || exit 1 would still classify it as bad, so I check the exit code explicitly. And a good commit that fails by chance is marked bad, which sends bisect to the wrong half. The arithmetic: at a 0.2 failure rate, one run misses the bug 80% of the time, nine runs miss it 13% of the time, and fourteen runs miss it 4.4% of the time. At 40 seconds a run, fourteen runs per point is about 9 minutes, and 11 points is about 100 minutes before builds. That cost decides whether I narrow the test first, for example by shrinking the fixture or pinning the seed, so the failure rate rises.

Blame and cause are different questions. git blame -w -C -C -M on the failing lines at v4.3 gives the last commit to touch each line, and --ignore-revs-file removes formatting-only commits that would otherwise take the blame. But a refund rounding bug is often caused by a commit that changed a caller, a default, or a constant somewhere else. For that I search for the commit that changed the behaviour: git log -S'ROUND_HALF_EVEN' --oneline v4.2..v4.3, or -G for a regex when the string is built.

Confirming the cause. For the commit bisect returns, X, I run the test 30 times on X and 30 times on its parent. I expect roughly six failures on X and none on the parent. Then I revert X on a tip worktree and run the test again. A pass after the revert is strong evidence. A bisect result that is a merge commit with an empty diff means the culprit sits inside the branch, so I bisect again within that branch.

Where the assistant fits. I give it the bisect log, the culprit diff, the failing test and the blame output, and I ask it to trace how the diff changes the code path the test exercises. I do not let it pick the commit. Its first answer is usually the blame commit, and that is the one most likely to be wrong.

Curated: · Written: · Reviewed:

Q&A-8

An AI assistant's fix for a refund bug adds a `mode` parameter with a default to `refund_amount` and calls `requests.get(url, retries=3)`. How do you decide whether the diff is safe to merge?

Answer

The 90-second answer. I start with the hunk that changes behaviour for callers who never touched it. A new default argument does that by definition: every existing call now runs the new branch. So I read the diff as a list of changes to what existing callers receive, and then I check each library call against the pinned version, following it to where the keyword is actually validated.

The rounding change is the one I would stop on. Before, the function was round(paid_cents * rate). On Python 3.11, round() on a float rounds ties to even, so refund_amount(125, 0.5) returns 62, because 62.5 is exact in binary and the tie goes to the even neighbour. If the assistant's version uses Decimal(...).quantize(Decimal('1'), rounding=ROUND_HALF_UP) as its default, the same call returns 63. Every caller that never passed mode now gets 63 on every tie. Some of those refunds may have been the cent the bug report describes, and some may have been correct. The diff does not say which.

# before, Python 3.11
refund_amount(125, 0.5)  # 62
refund_amount(124, 0.5)  # 62

# after the assistant's version, default mode=ROUND_HALF_UP
refund_amount(125, 0.5)  # 63
refund_amount(124, 0.5)  # 62

The second hunk is the requests call. inspect.signature(requests.get) shows **kwargs and nothing else, so it looks valid. Follow the chain: requests.get passes kwargs to Session.request, which has explicit parameters and no catch-all. retries is not one of them, so the call raises TypeError, but only on the code path that makes this request. The existing suite can stay green if no test reaches it.

The lockfile and the installed environment answer different questions. The lockfile says what the project pins. The installed environment says what runs. I check both, because they diverge when someone installs outside the lock.

Failure modes of the review itself. (1) Reading only the changed lines and missing callers whose result changed through the new default. Detect with a grep for every call site. (2) Accepting the assistant's claim that a change is safe without writing the test that claim implies. (3) Fixtures that cannot see the change, such as integer inputs with no ties. Detect by reverting each hunk and checking whether any test fails.

Trade-off. Reading the whole diff catches intent problems. Walking each changed function's callers catches edge problems that a read misses. I do the second because it produces evidence.

Curated: · Written: · Reviewed:

Q&A-9

An AI assistant proposes a fix for a one-cent rounding bug in a legacy Python invoice function, and the diff also renames helpers and reformats the file. Which change do you keep, and why?

Answer

The first 90 seconds. I split the assistant's diff into the hunk that changes the number and everything else. Only the first is a candidate for this change. Then I check whether that hunk sits where the wrong value is first produced. A rounding bug usually starts where the total is computed, not where it is printed, because every later reader inherits the value. So I want the fix in the function that builds the line total, and I want it to be the only behaviour change in the patch.

Mechanism. Suppose the total is round(amount * qty * (1 + tax), 2) with amount sometimes a float and sometimes a Decimal. Python's round() on a float works on the binary value with round-half-to-even, so round(2.675, 2) returns 2.67, because 2.675 is stored slightly below the midpoint. A fix that moves the rounding to the display layer leaves the stored total wrong for every other consumer, such as the ledger write or the tax report. The narrow version fixes the arithmetic at the source with an explicit rounding mode:

# Python 3.12
from decimal import Decimal, ROUND_HALF_UP

def line_total(amount: Decimal, qty: int, tax: Decimal) -> Decimal:
    raw = amount * qty * (Decimal(1) + tax)
    return raw.quantize(Decimal('0.01'), rounding=ROUND_HALF_UP)

The other changes. The renames and reformatting do not affect the arithmetic, but they make the diff harder to review and can hide a real change inside a large block. I keep them out and record them. The retry-loop change is different: it alters control flow, so it needs its own review even if it looks harmless.

Alternatives. Converting every caller to Decimal in one pass is the right end state, but it is a migration with its own tests. Patching only the caller that reported the bug is faster, but the same defect remains in the other call sites. I take the root-cause fix when the call sites share the function, and the caller-only patch only when a release forces it, with a ticket for the rest.

Verification and the note. The risk is a behaviour change hidden in the cleanup. I catch it by running the existing suite on the original and patched trees, and by comparing outputs for a fixed set of invoices. I add a half-cent test that fails on main and passes on the branch. For the naming and formatting problems I write a note with the file, the reason it was left, and the trigger for doing it, such as the next change to the same function.

Curated: · Written: · Reviewed:

Q&A-10

What must a regression test do before you can count it as evidence that a bug fix works?

Answer

The 60-second answer. A regression test earns its place by failing on the code that has the bug and passing on the code with the fix. If it passes on the parent commit, it does not reproduce the bug, so it proves nothing about the fix. I run the new test against the parent commit first, and I expect it to fail on the assertion that encodes the broken behaviour.

git checkout <parent-sha>
git checkout <fix-sha> -- tests/test_refunds.py   # bring only the test across
pytest tests/test_refunds.py::test_partial_refund_is_not_one_cent_short
# expected on parent: AssertionError, 998 != 999

The rule has three parts. The test fails before the fix. It passes after the fix. The failure is the assertion for the reported behaviour, not an import or fixture error, because those fail for reasons that have nothing to do with the bug.

A test that asserts the specific value the customer was owed meets the rule more cleanly than one that compares a whole snapshot of the output. A snapshot breaks on unrelated fields such as timestamps or ordering, so its failure does not tell you which behaviour changed.

If the parent run passes, the test misses the defect and I change its input. I still run the full suite on the fix commit, because a passing new test does not show that the rest of the behaviour is unchanged.

Curated: · Written: · Reviewed:

Q&A-11

Your fix makes the targeted test pass, but the module's suite now shows three failures you did not touch. How do you decide which of them your change caused, and what do you run to prove a failure was already there?

Answer

The 90-second answer. I widen in rings: the targeted test, then the module's suite, then tests that import anything I changed, then the full suite. I classify a failure as pre-existing only when a clean checkout of the base commit reproduces it under the same environment. 'The stack trace doesn't touch my diff' is a hypothesis to test, not a finding.

The base comparison uses a worktree, not a stash: git worktree add --detach /tmp/base origin/main. A stash leaves ignored files in place (.pyc caches, generated fixtures, local .env), so a 'clean' run can still share state with my tree. The worktree has only tracked files, so I build a fresh virtualenv from the same lockfile and diff pip freeze between the two environments before trusting any comparison. If base passes and my tree fails, a dependency or a cached artefact differs far more often than my change is innocent.

Flaky tests need arithmetic. Suppose a test fails about one run in five on main, so p = 0.2 per run. The chance of seeing no failure in n runs is 0.8^n: 0.33 at n = 5, 0.11 at n = 10, 0.044 at n = 14. Fourteen clean runs on base are the minimum before 'passes on base' means anything at 95% confidence. I run once with -p no:randomly to rule out ordering, then with randomized order to test whether the failure depends on order.

Base (clean worktree)My branchClassification
fails, same messagefailspre-existing; record the failure rate
passes in 14 runsfailsintroduced, or env/order dependent; bisect
failspassesmy change masked it; do not report it as fixed
fails, different messagefailstwo problems; treat as new until explained

The third row is the one candidates miss. A fix that stops a shared fixture from raising can hide a failing assertion downstream, so a green run is not evidence the failure disappeared.

An assistant will often label failures 'unrelated' from reading the trace. I check the first frame inside the repo, and whether the failing test shares a fixture, a database schema or a module-level global with my diff. If the test passes alone on both trees but fails in the suite, the cause is shared state, and I look for the fixture that leaks.

When no clean checkout is reachable, I say 'failing on main per CI run N' and name that as the limit of the evidence, rather than 'pre-existing'.

Curated: · Written: · Reviewed:

Q&A-12

A targeted test for load_rate now passes after the assistant's patch, but you have not read its callers. What do you check before you accept the change, and what does the answer look like when a caller depends on the old behaviour?

Answer

The first 90 seconds. The targeted test shows that the function now meets the one case it names. It says nothing about callers that rely on behaviour the test never mentions. Before I accept the patch I find every caller of the changed function and ask which of their assumptions the old code satisfied.

Assume a legacy module on Python 3.12:

RATES = {"EU": 0.20, "UK": 0.20, "US": 0.0}

def load_rate(region):
    return RATES[region]        # raises KeyError for unknown regions

The failing test expects load_rate("JP") to return 0.0. The assistant changes the body to return RATES.get(region, 0.0), and the test passes. Then a grep finds a caller in refunds.py:

try:
    rate = load_rate(order.region)
except KeyError:
    queue_for_manual_review(order)
    return

Unknown regions used to reach a person. After the patch they refund at 0% tax and never reach the review queue. The refund tests only use EU, UK and US, so they stay green. The problem surfaces later as tax-free refunds in reconciliation.

Two ways to fix it. Option A keeps load_rate strict and gives the defaulting behaviour its own name, such as load_rate_or_default, used only by the code that wants it. Option B keeps the lenient function and changes refunds.py to test membership explicitly with if order.region not in RATES. Option A keeps the exception contract for every existing caller, so nothing changes behaviour unless someone opts in. Option B makes the dependency visible, but it only works if every caller is found and edited, and a caller in another team's module would break with no test to warn anyone. The main trade-off is the blast radius of changing a shared function against the cost of one extra name. I would choose A when I cannot see every caller from this repository.

Why the green test is weak. It checks one input against one expectation. Line coverage would mark the refunds.py lines as executed even though their outcome changed. The evidence I want is differential: run the old and new versions over a replay of region values from the last quarter, and report every input where the return value or the exception path differs. Then I confirm that each difference is intended, and I check that no unknown region reaches a refund without a review record.

Failure mode to name. Exception-based control flow broken by a silent default. It is detected by the differential replay, and by comparing manual-review queue volume before and after the change.

Curated: · Written: · Reviewed:

Q&A-13

A unit test in a legacy Python service fails roughly one run in ten with a timeout, and the assistant suggests time.sleep(2) before the assertion. Why is that a poor fix, and how would you measure the flake rate before changing anything?

Answer

Start with the sleep. It does not remove the race; it changes the odds. The test is waiting on something, and a fixed two-second window is a guess about how long that takes on one machine. On a slower CI runner it may still be too short, and on a fast laptop it adds two seconds to every run and hides the timing dependence rather than naming it. So before touching the test, I want to know what it is waiting for.

Next, measure. One failure in ten is a rate, and I can't read a rate from a handful of runs. If the true failure rate is 10% and I run the test 20 times, the 95% interval around the observed rate is roughly 1% to 31%, so twenty runs cannot tell me whether a change moved anything. For a before-and-after comparison I would run 200 times per condition, which gives an interval of about 5% to 16% at the baseline, and that is enough to see a drop from 10% to 2%.

# Python 3.12, pytest 8
# Run the test N times, unpinned, and record outcomes.
import subprocess
fails = 0
N = 200
for _ in range(N):
    r = subprocess.run(['pytest', '-q', 'tests/test_refunds.py::test_settles_async'], capture_output=True)
    fails += r.returncode != 0
print(f'{fails}/{N} failed')

Then I separate the cases. Run the test alone, many times. If it fails only in the full suite, suspect shared state left behind by an earlier test, such as a module-level client, a cache, or a patched global that is not restored. If it fails alone but never at a fixed seed, the nondeterminism comes from the clock, random input, or unordered iteration. If it fails the same way at the same seed every time, the input controls the outcome and I can bisect on it. A real intermittent bug in production code shows up under the same seed too, so a seed alone does not tell test harness from product.

The principled fixes are removing shared state, injecting the clock (pass now into the function instead of calling time.time() inside it), and making ordering explicit (sort before iterating, or use a list instead of a set). Each one changes a signature or a fixture, which is a bigger diff than a sleep. That cost buys a test that behaves the same at any CI speed. A retry is the wrong substitute: a test that passes on its second attempt has told you nothing about correctness, and on a real bug it turns a failure into an occasional silent one.

Curated: · Written: · Reviewed:

Q&A-14

A bug report says a counter in a legacy Python service sometimes ends one short under load, and an AI assistant reading only that function suggests wrapping its increment in a lock. What symptom pattern points to a race, and what does the assistant's single-file view miss?

Answer

The first 60 seconds. Intermittent short totals that appear only under concurrent load, and vary between runs, point to a lost update. A read-modify-write such as self.total += 1 is not atomic in Python: two threads can read the same value, both add one, and one write overwrites the other. The result is a total that is short by a few, not a wrong formula, which is why the symptom is intermittent.

The assistant reads one function. It cannot see the caller. If the caller already takes a lock before calling the increment, the function is safe on the normal path, and a second lock inside it can deadlock when threading.Lock is not reentrant. So the first question is where the write path is protected, and whether every writer goes through that protection.

import threading

class Counter:
    def __init__(self):
        self.lock = threading.Lock()
        self.total = 0
    def bump(self):
        self.total += 1          # not atomic: read, add, write

def handle(counter):
    with counter.lock:           # caller already serialises writers
        counter.bump()

Confirm the race. Shorten the switch interval so threads interleave often (sys.setswitchinterval(1e-6), default 5e-3), then run eight threads each calling bump() directly, bypassing handle(). Repeat the run. Totals below the expected sum mean the unlocked path races. Run the same loop through handle(); it should always hit the expected total. That tells you which path is unprotected.

After the fix. Add the missing lock on the path that bypasses the caller, or make the write path reach one lock, and rerun both loops. A regression test should call every public write path concurrently and assert the exact total.

Curated: · Written: · Reviewed:

Q&A-15

Two tests pass alone and fail together, and the failing assertion shows another user's preferences. Walk me through how you reproduce the leak, why test order changes the result, and which fix scopes the state.

Answer

The 90-second answer. Before I touch the code I want two facts: does the failure reproduce when the test runs alone, and is the leaked value the same object across calls? A test that fails only after another test has run is a shared-state bug until proven otherwise. The usual sources are a mutable default argument (evaluated once, at def time), a class attribute mutated through an instance, a module-level cache or memo, and a singleton holding per-request data. The reproduction tells them apart, and the identity check is the diagnostic that names the object.

# session.py, Python 3.12
class Preferences:
    defaults = {'locale': 'en-GB', 'tags': []}

    def __init__(self, overrides=None):
        self.values = self.defaults          # alias, not a copy
        if overrides:
            self.values.update(overrides)    # writes the class-level dict

# test_a.py
def test_override_locale():
    assert Preferences({'locale': 'fr-FR'}).values['locale'] == 'fr-FR'

# test_b.py
def test_default_locale():
    assert Preferences().values['locale'] == 'en-GB'

Run the pair in both orders and then in isolation:

pytest test_a.py test_b.py   -> FAILED test_b: 'fr-FR' != 'en-GB'
pytest test_b.py test_a.py   -> 2 passed
pytest test_b.py             -> 1 passed

Order matters because the leak has a writer and a reader. The failure appears only when a writer runs first in the same interpreter. For a whole suite, pytest -p randomly --randomly-seed=N (pytest-randomly) makes the failing order repeatable, and then I bisect the list of preceding tests. The identity check, assert Preferences().values is not Preferences.defaults, fails at once and tells me the instance is holding the class's own object.

Fixes, and what each one scopes. The tempting patch is self.values = dict(self.defaults). It fixes the top level, but tags is still the same list in every instance, so Preferences().values['tags'].append('x') leaks again. Use a factory that builds fresh containers, or copy.deepcopy if the nested structure is large and fixed. For a default argument, def f(items=None): items = [] if items is None else items is the standard form.

A module-level cache is different. functools.lru_cache on a function that returns a mutable value shares that value across every caller, so the fix is to key the cache by the thing that owns the state (tenant, request, or test) or to hold it on a request-scoped object. For per-request state in async or threaded code, contextvars.ContextVar scopes the value to the task, but a task created inside a context inherits a copy of its parent's values, which surprises people who expected a fresh slate.

The trap I watch for is an autouse fixture that clears the cache between tests. It makes the suite green and leaves production exactly as it was: two concurrent requests with different locales still read one dictionary. A fix is correct only when the state has one owner per request, and I verify it with a test that interleaves two owners, not only with a green run.

Failure modes to name. Shallow copies that stop at the top level. Clearing in fixtures that hides the leak from the suite. Locks added around a cache that was never meant to be shared, which serialise requests and show up as p99 growth under load. Detection: a unit test that mutates nested state through one instance and reads through a second, plus a concurrent test with distinct inputs.

Curated: · Written: · Reviewed:

Q&A-16

A Python job logs a ValueError from deep in a large service, and the traceback has a chained KeyError above it. Which frame do you read first, and how do you tell the frame that raised from the frame that supplied the bad value?

Answer

The 90-second answer. Python prints the oldest frame first and the frame that raised last, so the bottom of the traceback is where the check failed. I read that line first, then the exception it was handling. The raising frame is where an invariant was checked. The frame that supplied the bad value is usually higher up, and I find it by following the argument that failed back to where it was built.

Take this case, on Python 3.12:

def load_rate(region):
    return RATES[region]          # KeyError: 'EU-WEST'

def refund_amount(order):
    try:
        rate = load_rate(order.region)
    except KeyError as e:
        raise ValueError(f'no rate for {order.region}') from e

The output has two blocks joined by a fixed sentence:

KeyError: 'EU-WEST'

The above exception was the direct cause of the following exception:

ValueError: no rate for EU-WEST

The ValueError is the raising frame, in refund_amount. The KeyError is the cause. Neither tells me where 'EU-WEST' was written. That value came from order.region, so the producing frame is whichever code set that attribute, probably ingestion. I find it with a grep on the attribute name, or by stopping at the raise under pdb.post_mortem and reading order.__dict__ and the caller's locals.

Mechanism of the chain. raise ... from e sets __cause__ and prints the 'direct cause' sentence. A raise inside an except block with no from sets __context__ and prints 'During handling of the above exception, another exception occurred'. So the printed text tells me which link I am reading. Walk exc.__cause__ or exc.__context__ until it is None, and treat the oldest exception as the first observed symptom, not the origin of the data.

Trade-off between two readings. Reading from the top gives the call path that was taken but buries the failing check under library frames. Reading from the bottom gives the failing check quickly but can point at code that merely enforced a value someone else broke. I use the bottom to find the check and the chain to find the first symptom, then decide which of the two intermediate frames to trust. A wrapper that uses from None hides the first block, so the chain stops being evidence and I need the logs or a reproduction instead.

Curated: · Written: · Reviewed:

Q&A-17

An AI assistant says a race on the balance cache explains zero balances on about 2% of export rows and proposes a lock. The zeros cluster in one five-minute window. Before you accept that story, what do you check in the logs, and what result would make you drop it?

Answer

Turn the assistant's story into predictions before reading the lock code. A race on a balance cache predicts failures that depend on timing and on the interleaving of reads and writes. Each candidate mechanism predicts a different pattern in the logs, and each prediction can be checked in minutes.

1. Timing. The zeros sit in one five-minute window. I line that window up against the cache TTL, any deploy or config push, and any traffic spike. A TTL-driven race has no reason to cluster in one window unless the TTL or traffic pattern does too. If the window matches a rollout, the race story is a weak fit.

2. The fields on each failing line. I pull the zero-balance rows and group them by build, account, currency and cache_hit. Hypothetical lines in the shape of the real export (labelled as illustrative):

ts=02:00:14 build=4.7.2 account=A118 currency=None cache_hit=true balance=0.00
ts=02:00:14 build=4.7.3 account=A231 currency=EUR cache_hit=true balance=412.50

The cache_hit flag does not rule the cache in or out. A stale or poisoned entry can be served as a hit, and a read-modify-write race can write a zero that later hits read back. So hits and misses both need explaining. What matters is whether the zeros concentrate on one build or one field value. If every zero shares one build and currency=None, the trigger sits in the data path. If the zeros spread across builds and currencies, a concurrency or cache-state fault becomes more likely.

3. Alternatives I write down before testing. At least three:

  • The older build reads a field the newer build renamed, so a missing value falls back to zero. This predicts a build split and a null pattern.
  • Some pods run a stale configuration. This predicts a build split without the null pattern.
  • A read-modify-write on the balance cache loses an update, or a poisoned entry is written and then served as hits. This predicts zeros spread across builds, with a write-side log line showing a zero or a missed update just before the read.

Each prediction becomes a cheap test. For the rename, replay the older build against a fixture with the field set to None and diff the lookup output between the two tags, and state the expected output before running it. For the race, check whether the writer logs a zero or a stale version before the first failing read, and whether the failing accounts have concurrent writes inside the window.

4. The fix. I do not accept a null guard that returns zero. A silent zero is worse than a visible error, because the export looks correct while the balance is wrong. The fix belongs at the field read, at the rollout, or at the cache write path. Detection should flag any balance of exactly 0.00 where the ledger sum for that account is non-zero.

What would make me drop the field-rename story: the fixture with the field populated returns the right balance, and the fixture with the field set to None still returns zero on the newer build. That means the null path is shared across builds and the rename is not the cause, so I return to the other candidates.

What would make me drop the race story: the zeros appear with no concurrent write to the same account in the window, and the cache entry read at failure time matches the ledger at the same timestamp.

Falsification. I ask the assistant which log line would disprove its explanation. If it cannot name one, I treat its story as a guess and not a diagnosis.

Curated: · Written: · Reviewed:

Q&A-18

A test passes on your laptop and fails in CI on the same commit. What is the first thing you check before you change any code?

Answer

The first thing I check is whether the two runs really started from the same inputs. A commit pins only the source, so before I read any logic I compare the environment, the exact commit under test, and the test selection.

python -VV
python -m pip freeze | sort > local-freeze.txt
python -m pip check
# repeat in CI and save as ci-freeze.txt, then:
diff local-freeze.txt ci-freeze.txt
git rev-parse HEAD

The freeze diff is the fastest signal. A different interpreter version, or a package that resolved to another release, explains most of these failures. An unpinned range such as requests>=2 is the usual cause, because each machine installed whatever release was current when it ran.

Some mismatches do not show up in the freeze. Illustrative values for a comparison:

InputLaptopCITypical cause
Interpreter3.12.43.11.9Different pyenv or setup-python version
Resolved packages41 lines in freeze42 linesFloating range or transitive dependency
TZEurope/LondonUTCCode that reads local time
PYTHONHASHSEEDunsetunsetIteration order over sets changes per process
Commityour branch tipGitHub merge commitpull_request events check out refs/pull/N/merge

The last row matters more than people expect. On a pull request, CI often builds a merge of the PR into the base branch, so GITHUB_SHA differs from the commit on your laptop even though the PR looks identical. Compare the SHA CI reports, not the one you pushed.

If the environment matches, I check three things next. Test order: a plugin such as pytest-randomly may be installed locally only, and it shuffles tests by default. Ignored files: git status --ignored on the laptop can show generated fixtures or cached data that CI does not have. Native libraries: pip freeze does not list system packages, so I compare the OS and shared libraries the wheel loads against.

The trade-off is that the freeze diff is a diagnostic, not a fix. The durable fix is a lockfile (pip-tools or uv lock) committed with the code, so both machines install the same resolved set. I rerun the diff after every pin, because pinning a direct dependency can leave a transitive one floating.

The check does not help when the failure is timing-based. If the test passes or fails depending on wall-clock reads or sleep-based waits, the environment will match and the flake will remain, so I move on to the clock and the synchronisation instead.

Curated: · Written: · Reviewed:

Q&A-19

A loyalty-points allocator in a legacy Python service gives some customers one point too few. Walk me through turning that report into a test that would catch it, and how you tell whether an accepted fix covers the input class or only the reported example?

Answer

Start with the input, not the assistant. A data-dependent bug is a claim about a set of inputs, so I make that set concrete before anyone edits code. I shrink one failing record from the logs to the smallest input that still fails, name the property that matters (empty list, boundary value, missing key versus null, non-ASCII text, integer above 2^53), and then write a property that states the invariant the function must keep. A fix is judged against that property, not against the one reported example.

Hypothetical legacy code (Python 3.12):

# legacy/loyalty.py
def allocate_points(total, parts):
    share = round(total / parts, 2)
    return [share] * parts

Shrinking by hand: (10.0, 1) and (10.0, 2) conserve the total. (10.0, 3) returns [3.33, 3.33, 3.33], which sums to 9.99. That is the minimal failing input, and the violated invariant is conservation: the parts must sum to the total. Saying "3.33 is wrong" would only fix that case. Saying "the parts must sum to the total" generalises.

The property:

from hypothesis import given, strategies as st

@given(st.integers(min_value=0, max_value=10**9), st.integers(min_value=1, max_value=200))
def test_allocation_conserves_total(total, parts):
    assert sum(allocate_points(total, parts)) == total

I add named cases beside it: (0, 1), (1, 3), (10, 3), (10**15 + 1, 7). Hypothesis shrinks any failure to a minimal case, and the table pins the inputs I already know matter. Then I check that the test can fail: I reintroduce the bug on a branch and confirm the property fails within seconds. A test that cannot fail on the old code is not evidence.

An integer-unit fix addresses the invariant directly by distributing the remainder one unit at a time:

def allocate_points(total: int, parts: int) -> list[int]:
    base, rem = divmod(total, parts)
    return [base + (1 if i < rem else 0) for i in range(parts)]

Two traps remain. Changing the contract to integers can break callers that pass floats, and divmod on floats returns floats without raising, so that failure is silent. I grep the callers and run the property with float inputs. parts=0 raises ZeroDivisionError, so the contract for zero parts has to be decided explicitly rather than left to fail.

Other input classes need the same treatment. An NFD-accented discount code misses an NFC lookup. payload.get('coupon') and payload['coupon'] is None behave differently. date.replace(month=2) on the 31st raises. A 64-bit ID that passed through a JavaScript gateway was already rounded before this code saw it. An assistant can list these from the code it has read, but a class introduced upstream is invisible to it, so I check where each input originates.

To tell whether an accepted fix covers the class, the minimal case stays as a fixture, the property stays in the suite, and I run the old and new functions side by side over a generated corpus. Any disagreement that the invariant does not explain means the fix covers the example, not the class.

Curated: · Written: · Reviewed:

Q&A-20

An order is marked paid and then cancelled by two async handlers in a legacy service, and the assistant says the consumers ran out of order. How do you trace one order across the queue boundary with correlation IDs, and what evidence would confirm the ordering hypothesis before you touch any code?

Answer

The first 90 seconds. I would not ask the assistant for a fix yet. I would give it one order ID, the producer function that publishes the events, the consumer handlers that subscribe to them, and the log lines that mention that ID. A whole-repo view tells it nothing about which message reached which handler. The ordering claim is testable, so I would test it before believing it.

Mechanism. A correlation ID tells me which messages belong to one causal chain. It does not tell me their order. For order, I need a per-entity sequence number assigned by the producer, such as an order_seq incremented on each state change, and a log field for the partition and offset the consumer actually read. Wall-clock timestamps are weak evidence because hosts drift and log lines are written after handling finishes.

# Python 3.12, consumer loop; headers are a list of (str, bytes) tuples
for msg in consumer:
    hdrs = dict(msg.headers or [])
    log.info("consume", extra={
        "cid": hdrs["correlation_id"].decode(),
        "order": msg.key.decode(),
        "seq": int(hdrs["order_seq"]),
        "partition": msg.partition,
        "offset": msg.offset,
    })

From a trace like this for one order:

cidorderseqpartitionoffsethandler finished
c-91O-10422787910:02:01.134
c-91O-10423788010:02:01.120

The offsets are in order, so the broker delivered them in order. The handlers finished out of order, so the reordering happened in the consumer, probably because a thread pool processes records from one partition concurrently. That is a different fix from a producer or partition-key bug, and it is the distinction the evidence has to make.

Deciding between hypotheses. Three checks separate them:

  1. Compare publish order (order_seq) with partition offset per order. If offsets are inverted for the same key, the problem is upstream: a non-order-preserving partition key, or a producer with retries that can reorder. Kafka clients since 3.0 default to idempotent producers, which keep per-partition order with up to five in-flight requests, but only when the key is the order ID.
  2. If offsets are in order and handler completion is inverted, check consumer concurrency.
  3. Count inversions across all orders in the window. A few inversions near a deploy or a rebalance point to a different cause than inversions spread evenly.

What would make me drop the ordering story: the inversions disappear once I sort by order_seq, meaning the handlers are order-independent and the bug is a duplicate or missing event, which I would test next with the idempotency key.

Failure modes. A consumer that retries a failed message after later messages succeeded will look like reordering with no concurrency. A duplicate delivery after a rebalance will look like a second cancellation. Both show up as the same cid with a repeated seq, which is why I count duplicates before I count inversions.

Curated: · Written: · Reviewed:

Q&A-21

You are fixing a one-cent refund bug in a legacy Python service and the AI assistant's diff also changes a search query so it no longer concatenates user input into SQL. Do you keep that change in the refund pull request, and why?

Answer

The 90-second answer. No, I keep it out of the refund pull request. The refund fix and the query change have different owners, different risks and different ways to fail, so they go through separate reviews. I apply the refund fix, leave the search endpoint unchanged in that branch, and open a second change for the query. That second change is small, has its own test, and is labelled as a security change so the reviewer reads it as one.

Why it matters. The assistant reads the files I gave it. It does not know whether the search endpoint sits behind authentication, which callers pass what into term, or whether a downstream component already escapes it. Its diff is a proposal, and a proposal that touches a second concern gets reviewed as if the refund change were the whole story.

# Current: term is concatenated into the SQL text
sql = "SELECT id, email FROM customers WHERE name LIKE '%" + term + "%'"

# Proposed: the driver sends term as data
cur.execute("SELECT id, email FROM customers WHERE name LIKE ?", (f"%{term}%",))

The two changes have different revert behaviour. If the refund fix misbehaves in production and I revert the combined PR, I also undo the query change, which leaves the injection live for the length of the rollback. If I keep them apart, reverting the refund fix touches nothing else.

Alternatives and their trade-off.

  1. Combined PR. Fewer merges and one deploy. It reduces review quality for both concerns, because the refund reviewer is unlikely to check parameter binding, and the security reviewer is unlikely to read refund rounding.
  2. Separate PRs, sequenced. Two reviews and two deploys. The refund fix ships on its own schedule. The query fix gets a test that asserts the SQL text and the bound parameters for normal input, plus a case with a quote character in the term.
  3. Refund PR only, query filed as a ticket. Lowest risk to the refund change, but the injection stays in production until someone picks up the ticket.

I pick option 2 unless the owner says the query is unreachable from outside, in which case option 3 becomes reasonable with the ticket marked high priority.

Failure modes I watch for. A parameterised query that changes the result ordering or the LIKE wildcard handling changes behaviour for callers who never asked for it. The test for the query therefore compares results on a fixed fixture, not just that the code runs.

What I tell the interviewer. Which change goes where, why the refund diff stays narrow, what the query test proves, and what I did not verify, such as whether the endpoint is reachable without login.

Curated: · Written: · Reviewed:

Q&A-22

When an AI assistant is still generating an answer during a timed debugging round, what should you say aloud before its output arrives?

Answer

The short answer. Say your current hypothesis out loud, with a rough confidence level, and name the next check you will run once the output arrives, along with what result would support or reject the hypothesis. The interviewer can only score reasoning they can hear, so the wait is when you make it visible.

For example: "I'm asking whether the refund total is computed before or after the partial-shipment adjustment. My working guess is that the adjustment runs after rounding, which would explain a one-cent shortfall on split orders. I'm not confident yet. When the answer comes back I'll check the call order in the trace before reading the diff."

What to avoid. Don't narrate the tool ("it's thinking now") and don't read its output back as if it were your reasoning. Don't go silent while typing. Don't announce a root cause you have not checked.

If the hypothesis is wrong. Say so aloud and name the observation that changed your mind. A revised hypothesis with a stated reason scores better than a silent pivot.

Curated: · Written: · Reviewed:

Q&A-23

Your AI assistant's patch turns the full suite green for a half-cent discount rounding bug, but the affected branch sits behind a feature flag. What do you verify before the patch reaches review, and what evidence do you accept?

Answer

Say first that a green run only tells me about the code it executed. Before I read the patch, I want to know which configuration the failing path needs and whether CI ever runs it.

The situation I would check: the report says half-cent discounts round down on orders above 500 units. The patch changes _round_discount in pricing/discount.py, and the assistant says all 412 tests pass.

  1. Find the gate. grep -rn "discount_v2" --include=*.py . shows the new path runs only when FEATURE_FLAGS["discount_v2"] is true, and that flag reads DISCOUNT_V2 from the environment.
  2. Find what CI sets. grep -rn DISCOUNT_V2 .github ci deploy returns nothing, so CI runs with the flag off and the flagged branch never executes.
  3. Check coverage, not the pass count. coverage run -m pytest followed by coverage report --show-missing pricing/discount.py should list the flagged lines as missing.
  4. Write a regression test that runs under the flag: DISCOUNT_V2=1 pytest tests/test_discount.py::test_half_cent_over_500_units. Use the report's own order size, not a round number.
  5. Prove the test can fail. Revert only the patch, keep the test, run it with the flag set. It must fail with the reported symptom, one cent short. Reapply the patch and it must pass. A test that passes on both versions is no evidence.

What this buys: the assistant's claim was true and irrelevant. The suite could neither expose the bug nor confirm the fix, because the branch was never run.

Failure modes I would watch for. The assistant may enable the flag through a local env file that is not checked in, so the pass exists only on its machine. The production default may differ from the code default, so I check the deployed configuration, not the repo. If the flagged test passes on old and new code alike, it probably uses an input the bug does not trigger, such as exactly 500 units, so I reread the report's examples.

I do not accept the patch on the assistant's word. If the flagged path has no test, adding one is part of the fix, and I say so in review instead of letting the green check stand in for it.

Curated: · Written: · Reviewed:

Q&A-24

A refund bug is short by a cent on some partial shipments, the module has no tests, and the deadline is tomorrow. Do you ship a narrow workaround with a follow-up, or go after the root cause first? What evidence would change your answer?

Answer

The 90-second answer. I ship the narrow workaround only if I can bound the input class it touches, switch it off in one step, and detect any money it still gets wrong. If I cannot bound it, the workaround is just a second, untested copy of the formula, and I pursue the root cause first, accepting the delay. Either way I write down what is covered, what is not, and who reconciles the difference.

First I restate the goal in money terms. A refund is wrong when the customer receives a different amount from the one the order rules define, and the ledger records the difference. Before choosing, I establish three facts: how often the bad path runs, how far the bad output travels, and whether I can observe it.

Worked example (hypothetical figures). An order totals 10.00 across three lines priced 3.33, 3.33 and 3.34. The customer ships two lines and the rest is refunded. The legacy code computes the refund as total times refunded share, rounded once:

MethodCalculationRefund
Current code (hypothetical)10.00 x 1/3 rounded3.33
Line-item ruleunshipped line3.34

The one-cent gap is the bug. A workaround that recomputes from line items inside the partial-shipment branch fixes the reported case. Before I trust it, I check whether the same rounding path serves non-partial refunds, multi-currency orders or the loyalty adjustment. If it does, the workaround leaves money wrong on paths I have not enumerated, and the root cause moves up in priority.

Evidence that justifies the workaround. The failing branch is identifiable by a flag or a field on the order. The change sits behind a feature flag that defaults off, so rollback is one config change. I have a characterization test that records current outputs for a few hundred historical orders outside the branch, and the outputs do not change. A nightly reconciliation compares ledger totals with line-item totals for flagged refunds, so residue is visible within a day.

Evidence that justifies the root cause. The bad rounding is shared by paths the workaround cannot isolate. Reconciliation shows drift on orders the bug report does not mention. Or the characterization test cannot be written in the time available, which means I cannot tell what the root-cause change will alter.

Failure mode of the workaround. The flag stays on, the follow-up ticket has no date, and the branch becomes the real code path that nobody owns. I detect this with a count of flag-on refunds per week and a ticket with an expiry date set at merge time.

What I tell the business. Covered: partial shipments with three or more lines, about the affected share of refunds. Not covered: any other rounding path, with a stated reason. Residual risk: at most one cent per affected refund, reconciled nightly, owner named, root-cause work scheduled with a date.

Curated: · Written: · Reviewed:

Q&A-25

Your AI-assisted fix for a one-cent refund bug passes the targeted test, and the module suite is green. Walk me through the verification statement you send before anyone merges, and why you label the unverified parts.

Answer

The 90-second answer. I send a statement in five parts, in this order: what reproduces now, what test was added, what suite ran and what it returned, what was not verified, and the next check. Every sentence is either a command I ran with its output or a claim I have explicitly labelled as inference. I do not say the bug is fixed unless part one shows the reproduction passing.

For a refund fix in a Python 3.11 service it looks like this:

Reproduces now: replay of order 4471 (3 lines, partial shipment) returned 19.99 before the patch and 20.00 after. Run: scripts/replay_refund.py at 3e9a1f2.
Test added: tests/refunds/test_partial_shipment_rounding.py. It fails at the parent commit (verified by checkout and run) and passes after.
Suite: pytest tests/refunds -q, 212 passed, 0 failed. Full repo suite not run in this session; last full run on main was 1,904 passed, 3 skipped, two days ago.
Not verified: refund_amount is read by the export job. I found those callers by grep; I have not executed the export job.
Next: run the export job against a staging copy with a JPY order and diff its output against main.

The scope of each suite result matters. "Tests pass" could mean one file, one package or the whole repository, and a reviewer will assume the widest. I name the scope every time.

Why I label the gaps rather than leave them out: the reviewer decides based on what they think I checked. An omitted caller gap reads as a complete fix. A labelled gap costs one sentence and tells them exactly where the risk sits.

The failure I watch for is drift. Late in a session the statement gets vaguer ("should be fine", "looks correct"). I reread each sentence and ask which command or observation supports it. If none does, it moves to the not-verified list.

Curated: · Written: · Reviewed: