How to Verify an AI-Generated Bug Fix

Verify an AI-generated fix by showing that the original failure is detectable before the change, disappears after it, and stays absent on affected paths. Record the exact version, inputs, observations, and remaining gaps. A successful build is useful evidence; it does not establish that the reported bug is fixed.

Start with one testable claim

Write the expected behavior before evaluating the patch: “After the permitted user saves this field and reopens the record, the saved value is returned.” State the role, normal entry point, input, environment, and version. If the expected rule is disputed, resolve that product decision first.

Keep the scope explicit: diagnosis, verification, code repair, test editing, historical-data repair, and deployment are different actions. A verification request does not authorize all of them. Read the repository's instructions and inspect test-command side effects before running anything.

1. Preserve the failure before the fix

Save the original steps and the actual result. Run a check that asserts the reported symptom, not merely that a response exists. For a browser failure, start from the user's normal page and role; an internal API call alone cannot replace that journey. For an API or background-job failure, use the relevant execution boundary without adding an irrelevant browser step.

Where safely available, run the same regression check against the known-bad version in an isolated environment. Keep the command, version, inputs, expected result, and output. A reduced test can aid diagnosis, but retain the unreduced scenario for final verification. Do not run destructive historical behavior against production just to obtain a red test.

2. Connect the repair to the cause

Trace the actual path: entry, validation or transformation, storage or external operation, readback, and user-visible result. Identify the first boundary that breaks the agreed rule. A plausible explanation or changed line is a hypothesis until the observations support it.

Inspect the final diff for a repair at the responsible layer. Check other effects of the same cause and other places using the same faulty pattern. Avoid unrelated refactoring. A code fix does not itself repair existing bad records; keep historical-data correction separately scoped and authorized.

3. Replay the original path after the fix

Use the repaired version with the same relevant inputs, role, and conditions. Record any necessary differences. Rerun the original complete scenario and the regression check. For saved data, refresh or reopen and compare against the authoritative readback; a success toast is insufficient. For rejected operations, verify that no prohibited write or external side effect occurred.

Do not report a local result as deployed. Keep implementation, local verification, deployment, and acceptance on the live path distinct. If the patch changes after review, rerun affected checks against the final version.

4. Regress the paths the cause can affect

Select checks from the mechanism, not a fixed number of cases:

• Shared validation: valid input, the relevant boundary, and rejection behavior
• Permissions: the allowed role and applicable denied role or resource, including side effects
• Persistence: save, readback, reopen, and applicable stale-state behavior
• Async behavior: relevant retry, cancellation, duplicate, or out-of-order paths
• Shared callers: other entry points using the changed rule

Reuse evidence only when the implementation, data binding, and failure conditions are genuinely equivalent. Similar-looking pages do not establish equivalence. Keep independent branches uncovered unless checked. Run required lint, build, integration, or end-to-end checks according to the repository and change impact. Review the latest diff; disclose missing independent review.

A synthetic walkthrough: save succeeds, old value returns

This is an invented teaching scenario, not a Boloop incident or an executed test.

Assume the confirmed requirement says an authorized profile edit must persist. Before the patch, the planned reproduction is: open the profile through the normal route, change the field, save, refresh, and compare the returned field. Capture the request and authoritative readback for the same record and timeline. Do not assume caching is the cause merely because refreshing exposes the symptom.

After a proposed repair, repeat those steps. Add the relevant rejected-save check and another caller of the changed rule if one exists. Leave every result NOT RUN until observed. This example supplies a verification plan, not a diagnosis, patch, or success claim.

A real Boloop case: a six-second timeout looked like a Google failure

On October 20, 2026, Boloop's analytics email could read sitemap information but could not return the six page-indexing checks. The honest UI said the page checks were unavailable. That was a reporting failure, not evidence that Google had indexed zero pages.

The first question was where the request stopped. The same read-only account and the same six URLs were used for the comparison:

• With the original deadline, all six requests were cancelled after roughly 6,002–6,011ms with AbortError.
• In a temporary diagnostic run with a longer deadline, the same six requests returned HTTP 200 after roughly 7,042–7,907ms.

Those observations supported a local timeout as the immediate cause. They did not prove that every earlier provider failure had the same cause. There was no evidence from this comparison that broader Google permissions were needed.

The repair was deliberately narrow. URL Inspection received its own 15-second deadline. Token requests, GA4, search-performance reports and sitemap reads kept their six-second deadline. The six inspections remained parallel; response-size limits, partial results, caching and read-only scopes stayed in place. No retry loop or indexing submission was added.

Before calling the repair complete, the checks covered three different outcomes:

• An eight-second inspection response now returns all six page results.
• A request that never returns is still cancelled at fifteen seconds, while the successful sitemap result remains available.
• Other reporting requests still stop at six seconds.

These are controlled automated regressions with simulated provider responses and a test clock. They test timeout behavior; they do not measure Google's current response time. The eight-second regression failed against the earlier implementation and passed after the repair. It was rerun for this write-up along with the stalled-request check.

The real-provider readback was separate: the repaired call returned all six results in 8,188ms. After the authorized deployment, the normal production analytics screen also displayed all six rows. The English homepage was indexed; the remaining fixed URLs were reported as unknown to Google at that time. Fixing the report did not itself get those pages indexed.

This case closes a bounded claim: the short local deadline incorrectly rejected valid responses in the observed runs, and the dedicated deadline restored those checks. It does not establish a general speed improvement, guarantee future provider availability or prove a successful scheduled email after deployment. Those outcomes require their own observations.

To reuse the method, write down the original symptom, the first boundary that fails, the before/after evidence and the nearby failure paths. Keep local regression, real-provider readback and production acceptance separate in the result template below.

When verification fails or cannot run

• Original symptom still present: record FAIL and return to diagnosis; do not weaken the expected result to make the test pass
• Missing access, data, or environment: record BLOCKED with the exact prerequisite and next action
• A planned check not attempted: record NOT RUN, not PASS or N/A
• Intermittent failure: record attempts, failures, timing conditions, and uncertainty; a clean sample does not prove absence
• Original reproduction unavailable: keep the root cause provisional, preserve available logs and inputs, and seek independent supporting evidence. Without a regression signal sensitive to the defect, stop at diagnosis rather than claiming a verified repair

Mocks can isolate a mechanism but cannot establish an unobserved real-system outcome. Do not substitute synthetic acceptance data without permission. Redact secrets and personal data from evidence before sharing it.

Stop when the evidence supports the scoped claim

Close verification only when the original scenario and applicable affected paths have matching evidence for the final version, required checks and review are complete, and no unresolved defect invalidates the claim. Record any remaining limits and the shortest rollback. Passing a subset supports only that subset.

If an important prerequisite is missing, stop dependent work and say what would unblock it. If evidence is complete, stop adding repetitive checks that cannot change the conclusion. The goal is a bounded, defensible result, not an endless green-test count.

Record the result and take the next step

Use the companion Markdown result template to record before/after evidence, regression results, review, status boundaries, and rollback.

For an existing failure, start with $bug-fixing. For an independent read-only review of the final diff, use $code-review-multiagent. For authorized validation against confirmed requirements, choose the explicit $test-case mode. These are workflow instructions; they supply neither permissions nor execution tools.

Next: fill in the expected behavior, known-bad version, and original replay steps before asking an agent to assess the repair.

Skill pages: bug-fixing · code-review-multiagent · test-case

Download the Markdown template