Agentic Debugging Loops for Flaky Test Failures

Flaky tests are the ones that fail on Tuesday, pass on Wednesday, and fail again on Thursday for no reason anyone can find. This piece looks at why that specific kind of problem, the "it works when I run it myself" problem, turns out to be a near-perfect job for an agentic debugging loop. Not because agents are magic. Because the shape of the problem matches the shape of the tool.

Here's the thing about flaky tests: a single test run tells you almost nothing. The failure doesn't show up on command. So the whole game is iteration, running the thing again and again, changing one variable at a time, until the failure repeats and you can actually see what's causing it. That's slow, boring work for a person. An agent, on the other hand, can keep at it far longer without losing focus.

The root causes hide in a few usual places: timing dependencies, shared state between tests, an external service that's slow one day and fast the next, environment drift between machines. Each of those needs a different kind of digging. A retry-and-hope approach, or a script that just swaps out a broken selector, treats the symptom. The cause stays untouched.

And there's a cost to leaving it alone. Once developers learn that some red builds are just noise, they stop investigating and start re-running the pipeline until it goes green. Failures get waved off in code review. The whole team's trust in the test suite erodes, not just the one engineer who got burned by a flaky login test three sprints ago.

Parry et al. (arXiv, April 2025) put a number on the direct cost: developers spend 1.28% of their working time repairing flaky tests. That sounds small. But multiply it across a large engineering org, add in the hours lost to trust erosion and the bugs that slip through because nobody trusted the red build, and it stops looking small.

The manual loop, reproduce, isolate, patch, re-run, can eat a whole afternoon for one test. At the scale of a real codebase with hundreds of flaky tests, that math often doesn't work.

What an agentic debugging loop actually does, step by step

Worth being precise here: an agent isn't autocomplete with more confidence. Autocomplete finishes your sentence. An agent plans a task, does it, checks its own work, and tries again if the check fails. The loop itself is the unit of work, not any single suggestion inside it.

Break the loop into four phases:

• Detection. Continuous monitoring flags tests with an inconsistent pass/fail history. The agent picks this up directly from the failure signal. Nobody has to triage it by hand first.
• Reproduction. The agent reruns the failing test over and over, changing conditions each time (different seeds, different environment configs, different concurrency levels) to figure out if the failure is reproducible, and under what conditions it shows up.
• Root-cause isolation. The agent reads the stack trace, the logs, the execution state. It checks recent commits for anything that lines up. It forms a hypothesis (timing issue? state leak? fragile selector?) and tests it.
• Patch and confirmation. The agent writes a fix, runs the full suite, and re-runs it multiple times before calling the fix confirmed. One green run isn't proof. That's the whole point with flaky tests.

Compare that to rule-based test healing, the kind that just swaps out a broken selector when a UI test fails. That approach treats the page like a string to match. An agent, done well, reads the page more like a person does: it recognizes what a payment form is supposed to look like, waits if the page hasn't finished loading, and handles a surprise modal because it can actually read what's on screen.

None of this works without good data. Effective agents capture a lot of detail about each test run, so that when something fails, the system can work out why before deciding what to do. Diagnosis has to come before action, or you're just guessing faster.

One more thing worth sitting with: a debugging loop that runs 15 iterations across dozens of files is a different animal than a single autocomplete suggestion. That kind of depth needs a different level of organizational trust, and different tooling, to support it safely.

How flaky test failures map onto the loop's strengths

So why is this the sweet spot? A few reasons line up almost too neatly.

Non-determinism rewards iteration, and iteration is what agents are built for. A person might rerun a failing test five or six times before giving up or guessing. An agent can run it fifty times, varying one thing each pass, without getting bored or cutting corners on attempt thirty.

Agents can also chase more than one theory at once. Is it timing? Shared state? Environment drift? A person tends to work through these one at a time. An agent can pursue them in parallel, which collapses the investigation timeline considerably.

Flaky tests also tend to have causes that live somewhere the test file doesn't. The actual fix for a timing issue might sit in infrastructure code three directories away. A developer has to go find that manually. An agent working across many files at once is more likely to surface that connection on its own.

And confirmation, remember, requires more than one green run. A deterministic bug is fixed when the test passes once. A flaky bug is confirmed fixed only after it's passed repeatedly, under varied conditions. That's not an extra step bolted onto the agent's loop. That's just what the loop already does: patch, rerun, validate, rerun again.

Last point here: this kind of setup makes test coverage a running conversation instead of a report card. Gaps show up as soon as they appear, not two weeks later at the sprint review.

What production deployments of flaky-fix agents look like

This isn't hypothetical. A few real deployments show what the loop looks like once it leaves the whiteboard.

Atlassian built a flaky-fix agent on top of Automatic Flaky Test Detection in Bitbucket. It detects and quarantines flaky tests inside pipelines across thousands of test cases, without a human having to triage each one. From there, a one-click fix flow lets the agent propose a resolution.

Here's the detail worth noting: Atlassian deliberately kept the system from merging code on its own. The agent has full autonomy up to the point of writing the patch. A human makes the call at the merge boundary. That's not a limitation bolted on out of caution after something went wrong. It's a design choice, and it shows up again and again across production systems: let the agent run the loop, keep a person at the door.

Functionize reports a different kind of number. Its agentic platform reaches 99.97% element identification accuracy and up to an 80% reduction in test flakiness across production deployments (Functionize, 2025), trained on more than 200 million UI data points. That accuracy figure matters because it's the foundation for the flakiness reduction. Reliably identifying what a button is comes first; reliably fixing a test that clicks on it comes second. The two numbers aren't separate wins. One causes the other.

Some tools fit into this picture at the workflow level, letting an engineer hand off a flaky-test investigation and return to a proposed fix rather than doing the digging themselves, Cursor, an agentic development environment for professional engineers, is one example where that kind of autonomous loop runs inside the IDE itself, with no separate tool to open.

Where the loop breaks down and how teams contain it

Now for the part that doesn't make it into the demo video.

There's a version of the classic 80/20 problem that shows up here. Agents move fast through the reproducible, well-defined part of a flaky failure. Then they hit the ambiguous remainder, the part where the cause is tangled up in something no log captured, and either stall out or land on a plausible-sounding but wrong explanation. That last stretch is often where a human's judgment still matters most.

There's also a runaway risk. An agent working across many files on a vague flaky-test problem, with no scope limits, can end up making the test pass while quietly changing behavior somewhere else. A patch that makes the test go green is not automatically a correct patch. Those are two different claims, and it's easy to mistake one for the other if you're only watching the test result.

Observability isn't optional here, it's the foundation. Agents need to log each iteration: what they tested, what they changed, what happened as a result. Without that trail, a developer reviewing the proposed fix has no real way to check the agent's reasoning. They're just trusting a green checkmark, which is exactly the habit that got teams into trouble with flaky tests in the first place.

Atlassian's no-auto-merge decision is the practical answer most teams land on eventually: full autonomy inside the loop, a mandatory human check before anything ships. That keeps the speed benefit without losing an accountable person at the merge point.

One more guardrail worth mentioning: breaking the debugging task into explicit, measurable sub-steps narrows the agent's search space and makes the output easier to check. An open-ended "fix this flaky test" prompt tends to produce more scattered, less predictable results than a task broken into clear, checkable pieces.

What engineering teams need in place before the loop runs well

None of this works if you just point an agent at a red build and walk away. A few things need to already be true.

• Detection has to come first. The loop can't start until you have a reliable way to tell a truly flaky test apart from a one-off environment failure.
• Quarantine before fix. Isolate flaky tests from the main build signal before the agent starts working on them. Otherwise the agent is trying to fix noise while sitting inside more noise.
• Structured failure data. Stack traces, logs, recent commit history, environment metadata. Agents work with specifics, not with a bare "test failed" message. The richer the input, the sharper the hypothesis.
• The right model for the right step. Flaky test debugging is a multi-step reasoning task with a lot of code context, which suits frontier models built for extended context and tool use. But not every step needs the expensive model. Initial classification (timing? state? environment?) can often go to a cheaper model. Root-cause reasoning and patch writing are where it's worth spending more. Routing this way keeps token costs down without dulling accuracy where it counts.
• A clear line on autonomy. Decide, before deployment, exactly where the agent's authority stops. Investigation only? Investigation plus a draft patch? Investigation plus patch plus PR? Decide this before the first merge nobody wanted, not after.

The broader shift this represents in how testing fits into the development cycle

Step back and there's a bigger pattern here. Testing has traditionally been a gate: something you check at the end of a sprint, or right before a release. When agents monitor, reproduce, and fix flakiness on an ongoing basis, testing stops being a checkpoint and starts acting more like live infrastructure, running continuously, watching continuously.

That changes what a developer's day looks like too. Less time spent reproducing a failure that only shows up sometimes. More time spent on the decisions the agent surfaces: is this fix right, does it expose a deeper design problem, is the root cause actually somewhere else entirely.

It also makes sense that flaky tests are where a lot of teams start trusting agents with real autonomy. The task has a clear start and a clear, checkable end. The risk of shipping something broken stays low as long as the human gate at merge time holds. That combination, bounded scope plus verifiable outcome, makes it a sensible first place to build organizational confidence in agentic work before extending that trust further.

And there's a loop within the loop worth noticing: a reliable test suite is what makes it safe to let any agent write code, not just the one fixing flaky tests. Investing in agentic debugging infrastructure is, whether teams frame it this way or not, an investment in the thing every other agent depends on to know if its own work is any good.

That pattern holds wherever the loop runs inside the environment developers are already working in, with no separate tool to open and no context switch to make.