SWE-Bench Verified Results Across Coding Agents

SWE-bench exists to answer a simple question: can an AI agent actually fix real software bugs, not just write clean little functions in isolation? That's a much harder test than what came before it, and it's why the benchmark became the reference point almost everyone in the field now uses.

Researchers from Princeton University built it, later maintaining it alongside Stanford University. Instead of inventing toy problems, they pulled real GitHub issues from real open-source repositories, including Django, Flask, and scikit-learn. An agent gets handed an issue and the state of the repository, then has to produce a patch. Success isn't a judgment call or a vibe check. The patch either passes the project's actual tests or it doesn't.

Three versions of the benchmark exist today. The original release had 2,294 tasks and barely gets cited anymore at the frontier. SWE-bench Verified trimmed that down to 500 tasks, each checked by a human, and it's the version almost everyone means when they casually say "SWE-bench score." SWE-bench Pro came later, built by Scale AI specifically to resist the contamination problems that eventually caught up with Verified, a story covered in detail further down.

Why did this format matter so much? Because the tools being evaluated had changed shape. Coding assistants used to be copilot-style: suggest a line, suggest a function, let a human decide. Agents now plan a fix, write the code, run the tests, read the failure, and try again, sometimes for an hour or more without a human in the loop. A benchmark like HumanEval, built around isolated function stubs, has no way to capture that looping, diagnostic process. SWE-bench was built for exactly that kind of work.

How fast scores have risen on the current leaderboard

Progress on this coding benchmark has been fast enough to make last year's leaderboard look dated. As of late 2026, the top of the field is packed tighter than almost anyone would have guessed a year earlier.

Seven of the 86 models evaluated now score 95% or better. The single leader sits only three percentage points away from a perfect run on 500 tasks. That alone says something about how saturated the top of this benchmark has become.

What's notable is who's up there. Open-weight models have pushed into the very top tier, not just the upper-middle. DeepSeek V4 Pro 0813 sits second overall at 96.40%, a mere 0.60 points behind the closed leader. Kimi K3 scores 93.40%, which puts it ahead of closed, proprietary models including Claude Opus 4.8 at 88.60% and Grok 4.5 at 86.60%. That's a real shift from the era when closed labs simply had the best numbers by default.

Below that top cluster sits a wide band of strong, closely matched models: Claude Opus 4.5 at 80.9%, GPT-5.2 at 80.0%, Gemini 3.1 Pro at 80.6%, and MiniMax M2.5 at 80.2%. Four different labs, four different architectures, all landing within a single percentage point of each other¹²³⁴. That's not one standout model pulling ahead of a weak field. It looks like broad, simultaneous progress across the industry.

A change in what the benchmark is actually testing produces these scores. Agents aren't limited to a single prompt and a single response anymore. They run for minutes, sometimes hours, exploring a codebase, writing changes, running the test suite, reading the errors, and adjusting. Performance differences appear most clearly on tasks that take between 15 minutes and an hour to complete, and models that handle that mid-length window well tend to score higher across the board. Genuine agent capability actually separates from noise not on the trivial one-line fixes, and not on the multi-day epics, but in that middle zone where planning and iteration actually get tested.

Why scores above roughly 80% are hard to interpret

The qualities that made this benchmark's verified set useful in the first place, a fixed, public set of real issues, are the same qualities now undermining its precision at the top of the leaderboard. A fixed dataset that sits in public view for years is a dataset that training data can eventually absorb.

Independent research has found that a substantial share of successful patches on Verified involve solution leakage. Models recall the correct file paths for a given issue from training data at a notably high rate. A high score is measuring memory as much as it's measuring problem-solving. An agent that has effectively seen the answer before isn't demonstrating the same skill as one reasoning through a codebase cold.

There's a separate problem, and it's arguably the more surprising one: the tests themselves aren't all trustworthy. An audit of the benchmark's hardest, still-unsolved problems found that more than half had flawed test cases, ones that rejected solutions that were actually functionally correct. The benchmark is grading correct answers as failures, not a contamination issue.

Then there's scaffolding. SWE-bench Verified doesn't isolate the model being tested, it evaluates the model and its surrounding agent harness together. Depending on which test harness gets used, top scores on the same model can diverge by more than ten points. Increasingly, the number a vendor reports reflects engineering choices in the scaffolding around the model as much as it reflects the model itself.

Even the company steering people away from Verified ran into reliability problems with the replacement.

What does this mean for anyone actually trying to use these numbers? Verified still works fine as a coarse filter: a score above roughly 80% suggests a model is worth testing on real work. What it can no longer do reliably is rank-order two models that are both above that line. The leaderboard's neat decimal points imply a level of precision the underlying numbers don't carry anymore.

The 35-point reality check: what happens when the same model faces tasks it hasn't seen

When the same class of model is tested on tasks it couldn't have memorized, performance can fall by something on the order of 35 points. It's the difference between a benchmark score and a real-world expectation, not a rounding error.

Think about what any actual engineering team is working with day to day. Their codebase isn't Django. Their bug tickets aren't sitting in scikit-learn's public issue tracker, waiting to be scraped into a training set. Every real assignment an agent gets handed at a company looks more like the harder commercial benchmark's task set than like the verified set: unfamiliar code, unfamiliar conventions, no chance the answer leaked into training data months earlier.

That commercial-set gap is the closest available proxy for the question that actually matters to a team evaluating these tools: how will this agent perform on code it has genuinely never encountered? Even for the strongest models on the public leaderboards, the answer comes in considerably more modest than a Verified score would suggest. None of this means these agents are bad at their jobs. It means the headline number on Verified was never a reliable estimate of real-world capability, and treating it as one has been the mistake.

SWE-bench Pro's own reliability problems

SWE-bench Pro was built to solve exactly the contamination described above. On that front, it's a real improvement. It is not, however, a clean or settled solution, and it comes with grading failures substantial enough to limit how much weight any single Pro score deserves.

The structure itself is harder to game. Pro covers 1,865 real-world software tasks across 41 professional repositories, scored Pass@1, spanning Python, Go, TypeScript, and JavaScript, with an average of 107.4 lines changed per task. That's a meaningfully bigger lift than Verified's typical task, and the breadth of languages and repositories makes memorization far less useful as a shortcut.

Scale AI also runs every model through identical scaffolding on its standardized leaderboard, which isolates model capability from harness quality, the exact problem that muddies Verified's rankings. That's the single biggest methodological upgrade Pro offers over its predecessor.

As of September 2026, Meta's Muse Spark 1.1 leads Scale's standardized public set at 61.5%, with GPT-5.4 (xHigh) close behind at 59.1%. On the harder commercial set, Muse Spark 1.1 again leads at 51.5%, followed by Claude Opus 4.6 (thinking) at 47.1% and Meta's Muse Spark at 44.7%.

For anyone trying to cite "the" score for the harder commercial benchmark, it gets genuinely confusing. Three different numbers all claim that title: 61.5% (Muse Spark 1.1, on Scale's standardized public set), 89.9% (Claude Opus 5.5, from the llm-stats vendor aggregate), and 51.5% (Muse Spark 1.1, on Scale's private commercial set). None of them is wrong, exactly. The spread between them comes down to which scaffolding and which data split produced it, and most pages that cite a Pro score never specify which one they mean. That's not a minor labeling issue. It makes "SWE-bench Pro score" close to meaningless as a phrase unless the source and split are named alongside it.

Pro also has its own grading accuracy problem. Audits estimate a substantial share of its tasks are broken, with automated graders accepting incorrect implementations and rejecting correct ones at rates serious enough that Epoch AI designated the benchmark "Flawed." Datacurve's figures put this at roughly 8.5% of trials where incorrect code was accepted, and roughly 24% where correct code was rejected, which works out to something close to one in three trials mis-graded.

The pattern across all three versions becomes clear when you zoom out. SWE-bench moved from original, to Verified, to Pro, and at each step it fixed real, documented problems with the version before it. At each step, it also introduced new ones. That's less a story about any one team's execution and more a structural fact: a static benchmark, evaluated against a frontier of models that keeps moving, is always going to keep breaking in new places.

What the benchmark's convergence reveals about model selection

Putting the contamination problems and the saturation together reveals a different, more useful story: at the frontier, the actual performance gap between strong models on general issue-resolution tasks is smaller than the leaderboard's decimal points suggest. The practical question for anyone picking a tool shouldn't be "which model has the highest score" so much as "which model fits this specific workflow."

For a long time, closed-source models simply scored better than open-source ones on Verified, and that gap felt reliable enough to plan around. It no longer is. DeepSeek V4 Pro 0813 at 96.40% and Kimi K3 at 93.40% now sit in the same tier as the best closed models, which compresses a gap that used to be a safe assumption into something much narrower.

It also helps to be clear-eyed about what Verified never measured. It doesn't test security vulnerability detection. It doesn't test code review quality. It's Python-centric, so it says nothing about performance in other languages. It doesn't test maintainability, compliance with a company's internal coding standards, behavior under production load, or performance on a private codebase that looks nothing like a public GitHub repo. That list covers most of what an enterprise engineering team actually cares about day to day.

So what's the right way to use the number? Treat it as a threshold, not a ranking. A model scoring above roughly 80% on Verified is worth putting through real evaluation on actual work. The exact order of finish among models above that line isn't a signal worth trusting.

Flexibility across models may matter more than commitment to any single leaderboard leader, since the best agent for one kind of task may not be the best agent for another. The best agent for a narrow, targeted code edit, the best agent for an unsupervised overnight run across a large codebase, and the best agent for reviewing someone else's pull request may turn out to be three different models. Building workflows that can route tasks to different models, rather than locking into whichever one currently tops a chart, looks like the more durable strategy given how quickly these rankings shuffle.

What better evaluation looks like

The direction researchers are already moving in says a lot about where this conversation goes next. Instead of treating a single aggregate score as the final word, newer analysis is breaking leaderboard results down task by task, checking how much of a top score comes from genuinely shared successes across models versus how much separates them, and checking what happens when the same model-and-scaffold pairing gets swapped out and re-tested.

That kind of granular auditing, counting which specific instances separate the top ten models rather than just comparing aggregate percentages, is a more honest way to read a crowded leaderboard. It also matches what the Pro grading audits already demonstrated: the quality of the test cases and graders underneath a benchmark matters as much as the task design itself.

None of this means this benchmark, in any of its three versions, should be thrown out. Verified remains a reasonable first filter, and Pro remains a harder, more contamination-resistant test, even with its grading issues. What's changing is the expectation placed on these numbers. A benchmark score was always meant to be a starting point for evaluation, not a substitute for it.