Key takeaways
- The top five SWE-bench Verified scores reported in March 2026 spanned 0.9 percentage points, from 80.0% to 80.9%. That gap is smaller than the noise.
- Identical model weights are reported to score 10 to 20 points apart depending on which evaluation harness runs them. The harness is a bigger variable than the model.
- OpenAI stopped reporting SWE-bench Verified in February 2026 after an audit found material problems in at least 59.4% of 138 audited items, as reported by benchmark trackers.
- A leaderboard position is a claim about one harness on one dataset on one date. It is not a claim about your workload.
- The only benchmark that predicts your production behaviour is one built from your own failed runs.
Does ChatGPT recommend your competitor instead of you?
Check how AI assistants describe your company, and whose name they give when someone asks for a recommendation in your category. Run an audit today, from $19.
In March 2026 the top five entries on SWE-bench Verified were separated by nine tenths of a percentage point. Claude Opus 4.5 at 80.9, then 80.8, 80.6, 80.2 and GPT-5.2 at 80.0.
Reported harness variance on identical model weights runs to ten or twenty points. Which means the ranking you are reading was decided by something other than the models.
What does a 0.9 point spread actually tell you?
That five labs have converged on the same measurement, and that the measurement has stopped discriminating between them.
| Model | Reported score | Gap to leader |
|---|---|---|
| Claude Opus 4.5 | 80.9% | - |
| Claude Opus 4.6 | 80.8% | 0.1 |
| Gemini 3.1 Pro | 80.6% | 0.3 |
| MiniMax M2.5 | 80.2% | 0.7 |
| GPT-5.2 | 80.0% | 0.9 |
| Reported swing from harness alone | 10 to 20 points | 11x to 22x the spread |
The last row is the article. Every difference in that table is an order of magnitude smaller than the difference your own test rig would introduce.
Why do identical weights score differently?
Because a benchmark is not a ruler. It is a program, and the program has opinions.
Prompt format changes what the model is being asked. Retry policy changes how many attempts a failure gets. Timeout changes whether a slow correct answer counts as correct. Tool availability changes what the agent can even attempt. Grading strictness decides whether a patch that works but differs from the reference is a pass.
None of those is the model. All of them are in the score.
What this means when a vendor quotes a number
Ask which harness, at which settings, on which date. A number without those three is a marketing claim wearing the costume of a measurement. Vendors who publish their harness configuration are making a checkable statement. Vendors who publish only the number are not.
How badly is the underlying data contaminated?
Enough that the benchmark’s most prominent user walked away from it.
Benchmark trackers report that an OpenAI audit of SWE-bench Verified found material test or task-description problems in at least 59.4% of the 138 items it examined, and that the company stopped reporting the benchmark in February 2026. The same reporting says every frontier model tested could reproduce gold patches or problem statement specifics verbatim for some tasks.
Verbatim reproduction is the tell. A model that can recite the answer is not solving the task, and a score built partly on recitation cannot be read as capability.
We have not seen OpenAI’s audit document. We say so because the figure is doing real work in this article and you should know how far it is from us.
What should you measure instead?
Something that costs you a day and predicts your outcome, rather than something free that predicts a leaderboard.
| Approach | What it predicts | Effort | Goes stale |
|---|---|---|---|
| Public leaderboard | Performance on that harness, that dataset, that date | None | Immediately on contamination |
| Vendor-published eval | What the vendor chose to show | None | Unknown, no methodology |
| Golden set you wrote once | Regression against problems you already solved | Days | Within weeks |
| Replay of your own failed runs | Behaviour on the cases that actually break you | A day, then continuous | Refreshes itself |
The last row is the only one that improves with age, because production keeps producing new failures and each one is a free test case. Harvesting them is unglamorous work that nobody puts in a launch post.
The question to bring to your next model evaluation
Not “which model scores highest”. Ask: on the fifty runs that failed us last month, which candidate fails fewest, graded the way we grade in production? If nobody on the team can produce those fifty runs, that is the finding.
Frequently asked questions
Is SWE-bench Verified still a reliable benchmark in 2026?
Less than its ubiquity suggests. Trackers report that an OpenAI audit found material test or task-description problems in at least 59.4% of 138 audited items, and that OpenAI stopped reporting the benchmark in February 2026. It remains useful as a rough capability signal and unreliable as a tiebreaker.
Why do the same models get different benchmark scores?
Because the harness is part of the measurement. Prompt format, retry policy, timeout, tool availability and grading strictness all move the number. Reported swings of 10 to 20 points on identical weights are larger than the gaps between the models being compared.
What is benchmark contamination?
It is when the answers are already in the training data. In agent coding benchmarks it shows up as a model reproducing a gold patch or problem statement verbatim, which means the score measures recall of the test set rather than the capability the test claims to measure.
How should you choose a model for an agent if benchmarks are unreliable?
Run your own tasks. Take fifty real runs that failed in production, replay them against each candidate with your actual tool definitions, and grade with the check you would apply in production. That takes a day and predicts your outcome better than any public leaderboard.
Method and sources
- Benchmark tracking write-ups published in 2026 reporting on an OpenAI audit of SWE-bench Verified, including the figure of material problems in at least 59.4% of 138 audited items and the February 2026 decision to stop reporting the benchmark. We have not seen OpenAI's audit document directly and cite these as reported.
- SWE-bench Verified leaderboard positions as of March 2026: Claude Opus 4.5 at 80.9%, Claude Opus 4.6 at 80.8%, Gemini 3.1 Pro at 80.6%, MiniMax M2.5 at 80.2%, GPT-5.2 at 80.0%, as published by benchmark aggregators.
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination, arXiv:2509.00072. Consulted on how contamination is detected and why release dates are a weak control.
- Reported harness variance of 10 to 20 percentage points on identical model weights, from 2026 benchmark methodology write-ups.
- No figures in this article are our own measurements.