A benchmark made of real GitHub issues from real Python projects: the model is given the bug report and the repository and must produce a patch that makes the hidden tests pass. It started at under two per cent in late 2023, and by 2025 the leading models cleared seventy on the verified subset. It is the number coding agents are sold on and the one they are most accused of training for.
Sighted in the launch chart of every coding product since 2024: a bar for SWE-bench Verified, a rival's bar slightly shorter, and a footnote about scaffolding that the record reads first. In 2025 the footnotes grew longer than the charts.
I remember in 2023 the top score being a couple of per cent and thinking the benchmark would stand for a decade. Eighteen months later a model fixed a bug in a library I maintain, correctly, from the issue I had written, and I had a strange sense of having been benchmarked myself.
Filed as the benchmark that made agents necessary. A multiple-choice test can be answered in one shot. Fixing a bug means reading files, running tests, and trying again, so SWE-bench scores are really scores for the loop around the model, and the labs know it.
There is not one SWE-bench. The full set has over two thousand issues; 'Verified' is five hundred that humans checked were solvable; 'Lite' is three hundred easier ones. Scores on different subsets are quoted as if they were the same test. They are not, and the difference is often the whole gap between rivals.
What does it mean? Write it the way you would say it out loud.
