What it measures
Solve real-world GitHub issues from 12 popular Python repos. The 'Verified' subset is human-validated to remove ambiguity and have working tests.
500-task verified subset. Run-time evaluation; can't be gamed by pure memorisation but agent harness affects results. Currency (2026-06-21): Verified is now effectively saturated/retired — OpenAI's now-live post "Why we no longer evaluate SWE-bench Verified" recommends the concrete named successor SWE-bench Pro (Scale AI, 1,865 tasks); frontier % resolved has moved well past the tracked 78.4% (Claude Opus 4.8 ~88.6%, May 2026; OpenAI cites 74.9%->80.9% in 6 months); saturationStatus updated to "saturated" to match the body, and OpenAI now points to the SWE-bench Pro successor (Scale AI) (iter-451 audit fix).
Contamination & gaming
SWE-bench Verified is a run-time benchmark — a candidate patch is applied and executed against the repository's tests — so it is not gameable by pure text memorisation in the way a multiple-choice exam is; this execution-grounded design aligns it with agentic-evaluation regimes that score multi-step tool-use behaviour rather than recall 1. The article's at-a-glance "medium" contamination rating nonetheless rests on a concrete temporal problem: the task instances are drawn from real GitHub issues and their resolving pull requests, and over 94% of the issues in the original SWE-bench and their pull requests were created prior to the training cut-off dates of current frontier models 2. The same study isolates leakage diagnostically: state-of-the-art models identify the buggy file path from the issue text alone with up to 76% accuracy on Verified, but only ~53% on issues from repositories absent from the benchmark — a gap consistent with memorisation rather than reasoning over the codebase 3. In an internal audit, OpenAI reports that every frontier model it tested could reproduce verbatim gold patches and problem details for some Verified tasks, indicating direct training exposure (OpenAI 2026). The standard mitigation is temporal decontamination: pipelines such as SWE-rebench continuously collect tasks created after model release dates and filter accordingly 4, and successor variants (e.g. "Pro"/private hold-outs) exist precisely so that headline numbers can be re-grounded on unseen code.
Critiques & limitations
The "Verified" subset was built to fix documented defects in the original SWE-bench: OpenAI engaged 93 professional Python developers to screen 1,699 sampled instances for underspecified issue text and over-strict tests that reject valid fixes, yielding the 500-task curated set (OpenAI 2024). Curation did not, however, resolve two deeper construct-validity problems. SWE-Bench+ finds that 32.67% of successful patches involve solution leakage — the fix appears in the issue report or comments — and that 31.08% of passing patches are "suspicious" because the test oracle is too weak to confirm correctness; both pathologies persist in SWE-bench Verified, not only in Lite 2. When problematic items are removed, SWE-agent+GPT-4's resolution rate collapses from 12.47% to 3.97%, implying that a substantial share of headline performance is attributable to flawed items rather than genuine issue-resolution 2. OpenAI's own audit of 138 tasks o3 failed across 64 runs, each reviewed by at least six experienced software engineers, found 59.4% contained flawed tests — 35.5% over-strict and 18.8% testing unspecified behaviour (OpenAI 2026). Such item-level fragility is why the capability-evaluation literature stresses that headline eval results should be interpreted cautiously and audited against the underlying items rather than taken at face value 5. In Policy Window's composite reading, a Verified score conflates true SWE capability with leakage and oracle artefacts; the construct it nominally measures — autonomously resolving real multi-file issues 6 — is only loosely identified by the score.
Saturation & score trajectory
The benchmark moved from near-floor to near-ceiling in roughly two years, a trajectory that itself motivated its retirement. In the original SWE-bench paper the strongest system, Claude 2, resolved only 1.96% of issues, and the authors noted that even frontier proprietary models solved "only the simplest issues" given the need to coordinate edits across multiple functions, classes, and files 6. The Verified curation reset the baseline upward — at launch GPT-4o resolved 33.2% of Verified tasks versus ~16% on the uncurated set, which OpenAI framed as evidence the original underestimated capability (OpenAI 2024). Vendor model cards subsequently reported figures in the high-70s, and the claim tracked on this article (claude-opus-4-7, 78.4%, 2025-05-22) sits in that band; the stakes of getting such a score right are high because software development is among the work-task domains most exposed to LLM automation 7. As scores compressed toward the top of the 0–100% range, the headline lost discriminative power, and the combination of saturation with the contamination and test-flaw evidence above led OpenAI to stop reporting Verified and recommend others do the same, pointing to harder, decontaminated successors (OpenAI 2026). The lesson Policy Window draws — an editorial reading — is that a metric near ceiling is best treated as a lower bound on the easiest tasks rather than a frontier-capability signal, since rankings among near-saturated systems are dominated by the artefacts the prior section documents; this is consistent with the broader eval literature's caution against reading capability headlines at face value 5.
Results & interpretation
Claimed scores
| Model | Score | Claim type | Reported | Citation |
|---|---|---|---|---|
| claude-opus-4-7 | 78.4 % solved | vendor card | 2025-05-22 | Anthropic model card |
How to read this number
Contamination risk: medium
Some test items may leak into training corpora; treat headline scores with mild skepticism and prefer evaluation runs with held-out subsets.
What a high score does and does not establish. A score evidences performance on this benchmark’s specific construct under its specific format; it is not, on its own, evidence of general capability, reliable real-world task performance, or safety. This benchmark is saturated, so small differences near the ceiling no longer reliably separate frontier from mid-tier systems.
The second silence. evidence: thin The evidence that a benchmark score predicts real-world deployment outcomes (construct-to-deployment validity) is sparse; benchmark performance and deployed performance are not established to be the same thing, and contamination can inflate the headline figure above true held-out ability.
Governance relevance
A benchmark measures a capability; governance attaches to the topicsthat capability bears on. These topic articles carry the instrument×dimension coverage matrix and the social-science so-what for this domain.
- Foundation Models / GPAI— coverage matrix + does-governance-work evidence
- Agentic AI Governance— coverage matrix + does-governance-work evidence
Further reading
50 academic & grey-literature sources on the governance questions this benchmark's results inform — catalogued metadata with a primary link; one-line findings are ✦ AI-generated summaries, labeled as such (charter §7.9). Browse the full literature index.
- Governing AI Agents Preprint✦ AIUses "agency law and theory to identify and characterize problems arising from AI agents" and proposes governance infrastructure built on inclusivity, visibility, and liability.
- Infrastructure for AI Agents Peer-reviewed✦ AIProposes "agent infrastructure": external technical systems for attributing actions "to specific agents, their users, or other actors," shaping interactions, and remediating harms.
- Multi-Agent Risks from Advanced AI Research institute✦ AIIdentifies three failure modes of advanced multi-agent systems — "miscoordination, conflict, and collusion" — plus seven risk factors, posing challenges distinct from single-agent AI.
- An interdisciplinary account of the terminological choices by EU policymakers ahead of the final agreement on the AI Act: AI system, general purpose AI system, foundation model, and generative AI Peer-reviewed✦ AITraces how the AI Act's legal text shifted across versions among the terms 'AI system, general purpose AI system, foundation model, and generative AI', exposing definitional instability in the regime.
- The EU model of AI governance: regulating artificial intelligence through law and policy Peer-reviewed✦ AIAnalyses how the AI Act's risk-based model handles general-purpose and foundation models whose 'autonomous content generation challenges legal categories of authorship, accountability, and control'.
- Generative AI and data protection Peer-reviewed✦ AIExamines friction between foundation-model training and the GDPR, noting models that 'memorize and leak pieces of training data' cannot be treated as anonymous.
- Authenticated Delegation and Authorized AI Agents Preprint✦ AIIntroduces a framework for authenticated, authorized, and auditable delegation to AI agents by extending OAuth 2.0/OpenID Connect, maintaining accountability chains for agent actions.
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents Peer-reviewed✦ AIProvides a 440-task benchmark across 11 harm categories measuring whether LLM agents resist or comply with harmful multi-step tool-use tasks, grounding safety-evaluation regimes for agents.
- Better together? Human oversight as means to achieve fairness in the European AI Act governance Peer-reviewed✦ AIExamines whether Article-14 human oversight of high-risk/autonomous AI can actually deliver fairness, probing the limits of human-in-the-loop as a governance mechanism.
- GPTs are GPTs: Labor market impact potential of LLMs Peer-reviewed✦ AIFinds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies".
- Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity Peer-reviewed✦ AIExamines how the EU AI Act, liability regimes, GDPR, copyright and cybersecurity rules apply to generative AI, identifying gaps and proposing targeted regulatory refinements.
- Visibility into AI Agents Peer-reviewed✦ AIProposes agent identifiers, real-time monitoring and activity logs to give governance actors visibility — "where, why, how, and by whom certain AI agents are used."
+ 38 more on these governance questions — see the literature index.
References
Sources cited inline in the analysis (linked from the superscript markers), then the primary instrument sources behind the classifications.
- Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, Xander Davies (UK AISI / Gray Swan) (2025) AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents, ICLR 2025. arXiv:2410.09024 — Provides a 440-task benchmark across 11 harm categories measuring whether LLM agents resist or comply with harmful multi-step tool-use tasks, grounding safety-evaluation regimes for agents. ↩
- arXiv:2410.06992 ↩
- arXiv:2506.12286 ↩
- arXiv:2505.20411 ↩
- Mary Phuong, Matthew Aitchison, Elliot Catt, et al. (Google DeepMind) (2024) Evaluating Frontier Models for Dangerous Capabilities, arXiv (cs.LG). arXiv:2403.13793 — Pilots dangerous-capability evaluations (persuasion, cyber, self-proliferation) on frontier models, finding 'early warning signs' but no strong present danger — grounding evaluation-based gating. ↩
- arXiv:2310.06770 ↩
- Eloundou, Manning, Mishkin, Rock (2024) GPTs are GPTs: Labor market impact potential of LLMs, Science. 10.1126/science.adj0998 — Finds around 80% of the U.S. workforce "could have at least 10% of their work tasks affected" by LLMs, which exhibit "traits of general-purpose technologies". ↩
- SWE-bench Verified methodology
- claude-opus-4-7 — 78.4 % solved (Anthropic model card, 2025-05-22)
How to cite this benchmark
Use the primary methodology source for academic citations; reference the Policy Window article for the cross-model leaderboard.
Cite this article 8 formats · BibTeX, RIS, APA, Chicago, … · 1-click copy
Persistent identifier: https://policywindow.org/wiki/swe-bench-verified — committed-stable URL with content-versioning via ?asOf= (rollout pending per methodology §7). DOIs via Zenodo are on the roadmap.
Article tools — track changes, suggest an edit
View history — every captured revision of this article · What links here