Free🇺🇸
Collaborative benchmark of 162 English legal reasoning tasks from 40 contributors, covering contracts, statutes and court opinions.
SWE-bench overview and key facts (October 2026)
Benchmark that tests whether AI systems resolve real GitHub issues, with Verified, Lite, Multimodal, Multilingual and Bash Only boards.
Free
What is SWE-bench?
SWE-bench tests whether AI coding systems can fix real issues from open-source GitHub repositories. Its leaderboards are a common reference when companies compare coding assistants and agents.
The benchmark gives a model a repository and an issue and checks whether its patch makes the tests pass. Variants include Verified, a set of 500 problems confirmed solvable by engineers and built with OpenAI, Lite, Multimodal with visual software tasks, Multilingual and Bash Only. Evaluation runs in Docker containers, locally or in the cloud via Modal and sb-cli. SWE-bench Multimodal v2 with 480 tasks became fully open source in September 2026. The code is MIT licensed.
SWE-bench AI features and use cases
- Compares coding agentsThe percentage of resolved issues gives engineering leaders a common measure when choosing between coding assistants and agents. Results on Verified avoid problems that are unsolvable by design. Scores are published per variant.
- Tests an in-house agentTeams building their own coding agent can run the containerized harness locally or in the cloud and track progress between versions on the same tasks. The Docker-based harness keeps runs reproducible.
- Checks visual and multilingual skillsThe Multimodal and Multilingual variants show whether an agent handles front-end tasks with screenshots or languages other than Python. This matters for teams with mixed technology stacks. The results help set expectations per language.
What sets SWE-bench apart
- Tasks come from real GitHub issues and are scored by running the repository's tests.
- Several leaderboard variants, including a human-verified subset of 500 problems.
Where SWE-bench is strongest
United States Available internationally
SWE-bench is maintained by researchers at Princeton University and Stanford University in the United States, according to the contacts in its README.
Questions to ask before you buy SWE-bench
Public issues and fixes may have been seen during model training, so strong scores can overstate performance on your own code base. Running the harness needs Docker and compute, or a cloud option. Most tasks come from Python projects; test agents on your own repositories before a purchase decision.
SWE-bench reviews
No reviews yet. Be the first to share how your team uses it.
Every review shows its verification level. We read each one before it goes live. How we check reviews
Questions about SWE-bench
What does SWE-bench measure?
The percentage of real GitHub issues an AI system resolves, checked by running the repository's tests on the proposed patch.
What is SWE-bench Verified?
A subset of 500 problems that software engineers confirmed are solvable, created together with OpenAI Preparedness in August 2024.
Can we run it without our own servers?
Yes. Evaluations can run in the cloud via Modal, and the sb-cli tool submits runs to the leaderboards.
SWE-bench update historyno updates yet
No changes recorded for SWE-bench yet.
Changes the vendor made and confirmed user reports, each checked against the vendor's own pages. We review every tool monthly.
Alternatives to SWE-bench
Tools in the same category (Domain benchmarks) with the most overlapping markets.
Contact for pricing · Free plan🇺🇸
Scale AI's leaderboards with expert-built and partly private evaluations such as Humanity's Last Exam, SWE-Bench Pro and safety tests.
Contact for pricing🇺🇸
Independent evaluator that benchmarks AI models on professional work in law, finance, tax, healthcare, coding and other fields.
Free🇭🇰🇨🇳
Chinese evaluation suite with 13,948 multiple-choice questions in 52 disciplines and four difficulty levels for foundation models.