Research paper🇺🇸
Meta paper on Llama Guard, a language model that classifies chatbot prompts and responses against a safety risk taxonomy.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? research paper summary (October 2026)
ICLR 2024 paper introducing SWE-bench: 2,294 real GitHub issues from 12 Python projects as a test bed for language models.
What is SWE-bench: Can Language Models Resolve Real-World GitHub Issues??
The SWE-bench paper introduces a benchmark that asks language models to fix real GitHub issues by editing a codebase. It became the standard reference that vendors of coding agents quote when they compare their products.
SWE-bench consists of 2,294 software engineering problems from real GitHub issues and pull requests in 12 popular Python repositories. A model gets the codebase and the issue and must produce a patch that passes the tests. The paper was an oral presentation at ICLR 2024.
The repository contains the evaluation harness, which runs fully in Docker containers, plus SWE-bench Multimodal, which became fully open source with 480 tasks in September 2026. SWE-bench Verified, a 500-task subset confirmed by engineers, was created with OpenAI. The repository is under MIT, had 5,928 stars and was updated on September 18, 2026.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? AI features and use cases
- Makes vendor claims comparableCoding agent vendors often report SWE-bench or SWE-bench Verified scores. Engineering leads can read those numbers correctly once they know which variant and setup was used.
- Tests agents on your own termsThe Docker-based harness lets teams run candidate models or agents on the same tasks themselves. Procurement decisions can rest on reproducible results rather than marketing figures.
- Checks visual software tasksSWE-bench Multimodal adds tasks from visual software domains with screenshots. Teams building front-end tools can see how agents handle UI bugs.
What sets SWE-bench: Can Language Models Resolve Real-World GitHub Issues? apart
- Tasks come from real repositories, not synthetic puzzles.
- Grading is automatic through the projects' own tests.
- Several variants exist, so scores are only comparable within one variant.
Where SWE-bench: Can Language Models Resolve Real-World GitHub Issues? is strongest
Available internationally
The paper is freely available on arXiv, and the accompanying code runs on your own hardware in any country.
Questions to ask before you buy SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
SWE-bench covers Python repositories only in its original form, so results may not transfer to your languages and codebase. Running the harness needs Docker and considerable compute. When a vendor quotes a score, ask which variant, which agent scaffold and how many attempts were used.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? reviews
No reviews yet. Be the first to share how your team uses it.
Every review shows its verification level. We read each one before it goes live. How we check reviews
Questions about SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
What is SWE-bench Verified?
A subset of 500 problems that software engineers confirmed as solvable, created in collaboration with OpenAI's Preparedness team, according to the README.
Can we run the benchmark ourselves?
Yes. The repository provides a Docker-based evaluation harness, and cloud evaluation via Modal and the sb-cli tool are also documented.
Which languages does it cover?
The original benchmark uses 12 popular Python repositories. SWE-bench Multimodal adds visual software domains.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? update historyno updates yet
No changes recorded for SWE-bench: Can Language Models Resolve Real-World GitHub Issues? yet.
Changes the vendor made and confirmed user reports, each checked against the vendor's own pages. We review every tool monthly.
Alternatives to SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Tools in the same category (Safety & evaluation) with the most overlapping markets.
Compare SWE-bench: Can Language Models Resolve Real-World GitHub Issues? with these alternatives →