Free🇭🇰🇨🇳
Chinese evaluation suite with 13,948 multiple-choice questions in 52 disciplines and four difficulty levels for foundation models.
OSWorld overview and key facts (October 2026)
Benchmark of 369 real computer tasks on Ubuntu, Windows and macOS that tests multimodal agents operating desktop and web apps.
Free
What is OSWorld?
OSWorld is an open benchmark that tests AI agents on real computer tasks in full operating systems. It shows how far computer-use agents are from handling office and web workflows on their own.
OSWorld contains 369 tasks, or 361 without some Google Drive tasks, with real web and desktop apps, file operations and workflows across several applications. Environments run in VMware, VirtualBox, Docker or cloud platforms such as AWS and Azure. OSWorld-Verified, released in July 2025, fixed reported issues and added parallel evaluation on AWS. In the original paper, humans completed more than 72 percent of tasks. The project is Apache 2.0 licensed.
OSWorld AI features and use cases
- Judges computer-use agents for office workBefore letting an agent operate spreadsheets, documents or browsers for staff, IT teams can check its success rate on comparable OSWorld tasks. The gap to human performance shows where supervision stays necessary.
- Tests own automation agentsTeams building agents for internal processes can run the benchmark in Docker or on AWS and compare versions on the same tasks with execution-based scoring. Parallel runs on AWS can bring evaluation time under an hour.
- Compares vendors on verified resultsOSWorld-Verified publishes results and trajectories, so buyers can inspect how an agent solved or failed a task instead of relying only on a score. This makes typical failure patterns visible.
What sets OSWorld apart
- Tasks run in real operating systems rather than only in the browser; the environment supports Ubuntu, Windows and macOS.
- Execution-based scoring checks the actual end state of each task.
Where OSWorld is strongest
Hong Kong Available internationally
OSWorld is led by researchers at the University of Hong Kong with Salesforce Research, Carnegie Mellon University and the University of Waterloo.
Questions to ask before you buy OSWorld
Running OSWorld needs virtual machines with enough compute, or a cloud account on AWS or Azure. Results from the original version and OSWorld-Verified are not directly comparable. Scores describe general desktop tasks, so pilot agents on your own applications and data before deployment.
OSWorld reviews
No reviews yet. Be the first to share how your team uses it.
Every review shows its verification level. We read each one before it goes live. How we check reviews
Questions about OSWorld
What does OSWorld test?
Whether multimodal agents can complete 369 real tasks with web and desktop apps, file operations and multi-app workflows. The project adds 43 Windows tasks for analysis, and the environment supports Ubuntu, Windows and macOS.
How well do people perform?
In the original study, humans completed more than 72 percent of the tasks, while the best model at the time reached about 12 percent.
Where can we run it?
On VMware or VirtualBox, in Docker with KVM support, or on cloud platforms such as AWS and Azure.
OSWorld update historyno updates yet
No changes recorded for OSWorld yet.
Changes the vendor made and confirmed user reports, each checked against the vendor's own pages. We review every tool monthly.
Alternatives to OSWorld
Tools in the same category (Domain benchmarks) with the most overlapping markets.
Free🇺🇸
Collaborative benchmark of 162 English legal reasoning tasks from 40 contributors, covering contracts, statutes and court opinions.
Free🇨🇳
Open-source LLM evaluation platform from Shanghai with CompassRank leaderboards, the CompassHub benchmark catalog and VLMEvalKit.
Contact for pricing · Free plan🇺🇸
Scale AI's leaderboards with expert-built and partly private evaluations such as Humanity's Last Exam, SWE-Bench Pro and safety tests.