Web pages were the training wheels. OSWorld hands the agent an actual operating system — real apps, real files, cross-application workflows — 369 tasks over five OSes, verified by machine-checked configurations.
The escalation chain: widgets → websites → the desktop itself.
OSWorld's environments run actual operating systems in sandboxes — real file managers, editors, terminals, browsers, and applications. Tasks are open-ended goals ("create the file X by extracting Y from this spreadsheet and Z from that PDF") whose success is checked by setup-configuration verification: the environment state itself is queried (files exist with the right content, app settings hold the right values). The agent interacts through the same interface humans use — screenshots in, keyboard/mouse actions out — making vision a first-class requirement, not a convenience.
What benchmarks couldn't see: the OS around the app.
A web benchmark is a microwave with a camera — one appliance, one interface, reheating only. OSWorld hands over the whole kitchen: the recipe spans fridge, stove, and pantry (files, apps, system settings); the judge tastes the final dish by testing it directly (configuration checks); and the agent must move like a cook — reading the room (screenshots), not the API manual.
Observe by screenshot, act by keyboard/mouse, verify by state — the full stack in one card.
The interface decision that made the benchmark matter.
Giving agents DOM or OS-API access would make tasks easier — and make the benchmark lie: deployed assistants face pixels, not APIs. By forcing the screenshot channel, OSWorld measures the capability humans actually need from "computer use": visual grounding (which button is that?), layout reading (where in the dialog am I?), and robustness to UI churn (the redesign didn't update your selector — because you don't have one). This is the same thesis SWE-agent (entry #79) argues for terminals: interface design is agent capability.
The 2024 numbers that set computer-use expectations.
| Layer | Web benchmarks | OSWorld |
|---|---|---|
| Arena | browser pages | full operating systems |
| Interface | DOM/accessibility | screenshots + keyboard/mouse |
| Cross-app workflows | no | yes — files, clipboard, installs |
| Verification | page/DB state | environment configuration checks |
| Frontier gap (launch) | large | agents < 22% even with aids |
The escalation that completed the environment ladder: widgets → web → desktop.
OSWorld became the proving ground for the computer-use agent wave.
Check your understanding of the key concepts from OSWorld.
Everything you need to remember about this paper.