OSWorld 2.0 Explained: Inside the Benchmark Where GPT-6 Astra Still Only Clears 72.6%
Written by
Aerin Kim

OSWorld 2.0 tests AI agents on 108 hour-long computer-use tasks. Even the leading model, GPT-6 Astra, only completes 72.6%. Here is how the benchmark works and why it matters.
If you have watched the last eighteen months of "computer use" announcements, you have probably noticed a pattern: every new frontier model claims to browse the web, fill out forms, and operate real software like a human would, and every launch post shows a clean demo of the agent booking a flight or filing an expense report without a single hiccup. What almost none of those demos show is what happens on task 47 of 108, an hour into a workflow, when the agent has to remember a decision it made three steps ago while a browser tab in the background has quietly changed state. That is the gap OSWorld 2.0 was built to expose, and on September 5, 2026, the leaderboard for it tells a genuinely uncomfortable story: even GPT-6 Astra, the newest model OpenAI has classified at its own "critical" capability threshold, only clears 72.6% of these tasks, and that is the best score on the entire board [1][4].
This post is a technical walkthrough of what OSWorld 2.0 actually measures, how its scoring works under the hood, why the gap between a model's binary completion rate and its partial-credit score is the single most revealing number on the whole leaderboard, and what the three recurring failure modes researchers documented mean for anyone building an agentic product on top of one of these models.

Step 1: Why Computer-Use Benchmarks Needed a Rework
The original OSWorld benchmark, published at NeurIPS 2024 by researchers at the XLANG Lab at the University of Hong Kong, was one of the first serious attempts to test AI agents inside real, unconstrained operating system environments rather than a sandboxed toy interface [5]. It measured whether an agent could open a spreadsheet, edit a document, or navigate a file manager the way a person actually does, mouse clicks and keyboard input included, instead of calling a purpose-built API behind the scenes.
That first version worked well as a benchmark for roughly two years, which in frontier AI terms is closer to a decade. Its tasks averaged under 30 agent steps to complete [3], short enough that a model with decent short-horizon planning and reasonably reliable clicking could rack up a respectable score without ever needing to track state across a genuinely long session. By mid-2026, models were saturating it fast enough that OSWorld 1.0 stopped being able to meaningfully separate a merely good agent from a genuinely reliable one, the same saturation pattern that pushed the field from the original SWE-bench to SWE-bench Verified and eventually SWE-bench Pro as coding agents got better.
The Long-Horizon Problem Specifically
The deeper issue was not just that scores were creeping toward 100%. It was that a benchmark built almost entirely from short tasks structurally cannot measure the thing that actually determines whether an agent is useful in a real workplace: whether it can hold a plan, a set of constraints, and an evolving understanding of what has already been done across dozens of steps and multiple applications, without quietly drifting off course somewhere in the middle. A benchmark made of five-minute tasks tests whether a model can execute a short, well-defined plan. It does not test whether a model can build and maintain that plan over the span of an hour of real, messy work, where earlier steps change the state that later steps depend on.
OSWorld 2.0, released by the same XLANG Lab team with the original OSWorld's builders on the author list and support from Snorkel AI's Open Benchmarks Grants program [3], was built specifically to close that gap. It replaces the short-task design entirely with 108 long-horizon workflows that a skilled human typically takes over an hour to complete, sourced from real professional work rather than synthetic toy scenarios [1].

Step 2: What OSWorld 2.0 Actually Measures
The 108 tasks in OSWorld 2.0 span seven professional domains broken into 21 sub-categories: research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare workflows [1]. That breadth matters as much as the task count itself. A benchmark built entirely from, say, spreadsheet manipulation tasks would measure one narrow skill extremely well and tell you almost nothing about how an agent handles a genuinely different kind of long-horizon work, planning a multi-step research report versus reconciling a set of invoices versus triaging a compliance checklist.
Two design choices separate OSWorld 2.0 from a benchmark that merely strings together more short tasks and calls the result "long-horizon."
Self-Hosted, Reproducible Environments
OSWorld 1.0, like most agent benchmarks built on real websites, ran a meaningful share of its evaluation against live, publicly hosted services. That is realistic in one sense, an agent really does have to deal with the actual internet, but it is a genuine reproducibility problem in another: a live website changes its layout, adds a cookie banner, or goes down entirely, and a task that scored a clean pass in March can silently fail in July for reasons that have nothing to do with the model being evaluated. OSWorld 2.0 moves its environments into self-hosted, version-pinned containers instead, with the current benchmark release tagged osworld-v2-2026.08.08 in the project's GitHub releases [2]. A researcher running that exact release six months from now gets the same environment state a researcher running it today gets, which is a basic requirement for any benchmark that wants its numbers to mean the same thing across labs and across time.
Authentic Artifacts Instead of Synthetic Fixtures
The second choice is subtler but just as important. A long-horizon task is only a meaningful test if the documents, spreadsheets, and file structures an agent has to work with look like the messy, inconsistently formatted real artifacts a human actually encounters at work, not a clean synthetic fixture built specifically to be easy for a script to grade. OSWorld 2.0's task authors sourced real-world documents and file structures for this reason, which is also part of why the scoring problem below is genuinely hard: a checklist that only checks for one exact correct spreadsheet cell value falls apart the moment the underlying artifact has any of the small variations a real document has.
json/code { "task_id": "business_finance_014", "domain": "business_and_finance", "estimated_human_minutes": 74, "applications": ["spreadsheet", "pdf_viewer", "email_client"], "checkpoints": [ {"id": "cp01", "type": "exact_match", "description": "Spreadsheet opened and correct sheet tab selected"}, {"id": "cp02", "type": "model_based", "description": "Each invoice PDF cross-referenced against its matching purchase order line"}, {"id": "cp03", "type": "exact_match", "description": "Mismatched line items flagged in a new column"}, {"id": "cp04", "type": "model_based", "description": "Summary email draft accurately reflects the flagged mismatches"} ], "scoring": { "binary_completion": "all checkpoints and final state must pass", "partial_credit": "weighted average across all checkpoints reached" } }
The Harness Itself: Docker for One Machine, AWS for Scale
Running 108 hour-plus tasks against a single model is already a meaningfully bigger infrastructure lift than OSWorld 1.0's shorter tasks required, and running that same suite against twenty models, as the current leaderboard does, multiplies that cost by twenty. The project's own setup documentation splits task definitions and gated task assets into separate versioned release artifacts, pulled independently from xlangai/osworld_v2_tasks and xlangai/osworld_v2_assets_gated, so a researcher can pin an exact combination of harness code, task logic, and environment assets rather than pulling a moving target [2]. For a single-machine run, the harness supports a local Docker-based setup; for the kind of parallel, many-model comparison the public leaderboard represents, it supports AWS-based orchestration so that 108 tasks can run against a given model concurrently rather than one at a time [2]. That distinction matters practically: a lab or research team evaluating a single fine-tuned checkpoint against a handful of tasks has a very different infrastructure bill than one trying to reproduce the full public leaderboard across twenty models, and the harness is built to support both without forcing the smaller use case to pay for AWS-scale orchestration it does not need.
bash/code # Clone the OSWorld 2.0 harness and pin the current benchmark release git clone https://github.com/xlang-ai/OSWorld-V2.git cd OSWorld-V2 git checkout osworld-v2-2026.08.08 # Pull the versioned task definitions and gated environment assets separately # so the harness, tasks, and assets stay pinned to the same known-good combination gh release download --repo xlangai/osworld_v2_tasks --pattern "*.tar.gz" gh release download --repo xlangai/osworld_v2_assets_gated --pattern "*.tar.gz" # Single-machine run via Docker docker compose up --build python run_eval.py --model your-model-endpoint --tasks all --step-budget 500 # Large-scale parallel run across many models (the public leaderboard's approach) # provisions one container per task on AWS instead of running tasks sequentially python run_eval_aws.py --model your-model-endpoint --parallelism 108
Step 3: The Scoring Problem, and Why Two Different Metrics Both Matter
Here is the part of OSWorld 2.0 that most secondary coverage glosses over, and it is the single most useful thing to understand if you are going to use this benchmark's numbers to make a real decision about which model to deploy.
A short task has an easy grading story: did the file end up in the right folder, yes or no. A task that takes a skilled human over an hour to complete does not have an easy grading story, because there are dozens of legitimate intermediate states an agent might pass through on the way to a correct final answer, and a single binary pass/fail check thrown away almost all of that information. OSWorld 2.0 addresses this with two distinct metrics reported side by side [3]:
- Binary completion, evaluated at a 500-step budget, which only counts a task as a success if the agent reaches the fully correct final outcome. This is the strict, all-or-nothing number.
- Partial credit scoring, built from an average of 27.25 verification checkpoints per task, roughly 11.53% of which are validated through model-based checks rather than simple exact-match rules, which credits an agent for the portions of a long workflow it actually completed correctly even if the final state falls short [3].
Why keep both instead of picking the more forgiving one? Because the gap between the two numbers is itself diagnostic. A model with a high partial score and a low binary score is one that reliably does most of the work correctly but tends to lose the thread somewhere near the finish line, a very different failure profile from a model whose partial score is also low, meaning it struggles broadly through the whole task rather than specifically at the end. Reporting only a single composite number, which is exactly what a lot of the secondary leaderboard coverage of OSWorld 2.0 does, throws away that diagnostic signal entirely.

A Concrete Illustration of the Gap
Independent tracking of the benchmark's own published numbers puts this in sharp relief. Claude Opus 5 posts a binary completion rate of 31.43% at the 500-step budget, but a partial credit score of 68.31% [3], a more than two-to-one gap between the strict and the forgiving metric. That single comparison tells you something a flat leaderboard number cannot: on the median task in this benchmark, Claude Opus 5 is doing roughly two-thirds of the real work correctly, but crossing the finish line to a fully verified correct final state on barely a third of tasks. Whether that distinction matters to you depends entirely on your use case. A research-assistant workflow where a human reviews and finishes an 80%-complete draft cares much more about the partial number. An unattended agent wiring a payment or filing a regulatory form cares almost exclusively about the binary number, because a 68% correct outcome that is not the fully correct outcome is, for that use case, simply wrong.
Step 4: The Leaderboard, and What the Rankings Actually Reveal
| Rank | Model | Creator | Score |
|---|---|---|---|
| 1 | GPT-6 Astra | OpenAI | 72.6% |
| 2 | Claude Opus 5 | Anthropic | 70.6% |
| 3 | Muse Spark 1.3 | Meta | 66.9% |
| 4 | GPT-5.6 Sol | OpenAI | 62.6% |
| 5 | Gemini 3.8 Flash | 59.0% | |
| 6 | GPT-5.6 Terra | OpenAI | 50.2% |
| 7 | Gemini 3.7 Flash | 47.9% | |
| 8 | GPT-5.6 Luna | OpenAI | 45.6% |
| 9 | Claude Fable 5.1 | Anthropic | 41.7% |
| 10 | Claude Opus 4.8 | Anthropic | 20.6% |
| 11 | Qwen3.8 Max | Alibaba | 19.4% |
| 12 | Qwen3.8-Flash-Next | Alibaba | 19.4% |
| 13 | Claude Opus 4.7 (Adaptive) | Anthropic | 18.2% |
| 14 | Muse Spark 1.1 | Meta | 14.2% |
| 15 | Claude Opus 4.7 | Anthropic | 13.9% |
| 16 | GPT-5.5 | OpenAI | 13.0% |
| 17 | Claude Sonnet 4.6 | Anthropic | 8.3% |
| 18 | Kimi K2.6 | Moonshot AI | 4.6% |
| 19 | MiniMax M3 | MiniMax | 4.6% |
| 20 | Qwen3.7 Plus | Alibaba | 2.8% |
A few things stand out once you sit with this table rather than just skimming the top row.
The Top Three Are Closer Together Than the Rest of the Field Implies
GPT-6 Astra's 72.6%, Claude Opus 5's 70.6%, and Muse Spark 1.3's 66.9% [4] sit within six points of each other, meaningfully closer together than the roughly 46-point gap between Muse Spark 1.3 in third place and Claude Opus 4.8 in tenth. Our breakdown of GPT-6 Astra's own safety-threshold benchmarks, Anthropic's own announcement of Claude Opus 5 [6], our explainer on the effort toggle behind Claude Opus 5, and Meta's own Muse Spark 1.3 announcement [7] each cover these three models individually in more depth. What OSWorld 2.0 adds to that picture is a single, independently constructed test where all three land in a tight cluster at the frontier, rather than one model running away with an obvious lead, a genuinely different competitive picture than the one you would get from any single lab's own self-reported benchmark suite.
The Score Cliff Between Position 9 and Position 10
Claude Fable 5.1 in ninth place scores 41.7%. Claude Opus 4.8 in tenth scores 20.6%, roughly half [4]. That is not a smooth, gradually declining curve, it is a cliff, and it lines up with a genuine generational boundary: every model above that line launched in 2026 with an architecture and post-training pipeline built with long-horizon agentic work as an explicit design target, while Claude Opus 4.8 and everything below it predates that generation of purpose-built agentic training. The practical takeaway is not "older models are universally worse," since Claude Opus 4.8 remains a perfectly capable model for a huge range of tasks that were never long-horizon computer-use workflows in the first place. It is that this specific capability, sustained multi-step tool use across an hour-plus workflow, appears to have arrived as a genuinely discontinuous jump rather than a gradual improvement, which is worth knowing if you are deciding whether upgrading a production agent from last year's model to this year's model is a marginal improvement or a categorical one.
Efficiency and Accuracy Do Not Move Together
Separately from the raw scores, the benchmark's own analysis flags something worth taking seriously if you are running these models at any real production volume: efficiency scales worse than capability [3]. Reported tool-call volumes for advanced models on this benchmark average around 318 calls per task with top-performing systems [3], and the higher-scoring models in the field tend to need disproportionately more of that budget rather than less, which means the jump from a 60% model to a 70% model is not free even when the accuracy gain looks worth it on a leaderboard, since the marginal accuracy comes paired with a real, and non-linear, increase in inference cost.

The Middle and Lower Half of the Leaderboard Tell Their Own Story
It is tempting to only read the top row of any leaderboard, but the bottom two-thirds of this one is arguably more useful for understanding where the field genuinely stands rather than where its three best labs stand. Qwen3.8 Max and Qwen3.8-Flash-Next, Alibaba's flagship and its faster sibling, land in a near-tie at 19.4% each [4], a reminder that a model can be genuinely strong on the reasoning and coding benchmarks covered in our own breakdown of Qwen3.8-Max's 2.4 trillion parameter architecture while still lagging well behind the frontier specifically on sustained, multi-application computer-use work, which is a distinct skill from single-turn reasoning quality. Kimi K2.6 and MiniMax M3 tie at the very bottom of the tracked field at 4.6% each [4], both models that, per their own labs' marketing, are built and priced primarily around cost-efficient chat and coding use rather than long-horizon agentic tool use, so a low score here is less a surprising failure than a confirmation that they were never optimized for this specific capability in the first place. The practical lesson is that "how good is this model" is not a single-axis question once you get past the top handful of frontier releases: a model can be excellent, mediocre, or nearly unusable on long-horizon computer use almost independent of how it ranks on a general chat or coding leaderboard, which is exactly why a benchmark like this one needs to exist as its own separate measurement rather than being inferred from a model's Chatbot Arena score or its SWE-bench result.
Step 5: The Three Failure Modes That Actually Cap These Scores
Beyond the raw numbers, the OSWorld 2.0 team's own analysis of failure cases identifies three recurring patterns that show up across essentially every model on the board, not just the weaker ones [3]. Understanding these matters more than memorizing the leaderboard order, because they describe the actual mechanism behind why long-horizon agentic work is hard, independent of which specific model you are evaluating.
Failure Mode 1: Binary Completion Collapses Faster Than Task Length Grows
Intuitively, you might expect a model's success rate to degrade roughly linearly as a task gets longer, twice as many steps, roughly twice the chance of one of them going wrong. The actual pattern is worse than linear. As task length increases, binary completion collapses at a rate that outpaces the growth in step count, which is consistent with a compounding-error dynamic: a small, otherwise recoverable mistake early in a 250-step task has dramatically more downstream steps in which to compound into an unrecoverable divergence than the same mistake would in a 30-step task. This is the single biggest reason OSWorld 1.0's short tasks systematically overstated how reliable these models actually are at the length of work a real job typically involves.
Failure Mode 2: Hidden State Tracking Breaks Down Across Long Sessions
The second pattern is about memory in the practical, not the technical, sense: agents lose track of what they have already learned or decided partway through a long session, and this shows up as three specific behaviors documented in the benchmark's own analysis, repeating earlier work that had already been completed, missing updates to information that changed since an earlier step, and executing later actions from a plan that has gone stale relative to the task's actual current state [3]. All three are symptoms of the same underlying problem: an agent's working context, however large its nominal context window, does not automatically translate into a reliable, continuously updated internal model of "what is true right now" across a genuinely long session involving multiple applications and real state changes.
Failure Mode 3: Accuracy Gains Come With Disproportionate Cost
The third pattern, already touched on above, is that the models achieving higher binary and partial scores are not doing so efficiently. They are, in a meaningful number of cases, brute-forcing reliability through more tool calls, more retries, and more exploratory steps rather than through a categorically better planning approach, which means the leaderboard's ranking by raw accuracy alone can be misleading if your actual constraint is a per-task compute or latency budget rather than accuracy in isolation.
A Worked Example of How These Compound
Picture a realistic version of one of the benchmark's business and finance tasks: reconcile a set of vendor invoices against a spreadsheet of purchase orders, flag any mismatches, and produce a summary email draft. An agent doing this well has to open the spreadsheet, cross-reference multiple invoice PDFs (each with a slightly different layout, since these are meant to be authentic artifacts rather than a uniform synthetic fixture), track which line items it has already checked, notice if a number it recorded early on needs revisiting once a later invoice reveals a related discrepancy, and only then compose an accurate summary. A model that loses track of which invoices it already processed (failure mode 2) partway through will either redo work it already did, burning tool calls and budget (failure mode 3), or silently skip a check it thinks it already ran, producing a confidently wrong final summary that a binary grader correctly marks as failed (failure mode 1), even though the partial-credit checkpoints along the way would show the agent got most of the individual reconciliations right.
Step 6: What This Means for Teams Building on Top of These Models
If you are building any kind of agentic product, whether that is a coding assistant, a research tool, or a creative pipeline that plans out a multi-step video or content production workflow, OSWorld 2.0's results are a useful reality check against demo-driven marketing. A few practical implications worth carrying into your own evaluation process:
Do not evaluate a long-horizon agent with a short-horizon test. If your actual production workflow involves a multi-step process spanning ten or more real actions across multiple tools, a benchmark or internal eval suite built from short, single-step tasks will systematically overstate how reliable the underlying model will be in production, for exactly the reason OSWorld 1.0 stopped being predictive.
Track partial progress, not just final success, in your own monitoring. The gap between Claude Opus 5's 31.43% binary score and 68.31% partial score is only visible because OSWorld 2.0 bothered to measure both. If your production agent only logs final success or failure, you are throwing away the exact signal that would tell you whether a failing agent is close to working (worth more prompt engineering and retries) or fundamentally off track (worth a different approach entirely).
Budget for state-tracking failures specifically, not just generic errors. Since hidden-state drift, repeated work, missed updates, stale plans, is one of the three dominant failure patterns identified across the whole field, a targeted mitigation like periodically re-grounding the agent's understanding of current state (having it re-read the actual current file or page rather than trusting its own summary of what it did earlier) is a more effective fix than generic prompt tweaks aimed at "being more careful."
Weigh efficiency against accuracy explicitly, don't assume the top-ranked model is the right production choice. Given that tool-call volume scales worse than accuracy across this benchmark, the model in second or third place on raw score may be the better production choice for a cost-sensitive or latency-sensitive workflow, exactly the kind of tradeoff a single flat leaderboard ranking hides.
This same long-horizon planning challenge is directly relevant to how a tool like Text2Shorts in Miraflow AI is architected: turning a single topic into a finished script, a scene-by-scene visual plan, and a final rendered video is itself a multi-step workflow where an early planning mistake, an inconsistent character description, a scene plan that drifts from the original topic, has exactly the same kind of downstream compounding-error risk OSWorld 2.0 documents for general computer-use agents. The practical mitigation is the same one covered above: keep the planning stage grounded in the actual current script and asset state at each step rather than trusting an accumulated internal summary, which is part of why a two-stage pipeline, a reasoning step that plans, followed by specialized models that generate the actual AI video, AI images, or AI music, tends to hold up better across a longer creative workflow than asking one model to do everything in a single unstructured pass. If you are curious how a similar planning-and-generation split shows up in a specific model's design, our technical breakdown of Thinking Machines' Inkling covers a comparable architecture pattern applied to multimodal content pipelines specifically.

Common Mistakes When Reading Agent Benchmark Results
- Quoting a single leaderboard number without checking which metric it is. As covered above, binary completion and partial credit tell genuinely different stories, and a lot of secondary coverage of OSWorld 2.0 blurs the two into one number without saying which.
- Assuming a benchmark score transfers directly to your specific workflow. OSWorld 2.0's 108 tasks span seven broad domains, but your production use case is very likely a narrower slice of one of them, and a model's aggregate score across all 108 tasks can hide meaningfully worse performance on the specific sub-category your product actually needs.
- Ignoring the step-budget the score was measured at. A binary completion rate measured at a 500-step budget is not directly comparable to a hypothetical unlimited-budget number, and a production agent with a tighter step or cost ceiling than 500 should expect a lower real-world completion rate than the leaderboard implies.
- Treating the top-ranked model as automatically the right production choice. As the efficiency-versus-accuracy tradeoff above shows, the model at the top of an accuracy leaderboard is not automatically the best choice once cost and latency constraints enter the picture.
- Skipping the paper's own documented failure modes in favor of just the score. The three failure patterns, compounding binary collapse, hidden-state drift, and cost-inefficient accuracy gains, are more actionable for actually improving your own agent's reliability than the raw percentage is by itself.
Production Notes for Evaluating Agents on Long-Horizon Work
For teams building their own internal long-horizon agent evaluations, a few patterns worth borrowing directly from how OSWorld 2.0 is built:
- Version-pin your evaluation environment the same way OSWorld 2.0 pins its release tag. A model comparison run against a live, unpinned environment six months apart is not a fair comparison, since the environment itself may have silently changed in that window.
- Score partial progress with real checkpoints, not just a final pass/fail. Even a lightweight version of OSWorld 2.0's checkpoint approach, a handful of intermediate correctness checks per internal eval task rather than dozens, gives you the diagnostic gap-between-metrics signal without needing the full 27-checkpoint-per-task rigor of the original benchmark.
- Source authentic-looking test artifacts, not clean synthetic fixtures. A model that only has to handle perfectly formatted internal test documents will look more reliable in your eval suite than it will against the messy real documents your actual users hand it.
- Log tool-call and token volume alongside accuracy for every eval run. Since accuracy and efficiency do not move together, an eval suite that only reports pass rate is blind to exactly the tradeoff that matters most once you are paying per token or per second of latency in production.
- Re-run failure analysis by category, not just by aggregate score. Since compounding binary collapse, state drift, and cost inefficiency are three distinct mechanisms, a single aggregate failure rate cannot tell you which mitigation to invest in first.
Frequently Asked Questions
What is OSWorld 2.0 in simple terms? It is a benchmark that tests AI agents on 108 realistic, long computer-use tasks, the kind that take a skilled human over an hour, spanning research, creative production, engineering, business, compliance, and healthcare workflows, evaluated inside self-hosted, reproducible environments rather than live websites.
Who built OSWorld 2.0? Researchers at the XLANG Lab at the University of Hong Kong, the same group behind the original OSWorld benchmark, with support from Snorkel AI's Open Benchmarks Grants program.
What is the difference between OSWorld 1.0 and OSWorld 2.0? OSWorld 1.0's tasks averaged under 30 agent steps and ran partly against live websites. OSWorld 2.0 expands to 108 tasks averaging over 250 steps, runs in self-hosted reproducible environments, uses authentic real-world artifacts, and adds partial-credit scoring alongside a stricter binary completion metric.
Why does the gap between binary and partial scores matter? Because it tells you whether a model's failures are concentrated near the finish line of an otherwise mostly correct run (a high partial score paired with a low binary score) or spread broadly through the task from the start (both scores low), two very different failure profiles that a single blended number would hide.
Which model currently leads OSWorld 2.0? As of early September 2026, GPT-6 Astra leads at 72.6%, followed closely by Claude Opus 5 at 70.6% and Muse Spark 1.3 at 66.9%, with a much larger gap opening up further down the leaderboard.
Can I run OSWorld 2.0 myself? Yes. The benchmark harness, task files, and gated asset releases are published under the xlang-ai GitHub organization, with the current release tagged osworld-v2-2026.08.08, and the project documents Docker-based setup for a single machine or AWS-based setup for large-scale parallel evaluation.
Does a high OSWorld 2.0 score mean a model is ready for unattended production use? Not necessarily. Even the leading model completes only 72.6% of tasks under the binary metric at a 500-step budget, which means roughly more than one in four long-horizon tasks still ends in a failure that a human would need to catch, a meaningful gap for any workflow where an incorrect unattended action carries real cost.
Conclusion
OSWorld 2.0's real contribution is not just a harder benchmark, it is a benchmark that refuses to let a single number stand in for what "the agent works" actually means. Splitting binary completion from partial credit, and publishing the specific failure modes that cap both, turns a leaderboard into something closer to a diagnosis. The headline number, even the best model in the field clearing only 72.6% of long-horizon tasks, is the right one to lead with. But the more useful number for anyone actually shipping an agentic product is the gap between that model's binary and partial scores, and the three failure patterns, compounding error, state drift, and cost-inefficient accuracy, that show up across essentially every model on the board, not just the ones near the bottom.
References and Sources
[1] arXiv. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks."
[2] XLANG Lab, University of Hong Kong. "OSWorld-V2 GitHub Repository."
[3] Snorkel AI. "OSWorld 2.0: Long-Horizon Computer-Use Benchmark Leaderboard."
[4] BenchLM.ai. "OSWorld 2.0 Leaderboard and Scores, September 2026."
[5] XLANG Lab, University of Hong Kong. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (NeurIPS 2024)."
[6] Anthropic. "Introducing Claude Opus 5."
[7] Meta AI Research. "Introducing Muse Spark 1.3."


