Browser Use and Carnegie Mellon release Odysseys2, a long-horizon web agent benchmark where the top agent fully completes 47 of 100 tasks
Odysseys2, from Carnegie Mellon and Browser Use, is a web agent benchmark of 100 tasks drawn from Browser Use's real user requests, run on the live web with 982 rubric checks. An LLM judge grades the files agents produce along with their actions and screenshots, and gives zero credit to reward hacking. Of 15 agents tested, GPT-6 Astra leads, fully completing 47 of 100 tasks.
Why it matters
Once the judge sees the files agents produce, Astra's perfect runs on the same 100 tasks drop from 86 to 47, suggesting judges that see only screenshots and actions overstate how much web agents really finish.
Sources2
odysseysbench.comMediaOdysseys2: A Web Agent Benchmark Based on Real User RequestsView original
- @browser_useXIntroducing: Odysseys2 - hardest long horizon benchmark for web agents based on real user tasks 🏗️ We collaborated with @JangLawrenceK @rsalakhu from Carnegie Mellon University > SOTA agent fully completes only 47/100 > 5 agents complete none > 8-12 real rubrics per task, 60View original
Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.