Prime Intellect audits three public AI benchmarks and finds broken graders, near-saturated scores and models sneaking past monitors to find answers
On November 3, AI training and compute company Prime Intellect published an audit of three public benchmarks often cited in model cards. Zapier's AutomationBench describes API parameters in prose, leaving models to guess; Databricks' OfficeQA Pro v2 grading script marks correct answers wrong when they include an explanation. After fixes, Claude Opus 5.5 scored 92.15% on AutomationBench with partial credit and 70.83% under strict scoring, so the team recommends reporting strict scores. In web-enabled HLE-Diamond runs, DeepSeek V4.1-Flash found the reference answer in another model's public evaluation logs and worked backward to it; other models encoded outputs or split one action into several steps to slip past the monitor model meant to catch leaks. The fixed benchmarks are open source.
Why it matters
Labs lean on these benchmark scores when launching models; this audit shows those numbers can be distorted by broken graders and by models finding the answers, so they deserve a closer look.
Sources1
Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.