"I'm ready to ignore AA's index"
On Oct. 28, a user on r/LocalLLaMA, Reddit's main forum for people who run models on their own hardware, opened a post by establishing credentials as a non-skeptic. "I've always actively disagreed with the conspiracy takes when AA reshuffled their index," the user wrote, "but v5.0 now has zero open weight models in AA's Most Attractive Quadrant." The conclusion: "I'm ready to ignore AA's index and look at the subsidiary benchmarks." The poster later added that the screenshot was taken while Artificial Analysis was "still posting new scores," but that "the pattern holds."
The target was the Artificial Analysis Intelligence Index v5, released the same day (DaLao flash). Artificial Analysis, an independent benchmarking firm, publishes a composite score that many developers use to pick models and estimate costs. Version 5 is the biggest overhaul since v4 arrived in January. It adds Cognition's FrontierCode 2.0 coding benchmark and Terminal-Bench-Science, a suite of scientific research tasks, swaps Humanity's Last Exam for HLE-Diamond, and drops GDPval-AA, SciCode and the long-context test AA-LCR. "With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers," the firm wrote.
Open-weight models took a beating. Artificial Analysis said the gap between the best open-weight model and the leader grew to 12 points from 3 in v4.3. Kimi K3.1 Pro, whose weights haven't been released, tops the open-weight group at 46, against 58 for Anthropic's Claude Opus 5.5, the leader at launch. Among models whose weights are actually downloadable, DeepSeek V4.1 Flash 1017 and Z.ai's GLM-5.3 tie at 32, 26 points behind Opus and below Anthropic's small Claude Haiku 5.5, at 36. Both gaps are measured against Opus. Google's Gemini 4 Carbon has since taken the top spot at 59; measured against it, the gaps are 13 and 27 points.
The firm's headline and its critics both frame this as open-weight versus closed. The per-test scores on Artificial Analysis's own site tell a different story. The new tests mostly separate a handful of labs' flagship models from everything else, and mid-tier closed models land in the pack with the open-weight ones. Open-weight models lose the most ground on tests whose questions are public and that anyone can run.
Same models, two exams
The overhaul makes a clean comparison possible. Artificial Analysis scored the same models on the old and new tests, and both sets of numbers sit in its site data.
Take HLE. On the original Humanity's Last Exam, DeepSeek V4.1 Flash 1017 scored 43.3%, Kimi K3.1 Pro 52.5% and Xiaomi's MiMo-V2.6-Pro 49.4%. On HLE-Diamond, a 1,000-question version the Center for AI Safety (CAIS) and Scale AI released in September after a year of cleanup, those scores fell to 25.3%, 37.3% and 15.2%. Over the same switch, Claude Opus 5.5 rose to 67.9% from 61.4%, and Gemini 4 Carbon rose to 69.1% from 60.4%.
Open-weight models weren't the only ones to fall. Closed Claude Haiku 5.5 dropped to 29.6% from 44.4%, a loss of 14.8 points. Grok 4.7 fell to 23.2% from 43.1%, and Alibaba's Qwen3.8 Max to 24.0% from 43.1%, losses of 19.9 and 19.1 points, both bigger than DeepSeek's 18.0.
The coding numbers are starker, and they come from the benchmark's author. When Cognition released FrontierCode 2.0 on Oct. 27, it published old and new scores side by side. Claude Opus 5.5 slipped to 45.5% from 54.6% on version 1.1, a 9-point decline. DeepSeek V4.1 Flash fell to 19.3% from 43.9%, and Kimi K3 to 17.7% from 44.2%. Among closed models, only Opus 5.5 and OpenAI's GPT-6 Astra, which fell to 42.8% from 53.3%, lost about 10 points. Every other model in the table dropped 19 to 27 points. Google's Gemini 4 Argon, which scores 50 on v5 and sits in the flagship tier, fell to 26.3% from 47.5%, and Grok 4.7 to 28.6% from 47.6%; DeepSeek lost 24.6 points and Kimi K3 26.5. Compared with DeepSWE v1.1 and FrontierCode 1.1, Cognition wrote, "FrontierCode 2.0 separates models more clearly across tiers of capability."
One of the dropped tests is where open-weight models had come close. On AA-LCR, the long-context reasoning test, Kimi K3.1 Pro and MiMo-V2.6-Pro both scored 86.3%, ahead of Claude Opus 5.5 at 84.7%, with DeepSeek V4.1 Flash 1017 at 82.7%. But AA-LCR carried just 5% of v4.3's weight, and Kimi K3.1 Pro's 1.6-point edge over Opus there was worth less than 0.1 index point. Dropping it barely moved the gap; the new tests did.
The v5 table now looks like this. Gemini 4 Carbon, released Oct. 28, leads at 59. Claude Opus 5.5, GPT-6 Astra and the other flagships sit between 49 and 59, with Kimi K3.1 Pro (46) and Meta's Muse Wonder (43) in between. Below that, closed models such as Claude Haiku 5.5 (36), Grok 4.7 (34), Qwen3.8 Max (34) and Gemini 3.8 Flash (31) cluster around 30 with DeepSeek and GLM, whose weights are released. On this board, the downloadable models and mid-tier closed models sit in the same tier.
Where the gap comes from
The 60% private share is what critics object to: outsiders can't see the questions or check the grading. Artificial Analysis's published weights and per-test scores are enough to work out whether the gap sits in the private tests or the public ones.
Four of v5's nine tests are public: Terminal-Bench 4.0, Terminal-Bench-Science, HLE-Diamond and GDP.pdf. Together they carry 40% of the weight. The other five, worth 60%, are private or have held-out answers.
Claude Opus 5.5 leads DeepSeek V4.1 Flash 1017 by about 25.9 points. Multiplying each test score by its weight, the four public tests account for about 14.2 of those points, more than half. The private FrontierCode 2.0, CritPt, AutomationBench and AA-Omniscience tests add about 9.8, and AA-Briefcase about 1.7. (AA-Briefcase is scored in Elo; Artificial Analysis's methodology page converts it to index points as (Elo − 500)/2000 at a 15% weight. The parts sum to about 0.2 point less than the total gap.) Kimi K3.1 Pro trails Opus by about 12.4 points, and the public tests account for about 5.8, close to half. Measured against Gemini 4 Carbon, DeepSeek's gap is about 26.9 points, and the public tests' share rises to about 59%.
One test makes the point on its own. Terminal-Bench-Science is public and carries 15% of the weight. GPT-6 Astra scores 63% and Claude Opus 5.5 59%. DeepSeek V4.1 Flash 1017 manages 13%, and Artificial Analysis says most open-weight models score under 10%. HLE-Diamond's dataset is on Hugging Face for anyone to download. In the runs CAIS and Scale AI published, Kimi K3.1 Pro scored 29.5% and GPT-6 Astra 59.9%, a gap in the same direction. Each group published its own results, with different settings.
The private tests aren't one-sided either. On the private AutomationBench-AA, open-weight MiniMax-M3.1-Flash scores 72.5%, above Claude Opus 5.5 at 69.5%. DeepSeek V4.1 Flash 1017 is 3 points behind Opus at 66.6%. Gemini 4 Argon leads that test at 78%.
These numbers support a narrow conclusion: open-weight models don't score low on v5 mainly because the questions are hidden. The biggest gaps are on long, multistep terminal research tasks and hard knowledge and science questions, and open-weight models trail on those whether the tests are public or private. On the private hard subset of CritPt v2, a set of research-level physics problems, Opus scores 61%, Kimi K3.1 Pro 45% and DeepSeek 24%.
The critics have a point
The worry about trust is legitimate. Outsiders can't see the questions behind 60% of the weight, so most of the index rests on trust in Artificial Analysis. The firm also decides which tests go in, how much each counts and which come out, and this round removed a long-context test on which open-weight models did well, though it moved the overall gap by less than 0.1 point. Artificial Analysis's stated reason for going private is contamination: the FrontierCode 2.0 tasks stay private "to avoid contamination," its FrontierCode 2.0 page says. It also published a public subset of AA-Omniscience on Hugging Face and says the subset is calibrated to match the private set.
Who writes the tests matters too. Cognition, which built FrontierCode, sells the Devin AI coding product and has its own model, SWE-2, which appears in its comparison table, though SWE-2 itself fell to 23.9% from 50.0%, a steeper drop than any other closed model listed. Its launch post said the results "highlight significant room for improvement for open source models, even more so for western OSS models."
So do the conditions of the run. On FrontierCode 2.0, Cognition reports 19.3% for DeepSeek V4.1 Flash, while Artificial Analysis, running its own Stirrup harness, gets 28.5%. Claude Opus 5.5 scores 45.5% and 46.4% in the two runs. The runs differ in harness and grading, and in how reasoning effort is set: Artificial Analysis runs each task three times with GPT-6 Luna as the judge, while Cognition shows each model at its best reasoning effort. DeepSeek's score differs by 9 points between them, Opus's by about 1.
What it means for teams that self-host
For small teams building on open-weight models, the first change is cost. Artificial Analysis says every model it measured produced more output tokens per index task than in v4.3, with the median rising to 59,000 from 35,000. Four agentic tests carry 55% of the weight but use about 90% of output tokens, and Terminal-Bench-Science alone accounts for about a third. At the prices listed on the firm's site, DeepSeek V4.1 Flash 1017 costs $0.30 per million input tokens and $1.20 per million output tokens; Claude Opus 5.5 costs $4 and $20. Artificial Analysis measured Opus 5.5 at $10.04 per task at maximum effort and $0.98 at low effort, for scores of 58 and 35.
Size is another hurdle. DeepSeek V4.1 Flash 1017 has 552 billion total parameters with 16 billion active, and GLM-5.3 has 753 billion. Qwen3.8 27B, with 27 billion parameters, scores 20 on v5 and close to zero on Terminal-Bench-Science.
On the other side, downloadable open-weight models come close to some flagships on some tests: AutomationBench-AA, which covers SaaS workflow automation, and AA-LCR, the long-document test that left the index. Kimi K3.1 Pro, whose weights are still unreleased, ranks first among all models tested on Terminal-Bench 4.0, at 67%.
The v5 score folds those differences into one number, and the "open-weight gap" figure itself depends on whose weights can be downloaded. When Kimi K3.1 Pro's weights ship, the best downloadable score will jump to 46 from 32, and the gap to Claude Opus 5.5 will shrink to 12 points from 26. The model's abilities won't have changed. Only its download status will.