METR's review of Anthropic's R&D acceleration shows an independent audit is not the same as public evidence
METR spent about 11 weeks assessing Anthropic, with permission to view some internal material, and concluded that Anthropic is improving its models at roughly twice its earlier pace. That conclusion could be published. But METR says a majority of the evidence it found most informative was redacted by Anthropic. Readers can review the published methods, results and robustness checks, but that is not the same as being able to recompute them independently; for the rest, it comes down to how far you trust the evaluator's judgment.
Editorial8 min read
