AiderFrom the builder
o1 tops aider’s new polyglot leaderboard
Aider introduced a new polyglot coding benchmark to replace its nearly saturated Python-only one, with exercises in C++, Go, Java, JavaScript, Python and Rust. From 697 Exercism problems, it kept the 225 that three or fewer of seven top models could solve. OpenAI's o1 at high reasoning effort scored 61.7%, well ahead of Claude 3.5 Sonnet at 45.3%.
These problems are far too easy, and wouldn’t be good choices for the new benchmark.
Saved . A shortened excerpt from the official blog; the original publisher retains authorship.