Multilingual LLM Bench

Which AI speaks your language best. Every model answers the same 150 questions, 5 written from scratch in each of 30 languages, once, at low reasoning. Run 2026-10-02.

Leaderboard

#ModelLabScoreCost of one run
1GPT-6 AstraOpenAI70.7 ±7.2$0.75
2Gemini 3.1 ProGoogle67.3 ±7.4$1.34
3GPT-6 SolOpenAI63.3 ±7.6$0.16
4Claude Opus 5.5Anthropic60.7 ±7.7$0.47
5Gemini 3.8 FlashGoogle51.3 ±7.9$0.12
6Claude Fable 5.1Anthropic50.0 ±7.9$0.66
7GPT-6 LunaOpenAI44.7 ±7.9$0.014
8Grok 4.7xAI44.0 ±7.8$1.93
9DeepSeek V4 ProDeepSeek44.0 ±7.8$1.30
10DeepSeek V4.1 FlashDeepSeek42.0 ±7.8$0.41
11Kimi K3Moonshot31.3 ±7.3$0.81
12Qwen3.8 MaxQwen30.7 ±7.3$0.82
13Claude Sonnet 5Anthropic23.3 ±6.7$0.30
14Gemini 3.5 Flash-LiteGoogle21.3 ±6.5$0.099
15GLM-5.3Z.ai19.3 ±6.3$0.42
16GLM-5.3 FlashZ.ai15.3 ±5.8$0.040
17Qwen3.8 FlashQwen12.0 ±5.2$0.19

Sample questions

One real question in each of the 30 languages, with the right answer and what every model replied: all 30 questions and answers.

How the bench works

Each language has 2 local knowledge questions, 2 idioms or proverbs, and 1 grammar, historical spelling or calendar question. Every answer key comes from a dictionary, a statute or a calendar table, and a second agent re-derives it without seeing it. No AI grades another AI, and an answer in the wrong language scores zero. Every model is told to reply with the answer alone: at low reasoning Claude Fable 5.1 used no thinking tokens on 139 of its 150 questions and worked through 32 of them in the reply itself, and 26 of those 32 replies scored zero.