Multilingual LLM Bench
Which AI speaks your language best. Every model answers the same 150 questions, 5 written from scratch in each of 30 languages, once, at low reasoning. Run 2026-10-02.
Leaderboard
| # | Model | Lab | Score | Cost of one run |
|---|---|---|---|---|
| 1 | GPT-6 Astra | OpenAI | 70.7 ±7.2 | $0.75 |
| 2 | Gemini 3.1 Pro | 67.3 ±7.4 | $1.34 | |
| 3 | GPT-6 Sol | OpenAI | 63.3 ±7.6 | $0.16 |
| 4 | Claude Opus 5.5 | Anthropic | 60.7 ±7.7 | $0.47 |
| 5 | Gemini 3.8 Flash | 51.3 ±7.9 | $0.12 | |
| 6 | Claude Fable 5.1 | Anthropic | 50.0 ±7.9 | $0.66 |
| 7 | GPT-6 Luna | OpenAI | 44.7 ±7.9 | $0.014 |
| 8 | Grok 4.7 | xAI | 44.0 ±7.8 | $1.93 |
| 9 | DeepSeek V4 Pro | DeepSeek | 44.0 ±7.8 | $1.30 |
| 10 | DeepSeek V4.1 Flash | DeepSeek | 42.0 ±7.8 | $0.41 |
| 11 | Kimi K3 | Moonshot | 31.3 ±7.3 | $0.81 |
| 12 | Qwen3.8 Max | Qwen | 30.7 ±7.3 | $0.82 |
| 13 | Claude Sonnet 5 | Anthropic | 23.3 ±6.7 | $0.30 |
| 14 | Gemini 3.5 Flash-Lite | 21.3 ±6.5 | $0.099 | |
| 15 | GLM-5.3 | Z.ai | 19.3 ±6.3 | $0.42 |
| 16 | GLM-5.3 Flash | Z.ai | 15.3 ±5.8 | $0.040 |
| 17 | Qwen3.8 Flash | Qwen | 12.0 ±5.2 | $0.19 |
Sample questions
One real question in each of the 30 languages, with the right answer and what every model replied: all 30 questions and answers.
How the bench works
Each language has 2 local knowledge questions, 2 idioms or proverbs, and 1 grammar, historical spelling or calendar question. Every answer key comes from a dictionary, a statute or a calendar table, and a second agent re-derives it without seeing it. No AI grades another AI, and an answer in the wrong language scores zero. Every model is told to reply with the answer alone: at low reasoning Claude Fable 5.1 used no thinking tokens on 139 of its 150 questions and worked through 32 of them in the reply itself, and 26 of those 32 replies scored zero.