Start for $0.99

Questions and answers

One public question from each quarter of 2024 and 2025, the right answer, and what every model replied. The other 124 questions stay private, including every 2026 question, so they cannot leak into training data.

What name did India give the first flight test of its Agni-5 missile with Multiple Independently Targetable Re-entry Vehicle (MIRV) technology, conducted on 11 March 2024? Answer with only the name, no explanation.

Right answer: Mission Divyastra. 16 of 17 models got it right.

  • GPT-6 Sol, right: Mission Divyastra
  • Claude Opus 5.5, right: Mission Divyastra
  • GPT-6 Astra, right: Mission Divyastra
  • DeepSeek V4.1 Flash, right: Mission Divyastra
  • GPT-6 Luna, right: Mission Divyastra
  • Claude Sonnet 5, right: Mission Divyastra
  • Kimi K3, right: Mission Divyastra
  • Qwen3.8 Max, right: Mission Divyastra
  • Gemini 3.5 Flash-Lite, right: Mission Divyastra
  • Claude Fable 5.1, right: Mission Divyastra
  • GLM-5.3, right: Mission Divyastra
  • GLM-5.3 Flash, right: Mission Divyastra
  • DeepSeek V4 Pro, right: Mission Divyastra
  • Gemini 3.1 Pro, right: Mission Divyastra
  • Grok 4.7, right: Mission Divyastra
  • Gemini 3.8 Flash, right: Mission Divyastra
  • Qwen3.8 Flash, wrong: I don't know

Anglo American confirmed on 25 April 2024 that it had received an unsolicited, non-binding all-share combination proposal from which mining company? Answer with only the company name, no explanation.

Right answer: BHP. 16 of 17 models got it right.

  • GPT-6 Sol, right: BHP
  • Claude Opus 5.5, right: BHP
  • GPT-6 Astra, right: BHP Group
  • DeepSeek V4.1 Flash, right: BHP
  • GPT-6 Luna, right: BHP Group Limited
  • Claude Sonnet 5, right: BHP
  • Kimi K3, right: BHP Group
  • Qwen3.8 Max, right: BHP
  • Gemini 3.5 Flash-Lite, right: BHP
  • Claude Fable 5.1, right: BHP
  • GLM-5.3, right: BHP
  • GLM-5.3 Flash, right: BHP
  • DeepSeek V4 Pro, right: BHP Group
  • Gemini 3.1 Pro, right: BHP
  • Grok 4.7, right: BHP
  • Gemini 3.8 Flash, right: BHP
  • Qwen3.8 Flash, wrong: Teck Resources

Which novel won the Hugo Award for Best Novel at the ceremony in Glasgow on 11 August 2024? Answer with only the title, no explanation.

Right answer: Some Desperate Glory. 16 of 17 models got it right.

  • GPT-6 Sol, right: Some Desperate Glory
  • Claude Opus 5.5, right: Some Desperate Glory
  • GPT-6 Astra, right: Some Desperate Glory
  • DeepSeek V4.1 Flash, right: Some Desperate Glory
  • GPT-6 Luna, right: Some Desperate Glory
  • Claude Sonnet 5, right: Some Desperate Glory
  • Kimi K3, right: Some Desperate Glory
  • Qwen3.8 Max, right: Some Desperate Glory
  • Gemini 3.5 Flash-Lite, right: Some Desperate Glory
  • Claude Fable 5.1, right: Some Desperate Glory
  • GLM-5.3, right: Some Desperate Glory
  • GLM-5.3 Flash, wrong: I don't know
  • DeepSeek V4 Pro, right: Some Desperate Glory
  • Gemini 3.1 Pro, right: Some Desperate Glory
  • Grok 4.7, right: Some Desperate Glory
  • Gemini 3.8 Flash, right: Some Desperate Glory
  • Qwen3.8 Flash, right: Some Desperate Glory

To what level did the Central Bank of Brazil's Monetary Policy Committee (Copom) set the Selic rate at its decision of 11 December 2024? Answer with only the number, in per cent, no explanation.

Right answer: 12.25. 15 of 17 models got it right.

  • GPT-6 Sol, right: 12.25%
  • Claude Opus 5.5, right: 12.25
  • GPT-6 Astra, right: 12.25
  • DeepSeek V4.1 Flash, right: 12.25
  • GPT-6 Luna, right: 12.25%
  • Claude Sonnet 5, right: 12.25
  • Kimi K3, right: 12.25
  • Qwen3.8 Max, right: 12.25%
  • Gemini 3.5 Flash-Lite, right: 12.25%
  • Claude Fable 5.1, right: 12.25
  • GLM-5.3, right: 12.25
  • GLM-5.3 Flash, right: 12.25
  • DeepSeek V4 Pro, right: 12.25
  • Gemini 3.1 Pro, wrong: 11.75%
  • Grok 4.7, right: 12.25
  • Gemini 3.8 Flash, right: 12.25%
  • Qwen3.8 Flash, wrong: 13.75

On 6 January 2025, Austrian Federal President Alexander Van der Bellen gave a party leader the mandate to form a new federal government. Who was that party leader? Answer with only the person's name, no explanation.

Right answer: Herbert Kickl. 12 of 17 models got it right.

  • GPT-6 Sol, right: Herbert Kickl
  • Claude Opus 5.5, right: Herbert Kickl
  • GPT-6 Astra, right: Herbert Kickl
  • DeepSeek V4.1 Flash, right: Herbert Kickl
  • GPT-6 Luna, right: Herbert Kickl
  • Claude Sonnet 5, right: Herbert Kickl
  • Kimi K3, right: Herbert Kickl
  • Qwen3.8 Max, right: Herbert Kickl
  • Gemini 3.5 Flash-Lite, wrong: Karl Nehammer
  • Claude Fable 5.1, right: Herbert Kickl
  • GLM-5.3, wrong: Christian Stocker
  • GLM-5.3 Flash, wrong: Christian Stocker
  • DeepSeek V4 Pro, right: Herbert Kickl
  • Gemini 3.1 Pro, wrong: I don't know
  • Grok 4.7, right: Herbert Kickl
  • Gemini 3.8 Flash, right: Herbert Kickl
  • Qwen3.8 Flash, wrong: I don't know

Which national team won the UEFA Nations League final played in Munich on 8 June 2025? Answer with only the country, no explanation.

Right answer: Portugal. 10 of 17 models got it right.

  • GPT-6 Sol, right: Portugal
  • Claude Opus 5.5, right: Portugal
  • GPT-6 Astra, right: Portugal
  • DeepSeek V4.1 Flash, right: Portugal
  • GPT-6 Luna, right: Portugal
  • Claude Sonnet 5, right: Portugal
  • Kimi K3, wrong: I don't know
  • Qwen3.8 Max, right: Portugal
  • Gemini 3.5 Flash-Lite, wrong: I don't know
  • Claude Fable 5.1, right: Portugal
  • GLM-5.3, right: Portugal
  • GLM-5.3 Flash, wrong: I don't know
  • DeepSeek V4 Pro, right: Portugal
  • Gemini 3.1 Pro, wrong: I don't know
  • Grok 4.7, wrong: I don't know
  • Gemini 3.8 Flash, wrong: I don't know
  • Qwen3.8 Flash, wrong: I don't know

Who won Malawi's presidential election held in September 2025, as declared by the Malawi Electoral Commission? Answer with only the person's name, no explanation.

Right answer: Peter Mutharika. 8 of 17 models got it right.

  • GPT-6 Sol, right: Peter Mutharika
  • Claude Opus 5.5, right: Peter Mutharika
  • GPT-6 Astra, right: Peter Mutharika
  • DeepSeek V4.1 Flash, right: Peter Mutharika
  • GPT-6 Luna, right: Arthur Peter Mutharika
  • Claude Sonnet 5, right: Peter Mutharika
  • Kimi K3, wrong: I don't know
  • Qwen3.8 Max, wrong: I don't know
  • Gemini 3.5 Flash-Lite, wrong: I don't know
  • Claude Fable 5.1, right: Peter Mutharika
  • GLM-5.3, wrong: Peter Kuwani
  • GLM-5.3 Flash, right: Peter Mutharika
  • DeepSeek V4 Pro, wrong: Lazarus Chakwera
  • Gemini 3.1 Pro, wrong: I don't know
  • Grok 4.7, wrong: I don't know
  • Gemini 3.8 Flash, wrong: I don't know
  • Qwen3.8 Flash, wrong: I don't know

On 1 December 2025, DeepSeek launched DeepSeek-V3.2 together with a second, API-only model focused on maximum reasoning. What was that second model called? Answer with only the model name, no explanation.

Right answer: DeepSeek-V3.2-Speciale. 8 of 17 models got it right.

  • GPT-6 Sol, right: DeepSeek-V3.2-Speciale
  • Claude Opus 5.5, right: DeepSeek-V3.2-Speciale
  • GPT-6 Astra, right: DeepSeek-V3.2-Speciale
  • DeepSeek V4.1 Flash, right: DeepSeek-V3.2-Speciale
  • GPT-6 Luna, right: DeepSeek-V3.2-Speciale
  • Claude Sonnet 5, right: DeepSeek-V3.2-Speciale
  • Kimi K3, wrong: I don't know
  • Qwen3.8 Max, wrong: I don't know
  • Gemini 3.5 Flash-Lite, wrong: I don't know
  • Claude Fable 5.1, right: DeepSeek-V3.2-Speciale
  • GLM-5.3, wrong: DeepSeek-R2
  • GLM-5.3 Flash, right: DeepSeek-V3.2-Speciale
  • DeepSeek V4 Pro, wrong: I don't know
  • Gemini 3.1 Pro, wrong: I don't know
  • Grok 4.7, wrong: I don't know
  • Gemini 3.8 Flash, wrong: I don't know
  • Qwen3.8 Flash, wrong: I don't know

Back to the Knowledge Cutoff Bench leaderboard