Home · Oh, That's Why

Transcript

1. Which AI is the smartest? And who decides?
2. Turns out, AI takes exams like students, then gets ranked.
3. One famous test has 57 subjects and over 15,000 questions.
4. But the questions are public online, and AI learns from the internet.
5. So it may have seen the answers in advance.
6. Researchers wrote new math problems just like the old ones.
7. Some AIs scored up to 8 points lower.
8. So people judge, too. Two hidden AIs answer, you vote for the better one.
9. But votes can be gamed. Meta got near the top with a special chat-tuned version.
10. The version it actually released ranked far lower.
11. So the top AI is a top test-taker. That's not the same as smart.
12. You're the teacher. This AI may have seen the test. 1: retake? 2: let it pass?

Sources

1. Quanta Magazine, "He Wants AI Benchmarking To Be a Science, Not a Dark Art" (2026-10-16)
2. Hendrycks et al., "Measuring Massive Multitask Language Understanding" (ICLR 2021, arXiv 2020-09)
3. GitHub, hendrycks/test (MMLU 공개 데이터)
4. Zhang et al. (Scale AI), "A Careful Examination of Large Language Model Performance on Grade School Arithmetic" (arXiv 2024-05)
5. NeurIPS 2024 Datasets and Benchmarks, 같은 논문 최종본
6. LMArena, "FAQ"
7. TechCrunch, "Meta's vanilla Maverick AI model ranks below rivals on a popular chat benchmark" (2025-04-11)
8. Simon Willison, LMArena 성명 인용 (2025-04-08)
9. The Register, Llama 4 LMArena 순위 논란 기사 (2025-04-08, 제목 미확인)