Evaluation
MMLU-Pro
2024ActivePublished
Key
innovation
Extended MMLU version eliminating trivial and noisy questions, expanding answer choices from 4 to 10, and enriching the dataset with reasoning-focused questions (not just knowledge), dropping model accuracy by 16-33% and restoring benchmark discriminative power.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)
Use cases
frontier LLM evaluationmulti-step reasoning testingfrontier model comparisonCoT assessment
How it works
The dataset extends MMLU by: (1) consolidating with external sources to remove trivial questions; (2) expanding options to 10 per question; (3) adding multi-step reasoning questions. Models are evaluated zero-shot and with CoT; results show CoT is more effective on MMLU-Pro than on original MMLU.
Problem solved
Saturation of the original MMLU by frontier models (>85-90%) and its sensitivity to prompt variations, making it impossible to distinguish capabilities between top models.
Implementation
Reference implementations
Implementation pitfalls
10 choices increase token cost in few-shotLow
A prompt with 10 answer choices is longer, increasing evaluation cost for few-shot with long examples.
Fix:Use zero-shot CoT or reduced few-shot (1-3 examples).
Evolution
Original paper · 2024 · NeurIPS 2024 (Datasets and Benchmarks Track, Spotlight) · Yubo Wang
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Xiang Yue, Wenhu Chen
2024
MMLU-Pro published (June 2024, NeurIPS 2024 Spotlight)
Inflection pointWang et al. publish the enhanced MMLU with 10 choices and reasoning questions; model scores drop 16-33%.
Hardware requirements
Primary
Text-based benchmark independent of evaluation hardware.