Evaluation
AGIEval
2023ActivePublished
Key
innovation
First benchmark grounded in real-world human qualification exams (college entrance, LSAT, SAT, math competitions, bar exams) rather than artificially constructed tasks, enabling evaluation of AI models in the context of tasks with genuine societal relevance.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)
Use cases
foundation model evaluationAI vs human comparisonknowledge and reasoning testingbilingual evaluation
How it works
The dataset contains questions from official exams, grouped by type: multiple-choice (MC), free-text, and math problems. Models are evaluated both zero-shot and few-shot. Results are compared against average human performance for each exam.
Problem solved
Artificial benchmarks do not reflect the difficulty of tasks AI models may encounter in real-world applications. AGIEval places evaluation in the context of human cognition and decisions by using exams designed to assess human competencies.
Implementation
Reference implementations
Implementation pitfalls
Chinese language in subset of tasksMedium
A subset of tasks is in Chinese, which may skew results for models weaker in that language.
Fix:Report scores separately for EN and ZH subsets.
Evolution
Original paper · 2023 · arXiv 2023 · Wanjun Zhong
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan
2023
AGIEval published (arXiv, April 2023)
Inflection pointZhong et al. from Microsoft Research introduce the qualification exam benchmark. GPT-4 surpasses human average on SAT and LSAT.
Hardware requirements
Primary
Text-based benchmark independent of hardware.