Evaluation

BIG-Bench

2022ActivePublished: 6 May 2026Updated: 6 May 2026Published

Key innovation

First crowdsourced LLM evaluation benchmark assembled by 450+ authors from 132 institutions, containing 204+ tasks designed to exceed current model capabilities and measure emergent abilities at scale.

How it works

1. Task crowdsourcing: researchers from 132 institutions contribute tasks in a standardized JSON format (multiple choice, generative, programmatic). Each task includes description, examples, metrics, and ground truth. 2. Validation: the central team verifies that the task is hard for current models (GPT-2, GPT-3 baseline) and has clear evaluation criteria. 3. Distribution: tasks published as a GitHub repository (Apache 2.0) with a library for running benchmarks via model APIs. 4. Evaluation: the model is prompted with each task (zero-shot or few-shot); output is scored against ground truth by task metrics (accuracy, ROUGE, BLEU, exact match). 5. Aggregation: results published on the BIG-Bench leaderboard, broken down by task category and analyzed for emergence as a function of parameter scale. 6. BBH (BIG-Bench Hard): a 23-task subset where standard CoT prompting yields clearly better results than direct prompting — the canonical reasoning suite.

Problem solved

Pre-2022 LLM benchmarks (GLUE, SuperGLUE) saturated quickly with larger models and did not test a broad spectrum of capabilities. The community lacked a benchmark sufficiently hard, diverse, and open to track progress across multiple model generations. BIG-Bench addressed this by crowdsourcing 200+ tasks specifically chosen as hard, spanning domains from mathematics to theory of mind.

Components

Task suiteMain evaluation task collection

204+ tasks in a standardized JSON format, each with metadata (author, category, metrics, prompt template, ground truth).

BIG-Bench Hard (BBH)Curated subset for reasoning evaluation

23 tasks where standard prompting underperforms humans; CoT prompting significantly improves performance. The canonical reasoning test (Suzgun et al. 2022).

Evaluation harnessLibrary for running evaluations

Python framework integrating with model APIs (OpenAI, Anthropic, HuggingFace), supporting multiple choice, generative scoring, and programmatic evaluation.

Lite subset (BIG-Bench Lite)Cost-efficient subset for fast evaluation

24 tasks optimized for low evaluation cost (small example counts) while preserving the diversity of full BIG-Bench.

Implementation

Reference implementations

BIG-bench (official GitHub repository)

Python · Google + BIG-bench collaboration

Official

BIG-Bench Hard (BBH) — prompts repository

Python · Mirac Suzgun et al.

Official

lm-evaluation-harness (BIG-Bench tasks integration)

Python · EleutherAI

HELM — Holistic Evaluation of Language Models (Stanford CRFM)

Python · Stanford CRFM

Implementation pitfalls

Data contamination — BIG-Bench tasks in pretraining corpusHigh

The BIG-Bench repository has been publicly available on GitHub since 2022. Models trained after 2022 may have tasks in their pretraining corpus, artificially inflating scores.

Fix:Decontamination pipeline on the pretraining corpus (Brown-et-al. style — 13-gram match). Evaluate on fresh tasks (held-out, post-training).

Heterogeneous metrics — aggregation difficultyMedium

Each task has its own metric (accuracy, ROUGE, BLEU, custom). Arithmetic mean is misleading — some tasks range 0–1, others 0–100.

Fix:Apply metric normalization (calibrated score, per-task z-score) or report per category/subset.

BBH saturation on frontier modelsHigh

GPT-5, Gemini 3, Claude Opus 4 reach 95–98% average accuracy on BBH. The benchmark loses its ability to discriminate top-tier models.

Fix:Use BBH only as a sanity check; for differentiating frontier models use GPQA, MMLU-Pro, FrontierMath, ARC-AGI.

Critique of emergence as a metric artifactMedium

Schaeffer et al. (2023) showed that some "emergence jumps" on BIG-Bench arise from discrete metrics (accuracy) — under continuous metrics (cross-entropy), behavior is smooth.

Fix:Report both accuracy and continuous metrics (negative log-likelihood). Be cautious when claiming phase transitions.

Evolution

Original paper · 2022 · TMLR (Transactions on Machine Learning Research) 2023 · Aarohi Srivastava

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, BIG-bench collaboration (450+ authors, 132 institutions)

2021

BIG-bench collaboration kicks off

Google announces an open call to crowdsource LLM evaluation tasks. Goal: hard, diverse, open tasks.

2022

BIG-Bench released (204 tasks, 442 authors)

Inflection point

First public benchmark release as a GitHub repository. Evaluation of GPT-3, PaLM 540B, and several open-weight models.

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (paper)

2022

BIG-Bench Hard (BBH) — 23 reasoning tasks (Suzgun et al.)

Inflection point

Curated subset of tasks where CoT prompting yields a clear gain over direct prompting. BBH becomes the canonical reasoning test.

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (paper)

2023

GPT-4 achieves breakthrough results on BBH

Inflection point

OpenAI reports that GPT-4 with CoT exceeds the human baseline on most BBH tasks — the first model generation to do so.

2023

Schaeffer et al. — critique of emergence as a metric artifact

Stanford NLP shows that some emergence jumps on BIG-Bench disappear after switching metrics from accuracy to continuous ones (e.g. cross-entropy).

Are Emergent Abilities of Large Language Models a Mirage? (paper)

2024

BBH starts to saturate — frontier models hit 90%+ accuracy

Claude 3.5, Gemini 1.5 Pro, GPT-4o reach 90%+ average accuracy on BBH; demand grows for harder benchmarks (GPQA, MMLU-Pro).

2025

BBH as a reference test for reasoning models

Reasoning models (o1, DeepSeek-R1, Gemini 2.5 Deep Think, Claude Opus 4 Thinking) use BBH as a standard reference alongside GPQA and AIME.

Hyperparameters (configurable axes)

Evaluation subsetCritical

Choice: full (204+ tasks), BBH (23 reasoning tasks), Lite (24 cost-efficient tasks), or a custom category-based subset.

fullAll 204+ tasks — full evaluation.

BBHBIG-Bench Hard — 23 hard reasoning tasks.

LiteBIG-Bench Lite — 24 cost-optimized tasks.

Prompting strategyHigh

Direct prompting vs Chain-of-Thought. BBH shows significant differences — CoT improves results by 10–30 pp on most tasks.

directDirect question, no reasoning step.

cotChain-of-Thought — model reasons step by step.

few-shot cotCoT with 3–8 demonstrations.

Number of shotsHigh

Zero-shot, 1-shot, few-shot (3–8). Most BIG-Bench benchmarks use zero-shot or 3-shot as the standard.

Metric typeMedium

Per task: exact match, multiple choice accuracy, ROUGE, BLEU, BLEURT, programmatic check, custom.

Parallelism

Parallelism level

Fully parallel

Each task and each example in the benchmark is independent — they can be evaluated in parallel on any number of devices. The bottleneck is model API rate limits, not the benchmark itself.

Scope

InferenceAcross devices