HellaSwag
How it works
Each example has a context (ctx, composed of ctx_a and ctx_b) and a list of four candidate endings; exactly one is correct (label field 0–3). The correct ending comes from the original source (ActivityNet or WikiHow), while distractors are machine-generated and selected via Adversarial Filtering: a discriminator is trained to tell human from machine text, generated examples that fool the model are kept, and the process iterates until the set is hard for models yet easy for humans (verified by annotators). A model is scored by accuracy in picking the correct ending; evaluation covers both in-domain and zero-shot categories (activities unseen in training).
Problem solved
Earlier commonsense benchmarks (e.g. SWAG) were quickly saturated by pretrained models like BERT, making it hard to measure real commonsense reasoning. HellaSwag constructs hard, adversarial examples that remain easy for humans, restoring the human–model gap as a measure of progress.
Components
Task format: a context (ctx_a + ctx_b) and four candidate endings, one of which is correct (label 0–3).
Data-collection paradigm: an ensemble of discriminators iteratively selects machine-generated wrong answers that are hard for models yet ridiculous to humans.
Contexts come from ActivityNet Captions video descriptions and WikiHow how-to articles; they include in-domain and zero-shot categories.
Implementation
Evolution
Zellers et al. introduce SWAG, a commonsense NLI benchmark built with Adversarial Filtering.
A harder successor to SWAG: longer contexts and examples from ActivityNet and WikiHow; humans >95%, SOTA models <48%.
GPT-4 (March 2023) reaches 95.3%, near the human level of 95.6%, effectively saturating the task.
Hyperparameters (configurable axes)
Each example has 4 possible endings, one of which is correct.
Commonly evaluated zero-shot or 10-shot in suites like lm-evaluation-harness.