GSM8K
How it works
Each record contains the problem statement and a reference solution written in natural language as a sequence of steps. Calculations are wrapped in calculator annotations <<calculation=result>> (e.g., <<50*3=150>>), and the final numeric answer appears after the #### marker on the last line. Evaluation works by having the model generate a solution (typically with chain-of-thought reasoning), extracting the number after ####, and comparing it to the reference answer (exact match). The Socratic variant injects automatically generated sub-questions before each step. In the original method the authors generated many candidate solutions and selected the one ranked highest by a trained verifier, which improved accuracy over plain finetuning.
Problem solved
There was no standardized benchmark that was hard enough yet still solvable for measuring multi-step arithmetic reasoning in language models. Earlier datasets were either too easy (saturated) or too templated, letting models guess answers without genuine reasoning. GSM8K provides linguistically diverse problems that require a chain of several steps, with an explicit answer format that enables automatic, unambiguous evaluation.
Components
A grade-school math word problem, linguistically diverse, requiring 2–8 reasoning steps.
Reference solution in natural language written out as a sequence of elementary calculations.
Calculations wrapped in <<calculation=result>> markers, enabling an external calculator to be injected and arithmetic to be verified.
Official
Numeric answer after the #### marker on the last line, enabling unambiguous automatic evaluation (exact match).
A format modification injecting automatically generated sub-questions before each solution step.
Official
A model that ranks many generated candidate solutions; selecting the highest-ranked one outperforms plain finetuning (the method from the original paper).
Official
Implementation
GSM8K has leaked into many models' pretraining corpora, inflating scores and undermining fair comparisons.
Incorrect parsing of the number after #### (formatting, commas, units) leads to under- or over-counted scores.
The <<...>> annotations must be stripped before text evaluation or consistently used as an external calculator; inconsistency skews comparisons.
The dataset covers only English grade-school arithmetic problems; a high score does not prove general mathematical ability.
Evolution
OpenAI releases GSM8K together with a method of training verifiers that rank multiple solutions.
Chain-of-thought prompting (Wei et al.) dramatically improves GSM8K scores, establishing it as a core LLM reasoning benchmark.
Self-consistency (Wang et al.) further raises GSM8K accuracy by sampling multiple reasoning paths and voting.
Frontier models exceed 90% accuracy on GSM8K, signaling benchmark saturation and the need for harder tests.
GSM-Symbolic (Apple) generates symbolic variants of GSM8K problems, revealing model sensitivity to surface changes and data-contamination risk.
Hyperparameters (configurable axes)
Number of elementary steps needed to solve a problem.
Split of the dataset into training and test portions.
How the correctness of a model's answer is scored.
How the model is queried, strongly affecting the score.