Robots Atlas>ROBOTS ATLAS
Evaluation

MultiArith

2015ActivePublished: 10 September 2026Updated: 10 September 2026Published
Key innovation
Isolated a compact set of 600 multi-step natural-language arithmetic word problems that became a standard probe for multi-step arithmetic reasoning in language models.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)Inference
Use cases
Evaluating multi-step arithmetic reasoning in LLMsResearch on chain-of-thought and zero-shot chain-of-thoughtComparing prompting strategiesTesting models on math word-problem solving

How it works

Each example is a pair (question, numeric answer). The model receives the question and must produce a final answer; evaluation compares the extracted number against the gold answer (exact match). LLM protocols use prompting variants: zero-shot (direct answer) and chain-of-thought (the model first writes reasoning steps, then gives the result). A key step is answer extraction from the generated text (e.g. "the answer is …"). A commonly used split is 420 training and 180 test examples.

Problem solved

It measures a model's ability to perform multi-step arithmetic reasoning — not a single operation, but a chain of operations derived from a natural-language description. It exposes the gap between pattern matching and actually carrying out step-by-step computation.

Components

QuestionInput — the natural-language arithmetic problem statement.

A short text (~108–217 characters) describing an everyday scenario requiring several arithmetic operations.

Final answerGold output — a single number that is the correct result.

A string representing the numeric final answer (typically 1–2 digits), used for exact-match comparison.

Implementation

Reference implementations
ChilleD/MultiArith (Hugging Face Datasets)
JSON / Python (datasets) · Hugging Face community
Zero-shot-CoT (Kojima et al. repository)
Python · Takeshi Kojima et al.
Implementation pitfalls
Benchmark saturationHigh

Modern LLMs reach near-100% accuracy on MultiArith, so the benchmark poorly discriminates between strong models.

Fix:Combine with harder sets (e.g. GSM8K, MATH) and report across multiple benchmarks.
Sensitivity to answer extractionMedium

The score depends heavily on the number-extraction rule; different heuristics yield different accuracies for the same model.

Fix:Use a consistent, documented extraction protocol (e.g. a two-stage "the answer is" prompt).
Small size and contamination riskMedium

Only 600 examples means high result variance and susceptibility to leakage into LLM training data.

Fix:Treat results as indicative; check for contamination and report confidence intervals.

Evolution

Original paper · 2015 · EMNLP 2015 · Subhro Roy
Solving General Arithmetic Word Problems
Subhro Roy, Dan Roth
2015
Introduction of MultiArith
Inflection point

Roy and Roth publish "Solving General Arithmetic Word Problems" (EMNLP 2015); the subset of 600 multi-step problems later becomes known as MultiArith.

2022
Zero-shot CoT on MultiArith
Inflection point

Kojima et al. (NeurIPS 2022) show that "Let's think step by step" lifts accuracy from 17.7% to 78.7% on text-davinci-002, making MultiArith a key LLM reasoning benchmark.

Hyperparameters (configurable axes)

Prompting strategyCritical

How the model is queried: zero-shot (direct answer) vs chain-of-thought / zero-shot-CoT (explicit reasoning steps).

zero-shotNo examples, direct answer.
zero-shot-CoT"Let's think step by step" — explicit reasoning.
few-shot-CoTIn-context examples with worked reasoning.
Answer extractionHigh

Rule for extracting the number from generated text before exact-match comparison; materially affects the score.

regex ostatnia liczbaTakes the last number in the output.
trigger „the answer is"Second prompt extracting the result.