Each example is a pair (question, numeric answer). The model receives the question and must produce a final answer; evaluation compares the extracted number against the gold answer (exact match). LLM protocols use prompting variants: zero-shot (direct answer) and chain-of-thought (the model first writes reasoning steps, then gives the result). A key step is answer extraction from the generated text (e.g. "the answer is …"). A commonly used split is 420 training and 180 test examples.
It measures a model's ability to perform multi-step arithmetic reasoning — not a single operation, but a chain of operations derived from a natural-language description. It exposes the gap between pattern matching and actually carrying out step-by-step computation.
A short text (~108–217 characters) describing an everyday scenario requiring several arithmetic operations.
A string representing the numeric final answer (typically 1–2 digits), used for exact-match comparison.
Modern LLMs reach near-100% accuracy on MultiArith, so the benchmark poorly discriminates between strong models.
The score depends heavily on the number-extraction rule; different heuristics yield different accuracies for the same model.
Only 600 examples means high result variance and susceptibility to leakage into LLM training data.
Roy and Roth publish "Solving General Arithmetic Word Problems" (EMNLP 2015); the subset of 600 multi-step problems later becomes known as MultiArith.
Kojima et al. (NeurIPS 2022) show that "Let's think step by step" lifts accuracy from 17.7% to 78.7% on text-davinci-002, making MultiArith a key LLM reasoning benchmark.
How the model is queried: zero-shot (direct answer) vs chain-of-thought / zero-shot-CoT (explicit reasoning steps).
Rule for extracting the number from generated text before exact-match comparison; materially affects the score.
