Robots Atlas>ROBOTS ATLAS
Evaluation

Chartography

2026ActivePublished: 18 November 2026Updated: 18 November 2026Published
Key innovation
The first benchmark measuring chart understanding in professional, domain-native formats (e.g. Kaplan-Meier curves, candlestick charts, Sankey diagrams, Bode plots), where questions and gold answers are authored by practicing domain experts rather than produced by automated label extraction.
Category
Evaluation
Abstraction level
Benchmark
Operation level
Evaluation (runtime)
Use cases
Evaluating vision-language models (VLMs)Measuring professional chart understandingTesting multi-step reasoning over graphicsBenchmarking frontier modelsAssessing visual perception and domain conventions

How it works

A model is given a chart image and a question, and must answer using only the chart itself and knowledge of the domain's conventions — no outside data. Tasks are authored by practicing professionals who attach a gold answer, a walkthrough of the calculation path and an expert-set acceptable range (e.g. '150 mph, acceptable 140-150 mph'), because chart-reading resolution varies and a single percentage tolerance is inadequate. A judge scores the model's answer against the gold answer; any value within the acceptable range counts as correct (pass@1 metric). Tasks are stored in the Harbor format. The most common model failure modes are: missing thin or overlapping elements, reading the wrong value despite following the correct feature, struggling with 3D surface plots, inconsistently applying unstated domain conventions, and compounding small visual errors across otherwise correct reasoning.

Problem solved

Existing chart-understanding benchmarks (e.g. ChartQA, CharXiv, ChartMuseum) have become saturated — extracting labeled data points is effectively solved and frontier models score highly on them. Chartography targets the gap in evaluating harder skills: estimating values against sparsely labeled axes, tracing overlapping flows to their sources, reading 3D surface plots through projected gridlines, and multi-step calculations where each step depends on a separate visual read and on domain-specific conventions.

Components

Expert-authored tasks

100 tasks (chart + question) across 12 professional domains, each written by a practicing specialist and verified by several other experts.

Gold answers with acceptable ranges

Each task has an expert gold answer, a calculation walkthrough and a defined acceptable range instead of a fixed percentage tolerance.

Judge (pass@1)

A judge compares the model's answer to the gold answer; a value within the acceptable range counts as correct.

Harbor format

Tasks stored in the Harbor framework format (instruction.md, task.toml, environment/, tests/).

Implementation

Implementation pitfalls
Data contamination risk (leakage)High

The benchmark is meant for held-out evaluation; the authors ask that tasks and derived materials be kept out of training, fine-tuning, RL, distillation and checkpoint selection.

Fix:Treat the set strictly as a test set and exclude it from training data.
Public samples are not a full scoreMedium

The public Hugging Face set contains only 2 sample tasks; results on them do not constitute a score on the full benchmark (100 tasks, access via benchmarks@surgehq.ai).

Fix:Request the full set from Surge AI; do not report a 2-sample result as a benchmark score.
Scoring via acceptable rangesLow

Scoring uses per-task expert acceptable ranges rather than a fixed percentage tolerance - account for this when comparing.

Fix:Use the provided acceptable ranges and the pass@1 metric per the methodology.

Evolution

Original paper · 2026 · BEAM 2 Workshop, ECCV 2026 · Suhaas Garre
Chartography: A Benchmark for Professional Chart Understanding
Suhaas Garre, Chris Mutty, Sushant Mehta, Edwin Chen
2026
Chartography benchmark released
Inflection point

Surge AI releases the benchmark (arXiv, 11 Aug 2026, accepted at the BEAM 2 workshop @ ECCV 2026): 100 tasks across 12 domains; the best frontier model reaches 45.0% pass@1.