Chartography
How it works
A model is given a chart image and a question, and must answer using only the chart itself and knowledge of the domain's conventions — no outside data. Tasks are authored by practicing professionals who attach a gold answer, a walkthrough of the calculation path and an expert-set acceptable range (e.g. '150 mph, acceptable 140-150 mph'), because chart-reading resolution varies and a single percentage tolerance is inadequate. A judge scores the model's answer against the gold answer; any value within the acceptable range counts as correct (pass@1 metric). Tasks are stored in the Harbor format. The most common model failure modes are: missing thin or overlapping elements, reading the wrong value despite following the correct feature, struggling with 3D surface plots, inconsistently applying unstated domain conventions, and compounding small visual errors across otherwise correct reasoning.
Problem solved
Existing chart-understanding benchmarks (e.g. ChartQA, CharXiv, ChartMuseum) have become saturated — extracting labeled data points is effectively solved and frontier models score highly on them. Chartography targets the gap in evaluating harder skills: estimating values against sparsely labeled axes, tracing overlapping flows to their sources, reading 3D surface plots through projected gridlines, and multi-step calculations where each step depends on a separate visual read and on domain-specific conventions.
Components
100 tasks (chart + question) across 12 professional domains, each written by a practicing specialist and verified by several other experts.
Each task has an expert gold answer, a calculation walkthrough and a defined acceptable range instead of a fixed percentage tolerance.
A judge compares the model's answer to the gold answer; a value within the acceptable range counts as correct.
Tasks stored in the Harbor framework format (instruction.md, task.toml, environment/, tests/).
Implementation
The benchmark is meant for held-out evaluation; the authors ask that tasks and derived materials be kept out of training, fine-tuning, RL, distillation and checkpoint selection.
The public Hugging Face set contains only 2 sample tasks; results on them do not constitute a score on the full benchmark (100 tasks, access via benchmarks@surgehq.ai).
Scoring uses per-task expert acceptable ranges rather than a fixed percentage tolerance - account for this when comparing.
Evolution
Surge AI releases the benchmark (arXiv, 11 Aug 2026, accepted at the BEAM 2 workshop @ ECCV 2026): 100 tasks across 12 domains; the best frontier model reaches 45.0% pass@1.