SDBench
How it works
A diagnostic agent receives a short case abstract and, in a loop, decides what information to request, which tests to order, and when to commit to a final diagnosis. Information is revealed by the Gatekeeper, a language model acting as an oracle for the case that discloses specific findings only when explicitly queried and can synthesize case-consistent results for tests not described in the original CPC narrative. Once a final diagnosis is submitted, the Judge (an o3 model) grades it against a rubric covering the core disease entity, etiology, anatomic site, specificity, and completeness on a 5-point scale (correctness threshold of at least 4). In parallel, a Cost Estimator sums the cost of ordered tests plus a fixed physician-visit cost ($300). The agent is scored simultaneously on accuracy and cumulative testing cost.
Problem solved
Most evaluations of language models in medicine rely on static vignettes and multiple-choice questions that fail to reflect the complexity of evidence-based medicine. In practice a physician iteratively formulates and revises hypotheses, adapting each subsequent question and test to newly obtained information while weighing the cost of the workup. SDBench emulates this sequential process and adds a cost dimension that static benchmarks ignore.
Components
A language model acting as an oracle for the case. It discloses specific clinical findings only when explicitly queried and can synthesize case-consistent results for tests not described in the original CPC narrative.
An evaluating agent implemented on the o3 model with a physician-authored rubric. It scores the diagnosis on the core disease entity, etiology, anatomic site, specificity, and completeness on a 5-point Likert scale; the correctness threshold is a score of at least 4.
A component that sums the cost of all ordered tests (in USD) plus a fixed physician-visit cost of $300. Cost is the benchmark's secondary metric alongside diagnostic accuracy.
Implementation
The Gatekeeper must not reveal findings that were not queried or hint at the final diagnosis; diagnosis-revealing cues must be removed from the original case while preserving clinical realism.
Accuracy scoring relies on a Judge built on the o3 model; the paper validated it by having in-house physicians independently grade the test-set diagnoses.
Evolution
Release of the benchmark over 304 NEJM-CPC cases together with the MAI-DxO orchestrator, which paired with o3 reached 80% accuracy (85.5% in a maximum-accuracy configuration) at a lower cost than physicians.
Parallelism
The benchmark enforces a sequential process by design: each subsequent question and test depends on previously revealed information, up to the final diagnosis.