RareBench
How it works
Each of the 122 cases provides a patient's phenotype and genome; the evaluated AI system (typically an agent making many tool calls) must return a ranked list of candidate variants. Scoring is based on top-k causal recall: it checks whether the correct causal variant appears within the first K ranked findings, where K is the number of required findings for that case. Reported metrics include Rank@1 (leading finding), Rank@3 (causal recall) and complete causal recall (all required variants within the top-K). Cases where the system found the correct variant but did not rank it highly enough to clear the top-k cutoff are tracked separately from outright misses, separating two failure modes: missing the relevant biology vs. poor prioritisation. Confidence is reported as 95% Wilson intervals; invalid outputs count as failures.
Problem solved
About 30% of newborns with a suspected rare disease receive a diagnosis, and genome interpretation is hard and time-consuming. The field lacked a standardised, verifiable metric to measure whether agentic AI systems are actually getting better at identifying the causal variant behind a rare disease.
Components
122 rare disease cases; each contains a patient's phenotype and genome.
Known causal variant / diagnosis for each case, making the task verifiable.
Classifies an outcome as top-k recall, a below-top-k detection, or a miss; reports Rank@1 and Rank@3 with 95% Wilson intervals.
Implementation
Evolution
Gamow Labs publishes RareBench 0.1 — a 122-case benchmark for agentic rare genetic disease diagnosis with a public leaderboard of frontier models.
Hyperparameters (configurable axes)
Metric variant: Rank@1 (leading finding), Rank@3 (causal recall) or complete causal recall.
K is the number of required findings for each case; it sets the ranking cutoff counted as a hit.