Robots Atlas>ROBOTS ATLAS
Safety

LatchBio Biosafety Benchmark

2026ActivePublished: 24 September 2026Updated: 24 September 2026Published
Key innovation
Separates two things that are usually conflated: a model's genuine biological competence and its propensity to refuse. A model can be safe because it understands the difference between research and misuse โ€” or because it knows nothing.
Category
Safety
Abstraction level
System
Operation level
Post-trainingSystem
Use cases
Biosafety auditing of language modelsAssessing model competence in molecular biologyMeasuring refusals on dual-use queriesReporting biological risk in model cards

How it works

The model answers closed-ended biology questions, designed so that some concern routine research work and some are potentially dangerous queries. The closed format allows fully automatic grading with no model-based judge. Results are aggregated as endpoint pass rates โ€” mean pass rates computed at the level of a single evaluation and averaged across multiple run configurations, which limits the influence of random variation. The competence dimension and the refusal dimension are reported separately, so it is visible whether a low risk-category score comes from contextual understanding or from the model's general ignorance.

Problem solved

Assessing models for biological risk is often reduced to the refusal rate alone, which rewards incompetence โ€” a model that does not understand biology will not help with misuse either, but is useless for research. The benchmark separates competence from safety and lets both be seen at once.

Components

Biological competence dimensionFirst axis of evaluation

Tests performance on routine research tasks, including detecting and fixing errors in molecular-biology protocols.

Refusal dimensionSecond axis of evaluation

Measures whether the model refuses dangerous dual-use queries.

Automatically graded closed-ended questionsGrading mechanism

The closed format avoids a model-based judge; results are reported as endpoint pass rates.

Implementation

Reference implementations
biosecbench-surveillance โ€” LatchBio repository
Python ยท LatchBio
Official
Implementation pitfalls
The closed format does not reflect open-ended workMedium

Closed questions give objective grading but do not test how the model behaves in a long, multi-step conversation about a protocol.

Fix:Supplement with red teams conducting multi-turn conversations.
Inconsistent score attributionLow

The 62.4% figure is attributed in sources sometimes to Grok 4.6 and sometimes to Grok 4.7 โ€” any citation must name the specific source.

Fix:Cite the model version and the source together with the number.

Evolution

2026
Public comparison of frontier lab models
Inflection point

The benchmark covers models from xAI, Anthropic, OpenAI and Google; reported endpoint pass rates span 14% to 50%, with a leading figure of 62.4%.