HackerBench v0.3
How it works
The model receives cybersecurity prompts of varying character — from unambiguously legitimate tasks, through dual-use cases, to outright malicious ones. The system records how many risky prompts were let through, meaning served rather than refused. In parallel it measures the refusal rate on lawful tasks, to detect excessive caution. Placing both numbers side by side distinguishes a model that genuinely discriminates intent from one that simply refuses a broad category of topics. Version 0.3 is the one xAI reported alongside Grok 4.7; numbering below 1.0 suggests the benchmark remains under development.
Problem solved
Safety evaluation in cybersecurity easily degenerates into one of two extremes: measuring only blocking, which rewards models that refuse everything, or only usefulness, which ignores misuse. HackerBench forces both indicators to be viewed together.
Components
Tasks that can serve both defence and attack; the share let through is measured.
Lawful security work; the rate of unnecessary refusals is measured.
Implementation
The set composition, task count and criteria for classifying a prompt as risky are not public — results are hard to reproduce independently.
A low pass-through rate alone says nothing — a model that refuses everything will score near zero and be useless.
Evolution
xAI reports that Grok 4.7 lets through 3.3% of risky dual-use prompts while keeping refusals low for legitimate work.