Robots Atlas>ROBOTS ATLAS
Evaluation

HELM

2022ActivePublished
Key innovation
First multi-metric LLM evaluation framework simultaneously measuring 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) across 42 scenarios, revealing model trade-offs invisible in single-metric rankings.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)
Use cases
holistic LLM evaluationmodel comparisonAI transparency researchsafety and fairness assessment

How it works

HELM defines a taxonomy of scenarios (domain x task x metric) and selects a representative subset. Each of 30 models is evaluated on the same prompts under standardized conditions. Results for 7 metrics are reported per scenario and aggregated into a model profile. The platform is hosted by Stanford CRFM with public access to raw data.

Problem solved

Fragmentation and selectivity in LLM evaluation – models were compared on different datasets with different metrics, making fair comparisons impossible and hiding important trade-offs (e.g. high accuracy with high toxicity).

Implementation

Reference implementations
HELM – official Stanford CRFM platform
Python
Official
Implementation pitfalls
Computational cost of full evaluationMedium

Evaluating 30 models across 42 scenarios is computationally and financially expensive, limiting access to full evaluation.

Fix:Use the subset of 16 core scenarios and a single reference model.

Evolution

Original paper · 2022 · Transactions on Machine Learning Research (TMLR) 2023 · Percy Liang
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Christopher D. Manning, Christopher Re
2022
HELM published (arXiv + TMLR)
Inflection point

Percy Liang and 49 co-authors introduce the framework; 30 models evaluated across 42 scenarios.

2023
HELM published in TMLR, extended with new models

Version v2 extends the benchmark with 2023 models and new scenarios.

Hardware requirements

Primary

Evaluation framework independent of hardware architecture – evaluation runs via API or local model inference.