Evaluation
AA-Omniscience
Category
Evaluation
Abstraction level
Benchmark
Operation level
Evaluation (runtime)
Use cases
Assessing LLM knowledge and hallucinationComparing model reliabilitySelecting a model for fact-critical applications
How it works
The benchmark poses factual questions across many domains and scores both correctness and whether the model abstains when it does not know. It rewards calibration (admitting ignorance) and penalizes confident hallucinations.
Problem solved
How to measure not only a model's knowledge but also its tendency toward confident yet false answers (hallucinations)?
Strengths & limitations
Strengths
✓Combines knowledge measurement with hallucination measurement
✓Rewards calibration and honest uncertainty
✓Broad domain coverage
Limitations
✗Results depend on question selection
✗Risk of test-set leakage into training data
✗Difficulty fully covering all domains