Security & Privacy
CTI-REALM
Category
Security & Privacy
Abstraction level
Benchmark
Operation level
Evaluation (runtime)Application
Use cases
Evaluating AI models for cybersecuritySelecting a model for threat-analysis tasksResearch on security agents
How it works
The benchmark includes tasks mirroring a CTI analyst's work — report analysis, indicator extraction and correlation, mapping to attack-technique taxonomies — and scores the accuracy and completeness of model or agent outputs.
Problem solved
How to measure AI models' usefulness for real cyber threat intelligence tasks, not just general security questions?
Strengths & limitations
Strengths
✓Realistic, domain-specific security tasks
✓Measures practical usefulness, not just general knowledge
✓Supports tool selection for SOC teams
Limitations
✗Narrow domain (cybersecurity)
✗Rapid obsolescence as threats evolve
✗Risk of test-data leakage