Robots Atlas>ROBOTS ATLAS
Evaluation

AA Briefcase v1.1

2026ActivePublished: 23 September 2026Updated: 23 September 2026Published
Key innovation
Evaluates not an answer in a chat window but the output files of multi-week business projects โ€” and does so on two tracks: a rubric for measurable task success, and pairwise comparisons for analytical and presentation quality.
Category
Evaluation
Abstraction level
System
Operation level
Agent runtimeSystem
Use cases
Evaluating models for knowledge work and business tasksComparing the quality of AI-generated documents and presentationsSelecting a model for long agentic projectsThe highest-weighted component of the Artificial Analysis Intelligence Index

How it works

The model receives a task mirroring part of a multi-week business project and must produce output files. Grading combines two mechanisms. The rubric checks verifiable task success โ€” whether what was asked for was delivered in the required form. Pairwise comparisons assess analytical quality and presentation quality: outputs from different models are set against each other in pairs, and the resulting order is converted into Elo ratings. The three components โ€” analytical quality, presentation quality and rubric score โ€” are then aggregated. The v1.1 set contains 91 tasks and enters the Artificial Analysis Intelligence Index with a 15% weighting.

Problem solved

Classic benchmarks measure single answers, whereas knowledge work consists of producing a document, spreadsheet or presentation whose quality cannot be reduced to one correct string. AA Briefcase evaluates real artefacts and separates what can be verified objectively from what requires qualitative comparison.

Components

91 tasks with file outputsTest set

The tasks mirror realistic knowledge work spread across multi-week business projects.

Rubric gradingObjective track

Checks verifiable task success โ€” whether the required elements were delivered in the expected form.

Pairwise comparisons and Elo scaleQualitative track

Analytical and presentation quality assessed by pitting outputs against each other in pairs, converted into Elo ratings.

Implementation

Implementation pitfalls
Elo depends on the field of compared modelsMedium

Elo ratings are only meaningful relative to the specific field of compared models; adding or removing a participant shifts the scale.

Fix:Do not compare Elo values across runs with different fields, or across benchmark versions.
Presentation quality rewards formMedium

Part of the score concerns presentation quality, which can reward visually polished outputs over substantively sharper but rawer ones.

Fix:Read the breakdown into analytical quality, presentation quality and rubric rather than the aggregate score alone.

Evolution

2026
Version v1.1 with a 15% weighting in the Intelligence Index
Inflection point

AA Briefcase v1.1 covers 91 tasks with file outputs and becomes the highest-weighted component of the index, level with AA-Omniscience.