Robots Atlas>ROBOTS ATLAS
Evaluation

Terminal-Bench

2025ActivePublished: 6 August 2026Updated: 6 August 2026Published
Key innovation
A standardized, executable benchmark measuring AI agents' ability to solve real-world tasks in a terminal (CLI) environment, with automated result verification in isolated sandboxes.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)
Use cases
Evaluating terminal agents (Claude Code, Codex, Muse Code)Comparing models on agentic operationMeasuring progress in automating software-engineering tasksTesting agents on ML, security and sysadmin tasksResearch on agents performing multi-step tasks

How it works

Each task is defined together with an isolated environment (container/sandbox) and a set of verifying tests. The AI agent receives the task description and terminal access, then autonomously runs commands, edits files and invokes tools to reach the goal. When finished, tests are run to check whether the success criteria were met; passing is binary (task resolved / not). A leaderboard aggregates results across many agents/models, reporting the fraction of tasks solved for a given version (e.g. terminal-bench@2.1).

Problem solved

Developing terminal-operating agents requires an objective, repeatable measure of their effectiveness on real tasks. Earlier LLM benchmarks often evaluate single answers rather than an agent's multi-step operation in a real execution environment.

Components

Tasks and success criteriaDefines what is measured.

A set of real terminal tasks with clearly defined pass conditions.

Isolated environment (sandbox)Reproducible, safe execution.

A containerized execution environment (Harbor) in which the agent safely runs commands.

Official

Verification testsAn objective correctness signal.

Automated tests checking whether the task was solved correctly (binary pass).

LeaderboardAggregating and comparing results.

A public ranking of the task-resolution rate for agents and models in a given version.

Official

Implementation

Implementation pitfalls
Saturation and training leakageMedium

Popular benchmarks saturate quickly, and their tasks can leak into training data, inflating scores.

Fix:New versions and tasks (2.0/2.1), held-out tasks, rotation and contamination audits.
Dependence on the agent harnessMedium

Scores depend not only on the model but also on the agent 'harness' (tools, loop), making pure model comparisons harder.

Fix:Reporting the agent configuration and standardizing the harness for comparisons.

Evolution

2025
Terminal-Bench 1.0 (80 tasks)
Inflection point

Release of the first version (Stanford x Laude Institute) with 80 terminal tasks and a public leaderboard.

2026
Terminal-Bench 2.0 / 2.1

Expanded and refined versions (2.0 - 89 high-quality tasks; 2.1 - current) plus Science and Challenges variants.