Each task is defined together with an isolated environment (container/sandbox) and a set of verifying tests. The AI agent receives the task description and terminal access, then autonomously runs commands, edits files and invokes tools to reach the goal. When finished, tests are run to check whether the success criteria were met; passing is binary (task resolved / not). A leaderboard aggregates results across many agents/models, reporting the fraction of tasks solved for a given version (e.g. terminal-bench@2.1).
Developing terminal-operating agents requires an objective, repeatable measure of their effectiveness on real tasks. Earlier LLM benchmarks often evaluate single answers rather than an agent's multi-step operation in a real execution environment.
A set of real terminal tasks with clearly defined pass conditions.
A containerized execution environment (Harbor) in which the agent safely runs commands.
Official
Automated tests checking whether the task was solved correctly (binary pass).
A public ranking of the task-resolution rate for agents and models in a given version.
Official
Popular benchmarks saturate quickly, and their tasks can leak into training data, inflating scores.
Scores depend not only on the model but also on the agent 'harness' (tools, loop), making pure model comparisons harder.
Release of the first version (Stanford x Laude Institute) with 80 terminal tasks and a public leaderboard.
Expanded and refined versions (2.0 - 89 high-quality tasks; 2.1 - current) plus Science and Challenges variants.
