Evaluation
TerminalWorld
2026ActiveUpdated: 17 September 2026Published
Key
innovation
A scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from real terminal recordings — instead of hand-curated tasks.
Category
Evaluation
Abstraction level
System
Operation level
Evaluation (runtime)Tooling
Use cases
Evaluating terminal / CLI agentsTesting coding and DevOps agentsVerifying improvement in self-improvement loops (e.g. DarwinX)
How it works
The data engine processes bulk terminal recordings, extracts recurring workflows and turns them into tasks with automatically verifiable success conditions. Tasks are validated, and the best-checked ones form the Verified subset. An agent is scored on whether it correctly completes a multi-step task in a terminal environment.
Problem solved
Hand-curated terminal benchmarks poorly reflect real, complex workflows; TerminalWorld scalably produces authentic tasks from real sessions, complementing (not duplicating) expert-curated benchmarks.
Evolution
Original paper · 2026 · arXiv (May 2026) · Zhaoyang Chu
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, He Ye, Federica Sarro
2026
TerminalWorld released
Inflection pointarXiv 2605.22535 (May 21, 2026); a data engine from 80,870 recordings → 1,530 tasks, Verified subset of 200.
2026
Used in DarwinX
DarwinX reports 68.3% (28/41) on its own held-out split of TerminalWorld (Opus 4.8 base).