The agent is run on the three component benchmarks. DeepSWE v1.1 covers implementation ability, Terminal-Bench 4.0 covers work in a terminal environment, and SWE-Atlas-QnA covers code repository understanding. The partial results are averaged without weighting โ all three components enter the index with equal share. The simplicity of the aggregation is deliberate: it lets anyone reproduce the index value from the published component results and see which component drives a model's change in ranking.
Coding agents are assessed with many benchmarks, each measuring a different slice of the work; results are often contradictory and hard to compare across models. The index solves this organizationally โ it provides a single, explicitly defined number whose composition can be inspected.
Measures the agent's ability to carry out software engineering programming tasks.
66 terminal-based tasks spanning software engineering, system administration and data processing; pass/fail scoring via a test suite, pass@1 averaged over 3 repeats.
Tests how well the agent understands the structure and content of a code repository, not merely whether it can write correct fragments.
Two models with the same index can have entirely different profiles โ one strong in the terminal and weak at repository understanding, the other the reverse.
The index is versioned (v1.5), and swapping a component benchmark shifts the values โ results from different versions are not directly comparable.
Artificial Analysis describes the Coding Agent Index as the average of DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas-QnA.
