APEX-Agents
How it works
Each task is embedded in an environment containing input files and available tools, and the agent must perform multi-step work until it delivers an output. The agent's output is graded against rubrics and gold outputs, and performance is reported with Pass@1 (the share of tasks solved on a single attempt). Agent execution and grading are handled by the open-source Archipelago infrastructure. Tasks are drawn from three professional domains: investment banking, consulting and corporate law.
Problem solved
Prior professional benchmarks (including APEX) evaluated models on single questions and did not test whether an agent can carry a realistic, multi-step task to completion using files and tools. APEX-Agents fills this gap by measuring agent productivity on tasks that resemble actual professional work.
Implementation
Evolution
Mercor releases APEX, a benchmark evaluating frontier models on single expert-level questions from four professions: investment banking, consulting, corporate law and medicine.
Mercor extends APEX to agentic tasks: 480 long-horizon, multi-step tasks in realistic environments with files and tools; the top Pass@1 score is 24.0%.