ProgramBench
How it works
1) Task input: the agent receives the compiled binary of the target program and its documentation. 2) Constraints: no access to source code, no decompilation, and no internet during inference; the agent chooses the language, project structure, and build script itself. 3) Production: the agent produces a complete codebase (submission.tar.gz) reproducing the original's behavior. 4) Evaluation: the ProgramBench harness builds the submission in an isolated Docker container (cleanroom images, linux/amd64 only) and runs hidden, per-branch behavioral test archives downloaded from HuggingFace. 5) Scoring: an instance counts as "resolved" when all tests pass; additionally "almost resolved" (≥95% of tests) and the average pass rate are reported. The programbench.com leaderboard ranks models by resolved rate, with the identical mini-swe-agent scaffold for all.
Problem solved
Most coding benchmarks (e.g. SWE-bench) measure small patches to an existing repository, which does not test the ability to independently architect and build an entire working system. ProgramBench fills that gap: it tests long-horizon, autonomous software construction from scratch and verifies it rigorously with a large body of hidden behavioral tests rather than shallow assertions.
Components
Each of the 200 instances provides only the compiled executable and the documentation of the target program — the sole allowed input for the agent.
The complete codebase produced by the agent, archived per instance, meant to reproduce the behavior of the original program.
Builds the submission in isolated cleanroom images (linux/amd64 only) and runs hidden per-branch test archives downloaded from HuggingFace.
Over 248,000 hidden behavioral tests verifying that the generated code behaves like the original; the basis for the resolved / almost-resolved metrics.
Implementation
The agent must not access the internet while solving a task; breaking this rule invalidates the submission as contaminated.
The benchmark's Docker images are built only for linux/amd64 (x86_64) and do not run natively on macOS/Windows; QEMU emulation is slow.
Evolution
Meta (FAIR) and collaborators release ProgramBench (arXiv:2605.03546), the facebookresearch/ProgramBench repository, a HuggingFace dataset, and the programbench.com leaderboard, with a mini-swe-agent based baseline.
Hardware requirements
ProgramBench is an evaluation harness run in Docker containers (linux/amd64) and is independent of the model and accelerator; hardware requirements depend on the evaluated agent, not on the benchmark itself.