Robots Atlas>ROBOTS ATLAS
Evaluation

ProgramBench

2026ActivePublished: 18 November 2026Updated: 18 November 2026Published
Key innovation
Instead of patching existing code (like SWE-bench), ProgramBench requires the agent to rebuild an entire program from scratch using only the compiled binary and its documentation — with no access to the source code, decompilation, or the internet.
Category
Evaluation
Abstraction level
Benchmark
Operation level
Evaluation (runtime)Agent runtime
Use cases
Evaluating autonomous coding agentsMeasuring long-horizon software engineeringProgram reconstruction from a binary and documentationComparing AI models on code generation tasksStudying the limits of from-scratch system planning and building

How it works

1) Task input: the agent receives the compiled binary of the target program and its documentation. 2) Constraints: no access to source code, no decompilation, and no internet during inference; the agent chooses the language, project structure, and build script itself. 3) Production: the agent produces a complete codebase (submission.tar.gz) reproducing the original's behavior. 4) Evaluation: the ProgramBench harness builds the submission in an isolated Docker container (cleanroom images, linux/amd64 only) and runs hidden, per-branch behavioral test archives downloaded from HuggingFace. 5) Scoring: an instance counts as "resolved" when all tests pass; additionally "almost resolved" (≥95% of tests) and the average pass rate are reported. The programbench.com leaderboard ranks models by resolved rate, with the identical mini-swe-agent scaffold for all.

Problem solved

Most coding benchmarks (e.g. SWE-bench) measure small patches to an existing repository, which does not test the ability to independently architect and build an entire working system. ProgramBench fills that gap: it tests long-horizon, autonomous software construction from scratch and verifies it rigorously with a large body of hidden behavioral tests rather than shallow assertions.

Components

Task: binary + documentationBenchmark input

Each of the 200 instances provides only the compiled executable and the documentation of the target program — the sole allowed input for the agent.

Agent submission (submission.tar.gz)Evaluated artifact

The complete codebase produced by the agent, archived per instance, meant to reproduce the behavior of the original program.

Dockerized evaluation harness (cleanroom)Scoring engine

Builds the submission in isolated cleanroom images (linux/amd64 only) and runs hidden per-branch test archives downloaded from HuggingFace.

Behavioral test suiteCorrectness criterion

Over 248,000 hidden behavioral tests verifying that the generated code behaves like the original; the basis for the resolved / almost-resolved metrics.

Implementation

Implementation pitfalls
No internet during inferenceHigh

The agent must not access the internet while solving a task; breaking this rule invalidates the submission as contaminated.

Fix:Run inference in an isolated, network-disabled environment and verify cleanroom integrity before submitting.
linux/amd64 images onlyMedium

The benchmark's Docker images are built only for linux/amd64 (x86_64) and do not run natively on macOS/Windows; QEMU emulation is slow.

Fix:Run inference and evaluation on a Linux x86_64 machine.

Evolution

Original paper · 2026 · arXiv (cs.SE) · John Yang
ProgramBench: Can Language Models Rebuild Programs From Scratch?
John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, Ofir Press
2026
ProgramBench released
Inflection point

Meta (FAIR) and collaborators release ProgramBench (arXiv:2605.03546), the facebookresearch/ProgramBench repository, a HuggingFace dataset, and the programbench.com leaderboard, with a mini-swe-agent based baseline.

Hardware requirements

Primary

ProgramBench is an evaluation harness run in Docker containers (linux/amd64) and is independent of the model and accelerator; hardware requirements depend on the evaluated agent, not on the benchmark itself.