ARC-AGI-3
How it works
An agent enters an interactive game with no instructions, rules, or stated goal. It perceives successive frames/states of the environment and takes actions in a turn-based manner, discovering the mechanics and the win condition on its own. Environments are novel, which prevents memorizing solutions. Scoring is based on action efficiency - the number of actions needed to complete levels compared to a human; 100% means beating every game as efficiently as humans. Access is via an API (in partnership with Hugging Face) and an agent harness, and the set is split into public and private (holdout) games to prevent overfitting to the visible tasks.
Problem solved
Static benchmarks (ARC-AGI-1/2) score single answers and are vulnerable to memorization and brute-force. They do not measure an agent's ability to act over time: exploration, long-horizon planning under sparse feedback, belief updating, and learning from experience. ARC-AGI-3 fills this gap by evaluating intelligence as a process rather than a single output.
Components
Hundreds of novel, turn-based games with no instructions; the agent perceives state and takes actions, discovering the goal and mechanics.
Interface (released in partnership with Hugging Face) letting agents observe state and submit actions in the game loop.
Official
Public games are used for agent development, private holdout games for fair evaluation on never-seen tasks.
Scores the number of actions needed to complete levels relative to humans; 100% = as efficient as humans.
Evolution
Francois Chollet publishes ARC-AGI alongside 'On the Measure of Intelligence' - static grid tasks.
Second generation of the static benchmark, harder for models; the ARC Prize Foundation is established.
Sneak peek with 3 public games (ls20, ft09, vc33) and 3 private holdout games, plus the first API version (Hugging Face).
Hundreds of interactive environments and thousands of levels; at launch humans 100%, frontier AI 0.51%; $2M prize pool, Kaggle competition.