Training

Open-Endedness

2019ResearchPublished: 24 October 2026Updated: 24 October 2026Published

Key innovation

Abandoning a fixed objective in favour of a process that endlessly generates ever harder problems together with their solutions, driving open-ended growth of agents' general competence.

How it works

Open-ended learning runs coupled loops of problem generation and problem solving. A generator (evolution, procedural environment generation, a game opponent, another agent) creates new tasks or environments; a population of learning agents tries to solve them, and their progress is transferred across tasks. The crucial piece is keeping difficulty well-calibrated — tasks must be novel and solvable but not trivial (e.g. the minimal criterion in POET, dynamic task selection and auto-curricula in XLand). Instead of maximising a predefined reward, the system rewards novelty, diversity and rising complexity, often using intrinsic motivation and curiosity as exploration signals. Solved challenges feed the generator of the next ones, so the frontier of competence keeps moving outward without end.

Problem solved

Classic learning toward a single, closed objective yields narrowly specialised agents that overfit a specific task or data distribution and fail to generalise. Open-ended learning addresses the exhaustion of the learning signal and the resulting stagnation: it provides an endlessly renewable source of novel, appropriately difficult challenges that sustain progress and push toward broad, general competence.

Components

Problem / environment generatorSupplies an endlessly renewable source of novelty and increasing complexity.

Mechanism that continuously creates new tasks, environments or challenges — e.g. via evolution, procedural generation or adversarial play.

Official

Population of learning agentsTurns challenges into competence; signals what is already solvable.

Agents (typically trained with reinforcement learning) that attempt the generated tasks and transfer acquired skills across them.

Official

Difficulty regulation / minimal criterionPrevents stagnation and trivialisation; sustains continual progress.

Rule that keeps tasks in the zone of proximal development — novel and hard enough yet solvable (e.g. the minimal criterion in POET, dynamic task selection in XLand).

Official

Novelty / diversity metricSteers exploration toward undiscovered regions of the solution space.

A signal rewarding the distinctness of solutions and behaviours instead of merely maximising a predefined reward; often grounded in intrinsic motivation and curiosity.

Official

Transfer and auto-curriculumMakes solved problems stepping stones to the next ones.

Mechanism for transferring skills across tasks and ordering them into an expanding sequence of ever harder challenges.

Official

Implementation

Reference implementations

POET (uber-research)

Python · Uber AI Labs

Official

Implementation pitfalls

Hard to measure progressHigh

Without a single objective it is hard to define whether the system truly progresses and how to compare runs.

Fix:Use measures of novelty, diversity and learnability plus broad transfer test suites instead of a single metric.

Very high compute costHigh

Population-based, massively parallel training across many environments is expensive and hard to reproduce.

Fix:Cap population size and environment space, rely on transfer and a sensible task-generation budget.

Stagnation or diversity collapseMedium

A poorly tuned difficulty criterion yields trivial or unsolvable tasks, and the population loses diversity.

Fix:Tune the minimal criterion and diversity-maintenance mechanisms; monitor the task-difficulty distribution.

Evolution

Original paper · 2019 · Rui Wang

Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley

2015

Why Greatness Cannot Be Planned

Stanley and Lehman argue that ambitious achievements often emerge from the search for novelty rather than directly chasing an objective — the intellectual foundation of open-endedness.

2019

POET

Inflection point

Paired Open-Ended Trailblazer operationalised open-ended learning: it co-evolves environments and the agents that solve them.

Paired Open-Ended Trailblazer (POET) (paper)

2021

XLand — Open-Ended Learning Leads to Generally Capable Agents

Inflection point

DeepMind's team showed that open-ended training across the vast XLand game space yields broadly, generally capable agents able to transfer zero-shot.

Open-Ended Learning Leads to Generally Capable Agents (paper)

2024

Open-Endedness is Essential for Artificial Superhuman Intelligence

Hughes et al. (DeepMind) propose a formal definition of open-endedness (novelty + learnability) and argue it is a necessary condition for artificial superhuman intelligence.

Open-Endedness is Essential for Artificial Superhuman Intelligence (paper)

Hyperparameters (configurable axes)

Population sizeHigh

Number of concurrently maintained environment–agent pairs or agents in the population.

Difficulty criterionCritical

Threshold defining when a generated task is novel and solvable enough to enter the curriculum.

Transfer frequencyMedium

How often agent skills are transferred across tasks/environments.

Environment generation budgetHigh

Rate and number of new environments/tasks the generator creates per unit of compute.

Parallelism

Parallelism level

Partially parallel

Open-ended methods (POET, XLand) are inherently population-based and massively parallel: many agents and environments train in parallel across many devices. Coordinating transfer and generation of new tasks nonetheless introduces sequential elements.

Scope

TrainingAcross devices