1) Initialisation: agent (policy neural network) with random weights. 2) Environment loop: copies of the same policy (or different history snapshots) are paired in interaction โ game, combat, dialogue, attack/defence. 3) Trajectory generation: the agent observes states, chooses actions, receives rewards. Each match generates trajectories (state-action-reward tuples) for both sides. 4) Reward assignment: reward from outcome (win=+1, loss=-1, draw=0 in games; in safety โ attacker=+1 for successful attack, defender=+1 for resist+complete task). 5) Policy update: standard RL algorithms (PPO, REINFORCE, DQN, Actor-Critic, Q-learning) update policy towards higher reward. Alternatively AlphaZero-style: Monte Carlo Tree Search + value/policy heads network. 6) Snapshot management: every N iterations, a copy of the current policy is saved to the "opponents" pool. PFSP samples from this pool proportionally to difficulty. League play maintains a population of diverse agents. 7) Repeat until the agent stops improving or reaches the target level. AlphaGo Zero: 40 days on 4 TPUs. OpenAI Five: 10 months on 128k CPUs. GPT-Red: 'weeks' on frontier compute (comparable to the largest post-training runs).
Classical RL requires an environment with external feedback (reward from the environment or opponent) and often human demonstrations. This scales poorly: (1) there are no infinite human demonstrations, (2) static opponents are quickly beaten, (3) manual curriculum requires engineering work. Self-play RL solves all three: the agent uses itself as an infinite data source, the opponent scales with the agent (natural curriculum), no human demos needed. Additionally: eliminates human demonstration bias (the agent can discover superhuman strategies, as in AlphaGo Zero โ move 37 against Lee Sedol that surprised Go masters). Introduces a self-reinforcing improvement loop โ the entire renaissance of capable AI systems (AlphaGo Zero, MuZero, GPT-Red) rests on this paradigm.
Latest: simple but cyclic policies. Uniform: robust but slow. PFSP (AlphaStar): most common good choice. League: for very complex games with multi-modal policies. Bad choice = collapse to rock-paper-scissors.
Zero-sum (Chess, Go): obvious. Cooperative (Hide-and-Seek): shared reward + potential shaping. Safety self-play (GPT-Red): critical second clause for the defender ("resist AND complete task") โ otherwise the defender learns to refuse everything.
Self-play scales extremely with compute. AlphaGo Zero: 4 TPUs * 40 days. OpenAI Five: 128k CPUs * 10 months. GPT-Red: frontier post-training scale. More compute = better policies, but diminishing returns for saturated tasks.
Single agent + snapshots: simplicity. League (AlphaStar): population with different hyperparameters and objectives โ prevents collapse and discovers diverse strategies. Increasingly important for very complex environments.
AlphaGo (2016): human games -> policy prior -> self-play refinement. AlphaGo Zero (2017): pure tabula rasa, no human data. Tabula rasa is cleaner but requires more compute. A compromise for real-world applications.
How often current policy snapshot enters the pool. Too often โ opponent pool fills with similar policies (redundant). Too rarely โ slow learning. Typically every few % of training iterations.
Self-play is stage-dependent โ the current policy evolves over time, the opponent may be a snapshot, current version, or population. In AlphaZero MCTS is active only during planning; policy/value heads dense. In LLM self-play (GPT-Red) full forward pass of both models โ dense.
Self-play RL is one of the most parallel-scalable approaches in AI. Each game/interaction is an independent transaction โ thousands can be run in parallel. OpenAI Five: 128k CPUs (rollout workers) + 256 GPUs (learners). AlphaGo Zero: distributed self-play on 4 TPUs per game, thousands of games in parallel. Learners and rollout workers separate (actor-learner split). Nearly linear scaling to cluster size.
