nanochat

nanochat is a minimal, open-source (MIT) training framework by Andrej Karpathy that lets you train your own ChatGPT-style language model from scratch on a single GPU node. The code covers every stage: tokenizer training (GPT-4-style BPE), base-model pretraining, supervised fine-tuning (SFT), optional reinforcement learning, and inference via a CLI and a web chat UI.

The repository is at once the training code and a recipe for a concrete resulting model. The reference speedrun trains a d20 model (depth 20 layers, ~561M parameters, model dimension 1280, 10 attention heads, 1024-token context, 65,536-token vocabulary) on ~11.2B tokens in about 4 hours on an 8×H100 node, at a cost of roughly $100. The model reaches GPT-2-grade capability (CORE score ~0.22 for the base model).

Model complexity is controlled by a single `--depth` dial; all other hyperparameters are set automatically in a compute-optimal way. nanochat uses a transformer architecture with an AdamW + Muon optimizer, a KV-cache inference engine, and Python code execution as a tool. The project maintains a 'time-to-GPT-2' leaderboard measured by the DCLM CORE metric.