Robots Atlas>ROBOTS ATLAS
GPT-2
AI Models›GPT

GPT-2

1.5B (1542M) · Family: GPT
OpenAI autoregressive language model (up to 1.5B parameters), decoder-only Transformer architecture, trained on the WebText dataset.
📦 Archived✓ Public access⚖ Open sourceLLM📁 GPT
Context window
1024 tokenów
tokens
Parameters
1,5 mld (największa wersja); warianty 124M, 355M, 774M, 1,5B
parameters
Release date
14 February 2019
🏢OpenAIProducer
Access:DownloadDeployment:💻 Local☁ Cloud

Overview

GPT-2 (Generative Pre-trained Transformer 2) is an autoregressive language model developed by OpenAI, announced on 14 February 2019. Its largest variant has about 1.5 billion parameters; smaller variants of 124M, 355M and 774M parameters were also released. The model uses a decoder-only Transformer architecture (largest version: 48 layers, hidden size 1600) with a context length of 1024 tokens and a vocabulary of 50,257 byte-level BPE tokens.

GPT-2 was trained on WebText - roughly 40 GB of text from 8 million web pages linked from Reddit (posts with at least 3 karma before December 2017). It was trained via self-supervised next-token prediction (causal language modeling) without task-specific fine-tuning. In the paper "Language Models are Unsupervised Multitask Learners", GPT-2 achieved state-of-the-art zero-shot results on 7 of 8 tested language modeling datasets.

OpenAI adopted a staged release strategy over misuse concerns: initially only the smallest model was published, the 774M model was released on 20 August 2019, and the full 1.5B weights were released on 5 November 2019 under a Modified MIT license.

Classification
LLM
Family: GPT
Access & deployment
Download
LocalCloud
Weights: Open source
Key parameters
📏 Context: 1024 tokenów
🧩 Parameters: 1,5 mld (największa wersja); warianty 124M, 355M, 774M, 1,5B
✓ Fine-tuning
📥 Input: text

Technical specification

Context window
1024 tokenów
tokens
Parameters
1,5 mld (największa wersja); warianty 124M, 355M, 774M, 1,5B
parameters
Knowledge cutoff
1 Dec 2017
Knowledge boundary
License
Modified MIT
Hardware requirements
Variants from 124M to 1.5B parameters; the largest model (1.5B) can run on a single GPU.
Features:✓ Fine-tuning
Modalities
⬇ Input
text
⬆ Output
text

Capabilities and applications

Native model capabilities
Language modeling
Ability to predict subsequent tokens and generate coherent natural-language text based on the preceding context.
Category: language
Zero-shot learning
The model's ability to perform a new task without dataset-specific training or hyperparameter tuning — prediction is produced in a single pass from context.
Category: other

Benchmark results

6 benchmarks
LAMBADA
accuracy · zero-shot
63.24%
📄 paper
1542M model, no training or fine-tuning (Table 3).
LAMBADA
perplexity · zero-shot
8.63
📄 paper
1542M model (Table 3).
WikiText-103
perplexity · zero-shot
17.48
📄 paper
1542M model (Table 3).
WikiText-2
perplexity · zero-shot
18.34
📄 paper
1542M model (Table 3).
Penn Treebank
perplexity · zero-shot
35.76
📄 paper
1542M model (Table 3).
CBT-NE
accuracy · zero-shot
89.05%
📄 paper
1542M model (Table 3).

Technical architecture

Sources and related pages