GPT-3

GPT-3 (Generative Pre-trained Transformer 3) is OpenAI's autoregressive language model introduced in May 2020 in the paper "Language Models are Few-Shot Learners" (Brown et al.). Its largest variant has 175 billion parameters — ten times more than any previous dense language model.

The paper's central finding was that scaling alone dramatically improves on-the-fly learning: GPT-3 performs new tasks purely from an instruction and a few examples given in the prompt (few-shot / in-context learning), with no gradient updates or weight fine-tuning.

The architecture is a Transformer decoder (96 layers in the 175B variant) with a 2,048-token context window, trained self-supervised on roughly 300 billion tokens from Common Crawl, WebText2, book corpora and Wikipedia. The weights are closed, and the model was offered exclusively through the OpenAI API.

GPT-3 popularized the "prompting instead of fine-tuning" paradigm and became the foundation for products such as ChatGPT and the later GPT-3.5 and GPT-4 families. The original completions models (davinci, curie, babbage, ada) have since been deprecated by OpenAI.