Llama 2 70B

Llama 2 70B is an open large language model (LLM) developed by Meta AI — the largest variant of the Llama 2 family, with about 70 billion parameters. It was released on 18 July 2023 alongside the smaller 7B and 13B variants.

The model uses a Transformer decoder architecture with RMSNorm pre-normalization, the SwiGLU activation function and rotary positional embeddings (RoPE). The 34B and 70B variants additionally use Grouped-Query Attention (GQA) for improved inference scalability. The context window is 4096 tokens, and the model was pretrained on 2 trillion tokens of publicly available data.

The Llama 2-Chat 70B variant, optimized for dialogue, was fine-tuned using supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). The model weights are available for download under the Llama 2 Community License, which permits commercial use with restrictions for services exceeding 700 million monthly active users.