Llama 2 13B

Llama 2 13B is a 13-billion-parameter language model from the Llama 2 family developed by Meta AI and released on 18 July 2023. It is an autoregressive model based on a decoder-only transformer architecture.

The model uses RMSNorm normalization, the SwiGLU activation function and rotary position embeddings (RoPE). Unlike the largest 70B variant, the 7B and 13B models use standard Multi-Head Attention rather than Grouped-Query Attention.

The base model was trained on roughly 2 trillion tokens of publicly available data, with a 4,096-token context window and a knowledge cutoff of September 2022. The Llama 2-Chat variant was fine-tuned with supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF).

The model weights are available for download under the Llama 2 Community License, which permits commercial use subject to the restrictions set out in the acceptable use policy.