Llama 2 7B

Llama 2 7B is the base (pretrained) language model from Meta's Llama 2 family, released on 18 July 2023. It is the smallest variant in the family, with 7 billion parameters, trained on 2 trillion tokens of publicly available data. The model uses an auto-regressive decoder-only transformer architecture with RMSNorm, RoPE positional encoding and the SwiGLU activation. The 7B variant uses standard multi-head attention (MHA) — grouped-query attention (GQA) is only used in the larger 34B and 70B variants.

This is the base variant, without instruction fine-tuning (SFT) or reinforcement learning from human feedback (RLHF) — those techniques were applied only to the Llama 2 Chat versions. The base model is intended primarily for text completion and as a starting point for custom fine-tuning. The context window is 4096 tokens and the knowledge cutoff is September 2022. Weights are distributed under the Llama 2 Community License (open weights, commercial use permitted with restrictions).