Grok-1

Grok-1 is xAI's base language model with 314B parameters, released on 17 March 2024 as open-weights under the Apache-2.0 license. It uses a Mixture of Experts architecture (8 experts, 2 active per token), 64 layers, 48 attention heads for queries and 8 for keys/values, embedding size 6144, a SentencePiece tokenizer with 131,072 tokens, and rotary positional embeddings (RoPE). Maximum sequence length is 8,192 tokens. This is the raw pretraining checkpoint (no fine-tuning, no RLHF) trained from scratch by xAI and finished in October 2023, with weights released in March 2024. Due to its size, a multi-GPU machine with substantial memory is required to run it.