About
The DBRX family consists of open-weights language models built by Databricks (originally the Mosaic team, acquired in 2023). All models in the family are based on a fine-grained Mixture of Experts (MoE) Transformer architecture using RoPE, GLU, and GQA. Models were trained using Databricks' own stack: MegaBlocks, LLM Foundry, Composer, and the Mosaic AI Training Service. Family members: - DBRX Base (132B / 36B active) — foundation model pretrained on 12T tokens, 32K context window - DBRX Instruct (132B / 36B active) — instruction-tuned variant; state-of-the-art among open LLMs at release (March 2024) - DBRX MoE-B (23.5B / 6.6B active) — smaller family member used for training efficiency research - DBRX MoE-A (7.7B / 2.2B active) — smallest member, compared against MPT-7B
