DBRX Base

DBRX Base is the foundation pretrained language model in the DBRX family, released on March 27, 2024 by Databricks under the open Databricks Open Model License. Unlike DBRX Instruct, it has not been instruction-tuned — it serves as a foundation for further fine-tuning or continued pretraining on enterprise data.

Architecture

DBRX Base is a decoder-only Transformer with fine-grained Mixture of Experts (MoE): 132B total parameters, 36B active per token, 16 experts of which 4 are selected per input. It uses Rotary Position Encodings (RoPE), Gated Linear Units (GLU), Grouped Query Attention (GQA), and the GPT-4 tokenizer from the tiktoken repository. Maximum context length is 32,768 tokens.

Training Data and Compute

DBRX Base was pretrained on 12 trillion tokens of carefully curated text and code data, with curriculum learning (mix changes during training). Data processing relied on Apache Spark, Databricks Notebooks, and Unity Catalog. Training ran on 3,072 NVIDIA H100 GPUs connected via 3.2 Tbps InfiniBand, using MegaBlocks, LLM Foundry, Composer, and Streaming. Databricks estimates the new pretraining data is at least 2x better token-for-token than the data used for the MPT family.

Use Cases

DBRX Base is intended primarily for advanced users: ML teams that want to perform their own instruction tuning, RLHF, or continued pretraining on domain-specific data. For typical chat and instruction-following workloads, Databricks recommends DBRX Instruct.