DBRX MoE-A

DBRX MoE-A is the smallest member of the DBRX family: an internal Databricks research model with 7.7B total parameters and 2.2B active. It was not released publicly — it serves as a benchmark for studying training efficiency of Mixture of Experts (MoE) architectures and validating the final DBRX training recipe.

Purpose

DBRX MoE-A serves as a head-to-head comparison against MPT-7B (Databricks/Mosaic's May 2023 model). The results demonstrate how the new DBRX training stack (MoE architecture, better data, GPT-4 tokenizer) improves training efficiency over dense 7B-class models.

Results

On the Databricks Model Gauntlet v0.3 it achieves 30.5% — comparable to MPT-7B (30.9%) — but at 3.7x lower compute cost (FLOPs). This demonstrates that the full DBRX training recipe is roughly 4x more compute-efficient than the previous-generation MPT pipeline.

Status

The model is not publicly available, neither on Hugging Face nor through APIs. Weights and checkpoints remain internal to Databricks. The public DBRX blog (March 2024) lists it only as a validation artifact of the training pipeline.