DBRX MoE-B

DBRX MoE-B is an internal Databricks research model in the DBRX family, intermediate between MoE-A (7.7B) and the flagship DBRX (132B). It has 23.5B total parameters and 6.6B active. It was not released publicly — it serves to validate scaling of the Mixture of Experts (MoE) architecture across growing model sizes.

Purpose

DBRX MoE-B serves as a head-to-head comparison against LLaMA2-13B and the full DBRX-132B. It demonstrates how training efficiency of the DBRX family scales into the 20B+ parameter range, and how fine-grained MoE (16 experts, 4 active) behaves at mid-scale.

Results

On the Databricks Model Gauntlet v0.3 it achieves 45.5%, outperforming LLaMA2-13B (43.8%) at significantly lower training cost. This is the key validation that the DBRX recipe (MoE architecture + curriculum learning + high-quality data + GPT-4 tokenizer) scales smoothly with model size.

Status

The model is not publicly available, neither on Hugging Face nor through APIs. Weights remain internal to Databricks. The public DBRX blog (March 2024) lists it only as a validation artifact — together with MoE-A — of the full DBRX-132B training recipe.