Active Parameters
How it works
In an MoE layer a gating network (router) scores the token and selects k experts out of a pool of N (e.g. top-2 of 8). The computation flows only through the selected experts plus the always-on weights (attention, embeddings, any shared experts). The active-parameter count is the sum of the weights used on that path; the per-token FLOP cost is roughly proportional to it. The total parameter count sums the weights of all experts — all must reside in memory even if a given token does not use them. Hence the split: compute cost ~ active parameters, memory cost ~ total parameters.
Problem solved
It lets model capacity scale (more parameters = more knowledge) without a proportional rise in inference compute cost. It also addresses misleading model comparisons: total parameter count alone overstates the expected cost of MoE models.
Components
All model weights, including every expert. Determines memory/VRAM cost regardless of how much is active per token.
Selects a subset of experts (top-k) for each token, deciding which parameters become active.
The selected feed-forward blocks that actually compute for the given token.
Attention, embeddings and any shared experts — counted as active parameters for every token.
Implementation
Comparing MoE and dense models by total params overstates MoE compute cost; by active params it understates memory needs.
Despite low active parameters, all experts must sit in VRAM; naively sizing memory from active params is wrong.
Different reports include or exclude attention, embeddings and shared experts, so figures can be incomparable.
Poor router load balancing makes effective cost and quality diverge from the nominal active-parameter count.
Evolution
Shazeer et al. introduce the sparsely-gated MoE layer: only a subset of the network is active per example, separating capacity from cost.
A sparse model with "outrageous numbers of parameters but a constant computational cost"; scaling to trillion parameters at constant compute.
Open SMoE model: access to 47B parameters but only 13B active at inference (8 experts, top-2).
Large MoE model with 671B total parameters and 37B activated per token.
Scout (109B total, 16 experts) and Maverick (400B total, 128 experts) each have 17B active parameters per token.
Hyperparameters (configurable axes)
Size of the expert pool; raises total parameters without directly changing active ones.
How many experts activate per token; directly sets the active-parameter count.
Parameter count of a single expert (FFN); scales both total and active.
Experts active for every token; add to the constant part of active parameters.
Computational complexity
Time complexity: O(P_active) per token. Space complexity: O(P_total).
Execution paradigm
Active parameters are meaningful only under conditional/sparse activation; in dense models active = total.
A router selects k experts per token, which directly determines the set of active parameters.
Parallelism
Experts can be sharded across devices (expert parallelism); which ones are active depends on each token’s routing.
Hardware requirements
Low active parameters give high per-token compute throughput, but total parameters demand large VRAM — the model is often memory-bound.