Robots Atlas>ROBOTS ATLAS
Architecture

Active Parameters

2017ActivePublished: 18 November 2026Updated: 18 November 2026Published
Key innovation
Decouples a model’s total capacity (its full parameter count) from the per-token compute cost, which is measured by the number of parameters actually activated.
Category
Architecture
Abstraction level
Pojęcie
Operation level
ModelInferenceArchitecture block
Use cases
Estimating inference cost and throughput of MoE modelsComparing MoE models against dense models at a similar compute budgetGPU/VRAM capacity planning (memory ~ total, compute ~ active)Designing MoE architectures (choosing expert count and top-k)

How it works

In an MoE layer a gating network (router) scores the token and selects k experts out of a pool of N (e.g. top-2 of 8). The computation flows only through the selected experts plus the always-on weights (attention, embeddings, any shared experts). The active-parameter count is the sum of the weights used on that path; the per-token FLOP cost is roughly proportional to it. The total parameter count sums the weights of all experts — all must reside in memory even if a given token does not use them. Hence the split: compute cost ~ active parameters, memory cost ~ total parameters.

Problem solved

It lets model capacity scale (more parameters = more knowledge) without a proportional rise in inference compute cost. It also addresses misleading model comparisons: total parameter count alone overstates the expected cost of MoE models.

Components

Total parameter poolCapacity and memory cost

All model weights, including every expert. Determines memory/VRAM cost regardless of how much is active per token.

Router / gating networkSelection of active parameters

Selects a subset of experts (top-k) for each token, deciding which parameters become active.

Activated experts (FFN)Source of per-token compute cost

The selected feed-forward blocks that actually compute for the given token.

Always-on parametersConstant component of active parameters

Attention, embeddings and any shared experts — counted as active parameters for every token.

Implementation

Implementation pitfalls
Confusing active with total parametersMedium

Comparing MoE and dense models by total params overstates MoE compute cost; by active params it understates memory needs.

Fix:Always report both figures (active / total) when comparing cost and memory.
Memory scales with total, not activeHigh

Despite low active parameters, all experts must sit in VRAM; naively sizing memory from active params is wrong.

Fix:Size memory from total parameters and the compute budget from active ones.
Ambiguous active-parameter definitionMedium

Different reports include or exclude attention, embeddings and shared experts, so figures can be incomparable.

Fix:Check what each model counts as active parameters before comparing.
Uneven expert utilizationMedium

Poor router load balancing makes effective cost and quality diverge from the nominal active-parameter count.

Fix:Use load-balancing losses and monitor expert utilization.

Evolution

Original paper · 2017 · Noam Shazeer
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer
2017
Sparsely-Gated MoE Layer — conditional computation
Inflection point

Shazeer et al. introduce the sparsely-gated MoE layer: only a subset of the network is active per example, separating capacity from cost.

2021
Switch Transformer — decoupling parameters from FLOPs
Inflection point

A sparse model with "outrageous numbers of parameters but a constant computational cost"; scaling to trillion parameters at constant compute.

2024
Mixtral 8x7B — 47B total / 13B active

Open SMoE model: access to 47B parameters but only 13B active at inference (8 experts, top-2).

2024
DeepSeek-V3 — 671B total / 37B active

Large MoE model with 671B total parameters and 37B activated per token.

2025
Llama 4 — 17B active at 109–400B total

Scout (109B total, 16 experts) and Maverick (400B total, 128 experts) each have 17B active parameters per token.

Hyperparameters (configurable axes)

Number of experts (N)High

Size of the expert pool; raises total parameters without directly changing active ones.

Experts per token (top-k)Critical

How many experts activate per token; directly sets the active-parameter count.

Expert sizeHigh

Parameter count of a single expert (FFN); scales both total and active.

Shared expertsMedium

Experts active for every token; add to the constant part of active parameters.

Computational complexity

Time complexity: O(P_active) per token. Space complexity: O(P_total).

Execution paradigm

Primary mode
Sparse

Active parameters are meaningful only under conditional/sparse activation; in dense models active = total.

Activation pattern
Top-K selected
Additional modes
ConditionalMixture
Routing mechanism

A router selects k experts per token, which directly determines the set of active parameters.

Parallelism

Parallelism level
Conditionally parallel

Experts can be sharded across devices (expert parallelism); which ones are active depends on each token’s routing.

Scope
InferenceAcross expertsAcross devices

Hardware requirements

Primary

Low active parameters give high per-token compute throughput, but total parameters demand large VRAM — the model is often memory-bound.