On 28 October 2026 NVIDIA released Nemotron-3.5-Transcribe-30B-A4B, the best open-weight speech model by mean WER on the Open ASR Leaderboard. The same day a pruned version appeared on Hugging Face: 10.0GB instead of 67.5GB, with no retraining at all. Accuracy fell from 4.12% to 5.16% WER, and the pruned model runs faster than the original.
Key takeaways
- Weight size: 67.5GB down to 10.0GB, a 6.75-fold reduction, with no retraining and no distillation
- Mean WER of 5.16% against 4.12% for the base model, measured on 300 clips (FLEURS en/de/es plus LibriSpeech test-clean)
- Intermediate 96-expert variant: 4.15% WER at roughly 53GB, a result within noise after removing a quarter of the experts
- Single-stream throughput on an H200: 50.4 RTFx?RTFx: real-time factor, how many seconds of audio the model transcribes per second of work against 39.3 for the base model
- Method: REAP from the paper arXiv:2510.13999, one calibration pass over roughly 2,500 FLEURS transcripts in 8 languages
Where this starts
A team wants to transcribe call recordings on its own hardware, without sending them to the cloud. Nemotron-3.5-Transcribe is the strongest open-weight model for the job, but NVIDIA's card lists a single 80GB H100 as the deployment configuration. The BF16 weights alone take 67.5GB.
A 12GB card, which is typical developer hardware, is ruled out from the start.
So the question becomes: how much of those 67.5GB is genuinely needed for the model to keep recognising speech correctly?
The pruned version published on Hugging Face by the akhaliq account answers with a number: 10.0GB fits the model on a 12GB card, at the cost of moving from 4.12% to 5.16% error.
What exactly was pruned?
Nemotron's language backbone is a mixture of experts, split into many small subnetworks from which a router picks only a few per token. Hence the gap in the parameter counts: NVIDIA's card lists 34B total but only about 4.3B active per token. The rest sits in memory waiting its turn.
That remainder is the candidate for removal.
Pruning touched only the routed experts in the MoE layers: 128 became 96 or 64. The audio encoder, speaker encoder and the CTC timestamp head were left alone. That is why the pruned model keeps the original's tasks: 40 languages, speech translation, speaker-attributed transcription, word timestamps and question answering over a recording.
How does REAP work?
REAP computes one number per expert: the router's assigned weight times the norm of its activation. The lowest-scoring experts in each layer are removed.
Calibration?Calibration: a pass of the model over a set of sample data, measuring how active each part of the network is is a single pass over roughly 2,500 FLEURS transcripts in eight languages. There is no recovery training and no distillation from the base model. Once experts are dropped, the matching router rows are sliced along with the correction bias, while shared experts, the MTP head and the dense layers stay untouched.
This is a simplified view of the procedure described in the model card, not a full account of the code.
W4A16 quantization comes last, run without calibration, with the router, embeddings, lm_head and Mamba scalars kept at full precision.
What do the numbers show?
The most telling result is not the 10GB variant but the intermediate one. After removing a quarter of the experts, from 128 down to 96, mean WER came in at 4.15% against 4.12% for the base model. That difference sits inside measurement noise: a quarter of the experts contributed nothing measurable.
Only the drop to 64 experts costs something: 5.08%, nearly a full percentage point worse. Stacking 4-bit quantization on top raises that merely to 5.16%, while taking the model from 38GB to 10GB.
| Variant | Experts | Weight size | Mean WER |
|---|---|---|---|
| Base model, BF16 | 128 | 67.5GB | 4.12% |
| Pruned, BF16 | 96 | roughly 53GB | 4.15% |
| Pruned, BF16 | 64 | 38GB | 5.08% |
| Pruned, W4A16 | 64 | 10.0GB | 5.16% |
Speed is the counterintuitive part. The 10GB variant proved the fastest configuration tested: 50.4 RTFx single-stream against 39.3 for the base model, and 171.3 against 142.1 across eight concurrent requests, measured on an H200 under vLLM with greedy decoding?greedy decoding: generation in which the model picks the single most probable token at each step, without exploring alternatives. A smaller model means less data read from card memory per token.
What do these names mean?
Four terms run through the procedure, and without them the numbers say little.
- Mixture of experts (MoE) — an architecture that splits the model into many parallel subnetworks, with a router sending each token to only a few. The equivalent of a firm with many specialists where every case calls in two, not all.
- REAP — the criterion for choosing which experts to remove, combining the router's assigned weight with the strength of the expert's activation. Comparable to clearing out warehouse parts nobody ever reaches for.
- W4A16 — weights stored in 4 bits, arithmetic run in 16. It changes the precision of storage, not the number of parameters.
- WER — word error rate, the share of words recognised incorrectly. Lower is better.
These terms split into two independent levers: REAP reduces the number of parameters, W4A16 reduces the bits per parameter, and WER measures what each one costs.
Why isn't quantization alone enough?
Quantization is simpler and well understood, and NVIDIA itself publishes FP8 and NVFP4 variants of the base model. It has a hard limit, though: it reduces the bits per parameter, not the number of parameters. A 128-expert model stays a 128-expert model whatever precision its weights use.
Pruning works on the other axis, removing whole subnetworks. That is why the two compose, as the figures show: moving from 64 experts in BF16 to 64 in W4A16 cuts size from 38GB to 10GB and worsens WER by 0.08 of a point.
There is a second problem quantization does not have and pruning does: selection. Removing the wrong experts can cut out competencies the test never probes. Calibration across eight FLEURS languages is meant to prevent that, but the published WER figures cover English, German and Spanish. For the other 37 languages, the claim that behaviour is preserved has no backing in measured data.
Is it working yet?
Yes, the weights are published and they run, but this is a community project rather than an NVIDIA release. The repository akhaliq/nemotron-3.5-transcribe-mini-64e was created on 28 October 2026 and holds three servable checkpoints?checkpoints: saved sets of model weights, ready to load and run: 96 experts in BF16, 64 experts in BF16, and the 64-expert W4A16 variant at 10.0GB. A public demo on Hugging Face Spaces is live too.
The model card states end-to-end verification on the fused-64e variant and serving through vLLM. It also names the limits of the measurement: the evaluation covered clips up to 30 seconds plus one of roughly 90 seconds, so hour-long recordings, which the base model supports, were not measured. The recommendation is split: 96 experts for datacenter cards, 64 experts in W4A16 for consumer hardware.
Why does this matter?
The distance between a model launch and its pruned derivative has shrunk to hours rather than months. What the 96-expert result shows matters more: in a model shipped by a vendor with full compute resources, a quarter of the experts proved redundant for speech recognition. That is a question not about compression technique but about how much MoE capacity goes unused in practice. For teams working locally it moves the entry barrier from a datacenter card to desktop hardware.
What next?
- The model card names an unmeasured area: behaviour on recordings approaching the one-hour length the base model handles inside its 65,536-token window
- The REAP paper reports near-lossless compression after removing half the experts in Qwen3-Coder-480B and Kimi-K2, while the 96-expert variant stopped at a quarter, leaving the margin for speech models untested
- NVIDIA states in the base model card that it aims to upstream its NIM serving optimizations into NeMo Speech and vLLM, with no date given
Sources
- Hugging Face — Nemotron-3.5-Transcribe-Mini (REAP-pruned)
- Hugging Face — NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
- arXiv — REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
- Hugging Face Spaces — nemotron-transcribe-mini
