About
Nemotron-H is a family of NVIDIA hybrid language models in which most Transformer self-attention layers are replaced by Mamba-2 and MLP layers, keeping only a small number of attention layers. The goal is to retain accuracy comparable to pure Transformer models while improving inference throughput and reducing memory use at long context. The family spans 8B, 47B and 56B models, with Base, Instruct and Reasoning variants. Described in the technical report "Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models" (arXiv:2504.03624).
