Gated DeltaNet

Gated DeltaNet is a sequence architecture from the linear-transformer family, developed by Songlin Yang (MIT CSAIL), Jan Kautz and Ali Hatamizadeh (NVIDIA Research). The paper "Gated Delta Networks: Improving Mamba2 with Delta Rule" (arXiv:2412.06464) was submitted on 9 December 2024 and accepted to ICLR 2025. The model combines gating — enabling rapid memory erasure — with the delta update rule, which allows precise, targeted state modifications.

The authors release only the (PyTorch) code under the NVIDIA Source Code License-NC (non-commercial); pretrained weights are not provided. The architecture has been integrated into the Flash Linear Attention library and adopted in models such as Qwen3-Next and OLMo Hybrid. Experiments in the paper used models at the 0.4B and 1.3B parameter scale trained on FineWeb-Edu and SlimPajama-672B.