Architecture
N-gram embedding
ActiveUpdated: 28 August 2026Published
Key
innovation
Representing sequences via embeddings of n-grams (groups of consecutive tokens), including to scale memory parameters on accelerators.
Category
Architecture
Abstraction level
Building block
Operation level
Architecture blockModel
Use cases
Parameter scaling in MoE/hybrid modelsEfficient models for memory-constrained acceleratorsSequence representations in NLPQwen-family models (Qwen3.8-Flash-Next)
How it works
The model maintains embedding tables for n-grams; while processing a sequence it injects information from the relevant n-gram embeddings into token representations. Much of the 'knowledge' resides in embedding parameters (cheap to access at compute time), increasing effective capacity with limited growth in active compute.
Problem solved
Increasing model capacity via more parameters usually raises compute cost and memory needs; accelerator-friendly parameter-scaling methods are needed.