The model maintains embedding tables for n-grams; while processing a sequence it injects information from the relevant n-gram embeddings into token representations. Much of the 'knowledge' resides in embedding parameters (cheap to access at compute time), increasing effective capacity with limited growth in active compute.
Increasing model capacity via more parameters usually raises compute cost and memory needs; accelerator-friendly parameter-scaling methods are needed.
