Gated Linear Units (GLU)

Replaces the standard Transformer FFN layer with a gated product of two linear projections, increasing modeling capacity without adding parameters.

DBRX Base

Title	Publisher	Type
Language Modeling with Gated Convolutional Networks (Dauphin et al., 2016)	—	—
GLU Variants Improve Transformer (Noam Shazeer, 2020)	—	—

—

—

Related models and families