SwiGLU vs GeLU

The FFN is half of every transformer block

Attention gets all the press, but every transformer block follows attention with a feed-forward network (FFN) — a two-layer MLP applied independently to each token position. FFNs account for two-thirds of the parameters in a modern transformer. Their design matters a lot.

Classical FFN:

where $\sigma$is some non-linearity. In the original Transformer it was ReLU. GPT-2 used GeLU (“gaussian error linear unit”), a smoother cousin. Modern models — Llama, Qwen, Phi, SmolLM — use a different construction entirely: the gated linear unit, specifically the SwiGLU variant introduced by Shazeer (2020).

SwiGLU — the multiplicative gate

Two projections are computed from the same input: $W_1 x$ is the candidate value, $W_3 x$ is fed through Swish and becomes the gate. An element-wise multiply combines them, then $W_2$ projects down. The Swish activation is $\text{Swish}(z) = z \cdot \sigma(z)$ where $\sigma$ is the sigmoid.

Why not just bigger ReLU?

SwiGLU introduces a third weight matrix, so at fixed parameter budget you need to shrink the FFN intermediate dimension by about 2/3 to compensate. Almost every ablation ever published (Touvron et al. 2023, Shazeer 2020, Noam Shazeer's GLU paper) finds SwiGLU still wins at fixed parameter count. The multiplicative interaction is worth the extra matrix.

You'll also see GeGLU (GeLU in place of Swish) in Gemma, ReGLU (ReLU) in older models, and plain SwiGLU in Llama, Qwen, Phi, Mistral, SmolLM. The differences between GLU variants are small; the main win is having a gated linear unit at all.