2017 → 2026: how attention evolved

The attention block has been the heart of every large language model since 2017. But the version running inside today's production systems looks almost nothing like the original. Over nine years, roughly a dozen independent advances have piled on top of one another — some making models smarter, many more making them faster or cheaper to run. Each advance pulled a different lever. Understanding which lever each one pulled, and why the mix shifted over time, is the fastest way to make sense of the landscape.

2017 — The transformer replaces recurrence

quality throughput Before the 2017 transformer, sequence models processed tokens one at a time, left to right, accumulating a hidden state. That recurrence made training inherently sequential — you could not start computing token t + 1 until token t was finished. The transformer replaced that loop with multi-head attention: every token attends to every other token simultaneously, so the whole sequence trains in parallel across GPU cores. Quality improved because attention can directly compare tokens thousands of positions apart without information decaying through a chain of state updates. Throughput improved because parallel matrix multiplications saturate accelerators in a way that sequential recurrence never could.

The KV cache — stop recomputing the past

speed During inference, an autoregressive model generates one token at a time. Without any caching, this means recomputing the keys and values for every previous token at every step — an O(n²) cost that compounds as the sequence grows. The KV cache is a standard technique, present in virtually every transformer inference stack, that stores those intermediate key and value tensors after they are first computed and reads them back on subsequent steps. Generation becomes O(n) per new token rather than O(n²), and per-token latency drops sharply. It is so fundamental that later advances — MQA, GQA, MLA, PagedAttention — are largely about managing the KV cache more cleverly.

2019 / 2023 — MQA and GQA compress the KV cache

memory speed A standard multi-head attention layer with h heads maintains h separate key/value pairs per token. Multi-query attention (MQA, 2019) collapsed that to a single shared KV head, slashing the cache size while keeping the full set of query heads. The quality trade-off was real but acceptable in many settings. Grouped-query attention (GQA, 2023) found a pragmatic middle ground: a small number of KV head groups shared across query heads, preserving most of the memory saving with less of the quality loss. Llama 2 70B adopted GQA, and all Llama 3 models followed — it is now the default architecture for open-weight models rather than an optional optimization.

2020 / 2023 — Sparse attention opens the context window

memory Full attention is quadratic in sequence length: doubling context quadruples memory cost. Longformer (2020) introduced a sliding-window approach where each token attends only to a local neighborhood, with a small number of global tokens able to attend everywhere. This let the model process documents of tens of thousands of tokens without the quadratic blowup. Mistral 7B (2023) brought the same idea into practical open-weight models with a 4096-token sliding window, demonstrating that local sparse attention could match dense attention at common task lengths while fitting comfortably on consumer hardware.

2021 — Mixture of experts scales without scaling per-token compute

quality Adding parameters to a dense model increases the FLOP cost of every forward pass proportionally. Switch Transformer (2021) broke that coupling: instead of one feed-forward network per layer, it trained a large bank of expert sub-networks and routed each token to only one of them. Total parameter count grew by an order of magnitude while per-token compute stayed roughly constant. The mixture-of-experts idea was not new, but the Switch paper showed it worked at transformer scale. By 2024, Mixtral 8×7B demonstrated that a 47B-parameter model with only around 13B active parameters per token could deliver capabilities well above what a dense 13B model would reach.

2021 — RoPE and the long road to longer context

quality Rotary position embedding (RoPE, 2021) encodes position by rotating key and query vectors so the attention dot product depends only on the relative distance between tokens, not their absolute indices. That seemingly small choice had large downstream consequences: relative distance generalizes more naturally to positions unseen during training. The methods that followed — position interpolation (PI, 2023), NTK-aware scaling (community work, no formal paper), and YaRN (2023) — all operate on top of RoPE, rescaling or mixing frequency bands to push inference context beyond the training length without full retraining. Virtually every open-weight model with a context above 32K today relies on some member of this family.

2022 — FlashAttention reorders the computation, not the result

speed memory FlashAttention (2022) did something conceptually striking: it made attention faster without changing what attention computes. The standard implementation writes the full N × N attention matrix to GPU high-bandwidth memory, reads it back to apply softmax, writes it again, reads it again for the weighted sum — four round-trips for a matrix that can be gigabytes at long sequence lengths. FlashAttention restructured the computation into tiles that fit in fast on-chip SRAM, fusing the steps so the large intermediate matrix is never materialized at all. The result is mathematically identical to standard attention — this is exact, not an approximation. FlashAttention-2 (2023) refined the tiling and work partitioning, pushing GPU utilization closer to the memory-bandwidth ceiling.

2022 / 2023 — Continuous batching and virtual memory transform serving

throughput Early inference servers processed one request to completion before starting the next, or padded all requests in a batch to the same length. Orca (OSDI 2022) introduced continuous batching: the server processes one generation step at a time and allows new requests to join the batch as soon as a slot frees — requests interleave at the token level rather than the sequence level. Server throughput increased substantially. PagedAttention (vLLM, 2023) attacked a different waste: KV cache for each request was allocated as a single contiguous memory block, leaving large fractions of GPU memory stranded as requests of varying lengths came and went. PagedAttention borrowed the OS idea of paged virtual memory, managing KV cache in non-contiguous pages and allowing far more concurrent requests on the same hardware. memory

2022 / 2023 — Speculative decoding recovers latency without changing the answer

speed Autoregressive generation is slow because each new token requires a full forward pass through the large target model. Speculative decoding (2022 / 2023) exploits an asymmetry: a small, fast draft model proposes several tokens ahead, and the large target model verifies them all in a single batched forward pass. Tokens the target model would have generated identically are accepted; the first mismatch is corrected and drafting resumes. The acceptance criterion is designed so the final output distribution is provably identical to running the target model autoregressively — there is no quality trade-off, only latency reductions of 2–4× depending on how well the draft model tracks the target.

2023 — Attention sinks let streaming go indefinitely

memory speed A transformer trained on sequences up to length L breaks when asked to generate beyond L: once older tokens scroll out of the cache window, positional indices shift and the model loses coherence. StreamingLLM (2023) found that keeping a small number of "attention sink" tokens — the earliest tokens in the sequence, which attract disproportionate attention mass — alongside the sliding window of recent tokens is sufficient to maintain stability indefinitely. With as few as four sink tokens, a model trained on 4096-token sequences can stream millions of tokens without degradation or retraining. The runtime cost is negligible.

2024 — Multi-head latent attention compresses the KV cache into a latent vector

memory throughput Multi-head latent attention (MLA), introduced in DeepSeek-V2 (2024), took a different route to a smaller KV cache. Rather than reducing the number of KV heads, MLA down-projects keys and values into a shared low-dimensional latent vector per token and stores that compressed form in the cache. The full per-head keys and values are reconstructed from the latent at attention time. The result: a 93.3% reduction in KV cache size compared to standard MHA, and a 5.76× improvement in measured throughput — without the quality penalty that typically accompanies head-reduction approaches like MQA.

2024 / 2025 — Multi-token prediction and tuned local-to-global ratios

quality speed Multi-token prediction (MTP, 2024) trains the model to predict not just the next token but several tokens ahead in parallel, using auxiliary output heads. At inference those heads provide cheap speculative drafts that the primary head can verify, recovering latency. Training with MTP also appears to sharpen the primary head's representations, because predicting multiple steps ahead forces the model to carry more information through its hidden states. Separately, the interleaving of local and global attention layers became quantitative: Gemma 2 (2024) used a 1:1 ratio; Gemma 3 (2025) shifted to 5:1 with a 1024-token local window, substantially cutting KV cache cost while preserving long-range coherence through the sparser global layers. memory

2024 — DeepSeek-V3 synthesizes the toolbox

memory throughput quality DeepSeek-V3 (2024) is a useful landmark because it deploys nearly every advance on this timeline simultaneously at scale: MLA for KV compression, a mixture-of-experts architecture with 671B total and 37B active parameters per token, and MTP for training and inference efficiency. It is not that DeepSeek-V3 invented new mechanisms — it demonstrated that the 2022–2024 efficiency toolbox, composed carefully, produces a model that is both state-of-the-art in capability and considerably cheaper to serve than contemporaneous dense alternatives.

The pattern across nine years

Stepping back, the timeline has a clear structure. The early wave — 2017 through roughly 2021 — was overwhelmingly about quality: replacing recurrence with parallel attention, learning better position representations, adding parameter capacity through mixture of experts. Models were simply not good enough, so every lever pointed at making them better.

Around 2022 the locus of difficulty shifted. Models had crossed a threshold of usefulness and the constraint moved from raw capability to cost of deployment. The 2022–2026 wave is dominated by speed, throughput, and memory advances — FlashAttention, continuous batching, PagedAttention, speculative decoding, attention sinks, MLA. Several of the most important advances in this era do not change what is computed at all: FlashAttention produces exactly the same output as the naive implementation; speculative decoding yields a provably identical output distribution. The innovation is entirely in how efficiently the GPU executes the identical math — a category of progress that is easy to underestimate because it leaves the model's behavior unchanged.

By 2024, the frontier stopped treating quality and efficiency as separate problems. DeepSeek-V3, Gemma 3, and their contemporaries co-design from the start: choosing attention head configurations, expert counts, and training objectives that are simultaneously good for capability and cheap to serve. That co-design posture — where the architecture decision and the serving decision are the same decision — is what the next wave will build on.