How a prompt becomes an answer

When you send a message to a language model, the path from your words to the model's first word back is not one continuous operation — it is two quite different phases of work, governed by different hardware constraints, and responsible for different parts of the experience you feel. Understanding the split between prefill and decode is the single most useful frame for reasoning about why LLMs behave the way they do at the hardware level.

From words to numbers

Before the model can reason about your prompt, it has to forget that the prompt is text. The tokenizer splits your string into a sequence of token IDs — small integers, each representing a subword piece. A 5,000-character message might become 1,300 tokens; a haiku in Japanese might tokenise very differently from an equivalent haiku in English, because the vocabulary was built to compress the most frequent patterns in the training corpus, not to respect word boundaries.

Each token ID is then looked up in an embedding table: a large matrix with one row per vocabulary entry. The lookup returns a dense vector of floating-point numbers — thousands of dimensions — that the rest of the model actually processes. From this point forward, your prompt is a matrix of numbers, and the original text is gone. Nothing about this step is specific to inference; the same table lookup happens during training. It is a lookup, not a learned computation.

Prefill: the sprint

When your tokenised prompt enters the model, something immediately different happens from what you might expect. The model does not process one token and then the next — it reads the entire sequence at once, in a single parallel forward pass. This phase is called prefill.

Prefill accomplishes two things simultaneously. First, it produces the first generated token — the opening word of the model's reply. Second, and more consequentially for everything that follows, it builds the KV cache: a stored record, layer by layer, of the Key and Value tensors for every position in your prompt. That cache is the model's working memory for every generation step to come.

Because prefill handles all prompt tokens in parallel, the GPU's math hardware has substantial work to do all at once: large matrix multiplications across the full sequence, attention computed over every position simultaneously. This makes prefill compute-bound: the bottleneck is how fast the hardware can execute floating-point operations, not how fast it can read data from memory. Modern GPUs have immense parallel arithmetic capacity, and prefill keeps that capacity fed.

The user-visible consequence is Time To First Token (TTFT) — the pause between submitting a prompt and seeing the first word of the response. Longer prompts mean more work in prefill, which means a longer wait before anything appears. If you have noticed that a model with a short system prompt feels snappier than one with a very long one, you have felt prefill latency.

Decode: the marathon

Once prefill completes, the model shifts into a fundamentally different rhythm. Where prefill processed the entire prompt in one sweep, decode generates one token at a time — attending over the full context, predicting the next token, appending it, and repeating until the model chooses to stop.

Each decode step looks deceptively small. The only new input is a single token. But to produce it, the model must stream its weights from high-bandwidth memory (HBM) — the GPU's on-chip DRAM — to the compute units. It must also stream the entire existing KV cache. The arithmetic involved in processing one new token is modest compared to a full prefill sweep; the data movement is not. Essentially the same weight data flows from HBM to the Tensor Cores on every single step, regardless of how many tokens have already been generated.

This is what makes decode memory-bandwidth-bound: the ceiling is not how fast the GPU can multiply matrices but how fast bytes can travel from HBM to the compute units. The math finishes quickly; then the cores wait for the next batch of numbers to arrive from memory.

A useful analogy: imagine a professor reading aloud while consulting an extraordinarily dense reference shelf. Before each sentence, they must reach to the shelf, pull the relevant volumes, find the right passage, return the books, and then speak. The professor's reading speed and recall are not the bottleneck. The limiting factor is the physical round-trip to the shelf. It does not matter that they could read ten times faster if only the books were already in hand. Every generated token is another trip to the shelf. More context (longer conversations, longer prompts) means heavier volumes on each trip, not just more trips.

The consequence is visible to anyone using a chatbot: the rate at which tokens appear — tokens per second — is governed by memory bandwidth during decode, not by arithmetic capacity. A reply ten times as long takes roughly ten times as long to stream out, because each token costs another full pass over the weights. Prefill governed how long you waited for the first word; decode governs how long you wait for the rest.

The KV cache

The mechanism that makes decode tractable at all is the KV cache. Without it, every decode step would need to recompute the Key and Value tensors for every token in context from scratch — re-processing the entire prompt on each of the hundreds or thousands of generation steps. That would make each step nearly as expensive as a full prefill, multiplied by the number of layers in the model.

The cache avoids this by computing each token's Key and Value tensors exactly once during prefill, storing them, and reusing them on every subsequent decode step. Each decode step extends the cache by one new row per layer — the Key and Value for the token just generated.

The cache's shape is worth picturing precisely. At each transformer layer, the model stores one Key vector and one Value vector for every token in the current context. Multiply by the number of layers (typically 24 to 80 depending on model size) and the cache occupies significant memory even for modest contexts. A context of 2,000 tokens fills the cache to that depth; each generated token adds one more row to every layer's table.

At long contexts, or when a single GPU is serving many users simultaneously, the KV cache can exceed the model's own weights in memory. A 7-billion-parameter model in FP16 sits at roughly 14 GB of weight data; a busy batch with long conversations can push total KV cache storage past that easily. This is why KV cache management — eviction, quantization, sharing — is one of the central engineering problems in LLM serving.

The memory wall

The NVIDIA A100 80 GB SXM puts numbers to the bottleneck. Its FP16 Tensor Core throughput is approximately 312 TFLOPS — 312 trillion floating-point operations per second. Its HBM bandwidth is approximately 2,039 GB/s, roughly 2 TB/s.

During prefill, the 312 TFLOPS are mostly in use. The large matrix multiplications across a long sequence give the hardware real work to do, and compute is the bottleneck in the way it should be.

During each decode step, the same 312 TFLOPS sit largely idle. A single new token requires far fewer floating-point operations than a full prefill sweep — but the weights and KV cache must still travel from HBM to the compute units, step after step. The ~2 TB/s memory bandwidth becomes the ceiling: you cannot decode faster than HBM can supply bytes, regardless of how much arithmetic capacity is theoretically available. This is the memory wall — hundreds of trillions of operations per second of compute waiting on a memory system that, while genuinely fast, cannot feed the cores fast enough during single-token generation.

Why it matters for serving

The two-phase picture explains most of what you see in LLM infrastructure decisions.

Batching strategies exist to keep the GPU busy across both phases. Many prompts can be prefilled together — their parallel matrix multiplications stack usefully, filling the hardware's appetite. But their decode phases may run at different lengths and finish at different times. Production systems manage this mismatch explicitly, slotting new requests into the decode phase as earlier ones finish, rather than waiting for a whole batch to complete before accepting new work.

Prompt caching — reusing the KV cache from a previous call that shared a long system prompt — is a direct application of this model. If the first 2,000 tokens of every call are identical, their KV cache never needs to be recomputed. TTFT and compute cost for that prefix drop to near zero on every subsequent request.

FlashAttention is worth a brief mention here as a foreshadow. It computes the exact same attention as the standard algorithm — not an approximation — but restructures memory access to reduce HBM round-trips. Because the output is mathematically identical, it can replace standard attention anywhere without changing model behaviour. It attacks the memory wall directly, which is why it became a near-universal component in modern transformer implementations almost immediately after publication. The evolution essay covers how this and similar advances reshaped the inference landscape.

The two-phase picture — a short sprint of parallel math, followed by a long marathon of sequential memory reads — is the foundation beneath nearly every optimization in modern inference systems. Once you see it, you start to see it everywhere.