An LLM response has two jobs that look similar from outside but behave very differently on a GPU. First, it reads the prompt and fills the KV cache. Then it generates one new token at a time while reading that cache. We call the first job prefill and the second decode.
Prefill reads the known prompt; decode grows the unknown answer token by token.
Prefill Reads What We Already Know
Suppose a user sends a 2,048-token document and asks for a summary. All 2,048 prompt tokens are already present. The model can process their positions in large matrix operations while the causal mask keeps each position from reading future prompt words.
During this pass, every layer creates the key and value entries that form the initial cache. Prefill ends when the model can choose the first output token.
At the last layer, the final prompt position supplies the hidden vector used for the first next-token distribution. We do not need a generated token as input before this first prediction. That is why prefill can produce the first output choice while leaving the cache filled only with prompt positions.
A causal mask still matters within prefill. Although all prompt tokens exist, an early prompt representation should be the same representation the decoder would have built before seeing later positions. That consistency is what makes the cache reusable during subsequent decoding.
The whole prompt is available before the forward pass begins.
Prefill is not free. Attention over a long prompt has many token-to-token comparisons, and the cache must be written. But it has useful parallel work because the prompt positions are known together.
The First User-Facing Metric: TTFT
Time to first token, or TTFT, is the wait from sending the request until the first generated token appears. Measured at the client, it can include request transport, prompt tokenization, queueing, prefill, first-token selection, and transport or buffering before that token is displayed.
Define the measurement boundary before comparing two TTFT numbers. A server-side metric might begin after the request arrives, while a browser metric begins when the user presses Send. Both can be useful, but they include different delays.
If the server finishes quickly but the browser shows nothing, inspect transport and buffering. If most time is spent waiting before GPU work begins, faster attention alone will not remove that queue. If prefill dominates, prompt length and its computation become more relevant targets.
A long prompt usually raises TTFT because prefill must finish before output can start.
For a chat interface, TTFT often feels like responsiveness. A user can forgive a longer answer if the first useful word arrives soon. That is why systems report TTFT separately instead of only reporting total request time.
Decode Writes the Answer One Step at a Time
After prefill, imagine the first token is Summary:. That selected token has not yet produced its own cache entries. To choose the second token, the model processes this newest input, appends its keys and values, and reads the allowed cached history. This repeats until an end token or length limit stops the answer.
Each decode step is small in token count but depends on the previous chosen token.
This step-by-step dependency means that decode cannot use the same position parallelism as prefill. A 128-token answer needs 127 further token selections after the first token, in the ordinary one-token-per-step loop.
The Second User-Facing Metric: Output Speed
During streaming, we measure tokens per second. At a steady 40 tokens per second, the remaining 199 tokens of a 200-token answer take about 4.975 seconds after the first token. This is a per-request output rate; it is not the server's total output throughput across all requests.
After the first token, users experience the spacing between later tokens.
Do not confuse this with prefill throughput. A server may process thousands of prompt tokens per second yet generate only tens of output tokens per second for one conversation. The two phases stress hardware in different ways.
Separate Token Gaps From Server Throughput
Inter-token latency, or ITL, is the gap between two successive output tokens. A steady 25 milliseconds per gap corresponds to 40 tokens per second because .
Real streams are not always steady. A service may pause to schedule another request or buffer several tokens before sending them. Reporting only an average can hide those pauses. A median gap describes the middle observed gap; a 95th-percentile gap is a value that 95% of measured gaps do not exceed.
Server throughput counts tokens across all active requests per unit time. Ten requests each producing 20 tokens per second could deliver 200 tokens per second in total, even though each reader sees a slower stream than a single request at 40. Always state whether a speed number describes one request or the whole server.
A Concrete Request Timeline
Take a request with a 2,048-token prompt and a 128-token answer. For an invented timing example, if prefill takes 0.8 seconds and the first selection takes 0.05 seconds, TTFT is about 0.85 seconds before queueing and tokenization. If decode runs at 40 tokens per second, the remaining 127 tokens take about 3.18 seconds.
A fast first token and a fast full answer are related but not the same target.
The word about matters. Timing changes with model size, active requests, prompt length, cache layout, hardware, and sampling settings. The point is to separate the components before trying to improve them.
For our invented example, the time from the start of prefill to the final token is seconds. Network and queue time would be added only if they are outside those measured intervals.
A small calculation makes the off-by-one rule explicit. The first token is already included in TTFT, so only output_tokens - 1 gaps remain.
output_tokens = 128
first_token_seconds = 0.85
output_rate = 40
remaining_seconds = (output_tokens - 1) / output_rate
total_seconds = first_token_seconds + remaining_secondsThe calculation assumes a constant rate and at least one output token. It is a model of timing, not a benchmark of an actual GPU. To diagnose a real service, record the request start, first token, later token timestamps, prompt length, output length, and active load.
Why Batching Is a Careful Trade-Off
Servers can prefill several prompts together. This keeps the GPU busier, but a request may wait until the batch is ready. Decode batching also helps total tokens produced per second, though every active sequence needs cache memory and has a different next-token step.
More requests can share a GPU step while each request still has its own cache history.
An inference engine therefore schedules prefill and decode work together with care. It cannot optimize only the average without considering slow first tokens and memory limits.
Continuous batching means an engine can add or remove requests between processing steps instead of waiting for a whole fixed batch to finish. Chunked prefill means it divides a large prompt into smaller pieces of work. Scheduling those pieces between decode work can reduce interruptions to existing streams.
Neither technique eliminates work. Smaller chunks may add overhead, and more active requests use more cache. The right policy depends on whether the service values quick first tokens, smooth ongoing streams, high total throughput, or some measured combination.
Prefill often has enough known positions to use large matrix operations well. A small-batch decode step often spends much of its time moving weights and cache data. These are common patterns, not universal laws: sequence length, batch size, kernel choice, and hardware can change the bottleneck.
The Same Cache Has Two Jobs
Prefill writes a large initial KV cache. Decode reads that cache repeatedly and appends one position at a time. A longer prompt makes prefill heavier and starts decode with a larger cache. A longer answer leaves prefill unchanged but makes decode run for more steps.
Prompt tokens mostly affect prefill; generated tokens mostly affect the number of decode steps.
This distinction helps when someone says an LLM is slow. Ask first: is the first token late, are later tokens slow, or is the server waiting for capacity?
What We Optimize Next
The large attention calculation during prefill often moves so much data between GPU memory and fast on-chip memory that memory traffic becomes the limit. Flash Attention changes how this calculation is organized without changing the attention result.
Recap
This is how an inference request moves through prefill and decode. Prefill processes the prompt that is already known and builds the KV cache. Decode uses that cache to choose and append one token at a time.
TTFT describes the wait for the first visible token. Tokens per second describes the pace after streaming starts. Day 24 explains why attention can be limited by moving data, even when a GPU can perform a great deal of math.
