How does an LLM generate a complete response one token at a time?

How does an LLM generate a complete response one token at a time?

A typical text-generating LLM builds a response autoregressively: each new token is conditioned on the prompt and the tokens already generated. A token can be a word, part of a word, punctuation, or another vocabulary item, so one generation step is not necessarily one complete word.

Start with the prompt

The model processes the input sequence and computes scores for the next token. A decoding strategy selects a candidate. That selected token becomes the first piece of the response and is appended to the context used for subsequent predictions.

Imagine the intended output is “A kettle heats water.” A toy tokenizer might produce separate pieces for “A,” “ kettle,” “ heats,” “ water,” and “.”. Another tokenizer could divide the same text differently. The model’s loop follows token boundaries rather than human word-count rules.

Reuse context as generation continues

Each step uses the growing sequence to compute the next continuation. Implementations often cache attention keys and values from earlier positions so they do not need to repeat every previous calculation. Caching improves efficiency without changing the basic causal relationship between earlier output and later predictions.

Because each selected token influences later steps, an early mistake can shape the rest of an answer. The process does not automatically verify the entire response before displaying it.

Stop and display the result

Generation ends when a stopping rule applies, such as an end marker, an output-length limit, or an application-defined condition. The tokenizer decodes generated IDs into text, which the interface may stream incrementally. More advanced inference methods can accelerate this workflow, but the standard explanation remains a sequence of conditional token predictions.