How does an LLM predict the next token in a sequence?

How does an LLM predict the next token in a sequence?

An autoregressive language model predicts a next token by processing the tokens already available in its context. A token is a unit from the model's vocabulary, which may represent a word, part of a word, punctuation, or another text fragment.

The model predicts one continuation position at a time during ordinary generation.

Represent the existing context numerically

A tokenizer converts text into token IDs. The model maps those IDs to learned numerical representations and processes them through its network.

In a transformer, attention and other transformations combine contextual information. The representation used for the next-token prediction therefore depends on more than the final visible word.

For example, "The bank approved the" and "The boat reached the bank of the" share a word but create very different contexts.

Score possible continuations

The model produces scores, commonly called logits, over its output vocabulary. A transformation such as softmax converts those scores into a probability distribution.

For an illustrative context like "Please close the", several continuations could make sense: a door, window, or application. The actual token options and probabilities depend on the model, tokenizer, and full context.

These probabilities describe potential text continuations. They are not automatically probabilities that a completed factual statement is true.

Select, append, and repeat

A decoding method selects a token from the distribution or chooses a highest-scoring option. That token becomes part of the context for the next step.

Repeating the process produces sentences and longer responses until a stopping condition is reached. The computation is based on learned parameters and current context; it does not require searching a database for a complete sentence at every step.