How does an LLM assign probabilities to possible next tokens?

How does an LLM assign probabilities to possible next tokens?

An LLM assigns next-token probabilities by turning its internal representation of the current sequence into vocabulary scores and normalizing those scores. The probabilities depend on the context and the model’s learned parameters. They describe possible continuations, not a direct probability that an answer is factually correct.

From hidden representation to logits

After processing the prompt, the network has a hidden representation relevant to the next prediction. An output layer maps that representation to one score per vocabulary item. These raw scores are called logits; they can be positive or negative and do not need to sum to one.

Imagine a toy vocabulary with three possible continuations and logits of 2, 1, and 0. The largest logit marks the most strongly favored candidate before any additional decoding adjustments.

Softmax creates a distribution

Softmax exponentiates the logits and divides each result by the sum across the vocabulary. For the toy scores above, the probabilities are approximately 0.665, 0.245, and 0.090. They are nonnegative and sum to one, apart from numerical rounding.

Actual models score much larger vocabularies. Temperature and filtering settings may modify the distribution used for sampling, so raw model probabilities and final selection probabilities need not be identical.

Interpret probabilities carefully

A token can be likely because it fits a familiar phrase, even if the phrase is false in the current situation. Similarly, a correct answer may have several equally valid phrasings, spreading probability across alternatives. Evaluating factual accuracy requires checking the completed claim against reliable evidence, rather than reading a single token probability as truth confidence.