What are parameters in a large language model?

What are parameters in a large language model?

Parameters are numerical values inside a large language model that influence how inputs become predictions. Training adjusts these values. They include the entries in embedding tables and the weight matrices used throughout attention and feed-forward layers, along with other learned quantities.

A model contains many adjustable numbers

A simple calculation such as output = weight × input + bias contains two parameters: a weight and a bias. A transformer combines many much larger calculations involving vectors and matrices.

For example, a matrix with 1,000 rows and 1,000 columns contains one million entries. If every entry is learned, that matrix contributes one million parameters. Real model parameter counts combine many components.

Knowledge is distributed across parameters

A parameter is not a labeled fact such as “Paris is the capital of France.” Useful associations and computational patterns are distributed across many values and their interactions.

Changing one number therefore does not usually edit one cleanly isolated fact. The same parameter can affect many inputs, while a particular behavior may depend on many layers.

Parameter count is only one comparison

More parameters can provide additional modeling capacity, but capability also depends on data, training methods, architecture, and computation. Some architectures activate only part of their parameter collection for each token.

Parameters also differ from hyperparameters, such as a learning rate, and from input tokens supplied during a conversation. Remembering those distinctions makes model specifications easier to interpret without treating one large number as a complete quality score.