What is a transformer architecture?

What is a transformer architecture?

A transformer is a neural-network architecture that processes representations using attention and other learned transformations. It can model relationships across a sequence without requiring the same recurrent step-by-step state update used by traditional recurrent networks. Transformers are used in language tasks and in architectures for other data types.

Attention is one part of the block

Self-attention allows a position to combine information from other permitted positions. Feed-forward layers then transform representations further. Residual connections help carry information across layers, while normalization helps control the numerical behavior of those computations.

Position information is also necessary because the order of inputs matters. “The child followed the dog” should not be treated as equivalent to “The dog followed the child.” Different transformer designs represent position in different ways.

Encoder and decoder roles differ

An encoder can build representations using context from across an input sequence. A causal decoder restricts attention so a position cannot use later positions. An encoder-decoder system combines an input-processing component with an output-generating component.

For translation, an encoder can represent a source sentence while a decoder generates the target sentence. For a decoder-only language model, the prompt and generated continuation are handled within a causal sequence.

Parallel training does not mean simultaneous generation

During training, known sequences allow many position-wise computations to be performed in parallel while attention masks preserve the intended information restrictions. Standard autoregressive inference still generates a continuation through dependent token choices.

The architecture provides a useful computational structure, but performance also depends on training data, optimization, scale, and later adaptation. Calling a model a transformer does not by itself describe how well it will handle a particular task.