Why is next-token pre-training called self-supervised learning?
The training targets are derived from the data itself: the next token is already present in each sequence. People need not label every prediction manually.
Explore This Question
Search for additional explanations using these tools:
More FAQs in What happens during the pre-training of a large language model?
No. Predicting varied text can require learning factual associations, discourse patterns, code structure, and useful representations, although those capabilities remain imperfect and uneven.
No. Models can differ in data mixture, architecture, objectives, modalities, and training procedure. Next-token prediction is common for autoregressive models, not a universal requirement.
Still have questions?
Browse all our FAQs or reach out to our support team