Does pre-training teach only grammar?
No. Predicting varied text can require learning factual associations, discourse patterns, code structure, and useful representations, although those capabilities remain imperfect and uneven.
Explore This Question
Search for additional explanations using these tools:
More FAQs in What happens during the pre-training of a large language model?
The training targets are derived from the data itself: the next token is already present in each sequence. People need not label every prediction manually.
No. Models can differ in data mixture, architecture, objectives, modalities, and training procedure. Next-token prediction is common for autoregressive models, not a universal requirement.
Still have questions?
Browse all our FAQs or reach out to our support team