Why do LLMs use subwords instead of whole words?
Subwords allow the model to handle typos, prefixes, and suffixes efficiently without expanding the vocabulary index into millions of words.
Verify This Answer
Cross-check this information using these trusted sources:
More FAQs in What is Tokenization and How do LLMs Read Text?
LLM providers charge per token processed, not per word or character. Long prompts cost more because they translate to more tokens.
A word or character the tokenizer doesn't recognize, which is replaced by a special [UNK] token.
Emojis are represented by multiple unicode bytes, which often get split into multiple tokens by the tokenizer.
Still have questions?
Browse all our FAQs or reach out to our support team