Does byte-pair encoding merge words while a model is answering?
The tokenizer typically applies an already learned vocabulary and merge rules during inference. Learning those rules is a separate preparation step from generating an answer.
Explore This Question
Search for additional explanations using these tools:
More FAQs in How does a tokenizer divide text into smaller units?
Some vocabularies include whitespace as part of a token or encode word boundaries explicitly. A leading space can therefore change the units selected for otherwise identical letters.
Many tokenizer libraries provide offset mappings that connect tokens to original character spans. The precise behavior depends on normalization, special tokens, and the library being used.
Still have questions?
Browse all our FAQs or reach out to our support team