The answer
Some vocabularies include whitespace as part of a token or encode word boundaries explicitly. A leading space can therefore change the units selected for otherwise identical letters.
How does a tokenizer divide text into smaller units?
Why does adding a space sometimes change a token?
More FAQs in How does a tokenizer divide text into smaller units?
The tokenizer typically applies an already learned vocabulary and merge rules during inference. Learning those rules is a separate preparation step from generating an answer.
Many tokenizer libraries provide offset mappings that connect tokens to original character spans. The precise behavior depends on normalization, special tokens, and the library being used.
Still have questions?
Browse all our FAQs or reach out to our support team
Want to upskill yourself?
Your learning library
Small steps. Big possibilities.
Find your next course. Build something great.