Can a tokenizer show which text produced each token?
Many tokenizer libraries provide offset mappings that connect tokens to original character spans. The precise behavior depends on normalization, special tokens, and the library being used.
Explore This Question
Search for additional explanations using these tools:
More FAQs in How does a tokenizer divide text into smaller units?
Some vocabularies include whitespace as part of a token or encode word boundaries explicitly. A leading space can therefore change the units selected for otherwise identical letters.
The tokenizer typically applies an already learned vocabulary and merge rules during inference. Learning those rules is a separate preparation step from generating an answer.
Still have questions?
Browse all our FAQs or reach out to our support team