Reference · Glossary
Tokenizer
Last updated
The piece of an LLM stack that **splits text into tokens** — the units models read, price, and count toward context limits.
When to use
You care about **cost estimates**, context fit, or why `"hello"` and `"Hello!"` tokenize differently.
When not to
End-user chat UX — users think in words, not tokens (unless you're building billing UI).
Example
"OpenAI" might be one token; a rare URL might become many small tokens — always measure with the model's tokenizer for production limits.