Tokens are chunks, not words

A token is one item from a tokenizer’s fixed vocabulary. Some vocabulary items look like complete common words. Others look like word pieces, punctuation, bytes, or a word with its leading space. Each item has an integer ID. The tokenizer turns text into IDs before the model processes it, and turns generated IDs back into readable text afterward.

Suppose a tiny demonstration vocabulary contains these entries:

• 241 = "Please"
• 88 = " summarize"
• 17 = " this"
• 9 = "."

The text “Please summarize this.” could become [241, 88, 17, 9]. The model calculates with those IDs and their learned numeric representations, not with the printed strings themselves. Real vocabularies are much larger, and real splits differ, but the sequence idea is the same.

Tokenization is usually deterministic for a specific tokenizer and exact input. Change capitalization, spacing, or punctuation and the sequence may change. Change models and the associated tokenizer may change too.

“Token” does not mean “word.” A common word may occupy one token because it appeared frequently in training data. An uncommon name may split into several subwords. A punctuation mark may stand alone. A run of spaces may be represented differently from one space. An emoji might be one token in one vocabulary and several byte-like pieces in another.

Consider three illustrative strings:

• predictable might stay whole or split as predict + able;
• unpredictability might split as un + predict + ability;
• invoice_2026.csv might split around the underscore, digits, and period.

These are examples of possible shapes, not promised outputs. You cannot infer an exact count just by looking at the words. Even a familiar-looking string can tokenize unexpectedly if it contains unusual Unicode characters or formatting.

This flexibility is useful. A finite vocabulary cannot contain every product code, surname, typo, URL, or newly coined term. Smaller pieces let the tokenizer represent text it has never seen as a whole.

During generation, a language model repeatedly estimates which token should come next. It selects one token, appends it to the sequence, then predicts again using the expanded sequence. A reply that appears as a smooth paragraph is therefore produced piece by piece.

This explains several visible behaviors. Streaming text arrives in small bursts because tokens are decoded as they are generated. A strict output limit can stop a response midway through a sentence because the limit counts tokens, not completed thoughts. Repeated text takes repeated computation even when a human sees it as redundant. The model also cannot directly “choose the next word” when one word may require several token decisions.

A token is not the same as meaning. The piece bank does not permanently mean a financial institution; surrounding tokens help determine whether the text concerns money or a river. Tokens are input units. Meaning arises from patterns the model learned across sequences of those units.

Token counts connect the text you write to three practical constraints:

1. Capacity: prompts, conversation history, retrieved documents, tool descriptions, and generated output must fit the model’s available context.
2. Cost: many hosted model APIs charge separately for input and output tokens.
3. Latency: more input usually means more work before generation, while more output means more generation steps.

Imagine two support requests that communicate the same facts. Request A includes a 2,000-token email thread with repeated signatures. Request B contains a 350-token summary plus the five relevant messages. B leaves more room for reasoning and a complete answer. It may also cost less and respond faster. Fewer tokens do not automatically mean better results, though: removing the refund amount or policy exception would damage the task.

The goal is useful density, not minimum length.

People often want a conversion between words and tokens. Any such conversion is only a rough planning estimate for a particular kind of text and language. Short English prose, source code, tables, Arabic, Japanese, emoji, and random identifiers can have very different relationships between visible length and token count.

For a real limit or invoice, use the tokenizer for the exact model or read the usage fields returned by the provider. Preserve the exact text, including system instructions and formatting. A word processor’s word count cannot tell you whether an API request fits.

Use rough estimates only for early planning, and include headroom. Replace the estimate with an actual count before production decisions.