How tokenizers learn subwords

Before a model is trained, its builders choose a tokenization method and train that tokenizer on a text corpus. The result includes a vocabulary that maps token IDs to text pieces and rules for turning new text into those pieces.

Two extreme designs show the tradeoff:

• A character vocabulary can spell nearly anything with a modest set of units, but a paragraph requires many model steps.
• A whole-word vocabulary makes common prose compact, but it needs an enormous list and still cannot cover every inflection, typo, identifier, or language.

Subword tokenization lies between them. It can represent walk, walking, and walked using reusable chunks rather than reserving an unrelated entry for every form. It can also break an unseen word into smaller known units. The exact units do not need to match school grammar. A statistically useful piece may be a stem, a suffix, a space plus a word, or just a recurring character sequence.

Vocabulary size is a design choice with consequences. A larger vocabulary can shorten some sequences, but it also means more token embeddings and does not guarantee fair efficiency across domains or languages. The training corpus matters as much as the number of entries.

One family of methods starts with small units and repeatedly combines frequent neighboring pairs. In a toy corpus containing low, lower, and lowest, the pair l + o may merge into lo, then lo + w into low. If er appears often, it may also become a reusable token. The final vocabulary might encode lower as low + er.

This describes the intuition behind byte-pair-style methods, not every production implementation. Other methods score candidate subwords probabilistically or begin with a broad vocabulary and remove pieces that contribute least. Some tokenizers operate from Unicode text; others use bytes as the reliable fallback layer. Normalization rules may also transform text before segmentation.

In every case, corpus frequency shapes what becomes cheap to express. Common phrases, programming symbols, or language fragments represented heavily in the corpus are more likely to receive convenient pieces. Rare combinations must be assembled from smaller ones.

An open-vocabulary system can encode a new string without requiring the complete string to exist in its vocabulary. Suppose the model encounters microorchard, a newly invented product name. A tokenizer might represent it using pieces resembling micro + or + chard, or with smaller byte-level units. Those pieces may not align with the intended meaning, but they preserve the text.

This is important for:

• names and place names;
• technical terminology;
• misspellings and slang;
• serial numbers and hashes;
• newly released products;
• languages with productive word formation.

Tokenization does not itself understand that microorchard is a product. It merely supplies a sequence the model can process. The model must infer meaning from context and learned patterns.

The fallback also prevents an “unknown word” from making input impossible. Older natural-language systems sometimes replaced unseen words with one unknown marker, losing their spelling. Modern subword and byte-level designs can usually retain enough units to reconstruct the original string.

Token boundaries are engineering artifacts, not a dictionary analysis. A tokenizer might keep a frequent misspelling whole while splitting a valid scientific term. It might attach a leading space to an English word because that pattern is common in running text. It may segment two grammatical forms differently because their character frequencies differ.

Do not interpret a token as a concept stored in the model. A single token may participate in many meanings, while one concept may require many tokens. Nor should you assume more tokens always means the model understands a phrase less well. Sequence length can affect efficiency, but understanding also depends on training examples, model capacity, and context.

The corpus can create unevenness. If one language or code style appears much less often, its ordinary text may require longer token sequences. That consumes more context and can increase cost for the same human-visible message. Measuring across the languages and data types your product serves is therefore part of product quality, not merely a technical curiosity.

The model learns parameters using token IDs from one tokenizer vocabulary. ID 241 has meaning only relative to that vocabulary. Sending IDs from an unrelated tokenizer would point to different learned entries and produce nonsense or an error.

This is why model libraries bundle tokenizer configuration with model checkpoints, and why API clients ask for a model name when counting. A newer model family may use a revised vocabulary even if its chat interface looks unchanged. Counts from one model can be a rough comparison, but they are not authoritative for another.

In production, record both the model/version and the count method. If a provider changes an alias to point to a new version, recheck representative prompts. Correct accounting depends on identifying the actual pair.