Spaces, punctuation, and special tokens

Many tokenizers learn pieces that include a boundary marker or leading space. A display might show Hello as one piece at the start of text but hello as a different piece after another word. That design makes common word boundaries compact. It also means copying a visible word into a tokenizer tool without its surrounding text can produce a different count.

Punctuation is not free. Commas, periods, brackets, Markdown markers, and repeated dashes may each be tokens or combine with nearby text. This matters when prompts contain large tables, deeply nested JSON, or decorative separators. Formatting can improve structure and accuracy, so the lesson is not to remove all punctuation. It is to recognize that structure occupies budget.

Whitespace can matter too. Indentation in source code carries syntax in languages such as Python. In other code, it supports readability. Tokenizers may encode runs of spaces efficiently or break them into several pieces, depending on their vocabulary. Minifying everything can save tokens but make examples harder for both people and models to follow. Measure the tradeoff with the exact material.

Some vocabulary entries do not represent ordinary user-visible text. Special tokens can mark the beginning or end of a sequence, separate messages, identify roles, represent padding, or delimit tool calls. Their names and behavior vary by model.

A chat request that looks like:

User: Add 2 and 3.
Assistant:

may be converted internally into a structured sequence with role markers and boundaries. Those wrapper tokens can count toward usage even though the chat interface does not show them. Tool definitions and structured response schemas may add more hidden or semi-hidden content.

Do not manually type a string that resembles a special token and assume it gains control powers. APIs usually apply chat templates and escaping rules. The authoritative sequence comes from the provider’s formatter or model library. Conversely, when running an open model yourself, applying the wrong chat template can seriously damage results because the model was trained to expect particular control markers.

Numbers do not necessarily map one number to one token. 7, 2026, 3.14159, and INV-004812 may split into digits or groups of digits. A long UUID, cryptographic hash, or random base64 string often lacks reusable linguistic patterns, so it can consume many tokens relative to its visible length.

Code has its own frequency patterns. Common keywords and operators may tokenize compactly, while rare library names, generated variable names, escaped strings, and long paths may fragment. These snippets are visually similar in size but can differ in count:

for item in items:
    total += item.price

customer_reconciliation_result_2026_07_18 = lookup["a8f3..."]

When asking a model to inspect logs, consider whether every timestamp, request ID, and stack-frame repetition is necessary. Preserve identifiers needed to correlate events, but remove unrelated high-entropy noise. Never shorten an identifier if exact matching is the task.

Languages differ in writing systems, word boundaries, and representation in tokenizer training data. Chinese and Japanese do not use spaces between every word. Arabic combines letters in ways that look different by position. Some languages build long words from many meaningful parts. A tokenizer designed from a multilingual corpus can represent all of these, but not necessarily with equal sequence efficiency.

Unicode adds another layer. The same visible accented character may have more than one underlying code-point sequence. Emoji can contain skin-tone modifiers or invisible joiners that combine several symbols into one displayed glyph. A copied nonbreaking space can look like an ordinary space. Depending on normalization and byte fallback, visually similar strings may receive different token sequences.

Avoid claims such as “one token is always four characters” or “one English word equals a fixed number of tokens.” Those shortcuts can mislead multilingual budgeting. Measure representative samples for each language, script, and content type your application serves. Compare not only average counts but also unusually expensive cases.

Token-aware editing asks what each piece contributes. Repeated email footers, duplicated navigation, irrelevant log columns, and copied legal boilerplate may be removable. Headings, delimiters, and schema labels may be worth their cost because they clarify roles and constraints.

For structured prompts, keep a stable compact format. If the model must return JSON, include only schema descriptions it needs, but do not delete field definitions that prevent ambiguity. If code indentation is semantically meaningful, retain it. If a table has twelve columns and the question uses three, select those three rather than flattening the entire table into unreadable text.

A useful test is: “Could removing this text change the correct answer?” If yes, keep it or replace it with an accurate summary. If no, it is a candidate for removal. Token efficiency should never silently alter data.