Tokenizer differences and practical measurement

Model families can use different vocabulary sizes, training corpora, normalization, byte handling, and segmentation algorithms. As a result, the same text can become different token sequences. A vocabulary trained with abundant code may keep common programming patterns compact. Another may split them more finely. A multilingual vocabulary may allocate pieces differently across scripts.

Even closely related model versions can differ. An alias such as “latest” may move to a version with a revised tokenizer or chat template. Open-weight models often publish tokenizer files with their checkpoints, while hosted APIs may expose counting tools or only return usage after a request.

The differences are not merely cosmetic. They affect:

• whether a request fits;
• input and output charges;
• how much history can remain;
• output caps;
• latency and throughput;
• comparisons between languages and content types.

Never reuse one model’s count as an exact guard for another model. It is acceptable for rough planning if clearly labeled and padded, but production enforcement should use the target model’s rules.

The visible prompt may be only part of the input. A chat SDK can add role markers and message boundaries. The application may add system instructions, examples, tool schemas, response schemas, retrieved passages, or safety text. Measure after these components are assembled whenever the API or library permits it.

Create a count record with:

model identifier and version
tokenizer or counting endpoint version
system/developer instruction tokens
history tokens
current user tokens
tool definition/result tokens
retrieval tokens
estimated output allowance
provider-reported input/output usage

If the local count and provider usage differ, retain both and investigate. The billed provider usage is authoritative for cost. The local count may still be useful as a preflight guard once you understand and bound the difference.

Avoid logging sensitive raw prompts merely to count them. Counts, hashes, component labels, and approved samples can support observability without storing private content. Follow the product’s data-handling rules.

A ten-sentence English sample cannot validate a global product. Assemble examples from actual request categories while removing or protecting sensitive data:

• short and long prose;
• every supported language and script;
• source code in common project languages;
• Markdown, JSON, CSV, and tables;
• URLs, email addresses, UUIDs, and product codes;
• emoji and accented characters;
• whitespace-heavy or minified inputs;
• unusually long retrieval and tool results.

For each sample, store character count, any meaningful word count, token count by target model, and component category. The purpose is not to discover a universal conversion. It is to understand your workload’s distribution.

Use percentiles, not only averages. If median input is 1,800 tokens but the 99th percentile is 18,000, a 4,000-token guard will reject an important tail. Inspect those cases: some may contain waste; others may be legitimate long-document tasks requiring a different route.

Suppose 500 support prompts produce these illustrative totals:

Tokenizer A: 1,200,000 tokens
Tokenizer B: 1,080,000 tokens

On this dataset, B uses:

(1,200,000 - 1,080,000) / 1,200,000 = 10% fewer tokens

That statement is limited to the measured dataset. It does not mean B always uses 10% fewer tokens. Break results down by category. B might save 18% on English prose but use 12% more on a language important to your users.

Token count alone also does not choose the better model. Apply each model’s rates, quality, latency, context rules, and operational reliability. A model with fewer tokens but a higher price per token may cost more. A cheaper model that fails the task and retries may cost more per successful outcome.

Use preflight measurement to reject, route, chunk, or compress over-budget requests before paying for a failed call. Return a useful error: identify the oversized component and suggest a safe action. “Context length exceeded” is less helpful than “retrieved evidence uses 22,400 of the 24,000-token input budget; narrow the date range.”

After each call, record provider usage by route and model. Alert on unexpected shifts in tokens per successful task. A sudden rise may indicate duplicated history, larger retrieval results, expanded tool schemas, or a model-version change. Re-run the representative suite when any of those components changes.

Keep safety margin between preflight limits and documented maximums. Counts may differ because of wrappers or conditional content. A request that fits by one token in a development counter is not a robust production design.