Don't Trust Your Token Estimator: We Tested 3 on 1,783 Texts, and Ours Missed Real JSON by 11%

We tested our own estimator and it failed one test

This week we published a 26-line token estimator and said it lands within about 2% of the real count. On the 951 samples we fitted it to, that is true. Then we did what we tell everyone else to do and measured how many characters a token really is on text it had never seen. It missed real package.json files by 11.5%.

That is the problem with every token estimate: it looks perfect on the text its author checked, and nobody checks the rest until a bill or a context-length error shows up. So we ran the same test on the three shortcuts people actually ship in JavaScript, and on two real tokenizers, to see where each one breaks.

• Characters divided by 4, the rule in almost every token guide.
• tokenx 2.2.0, a popular 2 kB estimator library on npm.
• estimate-tokens.js, our own 26-line estimator.
• gpt-tokenizer 4.1.0 and js-tiktoken 1.0.21, real o200k_base tokenizers, as the answer key.

Every count is OpenAI's o200k_base tokenizer, used by GPT-4o and newer OpenAI models. Both JavaScript tokenizers returned exactly the same counts as OpenAI's Python tiktoken on all 951 samples, 250,446 tokens in total, so we use them as the real count.

Round one: 951 samples we already knew

Each number below is the estimate divided by the real count, summed over every sample of that type. 1.00 is exact. 0.50 means the estimate saw half the tokens you will be billed for.

Read the last column with suspicion. Our estimator's weights were fitted on these exact samples, so it is grading its own homework. The useful numbers in this round are the other two. Dividing by 4 overcounts English and misses about half of everything with digits, punctuation or Chinese characters. tokenx is close on ordinary text but sees only 38% of the tokens in random strings and 43% in emoji-heavy messages.

Round two: 832 files none of them had seen

For a fair test we needed text that no estimator was tuned on. We took the first 4,000 characters of real files from the npm packages this website depends on: 325 README files, 338 package.json files and 169 index.js files, every one longer than 300 characters.

Two more things this round showed. Dividing by 4 undercounted even the Markdown READMEs, because they are full of links, code and badges, not plain prose. And the gap between round one and round two is the real lesson: an estimator is only as good as the text it was checked on. Ask yourself whether your text looks like that text.

What an 11.5% miss costs

Take an agent that sends 100 million input tokens a month of JSON that looks like these files: configs, API responses with URLs and version strings, nested objects. Our estimator would have planned for 88.5 million. Characters divided by 4 would have planned for 71.1 million.

On Claude Sonnet 5.5 input pricing, the tokens our estimator missed are $23.00 a month you did not budget for, and the divide-by-4 gap is $57.80. The money is the visible part. The quiet part is the context window: a payload you think is 110,000 tokens can be 124,000 in reality, and the request that fit yesterday gets cut off today. You can put your corrected token counts into the API cost calculator and see the monthly gap on your own model in under a minute.

Why not just run the real tokenizer?

You can, and on a server you should. The catch is size. We bundled each option for the browser with esbuild, minified, and gzipped it:

About a megabyte is the vocabulary itself, 200,000 entries, so no exact tokenizer gets much smaller. Speed was not the problem: in Node every option got through all 815,129 characters of our samples in well under a second, and the estimators in a few hundredths of a second. For a live counter in a text box, a megabyte download is the real cost. For a backend that decides whether a request fits, a megabyte is nothing.

Which one to use

• On a server, in a worker, or anywhere a wrong count breaks a request or a budget: use a real tokenizer such as gpt-tokenizer, or the usage numbers your provider returns.
• For a live counter in the browser on mostly English, code or Markdown: either small estimator is fine. Show the number as approximate.
• If your text carries API keys, hashes, UUIDs, base64 or emoji: do not use tokenx alone; it saw about 40% of those tokens.
• If your text is real-world JSON such as configs and API responses: tokenx did better than our estimator in our test. Leave at least 15% headroom either way.
• Never use characters divided by 4 for JSON, CSV or Chinese. It missed 29% to 65% of the tokens on every one of them.

Want to check your own data instead of trusting ours? Every sample, the comparison script and our estimator are free in the tokenizer-ratios repository on GitHub. Run npm install and npm run compare, swap in a folder of your own payloads, and you will know in a minute which estimator lies about your text.

Or skip the setup: paste a real payload into the free Token Calculator. It runs our estimator in your browser, nothing is uploaded, and it prices the result on the model you pick. Just remember round two before you trust it with JSON.

What this test does not cover

• Only OpenAI's o200k_base tokenizer. Claude, Gemini, Llama and DeepSeek tokenize differently, and every estimator here was built or checked against OpenAI's counts.
• The unseen files all come from one project's npm dependencies, so they are English, open-source and code-heavy. Business documents, chat logs and non-English text could rank the estimators differently.
• We kept the first 4,000 characters of each file. Very long files were not tested whole.
• Bundle sizes are esbuild output gzipped at level 9 on November 3, 2026. Speeds are from one Windows PC and changed noticeably between runs, so we only report the order of magnitude.

FAQ