Stop Dividing by 4: The Lazy Token Rule That Hides Almost Half Your JSON Bill

Where the 4-character rule comes from

Almost every token guide repeats the same line: one token is about four characters, or about three quarters of a word. It started as a rule of thumb for ordinary English, and people now use it for everything: prompt budgets, context-window checks, cost estimates for agents that stuff tool output into every call.

We wanted to know how far off it is, so we counted. On November 2, 2026 we ran 951 text samples through o200k_base, the tokenizer OpenAI publishes in its open tiktoken library (version 0.14.0). The samples: 416 paragraphs from this blog, 354 Python documentation strings, 53 slices of Python source, 40 CSV tables, 20 JSON files, 30 blocks of API keys, UUIDs and hashes, 15 short messages with emoji, 11 Chinese paragraphs, and four sentences each in Japanese, Korean and Russian.

How many characters is a token? The measured table

Why JSON, CSV and logs break the rule

Tokenizers are trained mostly on prose, so common English words often fit in one token, spaces included. Numbers do not. A price like 4821.37 or a date like 2026-07-14 splits into several short pieces, and every quote, colon, comma and bracket costs something too. Random strings are the worst case: an API key or a SHA hash has no common chunks at all, which is why that row sits at 1.5 characters per token.

The spread inside each type matters as much as the average. Our JSON files ranged from 1.7 to 2.8 characters per token depending on how many numbers they held, and CSV from 1.6 to 3.9. A table of prices behaves very differently from a table of product names, so one fixed divisor can be off even within the same file type.

Format choice adds a second multiplier on top. In our input vs output tokens guide, the same 1,000-row spreadsheet took 35,290 tokens as CSV and 92,289 as pretty-printed JSON, 2.6 times as many, because JSON repeats every column name on every row. If your agent passes records around as indented JSON, that is the first place to look for savings.

What the mistake costs: one agent, two budgets

Take an agent that reads tool results as JSON: 40,000 characters of JSON per call, 10,000 calls a month. Dividing by 4 says 10,000 tokens per call, or 100 million input tokens a month. At the 2.11 characters per token we measured for JSON, the real figure is about 18,970 tokens per call, or 189.66 million a month.

On Claude Sonnet 5.5 input pricing that is the difference between a planned $200.00 and an actual $379.32 a month, before a single output token. Nothing broke and nobody made a coding error; the estimate was just built on a ratio meant for prose. You can plug the corrected token count into the API cost calculator and see your own version of this gap in under a minute.

The same error shows up in context limits. A JSON payload that the 4-character rule puts at 100,000 tokens is closer to 190,000 in reality, which no longer fits a 128K context window. The first sign is usually a truncation error or an agent that quietly loses the start of its instructions.

How many tokens is 1,000 words?

The words-per-token rule holds up better than the characters rule for English, because it already ignores spaces and punctuation. It is still 8% high for everyday prose and 13% low for technical writing full of function names and version numbers. For Chinese and Japanese, which have no spaces between words, count characters and use about 0.73 tokens each.

Want page counts or the dollar side too? The words-to-tokens quick reference converts common word counts and page lengths, and shows what a 1,000-word prompt costs on three models at today's list prices.

A better rule of thumb you can copy

• Plain English (emails, articles, chat): divide characters by 5.
• Technical English and source code: divide by 4.2.
• JSON, CSV, logs and anything number-heavy: divide by 2.
• API keys, hashes, base64 and UUIDs: divide by 1.5.
• Chinese or Japanese: multiply characters by 0.73.
• Mixed text: count each part with its own divisor and add them up.

If you would rather not do this by hand, our Token Calculator now does exactly this: it weights words, digits, punctuation and Chinese characters separately, using rates fitted to the same o200k_base counts. It lands within about 2% of the real count on everyday English, technical English, code, CSV, JSON and Chinese, where a flat divide-by-4 misses JSON and CSV by close to half. It is free, needs no signup, and the text you paste never leaves your browser, so you can try it on a real production payload.

What this test does not cover

• Only one tokenizer. Claude, Gemini, Llama and DeepSeek use their own, and the same text can come out at a different count. Before you lock a budget on those models, run real samples through the provider's own token counting.
• The CSV and JSON files are generated order and metrics tables, not your data. Use the per-type range (1.6 to 3.9 for CSV) as a warning, not a promise.
• The Chinese set is 11 hand-written paragraphs, and Japanese, Korean and Russian are four sentences each. Treat those rows as directional.
• The technical docs are Python standard-library docstrings, which are heavier on code names than most product documentation.

FAQ