Token Counter

What is a token?

A token is a chunk of text the model processes as a single unit. It is not a character and not a word. In English, a typical token covers 3 to 4 characters, but the exact split depends on the tokenizer the model was trained with. Punctuation, whitespace, code symbols, and non-Latin scripts all push the count up or down. Tokens are the unit every API meters, so they drive both cost and context-window pressure on every call.

Here is the worked example we use to onboard new teammates. Run the string Hello, world! through OpenAI's o200k_base encoding (the tokenizer the GPT-5 family uses) and you get exactly 4 tokens: Hello, ,, world (with a leading space), and !. The same 13 characters land near the same count on Claude and DeepSeek, but with different boundaries because each provider trains its own vocabulary. This is why two models billed at the same per-token rate can produce different bills for the identical prompt. The tokenizer is the meter, and every meter is calibrated differently.

The character-to-token ratio also varies dramatically by content type. Plain English prose sits near the 4 chars-per-token average. Code is denser (more punctuation, more rare identifiers), so Python or TypeScript often runs closer to 3 chars per token. Chinese, Japanese, and Korean text is denser still: roughly 1 token per character on every major tokenizer, which makes CJK content 3-4x more expensive to process than the equivalent English. Knowing the ratio for the workload we plan to ship, not the marketing-blurb average, is the difference between a cost estimate that holds and one that ships a surprise invoice.

How token counting works

Every major LLM tokenizer is a flavor of byte-pair encoding (BPE). Training scans a huge text corpus, finds the most common adjacent character pairs, merges them into a single new token, and repeats the merge step tens of thousands of times. The result is a vocabulary where common subwords (like the or tion) are single tokens, while rare strings get split into smaller pieces. OpenAI uses o200k_base for the GPT-5 family and the GPT-4.1 line. Anthropic uses its own BPE variant for Claude. Google and Mistral use SentencePiece. DeepSeek and Meta use custom variants. Same algorithm class, different vocabularies, different counts on the same input.

Here is how that plays out in practice. Take a 1,000-character technical documentation snippet. On OpenAI it tokenizes to roughly 250 tokens (about 4.0 chars per token). On Google's Gemini family it lands around 285 tokens (about 3.5 chars per token). On DeepSeek it is closer to 278. We use exact js-tiktoken for the OpenAI count and per-provider char-per-token ratios from TOKENIZER_RATIOS for the rest. Why the split? OpenAI ships its tokenizer as an open library, so we run it in the browser and return the same number the API will bill. The other providers either gate the tokenizer behind an authenticated endpoint (Anthropic, Google) or do not publish a portable implementation, so we estimate. The tradeoff is real: estimates land within +/-5-15% of the provider's billable count, which is fine for capacity planning and budget math. For invoiced cost we always reconcile against the provider's own counter at the end of the month.

Common pitfalls

Most token-counting mistakes do not come from the math. They come from comparing numbers that were never apples-to-apples in the first place, or from copying assumptions across model generations that no longer hold. In our experience auditing production cost models, roughly 15-20% of cost projections we review contain at least one of the four errors below.

• Comparing OpenAI exact counts against Anthropic estimates as if they were the same metric. OpenAI counts come from tiktoken and match the bill exactly. Anthropic counts (in our tool and in most third-party tools) are estimates from a char-per-token ratio. The variance can swing cost projections by 10-15% on the same workload, which is the difference between a feature being profitable and being underwater. We label estimated counts with ~ est. in the UI for exactly this reason.
• Assuming output tokens equal input tokens. For prose tasks the output is typically 2-5x the input. For reasoning models (o3, o4-mini, DeepSeek R1) it can be 10x or more, because the hidden chain-of-thought tokens count toward billing on most providers. We always model input and output separately and we always check the provider's docs on whether reasoning tokens are billed at the input or output rate.
• Ignoring tokenizer changes within a single provider. GPT-4o and the GPT-5 family use o200k_base. Older GPT-4 used cl100k_base. The same prompt produces different counts under each, so an estimate copied from a 2023 cost spreadsheet onto GPT-5.5 today is the wrong number. Re-count when you migrate models.
• Forgetting that markdown formatting tokenizes more aggressively than plain text. Asterisks, backticks, headers, and table pipes each add tokens that prose alone would not. On heavily formatted RAG documents we have seen 15-25% inflation versus a plain-text equivalent. Stripping markdown before retrieval recovers most of that headroom.

When to use this tool

We built the counter for three concrete workflows we kept hitting ourselves. The first is an indie dev pricing a SaaS feature that calls Claude Sonnet 4.6 ($3/$15 per 1M tokens) 1,000 times per day. They paste an average prompt into the counter, see roughly 800 input tokens, plan for 600 output tokens, and confirm the per-call cost lands at about $0.0114 before deciding the unit economics work at their planned price point. The second is a prompt engineer iterating on a system prompt and wanting cost-per-iteration before deploying. They cut three versions of the same prompt, run each through the counter, and pick the one that hits the same accuracy at half the token count. Token math turns prompt engineering into a measurable optimization rather than a vibes exercise. The third is a team estimating context-window pressure for a long-document RAG pipeline. They count typical retrieved chunks plus the system prompt plus the user message, then confirm they have headroom under the 1M-token Gemini 3.1 Pro context before scaling ingestion. We use the same workflow ourselves whenever we add a new model row to the data, which is why the per-token color highlighting preview ships in the tool: the preview surfaces the surprise expansions (CJK text, dense markdown, base64 blobs) before they hit production.

Frequently asked

How accurate is this token counter?

Counts for OpenAI models are exact (computed via js-tiktoken with the official BPE tokenizer). Counts for other providers (marked ~ est.) are estimated from each provider's tokenizer ratios and vary ±5-15% based on content type. For exact non-OpenAI counts, use Anthropic's countTokens API or Google's Gemini countTokens endpoint.

Do input and output tokens cost the same?

No. Output tokens are typically 3-8x more expensive than input tokens across all major providers. This is why controlling response length is one of the most effective ways to reduce API costs.

What's the cheapest way to use LLM APIs?

Use prompt caching (saves up to 90% on repeated inputs), batch processing (50% discount on most providers), and route simple tasks to budget models like GPT-5 Nano or Gemini 2.0 Flash-Lite instead of flagship models.

Why does my prompt count differently across models?

Each provider trains its own tokenizer, so the same string fragments into different chunks. OpenAI's GPT-5 family uses o200k_base BPE; Claude uses Anthropic's BPE variant; Google and Mistral use SentencePiece; DeepSeek and Meta use custom variants. The same 1,000-character technical doc lands at roughly 250 tokens on OpenAI (4.0 chars/token) but 285 on DeepSeek (3.5 chars/token). We always price against the specific model you plan to ship on.

Can I count tokens for any model?

Yes, with two tiers of accuracy. For OpenAI we run js-tiktoken locally and return exact counts. For Anthropic, Google, xAI, DeepSeek, Mistral, and Meta we estimate from per-provider char-per-token ratios stored in TOKENIZER_RATIOS, accurate to roughly +/-5-15%. For exact non-OpenAI counts we recommend Anthropic's countTokens API or Google's Gemini countTokens endpoint, which return billable token counts directly from the provider.

How do I reduce my token count?

Four moves we use in production. Trim repeated context (system prompt, RAG snippets) and route shared instructions through prompt caching at 10% of input price. Switch verbose prose outputs to compact JSON or YAML. Strip stopwords and boilerplate from RAG retrievals before they hit the prompt. Cap response length with max_tokens. On a 4,000-token system prompt reused 1M times, caching alone saves roughly 90% of input spend.