Tokenizer Counter

Tokens are sub-word units used by LLM tokenizers. In English, 1 token ≈ 0.75 words, or ~4 characters. Common words are often single tokens ('hello' = 1 token). Rare or long words may split into multiple tokens ('unbelievable' = 3 tokens). Numbers, punctuation, and code have different tokenization patterns — a string like '2024-03-27' may be 4–5 tokens.

GPT-4o uses o200k_base, OpenAI's latest tokenizer with a 200K vocabulary (vs 100K for cl100k_base used by GPT-4). The larger vocabulary means fewer tokens for most text. You can test exact counts with OpenAI's tiktoken library: `import tiktoken; enc = tiktoken.get_encoding('o200k_base'); len(enc.encode(text))`.

Anthropic uses a custom BPE tokenizer. Claude's token counts are generally similar to GPT-4's cl100k_base but not identical. For exact Claude token counts, use the Anthropic API's token counting endpoint: `client.messages.count_tokens(model='claude-3-5-sonnet-20241022', messages=[...])`. This tool provides estimates accurate to within ~5-10%.

Gemini uses SentencePiece tokenization, which produces slightly different token counts than BPE-based models. For exact Gemini token counts, use the Google Generative AI library: `model = genai.GenerativeModel('gemini-1.5-pro'); model.count_tokens('your text')`. SentencePiece tends to be slightly more efficient (fewer tokens) for many languages.

Strategies: (1) Remove redundant whitespace and formatting. (2) Use concise instructions — 'Summarize:' vs 'Please provide a comprehensive summary of the following text:'. (3) Avoid repeating context — use references instead of re-pasting text. (4) Use few-shot examples only when necessary. (5) Compress system prompts — they count toward input tokens on every call. (6) For structured data, JSON is more token-efficient than XML.

For very long documents (100K+ tokens), Gemini 1.5 Flash is often most cost-effective at $0.075/M input tokens with a 1M token context. For shorter contexts where quality matters, GPT-4o-mini ($0.15/M) and Claude 3.5 Haiku ($0.80/M) are strong choices. Always compare actual token counts × price per million rather than assuming one model is always cheaper.