LLM Token Estimator

Tokens are not exactly words or characters — they're subword units created by tokenization algorithms like BPE (Byte Pair Encoding). On average, 1 token ≈ 4 characters or 0.75 words in English. Code and special characters can tokenize differently: a line of Python might use more tokens per word than plain prose. This estimator uses heuristics that approximate within ~10% of actual counts.

Each model uses its own tokenizer vocabulary. GPT-4o uses cl100k_base (100k vocab), Claude uses Anthropic's tokenizer, Llama uses a SentencePiece tokenizer, and Mistral uses its own variant. The vocabularies differ in how they split words, handle whitespace, and encode code symbols — leading to different token counts for identical text, usually within 5-15% of each other.

A context window is the maximum number of tokens a model can process at once — both input (your prompt + conversation history) and output combined. GPT-4o supports 128K tokens, Claude 3.5 Sonnet supports 200K, and Gemini 1.5 Pro supports up to 1M. Exceeding the context window means the model can't see earlier parts of your conversation. This is critical for long document analysis or multi-turn chats.

Common strategies: (1) Use shorter, more direct prompts — remove filler words. (2) Truncate or summarize long documents before sending. (3) Use system prompts efficiently — avoid repeating instructions. (4) For code review, send only the changed diff, not the full file. (5) Enable streaming to stop generation early if you have enough output. (6) Use smaller models (Haiku, GPT-4o-mini) for simple tasks.

LLM providers charge separately for input tokens (your prompt) and output tokens (the model's response). Output tokens typically cost 3-5x more than input tokens because generation is computationally expensive. For example, Claude 3.5 Sonnet charges $3/M input but $15/M output. If you're building a chatbot, minimizing output length (concise responses, structured data instead of prose) significantly reduces costs.