Prompt Token Counter

Tokens are chunks of text that LLMs process — typically 3-4 characters per token in English. Words like 'hello' are usually 1 token; longer words like 'programming' may be 2. Whitespace and punctuation are separate tokens. Non-English text, code, and special characters often use more tokens per word. OpenAI uses tiktoken (BPE), Claude uses a similar approach, Gemini uses SentencePiece.

LLM APIs charge per token (input + output tokens). Input tokens (your prompt) are typically cheaper than output tokens (the response). A 1M token context Gemini query at 1M input tokens costs around $1.25, while GPT-4o at the same would cost $2.50. Counting tokens before sending helps predict and control costs.

Input tokens are everything you send: system prompt + conversation history + current user message. Output tokens are what the model generates. For cost estimation, you must count both. Output tokens are usually 2-4x more expensive per token. A typical chat message might be 200 input tokens and generate 400 output tokens.

The context window is the maximum tokens the model can process at once (input + output combined). GPT-4o has 128K tokens, Claude 3.5 Sonnet has 200K, Gemini 1.5 Pro has 1M. If your conversation history + prompt exceeds the window, you must truncate history. At $0.002/1K tokens, a full 128K context costs $0.26 per call.

This tool uses an approximation: ~4 characters per token for English text, with adjustments for code, numbers, and punctuation. For precise counts, use the official tokenizers: tiktoken (pip install tiktoken) for OpenAI models, or the model's API to count tokens before sending. The estimates here are typically within 5-10% accuracy.