LLM Latency & Throughput Calculator

TTFT (Time To First Token) is how long the API takes before streaming the first token of the response. For interactive chat, users perceive TTFT as 'responsiveness' — anything above 1-2 seconds feels slow. For batch processing, TTFT matters less than total throughput (tokens/second). Most LLM APIs have TTFT of 200ms-1500ms depending on model size and server load.

TPS is the streaming rate after the first token arrives — how fast the model generates output tokens. Smaller models (8B) can hit 80-150 TPS via API. Larger models (GPT-4o, Claude 3.5 Sonnet) typically stream at 30-60 TPS. Self-hosted models on A100 can achieve 40-80 TPS for 70B models with optimized inference (vLLM, TGI). TPS directly determines how long a long response takes.

Key factors: (1) Model size — larger models are slower. (2) Server load — peak hours add 50-200ms. (3) Input length — longer prompts take longer to process before generation starts. (4) Output length — more tokens = more time to complete. (5) Region — API servers have geographic latency. (6) Batching — shared inference infrastructure adds queue time. For latency-critical apps, dedicated endpoints or caching are common solutions.

Techniques: (1) Use faster/smaller models where quality allows (Haiku, Flash, GPT-4o-mini). (2) Enable streaming so UX starts immediately. (3) Cache frequent prompts (OpenAI prompt caching, Anthropic cache_control). (4) Reduce input token count with prompt compression. (5) Use parallel calls for independent tasks. (6) Set max_tokens to limit output. (7) Use edge deployment (Cloudflare Workers AI, Vercel AI SDK) for lower geographic latency.

P99 latency is the worst-case latency that 99% of requests fall under — the 1% slowest requests. Average latency can be 300ms but p99 might be 3000ms due to cold starts, server congestion, or long queues. For SLA commitments, p95/p99 matter more than averages. Most LLM providers publish only average latencies; tools like LLMPerf or custom monitoring are needed to measure true p99.