Kafka Topic Planner

Partitions determine parallelism. A common rule: number of partitions = max(target throughput / single-partition throughput, num consumers in consumer group). Start conservatively (e.g. 6–12) because you can increase partitions later but cannot decrease them without recreating the topic. More partitions mean more file handles and leader elections, so avoid over-partitioning.

Replication factor controls how many broker copies exist for each partition. A factor of 3 means 1 leader + 2 followers — the industry standard for production clusters. The broker count must be >= replication factor. Higher replication improves fault tolerance (tolerate N-1 broker failures) but increases storage and network usage proportionally.

Kafka retains messages based on time (retention.ms) or size (retention.bytes). When a segment exceeds the retention window, it is eligible for deletion. Default is 7 days (604800000 ms). For event sourcing or audit logs you might set retention to weeks or months. Compacted topics use log compaction instead — keeping only the latest value per key indefinitely.

max.message.bytes controls the maximum size of a single message (default 1 MB). If your pipeline produces large payloads (images, documents), increase this alongside fetch.message.max.bytes on consumers and replica.fetch.max.bytes on brokers. For very large payloads consider storing data externally and passing a reference key via Kafka instead.

Storage per partition = (throughput MB/s per partition) × (retention seconds). Total storage = storage per partition × num partitions × replication factor. Add 20–30% overhead for index files and segment gaps. For a topic with 100 MB/s throughput, 10 partitions, replication factor 3, and 24h retention: (10 MB/s × 86400s × 3) ≈ 2.5 TB across the cluster.