Kafka Topic Config Generator

Partitions determine parallelism. A common rule of thumb: start with max(T/p, T/c) where T is your target throughput, p is the write throughput per partition, and c is the read throughput per partition. For most use cases, 3-12 partitions is a good starting point. More partitions increase parallelism but also increase leader election overhead.

For production, use replication factor 3. This tolerates one broker failure while still maintaining a majority quorum (2 of 3 replicas). Replication factor 1 means no fault tolerance — one broker failure loses data. Replication factor 2 means you lose quorum when one broker fails.

min.insync.replicas defines the minimum number of replicas that must acknowledge a write before it's considered successful (when acks=all). Setting it to replication_factor - 1 is common: e.g., for RF=3, set min.insync.replicas=2. If this value equals or exceeds the replication factor, all replicas must be in sync — meaning any single broker failure blocks writes.

cleanup.policy=delete removes messages older than the retention period (time or size based). cleanup.policy=compact retains only the latest message per key — useful for event sourcing and change data capture (CDC). Use compact,delete to apply both: keep only latest keys AND enforce a time-based retention on those records.

snappy offers the best balance of compression ratio and CPU cost for most workloads. lz4 is faster with slightly less compression. zstd provides the best compression ratio but requires Kafka 2.1+. gzip has good compression but higher CPU cost. none is zero overhead but increases network and storage costs significantly for text/JSON payloads.

Use kafka-configs.sh --alter --add-config: e.g., kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --alter --add-config retention.ms=604800000. Topic-level configs override the broker defaults. Use kafka-configs.sh --describe to see current effective config for a topic.