Apache Kafka vs RabbitMQ vs Redpanda

Apache Kafka is a distributed event streaming platform originally built by LinkedIn and open-sourced in 2011. It is designed for high-throughput, fault-tolerant, durable message streaming at massive scale. Kafka stores messages in partitioned, replicated logs on disk, allowing consumers to replay history — a feature no traditional message queue offers. This makes Kafka ideal for event sourcing, activity tracking, log aggregation, stream processing (with Kafka Streams or Apache Flink), and change data capture (CDC). Kafka is the backbone of real-time data infrastructure at companies like Uber, Netflix, Airbnb, and LinkedIn, handling trillions of messages per day. The operational tradeoff: Kafka requires ZooKeeper (legacy) or KRaft (Kafka 3.x) and is complex to tune and operate at scale.

RabbitMQ implements the AMQP protocol and is a traditional message broker with rich routing logic: topic exchanges, direct routing, fanout, and header-based routing. Unlike Kafka's log retention, RabbitMQ deletes messages once they are acknowledged — it is designed for task queues where each message is processed exactly once by one consumer. Use RabbitMQ when you need: complex routing logic between microservices, task queue semantics (background jobs, email sending, image processing), low message volume with complex workflows, or when your team already uses it and doesn't need Kafka's replay capabilities. RabbitMQ is simpler to operate for workloads under ~50k messages/second and integrates natively with Celery, Sidekiq, and most application frameworks.

Redpanda is a Kafka-compatible streaming platform written in C++ (not Java/Scala) with no ZooKeeper or JVM dependency. It implements the Kafka API 100% — any Kafka client works with Redpanda without code changes. The C++ implementation delivers 10x lower tail latencies (p99) and significantly lower resource consumption compared to Apache Kafka for equivalent throughput. Redpanda's built-in Wasm-based transforms, schema registry, and HTTP proxy ship in a single binary. Redpanda Cloud (managed) competes directly with Confluent Cloud. The main tradeoff: smaller ecosystem and community compared to Kafka, and some advanced Kafka features (e.g., certain compaction configurations) have behavioral differences. Ideal for teams wanting Kafka semantics with lower operational overhead.

For low-volume use cases, a database-backed queue (PostgreSQL SKIP LOCKED, Redis lists, or Amazon SQS) may be sufficient and much simpler to operate. The case for a dedicated message broker emerges when: (1) you need fan-out to multiple consumers simultaneously, (2) message volume exceeds what your database can handle without impacting primary workloads, (3) you need message replay or event sourcing, (4) you need stream processing (aggregations over time windows), or (5) you need to decouple services across different tech stacks or teams. As a rule of thumb: under 10k messages/day, use database queues. Between 10k and 1M/day, evaluate RabbitMQ or Redis Streams. Above 1M/day or with replay requirements, Kafka or Redpanda.