Kafka vs RabbitMQ vs Redis Streams: Message Queue Comparison 2026

Choosing the wrong message queue can cripple a microservices architecture. Too little throughput, unreliable delivery, or a system that can’t scale with your traffic — these are expensive mistakes. Kafka, RabbitMQ, and Redis Streams each solve message queuing from a different angle. This guide helps you pick the right one for your workload in 2026.

---

Quick Comparison: At a Glance

	Apache Kafka	RabbitMQ	Redis Streams
Architecture	Distributed log	Message broker	In-memory log
Throughput	Millions msg/sec	~50k–100k msg/sec	~500k–1M msg/sec
Persistence	Log-based, configurable TTL	Persistent (disk)	In-memory + AOF/RDB
Message Ordering	Per partition	Per queue	Per stream
Consumer Groups	Yes (native)	Yes (with plugins)	Yes (XREADGROUP)
Message Replay	Yes (offset-based)	No (consumed = gone)	Yes (stream ID-based)
At-least-once	Yes	Yes	Yes
Exactly-once	Yes (Kafka transactions)	Limited	No
Setup Complexity	High	Medium	Low (Redis already deployed)
Best For	Event streaming, audit log	Task queues, routing	Lightweight queuing
Managed Cloud	Confluent, MSK, Aiven	CloudAMQP, AmazonMQ	Redis Cloud, Upstash

---

Apache Kafka: The Event Streaming Standard

Kafka is a distributed commit log, not a traditional message queue. Messages (events) are written to topics divided into partitions, retained on disk for a configurable duration, and consumed by any number of consumer groups independently. This is the key architectural difference: consuming a message does not delete it.

Core Concepts

Topics and Partitions: A topic is like a database table. Each partition is an ordered, immutable sequence of records. Partitions enable horizontal scaling and parallel consumption.

Consumer Groups: Multiple consumers can form a group and divide the partitions among themselves. Each message is processed once per group — but different groups can each process every message independently.

Offsets: Each record has a numeric offset within its partition. Consumers commit offsets to track their position. On restart, a consumer resumes from its last committed offset.

Producer Code

import { Kafka, CompressionTypes } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['kafka-1:9092', 'kafka-2:9092', 'kafka-3:9092'],
  ssl: true,
  sasl: {
    mechanism: 'scram-sha-256',
    username: process.env.KAFKA_USERNAME,
    password: process.env.KAFKA_PASSWORD,
  },
});

const producer = kafka.producer({
  idempotent: true,         // exactly-once semantics
  maxInFlightRequests: 5,
  transactionTimeout: 60000,
});

await producer.connect();

// Send batch of events
await producer.sendBatch({
  topicMessages: [
    {
      topic: 'order-events',
      messages: [
        {
          key: order.userId,              // partition key
          value: JSON.stringify({
            eventType: 'ORDER_CREATED',
            orderId: order.id,
            userId: order.userId,
            items: order.items,
            timestamp: Date.now(),
          }),
          headers: { 'content-type': 'application/json' },
        },
      ],
    },
  ],
  compression: CompressionTypes.GZIP,
  timeout: 30000,
});

Consumer Code

const consumer = kafka.consumer({
  groupId: 'notification-service',
  sessionTimeout: 30000,
  heartbeatInterval: 3000,
  maxWaitTimeInMs: 5000,
});

await consumer.connect();
await consumer.subscribe({ topic: 'order-events', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());

    try {
      await processOrderEvent(event);
      // Offset committed automatically after successful processing
    } catch (err) {
      // Dead letter queue pattern — publish to DLQ topic
      await producer.send({
        topic: 'order-events-dlq',
        messages: [{
          key: message.key,
          value: message.value,
          headers: {
            'original-partition': String(partition),
            'error': err.message,
            'retry-count': '1',
          },
        }],
      });
    }
  },
});

Kafka: Exactly-Once Semantics

Kafka transactions enable exactly-once processing — no duplicates, no data loss, even across multiple topics:

await producer.transaction(async (transaction) => {
  // Read-process-write in a single atomic operation
  const events = await readFromSourceTopic();

  const processed = events.map(transform);

  await transaction.send({
    topic: 'processed-events',
    messages: processed.map(e => ({
      key: e.key,
      value: JSON.stringify(e),
    })),
  });

  await transaction.sendOffsets({
    consumerGroupId: 'processor-group',
    topics: [{ topic: 'source-events', partitions: [{ partition: 0, offset: lastOffset }] }],
  });
  // Commit transaction — either all messages sent + offsets committed, or nothing
});

Kafka Strengths

• Event replay. Consumer groups can reset to any offset — replay the last hour, day, or month of events. Critical for debugging, backfilling, and adding new consumers to existing event streams.
• Massive throughput. A single Kafka cluster handles millions of messages per second. Netflix, LinkedIn, and Uber run Kafka at truly industrial scale.
• Retention policy. Topics retain messages for a configurable duration (hours, days, forever as a compacted topic). This makes Kafka suitable as a system of record.

Kafka Weaknesses

• Operational complexity. Kafka requires ZooKeeper (or KRaft in newer versions), careful partition planning, and ongoing cluster management. Managed services (Confluent Cloud, Amazon MSK) hide this complexity but cost money.
• Not ideal for task queues. Kafka doesn’t support message acknowledgment in the traditional sense — you can’t selectively acknowledge or reject individual messages.
• High latency for small loads. Kafka optimizes for throughput via batching. For low-volume, latency-sensitive workloads, RabbitMQ or Redis Streams are faster end-to-end.

---

RabbitMQ: The Flexible Message Broker

RabbitMQ is a traditional message broker built on AMQP. It routes messages through exchanges to queues, supports multiple messaging patterns (pub/sub, work queues, RPC), and provides reliable delivery with per-message acknowledgments.

Core Concepts

Exchange → Queue routing. Producers publish to exchanges, not queues. Exchanges route messages to queues based on routing keys and binding rules. This enables sophisticated routing without changing producer code.

Message acknowledgments. When a consumer receives a message, it can explicitly acknowledge (ack) or reject (nack) it. Unacknowledged messages are requeued or sent to a dead letter exchange.

Four exchange types:

• direct — exact routing key match
• topic — pattern matching with * and # wildcards
• fanout — broadcast to all bound queues
• headers — route based on message headers

Producer and Consumer Code

import amqplib from 'amqplib';

const connection = await amqplib.connect(process.env.RABBITMQ_URL);
const channel = await connection.createConfirmChannel();

// Setup topology
const exchangeName = 'order-events';
const queueName = 'order-notifications';

await channel.assertExchange(exchangeName, 'topic', { durable: true });
await channel.assertQueue(queueName, {
  durable: true,
  deadLetterExchange: 'order-events-dlx',
  messageTtl: 3600000,  // 1 hour TTL
  arguments: { 'x-queue-type': 'quorum' },  // RabbitMQ 3.8+ quorum queues
});
await channel.bindQueue(queueName, exchangeName, 'order.#');

// Producer — publish with confirm
await new Promise((resolve, reject) => {
  channel.publish(
    exchangeName,
    'order.created',
    Buffer.from(JSON.stringify({
      orderId: '123',
      userId: 'user-456',
      amount: 9999,
    })),
    {
      persistent: true,           // survive broker restart
      contentType: 'application/json',
      messageId: crypto.randomUUID(),
      timestamp: Date.now(),
    },
    (err) => (err ? reject(err) : resolve())
  );
});

// Consumer — with manual acknowledgment
const consumer = await channel.createChannel();
await consumer.prefetch(10);  // max 10 unacked messages per consumer

await consumer.consume(queueName, async (msg) => {
  if (!msg) return;

  try {
    const event = JSON.parse(msg.content.toString());
    await sendNotification(event);

    consumer.ack(msg);  // explicit acknowledgment
  } catch (err) {
    console.error('Processing failed:', err);

    const retries = (msg.properties.headers['x-retry-count'] || 0) + 1;

    if (retries >= 3) {
      consumer.nack(msg, false, false);  // send to DLX
    } else {
      // Requeue with retry count
      consumer.nack(msg, false, true);
    }
  }
}, { noAck: false });

RabbitMQ Routing Patterns

Topic exchange for selective routing:

// Route different event types to different queues
await channel.bindQueue('payment-queue', 'order-events', 'order.payment.*');
await channel.bindQueue('shipping-queue', 'order-events', 'order.shipping.*');
await channel.bindQueue('analytics-queue', 'order-events', 'order.#');  // catches all

// Now publishing 'order.payment.processed' only reaches payment-queue and analytics-queue
// Publishing 'order.shipping.dispatched' only reaches shipping-queue and analytics-queue

Dead Letter Exchange:

// Setup DLX
await channel.assertExchange('order-events-dlx', 'direct', { durable: true });
await channel.assertQueue('order-events-dead-letters', { durable: true });
await channel.bindQueue('order-events-dead-letters', 'order-events-dlx', queueName);

// Failed messages automatically routed here after x-message-ttl or rejection

RabbitMQ Strengths

• Flexible routing. No other system matches RabbitMQ’s exchange/binding routing patterns for complex message distribution.
• Per-message acknowledgment. Unlike Kafka’s offset-based commits, RabbitMQ lets you ack or nack individual messages — perfect for task queues where some tasks fail and need special handling.
• Low latency. RabbitMQ delivers messages with sub-millisecond latency for small, latency-sensitive workloads.
• Stream queues (RabbitMQ 3.9+). RabbitMQ now supports stream queues with Kafka-style replay semantics, bridging the gap with Kafka.

RabbitMQ Weaknesses

• No built-in replay. Traditional AMQP queues — once consumed, messages are gone. (Stream queues fix this, but they’re a newer addition.)
• Throughput ceiling. RabbitMQ’s ~50–100k msg/sec is excellent but can’t match Kafka for high-volume event streams.
• Schema-free by default. No native schema registry; you bring your own Avro/Protobuf validation.

---

Redis Streams: Lightweight and Fast

Redis Streams (introduced in Redis 5.0) brings Kafka-like semantics — consumer groups, acknowledgments, message replay — to Redis. If you already run Redis, Streams add event streaming with zero new infrastructure.

Core Concepts

XADD / XREAD. Append entries to a stream (XADD) and read them (XREAD). Each entry has an auto-generated or custom ID in the format <milliseconds>-<sequence>.

Consumer Groups. XREADGROUP assigns entries to specific consumers within a group. Entries must be explicitly acknowledged with XACK.

Pending Entry List (PEL). Unacknowledged entries stay in the PEL. Consumers can claim stale PEL entries with XCLAIM to handle crashed consumers.

Producer Code

import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

// XADD — append to stream
const messageId = await redis.xAdd(
  'order-events',    // stream key
  '*',               // auto-generate ID
  {
    eventType: 'ORDER_CREATED',
    orderId: '123',
    userId: 'user-456',
    amount: '9999',
    timestamp: Date.now().toString(),
  },
  {
    MAXLEN: { count: 100000, strategy: '~' },  // trim to ~100k entries
  }
);

console.log('Published:', messageId); // e.g. "1648387200000-0"

Consumer Group Setup and Reading

// Create consumer group (once)
try {
  await redis.xGroupCreate('order-events', 'notification-service', '$', {
    MKSTREAM: true,  // create stream if it doesn't exist
  });
} catch (err) {
  if (!err.message.includes('BUSYGROUP')) throw err;
}

// Consumer loop
const consumerId = `worker-${process.pid}`;

async function consume() {
  while (true) {
    const messages = await redis.xReadGroup(
      'notification-service',  // group name
      consumerId,               // consumer name
      [{ key: 'order-events', id: '>' }],  // '>' = new, undelivered messages
      { COUNT: 10, BLOCK: 5000 }           // batch of 10, block 5s
    );

    if (!messages || messages.length === 0) continue;

    for (const { name: stream, messages: entries } of messages) {
      for (const { id, message } of entries) {
        try {
          await processEvent(message);
          await redis.xAck('order-events', 'notification-service', id);
        } catch (err) {
          console.error('Failed to process:', id, err);
          // Entry stays in PEL — will be reclaimed after timeout
        }
      }
    }
  }
}

Handling Crashed Consumers (PEL Reclaim)

async function reclaimStalePendingMessages() {
  const staleAfterMs = 60_000;  // 1 minute

  const pendingEntries = await redis.xAutoClaim(
    'order-events',
    'notification-service',
    consumerId,
    staleAfterMs,
    '0-0',  // start from beginning of PEL
    { COUNT: 100 }
  );

  for (const { id, message } of pendingEntries.messages) {
    try {
      await processEvent(message);
      await redis.xAck('order-events', 'notification-service', id);
    } catch (err) {
      // Send to DLQ after N retries
      const retries = await incrementRetryCount(id);
      if (retries >= 3) {
        await moveToDlq('order-events', id, message, err);
        await redis.xAck('order-events', 'notification-service', id);
      }
    }
  }
}

// Run every 30 seconds
setInterval(reclaimStalePendingMessages, 30_000);

Redis Streams Strengths

• Zero additional infrastructure. If you already use Redis, Streams adds zero ops overhead.
• High throughput. Redis Streams processes 500k–1M messages/sec on modern hardware.
• Simple API. Redis commands are simpler to reason about than Kafka’s consumer group protocol.
• Message replay. Stream IDs are time-sortable — read from any point in the stream history.
• TTL-based retention. MAXLEN trims streams automatically; XTRIM provides manual control.

Redis Streams Weaknesses

• In-memory primary storage. Redis is primarily memory-based. Large, long-retention event streams get expensive quickly. Persistence (AOF/RDB) adds durability but at I/O cost.
• Single-node write bottleneck. Redis Cluster shards across nodes, but a single stream key lives on one node. For multi-million msg/sec throughput, you need to shard streams by key.
• No exactly-once semantics. Redis Streams provides at-least-once delivery. Exactly-once requires application-level idempotency keys.
• Not a true event log. MAXLEN trimming means long-term event replay isn’t viable without external archiving.

---

Side-by-Side Feature Comparison

Message Delivery Guarantees

Scenario	Kafka	RabbitMQ	Redis Streams
At-least-once	✓	✓	✓
At-most-once	✓	✓	✓
Exactly-once	✓ (transactions)	Partial	✗
Message ordering	Per partition	Per queue	Per stream

Scalability

	Kafka	RabbitMQ	Redis Streams
Horizontal scale	Add partitions + brokers	Add nodes + queues	Shard by stream key
Consumer scale	Add consumers to group	Add consumers	Add consumers to group
Max throughput	Virtually unlimited	~500k msg/sec	~5M msg/sec (sharded)

---

When to Choose Each

Choose Kafka when:

• You need event streaming at scale — millions of messages per second
• You need long-term event retention and replay (audit logs, event sourcing)
• You’re building a data pipeline feeding analytics, data lakes, or multiple downstream services
• You need exactly-once semantics across distributed services
• Multiple independent consumer groups need to process the same event stream differently

Choose RabbitMQ when:

• You’re building task queues where per-message routing, prioritization, and selective acknowledgment matter
• You need complex routing — topic exchange pattern matching, headers-based routing, fanout
• Your workload is latency-sensitive (< 1ms delivery for low-volume queues)
• You need request-reply (RPC) patterns over messaging
• Your team understands AMQP and wants the operational familiarity

Choose Redis Streams when:

• Redis is already in your stack and you want to avoid adding a new service
• You need lightweight, high-speed queuing for moderate volumes (< 500k msg/sec)
• You want Kafka-like semantics (consumer groups, replay, ack) without Kafka’s operational overhead
• Your retention needs are short to medium term (hours to a few days)
• You’re building job queues, notification systems, or real-time activity feeds

---

2026 Managed Service Comparison

	Kafka	RabbitMQ	Redis Streams
AWS	Amazon MSK	Amazon MQ	ElastiCache
Google Cloud	—	—	Memorystore
Azure	Event Hubs (Kafka-compatible)	—	Azure Cache for Redis
Confluent Cloud	✓	—	—
CloudAMQP	—	✓	—
Upstash	✓ (Upstash Kafka)	—	✓ (Upstash Redis)
Aiven	✓	✓	✓
Redis Cloud	—	—	✓

---

Final Recommendation

For new event-driven microservices in 2026:

• Start with Redis Streams if Redis is already in your infrastructure, your volume is under 500k msg/sec, and you want fast implementation with minimal new ops.
• Use RabbitMQ if you need flexible routing patterns, per-message ACKs, or traditional task queue semantics.
• Use Kafka when you’re building a true event streaming platform — audit trails, event sourcing, multi-consumer fan-out at scale, or data pipeline ingestion.

The common mistake is over-engineering with Kafka when Redis Streams or RabbitMQ would do the job with 1/10th the operational complexity. Start simple, migrate when you genuinely hit limits.