Apache Kafka: Event Streaming Architecture & Setup Guide

Apache Kafka is the industry-standard distributed event streaming platform, handling trillions of events per day at companies like LinkedIn, Uber, and Netflix. Build real-time data pipelines and streaming applications with durable, replayable event logs.

What is Apache Kafka?

Apache Kafka is a distributed event store and stream-processing platform. Created at LinkedIn in 2010 and open-sourced in 2011, it is now an Apache top-level project. Unlike a traditional message queue that deletes events after delivery, Kafka persists all events in an ordered, immutable log for a configurable retention period. Any number of independent consumers can read the same events at their own pace, and events can be replayed from any offset.

Kafka's architecture consists of brokers (the servers that store data), topics (logical event streams), partitions (horizontal shards of a topic enabling parallelism), producers (applications that write events), and consumers (applications that read events). As of Kafka 3.3, ZooKeeper is no longer required — the KRaft mode uses Kafka itself for metadata management, simplifying deployment significantly.

Core Architecture

• Broker — a Kafka server that stores partitions and handles producer/consumer requests
• Topic — a named, append-only log; events are never modified once written
• Partition — ordered shard of a topic; enables parallel reads and writes across brokers
• Offset — unique sequential ID for each event within a partition; consumers track their own offset
• Consumer Group — a set of consumers sharing a group.id; each partition assigned to exactly one consumer
• Replication Factor — number of broker copies per partition; protects against broker failure
• Schema Registry — external service that validates and stores Avro/Protobuf schemas

Docker Compose Quickstart

# docker-compose.yml — Kafka + Zookeeper + Schema Registry
version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on: [zookeeper]
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

  schema-registry:
    image: confluentinc/cp-schema-registry:7.6.0
    depends_on: [kafka]
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:9092

Python Producer & Consumer (confluent-kafka)

# pip install confluent-kafka
from confluent_kafka import Producer, Consumer, KafkaError

# --- Producer ---
producer = Producer({'bootstrap.servers': 'localhost:9092'})

def delivery_report(err, msg):
    if err:
        print(f'Delivery failed: {err}')
    else:
        print(f'Delivered to {msg.topic()} [{msg.partition()}] @ {msg.offset()}')

for i in range(10):
    producer.produce(
        topic='user-events',
        key=f'user-{i}',
        value=f'{"event": "click", "user_id": {i}}',
        callback=delivery_report
    )
producer.flush()

# --- Consumer ---
consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'analytics-consumer-group',
    'auto.offset.reset': 'earliest',
})
consumer.subscribe(['user-events'])

try:
    while True:
        msg = consumer.poll(timeout=1.0)
        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            raise Exception(msg.error())
        print(f'Key: {msg.key()}  Value: {msg.value()}')
finally:
    consumer.close()

Kafka Connect Source Connector

# Kafka Connect — JDBC Source Connector config (REST API)
# POST http://localhost:8083/connectors
{
  "name": "postgres-source-connector",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:postgresql://db:5432/mydb",
    "connection.user": "kafka",
    "connection.password": "secret",
    "mode": "incrementing",
    "incrementing.column.name": "id",
    "topic.prefix": "db.",
    "table.whitelist": "orders,customers",
    "poll.interval.ms": "5000",
    "value.converter": "io.confluent.kafka.serializers.KafkaAvroSerializer",
    "value.converter.schema.registry.url": "http://schema-registry:8081"
  }
}

Kafka Streams vs Apache Flink vs Spark Streaming

Production Best Practices

• Set replication factor to at least 3 in production and configure min.insync.replicas=2 for durability
• Use Schema Registry with Avro or Protobuf to enforce schema compatibility across producer/consumer deployments
• Monitor consumer lag per partition — lag growth indicates consumers are falling behind producers
• Tune linger.ms and batch.size on producers to balance throughput vs latency
• Prefer KRaft mode over ZooKeeper for all new deployments (Kafka 3.3+)
• Use Kafka Connect for integrations — avoid reinventing offset management in custom consumers