Apache Kafka: A Working Introduction

Apache Kafka is a distributed, append-only log. Producers append records to the end of a topic; consumers read forward from a position they control, and nothing is removed when they read it. A topic is split into partitions, and the partition is the unit of ordering, storage and parallelism: records inside one partition are strictly ordered by an increasing offset, and records in different partitions have no defined order relative to each other. A consumer group divides a topic's partitions among its members, with each partition going to at most one member, so the partition count sets a hard ceiling on how many consumers can work at once. Kafka keeps records for a configured retention period (seven days by default) whether or not anyone has read them, which is what makes replay possible and what separates a log from a queue. Every other behavior in this guide follows from those four facts.

What follows is the model, a broker you can run in five minutes, working code, and an honest account of when a queue or a database table serves better. Defaults are those of Kafka 4.x.

What Kafka Actually Is: An Append-Only Log

A Kafka topic is a durable, ordered, replayable file that many readers scan independently. Writes only go to the end, reads are sequential scans from a position the reader owns, and there is no per-message acknowledgement.

Most confusion about Kafka comes from importing queue vocabulary. In a queue a message has a lifecycle: enqueued, delivered, acknowledged, gone. In Kafka a record has a location, partition 3 offset 918,442, and it stays there until retention expires. Any number of unrelated consumer groups read it at their own pace without affecting each other.

That buys sequential disk writes, free replay, and the ability to add a downstream service by starting a new group at the earliest offset with no coordination from the producer. It costs the queue features: no per-message retry, no built-in dead-letter routing, and head-of-line blocking whenever one slow record sits at the front of a partition.

Topics, Partitions, and the Partition Key

The record key decides the partition. The default partitioner hashes the key modulo the partition count, so every record with the same key lands on the same partition and stays ordered. Keyless records are spread across partitions in batches.

This is the most useful lever in Kafka. If ordering matters per bank account, key on the account id. You get ordered processing per entity and full parallelism across entities, a far better trade than a single-partition topic that orders everything and scales to one consumer.

Two consequences catch people. Partition counts only increase. And increasing them changes the hash result for existing keys, so a key that lived on partition 2 may start landing on partition 5 while its older records sit on partition 2. Plan the count once, at creation.

Real durability is replication factor 3, min.insync.replicas=2, and a producer using acks=all: a write is confirmed only once the leader and at least one follower hold it. Setting acks=all while leaving min.insync.replicas at its default of 1 is the most common false sense of safety in a new cluster, because the leader alone satisfies "all in-sync replicas" when it is the only one.

Run a Kafka Broker Locally with KRaft

Kafka 4.0 removed ZooKeeper. Metadata lives in a Raft-replicated internal log managed by controller nodes, and 3.9 was the last release supporting the old arrangement. A single-node broker that plays both roles is one Compose file.

The official apache/kafka image ships the standard scripts under /opt/kafka/bin, so what you learn locally transfers to a real cluster. This configuration combines the broker and controller roles, which is fine for development and wrong for production, where three dedicated controllers are the norm.

That last command prints current offset, log end offset and lag for every partition the group owns. Lag is the metric that tells you whether a consumer is healthy: steady growth means it cannot keep up, while a sawtooth usually means rebalances. If you run Kafka in containers, the Docker fundamentals guide covers the volume and networking details worth settling before you put state in one.

Producing and Consuming in Python

The confluent-kafka package wraps the C library and is the fastest and most complete Python client. A producer is asynchronous by default: produce() puts a record in an internal buffer, and only a delivery callback tells you whether the broker accepted it.

Forgetting that asynchrony is the first bug in most Kafka programs. A script that produces a thousand records and exits without calling flush() sends some fraction of them and drops the rest with no error at all.

Committing after the work is what makes this at-least-once. If the process dies between handle() and commit(), the record is redelivered, so handle() has to tolerate seeing it twice. Committing first would be at-most-once: faster, and it loses records on every crash.

Consumer Groups, Offsets, and Rebalancing

A consumer group is a set of processes sharing a group.id. Kafka assigns each partition to exactly one member and keeps the group's committed offsets in the internal __consumer_offsets topic. Any membership change triggers a rebalance.

Three rules cover most of it. A consumer beyond the partition count sits idle. Two groups reading one topic are fully independent, each with its own offsets. And a rebalance fires on joins, leaves, crashes, and on a member that took too long between calls to poll().

That last trigger causes the classic rebalance storm. If processing exceeds max.poll.interval.ms, five minutes by default, the broker declares the member dead and reassigns its partitions. The member finishes, tries to commit, is told it no longer owns the partition, and rejoins, starting another rebalance. The fix is not a longer timeout: lower max.poll.records from its default of 500, or move slow work off the polling thread.

Setting	Default	What it controls
max.poll.records	500	Batch size per poll. Lower this first when processing is slow
max.poll.interval.ms	300000	Time allowed between polls before eviction
session.timeout.ms	45000	Heartbeat deadline. Detects crashes
auto.offset.reset	latest	Where a new group starts. latest skips history
enable.auto.commit	true	Commits on a timer regardless of work

Delivery Semantics: At-Most-Once to Exactly-Once

Kafka offers three levels, and the choice comes down to where you commit the offset. At-most-once commits before processing. At-least-once commits after. Exactly-once uses transactions to make the write and the offset commit one atomic operation, and works only when both ends are Kafka.

Exactly-once in Kafka means exactly-once processing, not exactly-once delivery to arbitrary systems. The idempotent producer gives each producer a session id and a per-partition sequence number, so the broker discards duplicates from retries. Transactions extend that: the producer writes to one or more topics and commits the consumed offsets in the same transaction. Consumers set to read_committed never see aborted records.

Kafka Streams packages that loop behind processing.guarantee=exactly_once_v2, the right choice when a job reads Kafka and writes Kafka. Everywhere else, at-least-once with idempotent handlers survives production. The distributed systems guide covers why the stronger guarantee is hard to extend past one system.

Retention, Log Compaction, and Tiered Storage

Two cleanup policies exist. delete removes whole segments once they exceed retention.ms or retention.bytes. compact keeps the most recent value for every key forever, turning the topic into a durable changelog of current state.

Time-based deletion is the default and applies per segment, not per record, so a record survives until its segment ages out. Both limits are active and whichever fires first wins. retention.bytes is unlimited by default; setting it is the simplest protection against a topic filling a disk.

Compaction is a different tool. Send a key and a value to set state; send the same key with a null value, a tombstone, to delete it. The cleaner collapses the log so only the latest value per key remains, so a consumer reading from offset zero rebuilds current state without the full history. This is how Kafka stores its own consumer offsets and how change-data-capture topics are configured.

Long retention used to force a choice between broker disk and lost history. Tiered storage, production-ready in the 3.9 line, moves closed segments to object storage while recent data stays local, so a topic can hold months of history at object-storage prices. Cold reads are slower: right for backfills, wrong for a latency-sensitive consumer.

Schema Registry: The Contract Between Services

A schema registry stores versioned schemas, hands out an integer id for each, and enforces a compatibility rule before accepting a new version. Serializers prefix every payload with a magic byte and that four-byte id, so a consumer fetches the exact writer schema.

Kafka moves opaque bytes and has no opinion about their shape. That works until a producer adds a required field and three downstream consumers fail at once, at three in the morning, with a deserialization error that names no owner. A registry turns that outage into a rejected deploy.

Compatibility modes are the actual product. BACKWARD, the default, means a consumer on the new schema can read data written with the previous one, which lets you upgrade consumers first. FORWARD means old consumers can read new data, so producers move first. FULL requires both, and the _TRANSITIVE variants check every earlier version rather than only the last. Under BACKWARD, adding an optional field with a default is allowed; removing a required field is not.

Confluent Schema Registry is the reference implementation; Apicurio Registry and AWS Glue Schema Registry offer the same service under different licenses. Whichever you run, register schemas from CI, so an incompatible change fails a pull request rather than a deployment.

What Breaks First in a New Kafka Deployment

New clusters fail in a small, predictable set of ways, and almost none of them are broker problems. They are defaults chosen for compatibility rather than for correctness.

Three numbers cover most of the operational risk: consumer group lag per partition, under-replicated partition count, and broker request handler idle ratio. Rising lag is a capacity or code problem. Under-replicated partitions above zero for more than a moment means a broker or a disk is failing.

Kafka vs RabbitMQ vs SQS vs a Postgres Table

The honest comparison is not about throughput, since all four handle more than most applications produce. It is about replay, per-message acknowledgement, and operational surface.

Capability	Kafka	RabbitMQ	Amazon SQS	Postgres table
Replay after consumption	Any offset	Gone on ack	Gone on delete	Only if rows are kept
Many independent readers	One group each	Fanout exchange, copies	SNS fanout to queues	Cursor per reader
Ack a single message	No, offsets only	Yes	Yes	Yes, one row
Dead-letter handling	Build it yourself	Built in	Built in	A status column
Ordering	Total per partition	Lost on redelivery	FIFO queues, per group	Whatever you order by
Consumer scaling limit	Partition count	Unbounded	Unbounded	Row lock contention
Operational cost	Brokers and rebalances	One clustered service	None, it is managed	A database you already run
Best fit	Streams several teams read	Task routing with retries	Background jobs on AWS	Jobs that share your data

The Postgres column deserves more credit than it usually gets. SELECT ... FOR UPDATE SKIP LOCKED turns an ordinary table into a competing-consumer queue with per-row acknowledgement, retries, dead-lettering and SQL visibility, inside the same transaction as your business writes. That last property is the one Kafka cannot offer: no dual-write problem, no outbox.

Throughput depends on row size, index count and how long each worker holds its transaction. The PostgreSQL performance guide covers the index and vacuum behavior that decides where the ceiling falls, since a queue table generates dead tuples faster than almost anything else.

Redis Streams sits in the middle: consumer groups and per-message acknowledgement at low latency, bounded by memory rather than disk. Kafka 4.0 also shipped share groups as an early-access feature, queue-style competing consumers on top of the log; watch it rather than design around it.

When Not to Use Kafka

Kafka is the wrong tool when you need per-message retry semantics, when one team is both the only producer and the only consumer, or when the data must stay transactionally consistent with a database.

The clearest signal is the shape of failure handling. If a single bad record should be retried with backoff and eventually parked without holding up anything behind it, that is queue behavior, and building it on Kafka means retry topics, a dead-letter topic, and code to move records between them.

The second signal is ownership. A service publishing to itself gains nothing from a log and gains a cluster to patch, monitor and upgrade. Kafka earns its cost when several teams read the same stream at different speeds, when replay has real value, or when retention is itself the feature, as in an audit trail. If none of those apply, use a table, a managed queue, or a scheduled batch pipeline.

The Order to Work In

The bottom line: Kafka is a log with a partition-based concurrency model attached, and every guarantee it offers or withholds comes from that. Get the key and the partition count right, commit offsets after the work, put a schema contract in front of the topic, and it runs quietly for years. Reach for it because you need replay and independent readers.

Frequently Asked Questions

What is the difference between Kafka and a message queue like RabbitMQ?

A queue removes a message once a consumer acknowledges it. Kafka removes nothing on read: it keeps every record for the retention period, and each consumer group tracks its own position. Kafka fits when you need replay, several independent readers, or a consumer added six months later. A queue fits when you need per-message acknowledgement, redelivery of one failed message without blocking the records behind it, and a dead-letter queue, none of which Kafka provides.

How many partitions should a Kafka topic have?

Start from the parallelism you need, not the throughput you hope for. Each partition goes to at most one consumer inside a group, so the partition count caps concurrent consumer instances. Pick slightly above expected peak, commonly 6 to 24 for an application topic. You can add partitions but never remove them, and adding them changes which partition a key hashes to, breaking per-key ordering across the change.

Does Kafka guarantee message ordering?

Kafka guarantees order within a single partition and nothing across partitions. Records in one partition are read exactly in write order, identified by an increasing offset. To order a business entity, set a record key such as the account or device id: the default partitioner hashes the key, so those records share a partition. Different keys may be processed in any relative order.

Can Kafka really do exactly-once delivery?

Kafka supports exactly-once processing inside Kafka, which is narrower than exactly-once delivery to anything else. The idempotent producer, on by default since Kafka 3.0, drops duplicates from producer retries. Transactions commit records and consumer offsets atomically, so a read-process-write loop is exactly-once when both ends are Kafka and readers set isolation.level=read_committed. As soon as a consumer writes to a database or calls an API, the guarantee stops; use at-least-once plus an idempotent write.

How long does Kafka keep messages?

By default a topic keeps records for seven days, set by retention.ms, and deletes them whether or not anyone read them. retention.bytes caps by size and is unlimited by default; whichever fires first wins. Deletion is per log segment, so a record survives until its whole segment ages out. Log compaction, cleanup.policy=compact, instead keeps the latest value per key indefinitely.

Do I still need ZooKeeper to run Kafka?

No. Kafka 4.0 removed ZooKeeper and runs only in KRaft mode, where controller nodes store cluster metadata in an internal Kafka log. Version 3.9 was the last release supporting ZooKeeper and is the bridge release for migrating an older cluster. Format a storage directory with a cluster id, set the controller quorum, and start. Production clusters run three dedicated controllers.

References: Apache Kafka documentation, Kafka design notes, Broker, producer and consumer configuration reference, Confluent Schema Registry documentation