Streaming data architecture: when you need it and when you do not

The one question that decides it

The conversation is usually framed wrong. Someone says the data is stale, someone else says the warehouse cannot go faster, and the meeting ends with a proof-of-concept on Kafka. Six months later there is a broker cluster, a schema registry, a Flink job, an on-call rotation nobody volunteered for, and a dashboard a human refreshes twice a day.

The question that settles it: name the decision, the thing that makes it, and the deadline. Not the data — the decision. A payment is scored and either goes out or gets held. A sensor reading crosses a threshold and a ticket opens. An agent needs the current state of a case before it answers. In each of those the deadline is real and shorter than a batch window. If you cannot fill in all three blanks, you have a freshness preference — a legitimate thing to want, and a poor thing to buy a distributed log for.

A useful second cut: is the consumer a machine or a person? Machines genuinely consume events — fraud scoring, alerting, autoscaling, feature serving. People consume snapshots. If a dashboard gets opened twice a day, a fifteen-minute incremental load has already overshot the requirement by two orders of magnitude.

What a nightly batch is genuinely good at

Batch has a reputation problem its engineering properties do not deserve. The core advantage is that a batch job has a beginning and an end, and that boundary makes everything else tractable.

Because the job is bounded, it can fail loudly and be re-run. "Reprocess yesterday" is a command. The streaming equivalent is a project: you need retained source data, a replay from an offset, a plan for downstream state that already consumed the bad version, and a decision about whether consumers can tolerate seeing corrected records twice. Late-arriving data has the same asymmetry — in batch you re-run a partition and the answer changes; in streaming you are managing watermark thresholds and holding state open, and a correction may arrive after the window that owned it has closed.

Batch is also cheap in a specific way: cost tracks volume, not uptime. A job that runs twenty minutes costs twenty minutes. A broker cluster costs money at 3 a.m. on a Sunday whether or not a message flows. And batch is debuggable on a laptop against a file, which sounds trivial until you have tried to reproduce a bug that only appears under one interleaving of two partitions. Batch trades latency for boundedness, and boundedness is worth a great deal.

The costs of streaming that do not appear in the design review

The billing dimensions are shaped differently than you expect. Per the Amazon MSK pricing page, provisioned MSK bills broker instance hours and storage GB-months — the standing cost is a function of the cluster you keep alive, not the traffic through it. MSK Serverless bills cluster hours and partition hours, plus data in, data out, and storage. That partition-hour line surprises people: an over-partitioned topic created in a hurry keeps costing money for as long as it exists, and partition counts are easy to raise and awkward to lower.

Kinesis has a different trap. The Kinesis Data Streams pricing page documents provisioned mode as shard hours plus PUT payload units, where a record is counted in 25 KB payload chunks, with each shard supporting 1 MB/second or 1,000 records/second of write and 2 MB/second of read. Read that metering rule against a realistic workload: telemetry and log lines are often a few hundred bytes, and a 200-byte record still consumes a 25 KB unit. The fix, batching several logical events into one record at the edge, has to be decided early because it reshapes everything downstream. On-demand mode meters differently again, rounding data in to the nearest 1 KB.

The partition key is a decision you make once. Kafka's documented ordering guarantee is per-partition: per Confluent's Kafka introduction, events with the same key go to the same partition, and "Kafka guarantees that any consumer of a given topic partition will always read that partition's events in exactly the same order as they were written." One key choice sets three things at once — what is ordered relative to what, how much parallelism you can ever have, and whether one hot tenant can saturate a partition while the rest idle. Keying by customer_id when the natural unit of work is case_id is not a tuning mistake you fix later; re-keying means republishing history and accepting a window where ordering does not hold across old and new topics.

And the operational cost is a person. A batch job that fails at 2 a.m. can usually wait until 8 a.m.; a stream that stops is accruing a backlog against a retention window, and if retention expires before anyone notices, the data is gone.

Exactly-once: what it guarantees, and what you pay for it

This is where most streaming projects acquire their hardest-to-remove complexity, usually because the phrase sounds like a promise it does not make.

Confluent's delivery-semantics documentation gives the three options: at-most-once (messages may be lost and are not redelivered), at-least-once (never lost, may be delivered more than once), and exactly-once ("each message is delivered once and only once"). Since Kafka 0.11.0.0 the idempotent producer supports the last at the log level — the broker assigns each producer an ID and dedupes on a sequence number sent with every message, so "resending a message will not result in duplicate entries in the log, and that log order is maintained." KIP-679 made acks=all and enable.idempotence=true the producer defaults — worth verifying rather than assuming, since KAFKA-13598 records a config-validation bug that kept idempotence from actually being enabled until the 3.0.1, 3.1.1, and 3.2.0 fix releases.

All of that covers producer to broker. The hard boundary is the one to an external system, and the same Confluent page is candid: "When writing to an external system, it can be challenging to coordinate the data the consumer is receiving and the consumer's position in the data. Typically, this might be done with a two-phase commit."

Flink states the limit of the guarantee more plainly than most. Per the Flink fault-tolerance documentation, exactly-once does not mean each event is processed once; it means "every event will affect the state being managed by Flink exactly once," achieved by replaying source data on recovery. For that to hold end to end, "your sources must be replayable" and "your sinks must be transactional (or idempotent)." Spark's Structured Streaming documentation describes the identical structure — offset-tracked sources, checkpointing and write-ahead logs recording the offset range per trigger, and sinks "designed to be idempotent for handling reprocessing."

Exactly-once is therefore a property of a whole chain, not a switch on the broker, and it is bought with three costs:

• Latency — the thing you came for. The Flink Kafka connector documentation notes the guarantee "delays record visibility effectively until a checkpoint is written, so adjust the checkpoint duration accordingly." Your latency floor becomes your checkpoint interval. Teams routinely buy streaming for sub-second freshness, configure a thirty-second checkpoint, and never notice they reinstated a small batch window.
• Read-side configuration that is not the default. Per Confluent's consumer configuration reference, isolation.level defaults to read_uncommitted, which returns "all messages, even transactional messages which have been aborted." Setting read_committed limits reads to the last stable offset, and the docs are explicit that "read_committed consumers will not be able to read up to the high watermark when there are in flight transactions." An exactly-once pipeline read by a default-configured consumer is at-least-once with extra machinery and a false sense of security.
• Throughput overhead. Kafka Streams documentation shows the shape of it: when processing.guarantee is exactly_once_v2, the default commit.interval.ms is 100; otherwise it is 30000. A three-hundred-fold increase in commit frequency is not free.

There is also a failure mode where exactly-once is worse than at-least-once, which should settle most of these debates. The Flink connector docs warn that transaction.timeout.ms should be at least the maximum checkpoint duration plus the maximum restart duration, "or data loss may happen when Kafka expires an uncommitted transaction." Misconfigured exactly-once does not degrade into duplicates. It degrades into loss.

So before reaching for transactional machinery, ask whether the sink is naturally idempotent. If the write is an upsert keyed on a business key, duplicates cost nothing — the second write produces the same row. Debezium's documentation takes exactly that position: it guarantees at-least-once, notes that consumers will see events more than once when things go wrong, and expects them to be idempotent. Reserve exactly-once for operations that are genuinely not repeatable — incrementing a counter, appending to a ledger, moving money.

The middle ground almost nobody tries first

The batch-versus-streaming framing hides the option that fits most requirements. Spark's performance documentation puts both ends on one scale: continuous processing "enables low (~1 ms) end-to-end latency with at-least-once fault-tolerance guarantees," compared with "the default micro-batch processing engine which can achieve exactly-once guarantees but achieve latencies of ~100ms at best."

Hold that against a real requirement. If the deadline is five minutes, both numbers are enormous overkill — and so is anything needing a permanent broker cluster. An incremental load on a five- or fifteen-minute trigger keeps the property that made batch good, each run bounded and re-runnable, while collapsing staleness by two orders of magnitude against nightly. It will never be a conference talk. It is very often the correct answer.

The other high-value middle option is change data capture used as a replacement for the nightly extract rather than as an entry into event-driven architecture. Reading a database's transaction log is usually lighter on the source than a repeated full extract, and it makes freshness a property of the pipeline rather than of a cron schedule. It is still a streaming system to operate, but the scope is one well-understood pattern rather than a platform. Our guides on building data pipelines and data engineering in 2026 cover the surrounding ground.

What breaks in year two

First-year streaming projects usually work. The problems arrive later, and they are predictable enough to plan for.

State grows. Stateful windows and stream-to-stream joins hold data proportional to their window and cardinality. Checkpoint size grows with it, and so does restore time. The failure that matters is the one where restarting the job takes longer than the incident you restarted it for.

Schema change becomes a multi-party negotiation. One producer and one consumer is a conversation; one producer and eleven consumers is a governance process, and an enforced compatibility mode stops being optional — the same class of problem covered in why data pipelines break every week, with more parties and less room to fail loudly.

Someone asks what the value was at 3 p.m. last Tuesday. A stream is a moving present tense. Historical questions require that the events landed somewhere queryable — an open table format such as Iceberg or Delta over object storage is the common choice — which means operating a stream and a batch-shaped landing zone. Budget for both up front rather than discovering the second during an audit.

The partition key is now wrong. Requirements moved, the natural unit of work changed, and the ordering guarantee you built on hangs off a key that no longer matches the domain.

A decision procedure you can run this week

1. Write down the decision, the actor, and the deadline. One sentence. If the deadline is not shorter than your current batch window, stop and tune the batch.
2. Measure actual end-to-end lateness from event time to availability, and find where it is spent. It is common for the batch window to be a minor term and the source system's own once-daily export to be the real constraint. Streaming your half of a once-a-day handoff achieves nothing.
3. Classify every sink as naturally idempotent or not. If all are, plan for at-least-once and keep the design small. If one is not, decide whether it can be made so before buying transactions.
4. Price the platform at idle, not at peak — broker or shard hours, storage, partition hours where they apply — against the batch compute you would retire.
5. Name the person paged at 3 a.m. Out loud, in the meeting. If there is no name, do not run a stream.
6. If you still say yes, start with one topic, one producer, one consumer, and schema governance from day one. Enforcement is nearly free at the start and painful to retrofit across a dozen consumers.

Streaming converts a scheduling problem into a distributed-systems problem, and distributed-systems problems are paid for continuously rather than once. When a real deadline demands it, that trade is worth making. When it does not, the nightly batch everyone is slightly embarrassed by is the correct architecture, and saying so is the more senior position. For the vocabulary behind the consistency-versus-availability half of the argument, see the CAP theorem and our distributed systems guide.

Sources: Apache Flink — fault tolerance via state snapshots (exactly-once semantics, replayable sources, transactional sinks); Apache Flink — Kafka connector (DeliveryGuarantee.EXACTLY_ONCE, transaction.timeout.ms, record visibility); Confluent — Kafka message delivery semantics; Confluent — consumer configuration reference (isolation.level, last stable offset); Confluent — Kafka Streams configuration (processing.guarantee, commit.interval.ms); Confluent — Kafka introduction (per-partition ordering, keys); Apache Kafka — KIP-679 (producer defaults acks=all, enable.idempotence=true); Apache Kafka — KAFKA-13598 (idempotence default not applied until 3.2.0); Apache Spark — Structured Streaming fault-tolerance semantics; Apache Spark — continuous processing and micro-batch latency; Debezium — FAQ on delivery and ordering guarantees; AWS — Amazon Kinesis Data Streams pricing (shard hours, 25 KB PUT payload units, per-shard throughput); AWS — Amazon MSK pricing (broker instance hours, storage, serverless partition hours). Analysis and framing by Precision AI Academy.

Common questions

When do I actually need streaming instead of batch? When a decision has a deadline shorter than your batch window, and something acts on the data before the next batch would have run. If you cannot name the decision, the actor, and the deadline, you have a freshness preference rather than a streaming requirement. A useful second test is whether the consumer is a machine — fraud scoring, alerting, autoscaling, model feature serving — or a human who checks a dashboard twice a day.

What does exactly-once actually guarantee? Less than the name suggests. Apache Flink's documentation states that exactly-once does not mean every event is processed exactly once; it means every event will affect the state being managed by Flink exactly once, achieved by replaying source data on recovery. End-to-end exactly-once additionally requires that sources are replayable and sinks are transactional or idempotent. Apache Spark's Structured Streaming documentation describes the same combination: offset-tracked sources, checkpointing and write-ahead logs, and idempotent sinks.

What does exactly-once cost in latency? It reinstates a batch-shaped delay. Flink's Kafka connector documentation notes that the exactly-once delivery guarantee delays record visibility effectively until a checkpoint is written, so the checkpoint interval becomes the end-to-end latency floor. Consumers must also read with isolation.level=read_committed, and per Confluent's consumer configuration reference, read_committed consumers cannot read up to the high watermark when there are in-flight transactions.

Is at-least-once delivery good enough? For most pipelines, yes — if the sink is naturally idempotent. An upsert keyed on a business key absorbs duplicates for free. Debezium's documentation takes this position explicitly: it guarantees at-least-once rather than exactly-once delivery and expects consuming applications to be idempotent. Exactly-once machinery earns its cost when the operation is not naturally repeatable, such as incrementing a counter or triggering an external side effect.