A practical lesson on append-only streams, retention, replay, ordered partitions, and why durable history changes event-driven architecture significantly.
Event streams treat history as a first-class architectural asset. Instead of delivering a message once and forgetting it, a stream keeps an append-only log of events for some retention window or indefinitely. That changes how teams think about consumers, recovery, projections, and correctness. A retained stream is not merely a transport channel. It is part of the system’s long-lived memory.
Streams are especially powerful when consumers need replay, when projections may need rebuilding, or when derived systems depend on ordered per-key history. But that power comes with stronger contract responsibility. A weakly modeled event that lives for seconds in a transient queue is one thing. A weakly modeled event that remains replayable for months is much more expensive.
flowchart LR
A["Producer"] --> B["Append-only stream"]
B --> C["Consumer reads current flow"]
B --> D["Consumer replays from earlier offset"]
B --> E["Projection rebuild"]
What to notice:
In stream-based systems, new events are appended rather than old ones being rewritten in place. This makes the system good at preserving the sequence of change over time. It also makes streams especially useful for audit-like reconstruction, event sourcing, and downstream read-model rebuilding.
Append-only design does not mean events never need correction. It means correction usually appears as new events or controlled replay logic rather than by pretending the older record never existed.
Retention determines how long the platform keeps historical events. Replay allows consumers to re-read those events from an earlier point. Together, these two features make streams different from transient messaging. Consumers can rebuild projections, recover from bugs, or backfill analytics by walking historical flow again.
That capability is powerful, but it creates risk. Some consumers are safe to replay. Others produce external side effects and should not simply re-run against old data without control.
1stream:
2 name: order-events
3 retentionHours: 168
4 partitions: 6
5 replaySupported: true
The snippet highlights the kinds of properties that matter operationally. A stream is more than a topic name. Its retention and partition model define how consumers should think.
Streams often provide ordering within partitions rather than one global order across the whole system. That is still useful. If the partition key matches the entity whose order matters, consumers can reason safely about one order lifecycle, one account lifecycle, or one tenant’s flow without requiring impossible global guarantees.
This is another reason streams shape architecture: teams must think explicitly about partition keys, replay scope, and the unit of consistency that matters.
Streams change architecture because they make events durable beyond the first delivery. This affects:
A transient queue can often be treated mainly as workload plumbing. A durable stream becomes part of the domain and operational model.
A team wants to store all customer-affecting workflow events in a retained stream, but one consumer sends external partner calls with no replay guard. What should you challenge?
Challenge replay safety first. If the stream becomes part of the recovery and rebuild story, then consumers that perform external side effects need explicit controls so replay does not resend actions blindly.