The pod restarted.

The collector came back up, picked up its last recorded position, and requested the Oracle redo history starting from a historical SCN. It needed to resume from the exact point where it had stopped, without missing changes that had happened while it was down.

Oracle returned an error: missing logfile.

The collector knew where it needed to continue from. The database simply no longer had the log it needed.

This is an interesting kind of failure because nothing necessarily went wrong with the database itself. Oracle had been doing its job. The application could continue writing data, transactions could commit, and the database could remain perfectly healthy. But the CDC pipeline had a different requirement: it needed access to the history of changes that had already happened.

The distinction is easy to miss when the system is working. A collector reads changes, publishes them to Kafka, and downstream consumers process them. It can feel as though the database is simply another source of events.

But the database was never designed around the assumption that an external consumer might return days later and ask to replay an arbitrary point in its history.

The integration worked until the collector needed something that the database was no longer obligated to retain.

And that made me think about what a database log actually represents, and why treating every log as a durable event stream can lead to some fairly fundamental architectural mistakes.

Two systems, two different relationships with the log

To understand the problem, it helps to look at how two different storage architectures use their logs.

The B-tree: the log supports the database

Traditional relational databases commonly use B-tree indexes and data pages. When a transaction modifies a record, the database must eventually persist the changed pages to disk. Updating pages in place can involve random writes, page contention, and write amplification.

The write-ahead log, or WAL, helps make this safe. The database records the changes in its log before the corresponding data pages need to be persisted. If the system crashes, it can use the log to recover changes that had been committed but were not fully reflected in the data files.

Once the relevant changes have been made durable in the database files and the necessary recovery requirements have been satisfied, older log records can become eligible for reuse or removal, subject to the database's other retention requirements.

This is the important part: the log has a lifecycle governed by the database's own recovery and operational requirements.

Oracle's redo logs serve this purpose. Checkpoints, archival, log switching, and recovery requirements all influence when redo can be reused. Archived redo may remain available for recovery, but that does not mean every historical log is retained indefinitely.

The database is responsible for preserving enough history to satisfy its configured guarantees. It is not automatically responsible for preserving every change for every external consumer.

The LSM-tree: the log is part of a different write strategy

Log-structured merge-tree architectures approach writes differently. Instead of repeatedly updating data pages in place, they generally write changes sequentially and accumulate them in memory before flushing them into immutable, sorted files.

This makes the write path efficient for workloads where sequential writes are preferable to frequent random updates.

Here, the log is closely tied to the write path itself. It records recent changes so that acknowledged writes can survive a crash, while the durable sorted files hold the persisted representation of the data.

It is tempting to conclude that an LSM log is therefore permanent. It isn't. Once changes have been safely flushed and the relevant durability requirements have been met, the system can also discard older log segments.

The difference is not that one architecture keeps its log forever and the other deletes it. Both can reclaim logs.

The difference is the role the log plays in each system, and the guarantees made to the components consuming it.

A Kafka topic, for example, is explicitly designed to retain an ordered sequence of records according to configured retention or compaction policies. Consumers can fall behind, restart, and replay records as long as the required history remains available.

A database redo log has no equivalent promise to an arbitrary CDC consumer unless the surrounding configuration and operational design establish one.

The integration boundary is where things get interesting

CDC systems sit between these two worlds.

On one side, there is a transactional database that maintains its own data and manages its own recovery history. On the other, there is a streaming pipeline that expects to consume a sequence of changes, recover its position after a failure, and continue without gaps.

The CDC collector translates database changes into events that the rest of the system can consume. It uses a position such as an SCN to identify where it needs to resume reading Oracle redo.

The SCN is a useful anchor, but it is not a guarantee that the redo required to reconstruct changes from that point will still exist.

Consider what happens when the collector falls behind.

The database continues processing transactions. Redo is generated, logs switch, archived logs accumulate, and eventually older logs may be removed or become unavailable according to the configured retention and storage policies.

Meanwhile, the collector is still trying to catch up.

If the required redo disappears before the collector reads it, the pipeline has lost the ability to reconstruct the missing changes from its source history. Restarting the pod, increasing its throughput, or retrying the same SCN does not bring that history back.

The problem is not simply that the collector is slow. It is that the system has allowed the producer's history to expire before the consumer has finished reading it.

This is where a CDC integration can accidentally turn a recovery log into a virtual commit log without establishing the retention guarantees that a real streaming log would require.

The interface looks like a stream. The underlying contract is different.

What should the architecture do about it?

There are two reasonable approaches, depending on how much responsibility you want the source database to carry.

1. Retain the history until the consumer is done

The first approach is to make redo retention aware of the CDC consumer's progress.

The system tracks the oldest SCN that the collector might still need. Retention and archival policies must ensure that the redo covering that position remains available until the collector has safely consumed it.

This is more than configuring Oracle to retain archived logs for a few days. A fixed retention period is a time-based assumption about how long recovery will take. A consumer watermark is a statement about what history is still required.

If the collector falls behind because of an outage, the retention policy must account for that lag. If the source cannot safely preserve the required history, the system needs to raise an alert before the collector reaches a point where recovery becomes impossible.

There are practical complications. The consumer's checkpoint must be trustworthy, the relevant logs must actually be retained and accessible, and the system must account for the distinction between a log being generated, archived, available, and safe to remove. Oracle-specific recovery and archiving behaviour also matters.

The benefit is that the source database remains the place from which changes are read, without introducing another durable storage layer.

The cost is that the CDC pipeline now depends on coordinating its progress with the source database's log lifecycle. The database must retain history for a consumer that may have very different availability and recovery requirements from the database itself.

2. Decouple the source from the consumer

The second approach is to introduce a durable log boundary as early as possible.

Instead of expecting the CDC collector to remain continuously caught up with Oracle, the architecture captures the required redo changes and writes them into a durable, replayable log such as Kafka. Downstream processing then consumes that log independently of the source database.

The idea is straightforward: separate the time at which a change must be captured from the time at which it must be processed.

If the downstream consumer is unavailable for a day, Kafka can retain the events while the consumer recovers. If a downstream transformation needs to be rebuilt, the retained history can provide an opportunity to replay the changes.

The source database no longer needs to retain its redo indefinitely just because a downstream consumer is temporarily unavailable.

However, this only works if the decoupling boundary is real. The CDC connector must still read the necessary Oracle redo and successfully persist the corresponding events into Kafka before that source history becomes unavailable. Kafka cannot recover changes that were never captured.

The design therefore needs a clear durability contract: when is a change considered safely captured, how is the source position coordinated with the published records, and what happens if the collector fails between reading redo and committing its progress?

The broker moves the retention boundary; it does not eliminate the need for one.

The part that matters most

Neither approach is universally better. A well-managed source-retention strategy can be sufficient for a pipeline with modest recovery requirements. A durable broker is often more attractive when several independent consumers need the same history, processing can lag significantly, or downstream systems need replay capabilities.

The important decision is where the system guarantees that history will remain available.

If the database is expected to provide that guarantee, retention must account for the slowest relevant consumer and its recovery requirements. If the streaming platform is expected to provide it, the architecture must capture changes into that platform before the source's retention window closes.

What cannot work reliably is leaving that boundary implicit and hoping the collector stays ahead of log reclamation.

This is also why I think throughput and recoverability have to be considered together. A pipeline that processes changes quickly under normal conditions can still be fragile if a short outage causes it to lose access to the history it needs. Average lag is not enough to establish that the architecture is safe.

The relevant question is not just how quickly the system processes changes, but how much time it can afford to stop processing them before recovery becomes impossible.

A streaming architecture is only as replayable as its earliest point of guaranteed history.

And if that point is still controlled by a database that was never configured to retain the required logs for the consumer, then the architecture is relying on a guarantee it doesn't actually have.