Keeping Redis Honest Against a System of Record: RDI Architecture and When to Use It

Most Redis caches start honest and drift. Someone writes to the database directly. A TTL is longer than anyone remembers agreeing to. A batch job updates a table nobody wired into the invalidation path. And now your cache is confidently serving something that stopped being true an hour ago.

Redis Data Integration exists to take that problem out of your application code. It watches the source database's change log and applies those changes to Redis for you, so the cache stays close to the system of record without anybody having to remember to invalidate anything.

That much is easy to find. What is harder to find written down is how RDI is actually built, and which workloads it is the wrong answer for. The second list is longer and more specific than most people expect, which is why it gets most of the space below.

What it is

RDI implements change data capture. It tracks changes in a non-Redis source database and makes corresponding changes in a Redis target. You define a dataset, meaning which tables and columns to capture, and how you want them represented in Redis. The target shape does not have to mirror the source, because RDI applies transformations after capture, so relational rows can arrive as hashes or JSON documents in whatever form your application actually wants.

Two properties frame everything else. It is one-directional, and it is a cache. The source stays the system of record, your application keeps writing there, and Redis is downstream.

The architecture

Here is the whole thing on one page. Everything after this is an explanation of a box in it.

The data path is three steps. A CDC collector captures changes from the source, using Debezium via Debezium Server connectors. The collector writes those changes into Redis streams held in RDI's own Redis database. Then a stream processor reads the streams, applies your transformations, and writes to the target.

The lifecycle has two phases. On first start the target is empty, so everything in the source counts as change data, and that initial cache loading takes minutes or hours depending on volume. Once the snapshot lands, RDI switches automatically to change streaming, where changes reach the target within a few seconds of being captured.

Three planes

RDI splits its processes into three planes, and the distinction matters because they fail differently. Lose the data plane and replication stops. Lose the control plane and you are flying blind while data keeps moving.

The control plane is an API server exposing REST for observation and control, an operator managing the data plane processes, and a metrics exporter that surfaces RDI metrics as Prometheus metrics. The data plane is the CDC collector and the stream processor, the parts that actually move data. The management plane is the CLI for install and pipeline administration, plus the pipeline editor built into Redis Insight.

Where it runs

This is the detail most diagrams get wrong, and it affects sizing. RDI's control processes run on dedicated VMs outside the Redis Enterprise cluster, but RDI keeps its state, its configuration and the change streams themselves in a Redis database on the same cluster as the target.

So RDI is neither a sidecar to your target database nor fully external to it. Budget for both the VMs and the memory that the RDI database consumes on your cluster.

On your own VMs you provide two. Collector and stream processor are active on one and standby on the other, with the two operators running a leader election to decide which is active. On Kubernetes there is a Helm chart, which creates an rdi namespace, deployments and services for the operator, metrics exporter and API server, a service account with RBAC, a ConfigMap for the RDI database details, and secrets for credentials and TLS certificates.

Classic or Flink

There are two stream processor implementations, selected per pipeline through processors.type in config.yaml:

processors:
  type: classic   # or: flink

classic is the default, so existing pipelines keep their behaviour. Mostly worth knowing the choice exists before somebody asks about it in a design session.

Component by component

Source database

Your existing relational or document database, and the system of record. Oracle, PostgreSQL, MySQL, SQL Server, MongoDB and others.

RDI does not query it on a schedule. It reads the database's own change log, which is the WAL on PostgreSQL, the binlog on MySQL, LogMiner on Oracle. That is why CDC is cheap on the source compared with polling, and also why it needs privileges and log-retention settings that a DBA has to grant.

Which brings the practical consequence: the source has to be configured for CDC before anything works at all. In my experience that configuration is the single most common reason a proof of concept slips. Not the pipeline, the ticket to enable logical replication.

CDC collector

Debezium, running as a Debezium Server connector. Open source, widely deployed, and not written by Redis, which is worth knowing because it means the failure modes are well-documented elsewhere.

It tails the source change log and turns each committed row change into an event. On first run there is nothing in the target, so it treats the entire source as change data and produces a full snapshot, then switches automatically to change streaming when that finishes.

What breaks it is long, large transactions. A batch job rewriting millions of rows inside one transaction is the documented failure case, and RDI will fail processing those changes rather than merely lagging behind them.

RDI database

A Redis database that RDI uses for itself, holding the change event streams plus RDI's own state and configuration.

It lives on the same Redis Enterprise cluster as your target database, not on the RDI VMs. That is worth repeating because it means RDI consumes memory on the cluster you are sizing for your application, and that consumption grows when the pipeline backs up.

Streams are the right structure for this because they give ordered, replayable event storage with consumer tracking, which is exactly what a checkpointing processor needs in order to note its position and resume after a crash. It also makes backpressure observable: if the streams are growing, the processor is not keeping up.

Stream processor

The component that reads events from the streams, applies your transformations and writes to the target. It reads in order, transforms each record from the source's shape into whatever you defined for Redis, writes it, and records a checkpoint marking the last event it durably stored. If it dies it resumes from that checkpoint and may reprocess events, which is where at-least-once delivery comes from.

It also decides your target data model, and that is the part worth spending real design time on, because the transformation is where a relational row becomes something your application can read in one operation instead of three.

Target Redis database

The database your application reads from, where rows arrive as hashes or JSON documents in the shape you specified.

The rule is that your application reads here and writes to the source. RDI never carries writes in the other direction. If somebody proposes writing to Redis and letting RDI push those writes back to the database, they are describing write-behind, which is a different pattern that RDI does not implement.

Control and management processes

None of these touch your data. They keep the pipeline running and let you see it.

Component	What it is	What it does	If it fails
Operator	Supervisor process	Manages data plane processes; leader election between the two VMs	Failover is not clean; data plane unmanaged
API server	REST endpoint	Observe and control RDI programmatically	You lose automation, not replication
Metrics exporter	Prometheus exporter	Exposes RDI metrics for scraping	You go blind while the pipeline keeps running
CLI	Command-line tool	Install, upgrade, deploy, manage pipelines	Nothing at runtime
Redis Insight	GUI	Visual pipeline editor	Nothing at runtime

The asymmetry is useful when you set up alerting. Control plane failures cost you visibility, data plane failures cost you freshness. Alert on both and page on the second.

The guarantees

RDI guarantees at-least-once delivery. A change will never be lost, but it may be applied to the target more than once. That is fine in normal use because the writes are idempotent, so every write after the first makes no difference to the final state, and the cost is a small performance overhead rather than a correctness problem.

The word idempotent is doing real work in that sentence, though. It holds because RDI writes whole keys. If you were tempted to add a transformation that increments a counter, you would be building something that at-least-once delivery quietly breaks, and it would break in a way that looks like a slow drift rather than an error.

Checkpointing is what produces that guarantee. The stream processor tracks the last event it successfully processed and stored, and on restart it resumes from there and reprocesses anything that might not have landed. Duplicates are therefore the expected failure mode, and gaps are not.

Backpressure handles the case where change records arrive faster than RDI can process them, whether from a slow target, a disconnection, or simply a burst of writes at the source. Left alone the streams would consume all available memory, so RDI detects the condition and holds change data at the source until it has cleared the backlog.

One operational note here will save you a support ticket. The Debezium log sometimes reports that RDI has run out of memory, usually during the initial snapshot, and that is not an error. It is an informational message telling you backpressure engaged. Worth knowing before somebody raises a priority ticket about it at 2am.

Finally, the consistency model is eventual, measured in seconds. Changes land in the target within a few seconds of capture, and every fit decision below follows from that one sentence.

When it fits

The criteria, with the ones that actually decide it marked:

• Your app or microservices read from Redis to scale reads at speed.
• You are pulling from a single source database.
• You must use a slow, disk-based database as the system of record.
• The app always writes to that source database, never to Redis.
• The app can tolerate eventual consistency in the cache.
• Source data changes frequently, in small increments.
• You want a self-managed or AWS-based solution.
• Your caching needs are too complex to hand-roll and maintain.
• Your DBA has reviewed RDI's requirements against the source and accepted them.

That last one is not a formality. RDI needs CDC enabled on the source, which means log retention settings, a replication slot or equivalent, and a user with the right privileges. If your DBA will not grant them then this architecture is finished, so settle that question before you commit to a delivery date rather than after.

When it does not fit

The more useful list.

A one-time migration is the wrong job for RDI; use RIOT instead. Anything needing immediate consistency or a hard latency bound is out, because eventual means eventual. Transactional consistency between source and target is not on offer at all.

Write-behind and write-through patterns, where the application writes to Redis and Redis updates the source, are a different architecture entirely and RDI is one-directional.

A permanently small dataset will not repay the operational overhead.

Batch and ETL sources with long, large transactions are the case where the documentation is blunt: RDI will fail processing these changes. If your source is fed by an overnight ETL that rewrites millions of rows in a single transaction, stop here.

Complex stream processing with aggregations, sliding windows or custom logic belongs in a different layer. Multiple targets from one pipeline is not supported, and replicaOf is the way to fan out between Redis databases instead. Active-Active is not supported as an RDI Cloud target topology.

And joins into nested JSON, meaning denormalising one-to-many relationships across several tables into one nested document, is not something to assume works. Test your specific case.

Numbers worth keeping on a card

The throughput envelope, assuming roughly 1KB average records and a pipeline without transformations:

Limit	Value
Source change rate	≤ 20K changes/sec
Full sync throughput	< 60K records/sec
CDC throughput	< 20K records/sec
Total data size for a sub-hour full sync	≤ 200 GB

RDI can ingest more than 200 GB. It just will not finish the initial load inside an hour, which matters for cutover planning far more than for steady state. That is the number I have seen surprise people most often.

Transformations move all of these figures, and the table assumes you are not doing any.

Redis Cloud specifics

If the target is Redis Cloud rather than self-managed Software, the constraints tighten considerably, and these are worth having to hand before a design conversation.

The target has to be a Redis Cloud Pro database on AWS, since Essentials does not support RDI and neither does Redis Cloud on Google Cloud. It must have high availability enabled, single-zone or multi-zone. It can use TLS but not mutual TLS. One source database syncs to one target database, with no fan-in or fan-out.

The source must be publicly accessible or on AWS EC2, RDS or Aurora, and MongoDB Atlas and Snowflake sources have to be on AWS. Private connectivity is AWS PrivateLink only, not VPC peering and not anything else. The instance hosting the database has to be created with a custom AWS KMS key. And mTLS is not supported for RDS or Aurora sources.

On security, source credentials and TLS secrets live in AWS Secrets Manager and are shared via the Kubernetes CSI secrets driver. Connections to the source use JDBC over PrivateLink, so the pipeline is only exposed to that specific database endpoint, and all network connections are TLS-encrypted.

One more operational detail that catches people: RDI Cloud maintenance follows your subscription-wide maintenance window, and the pipeline may be briefly interrupted while updates apply. If you care when that happens, set a manual window, which covers both databases and the pipeline.

Supported sources

Oracle 19c, 21c and 23ai via LogMiner. MySQL, MariaDB, PostgreSQL 10 through 18, SQL Server 2017 to 2022, MongoDB, Supabase, AWS Aurora PostgreSQL, AlloyDB, Neon, Spanner, and Snowflake in preview.

Version support varies across self-managed, AWS RDS and GCP SQL, and the columns do not always match. Check the matrix for your specific combination rather than assuming your version is covered.

Failure modes to plan for

The documentation covers what RDI guarantees. These are the things I would put on a risk list before committing.

Schema changes at the source are the first one. Your pipeline is defined against specific tables and columns, so a migration that renames or drops one is a change nobody told the pipeline about. Find out who owns source schema changes and whether they know a pipeline is watching, because this is an organisational problem as much as a technical one.

Drift you cannot see is the second. At-least-once plus idempotent writes means the target should converge, but should is not monitoring. Decide now how you would detect a divergence, whether that is a periodic row-count comparison or checksums over a sample. Nobody builds this until after the first incident.

Initial sync duration on the real dataset is the third, and it is the most avoidable. People size a proof of concept on 5 GB and then deploy against 400 GB. Measure the full sync against production volume before you commit to a cutover window.

The fourth is a conversation rather than a technical risk: eventual consistency with whoever owns the read path. A few seconds is fine for a product catalogue and not fine for a balance check. That is a product decision and it should be made explicitly, not discovered in UAT.

Where this leaves you

RDI answers a narrow and common question well. My system of record is too slow for my read volume, and I do not want to hand-write cache invalidation. Inside those bounds it is less work than the alternative and the guarantees are clearly stated.

Outside them it fails in specific, documented ways: one-time migrations, batch ETL sources, write-through patterns, Active-Active targets, anything needing transactional consistency. Knowing that list in advance is the difference between a design that holds and one that unravels in UAT.

One question settles it faster than any of the above. Does your application write to the database, and only to the database? If the answer is anything other than a clean yes, resolve that before you look at pipelines at all.