tanujtyagi.com

Migrating from ElastiCache to Redis Enterprise: A Comprehensive Guide

Import and export for a maintenance window, RIOT for near-zero downtime. But the data copy is the easy half; the client-side assumptions are what decide the cutover.

Moving Redis workloads off ElastiCache is usually straightforward. Getting the data across is a solved problem with two well-worn paths, and neither is difficult.

What tends to catch teams out is everything after that. Redis Enterprise shards data across nodes, and if the application was written against a single ElastiCache node with cluster mode off, some of its query patterns will stop working the moment it points at the new endpoint. That part deserves more planning than the copy does, so it gets the second half of this post.

Option one: export and import

Good for small datasets, or anything that can take a maintenance window. It is boring in the way you want a migration to be.

Getting the data out

Use AWS's own backup and export rather than pulling from a live node. Create a manual backup of the ElastiCache cluster, then export it to an S3 bucket in the same region. AWS writes one or more RDB files and you never touch production.

If you would rather pull the snapshot directly:

redis-cli -h <elasticache-endpoint> --rdb dump.rdb

Be aware that this issues a replication command against the source, so point it at a replica rather than the primary.

Getting the data in

Use Redis Enterprise's built-in import. In the admin console, open the target database, choose Import, and point it at your source: S3, FTP or SFTP, a mounted volume, or a URL. Over the REST API it is POST /v1/bdbs/<uid>/actions/import.

Two things to know before you run it. Import replaces the contents of the target database, so import into a fresh one if you want the step to be reversible. And it is asynchronous, with a duration that depends on dataset size and the bandwidth between source and cluster, so poll the action rather than assuming it finished.

One thing that does not work, and that people try because it works on Redis OSS: copying an RDB file into a node's data directory. Redis Enterprise manages its own storage layout and will not pick up a hand-placed file.

Checking it worked

DBSIZE on both sides is the first sanity check. After that, sample some keys and confirm their TTLs came across, since expiry behaviour is where inconsistencies tend to hide:

redis-cli -h <endpoint> DBSIZE
redis-cli -h <endpoint> --scan --count 1000 | head -100

Option two: live migration with RIOT

When you cannot take the downtime, the Redis Input/Output Tool does an initial snapshot and then keeps the target in sync with the source until you cut over.

Create the target cluster and database first, and confirm the RIOT host can reach both endpoints. Install RIOT somewhere with good network access to both sides, following the install guide. Put it near the source if you have a choice, because the initial pass is bandwidth-bound.

Before anything else, enable keyspace notifications on the source. RIOT's live mode subscribes to them to pick up ongoing changes, and without them the initial snapshot will succeed and then nothing further will replicate, which is a confusing failure to debug. On ElastiCache that means setting notify-keyspace-events in the parameter group.

A snapshot copy is the default:

riot replicate redis://<elasticache-endpoint>:6379 redis://<redis-enterprise-endpoint>:6379

Adding --mode live keeps it streaming changes after the first pass:

riot replicate --mode live \
  redis://<elasticache-endpoint>:6379 \
  redis://<redis-enterprise-endpoint>:6379 \
  --threads 4 \
  --batch 500

Tune --threads and --batch to the dataset. There are --key-pattern and --key-type filters if you are only moving a subset, which is often worth doing to leave genuinely disposable cache data behind rather than paying to copy it.

Then let replication settle, verify consistency, repoint the application, watch it for a while, and only stop RIOT once you are confident. Keeping the source alive and replicating for a day after cutover costs very little and gives you somewhere to fall back to.

The part that actually needs planning

Redis Enterprise can run with Cluster Mode Enabled, sharding the keyspace across nodes. That is what you want for scale, and it also means any command touching several keys requires those keys to be in the same hash slot. Miss that and you get:

CROSSSLOT Keys in request don't hash to the same slot

Hash tags are the fix. Whatever appears inside braces determines the slot, so tagging related keys puts them on the same shard:

{user:123}:profile
{user:123}:settings

Choose the tag carefully, because you are making a permanent co-location decision. Tag on something with even distribution. Tagging on a tenant id is fine with thousands of comparable tenants and painful when one of them is most of your traffic.

Where the pattern was really about saving round trips rather than atomicity, pipelining does the job and works across shards with a cluster-aware client:

Pipeline pipeline = jedis.pipelined();
pipeline.get("key1");
pipeline.get("key2");
List<Object> results = pipeline.syncAndReturnAll();

It is also worth heading off a common assumption here: Lua scripts do not get you around the slot rule. A script is routed by the keys declared in KEYS, so if those span slots, EVAL fails with the same CROSSSLOT error. Lua gives you atomicity for multi-step logic on keys that already share a slot, which is useful, but it does not extend your reach across shards.

Often the better answer is to stop splitting the data. If several keys are always read together, store them as one Hash or one JSON document and the problem disappears rather than getting worked around.

Finally, use a cluster-aware client. This is a one-line change that people forget until the first resharding event:

JedisCluster jedisCluster = new JedisCluster(new HostAndPort("redis-cluster-endpoint", 6379));

Before you cut over

Size a connection pool to your workload instead of opening connections per request. Set explicit command and connection timeouts, make sure the client retries idempotent operations through a failover, and put a circuit breaker in front of Redis so a slow cluster degrades your application instead of hanging it.

Wire up monitoring before the migration rather than after. Redis Enterprise reports throughput, memory and latency, and those feed into Prometheus or Datadog easily enough. The reason to do it early is baselines: if you only start measuring after cutover, you have nothing to compare against when somebody asks whether the new cluster is slower.

Where the risk really sits

I have seen the data copy go wrong perhaps once. I have seen the client-side assumptions go wrong many times, usually as a CROSSSLOT error in an endpoint nobody tested because it was not on the critical path.

So audit the application for multi-key commands before you schedule anything. Get the hash tags decided, run the whole test suite against a small Redis Enterprise database with cluster mode on, and treat that as the real go/no-go rather than the snapshot restore. And know your rollback path, in writing, before the day.

If you are planning one of these and want a second opinion on the approach, I am happy to talk it through.