The Redis That Wouldn’t Come Back: Implementing a Cluster Reconnect Strategy
Wherein we write sophisticated retry logic, exponential backoff, and cluster error handlers — then fix everything by changing one environment variable.
Redis cluster unavailability is a normal operational event. Nodes restart, failovers happen, maintenance windows fire at inconvenient times. A well-behaved client handles this transparently: detects the disconnection, waits, reconnects, carries on. Ours did not do this, and the reasons why are instructive.
The Setup
The application connects to an AWS ElastiCache Redis cluster using ioredis in cluster mode. A shared singleton client is initialised on first use and reused across Lambda invocations via module-level caching. The client handles distributed locking, real-time state caching, and atomic multi-key updates via Lua scripts. It is, in other words, not optional.
Connection is initialised with a single seed node — one host and port — from which ioredis discovers the full cluster topology:
REDIS_CLUSTER_ENDPOINT=<host>
REDIS_CLUSTER_PORT=6379
This is a perfectly standard way to connect to a Redis cluster. It is also, it turns out, the problem.
What Was Already There
The codebase had all the right pieces. Retry logic with exponential backoff, capped at 10 seconds, up to 5 attempts. An error event listener on the cluster client that caught connection errors and called disconnect() followed by connect(). A connection state check — client.status === 'ready' — before returning the client to callers. An isConnecting flag to prevent concurrent initialisation attempts from stomping on each other.
The logic looked correct. The code reviewed fine. The reconnection just didn’t work.
Why It Didn’t Work
When ioredis connects to a Redis cluster via individual host and port properties, it resolves the hostname at initialisation time to discover cluster topology. On reconnection after a failure, it attempts to reuse that resolved topology. If the cluster has failed over — new primary, different node configuration — the stored topology is stale and the reconnection fails silently or lands on a node that is no longer primary.
The connection string format (rediss://host:port) changes the behaviour: ioredis re-resolves the endpoint on each reconnection attempt rather than relying on the cached topology from the initial connect. For a managed cluster with automatic failover, this matters: the endpoint doesn’t change, but what it points to does.
The fix was changing from:
REDIS_CLUSTER_ENDPOINT=<host>
REDIS_CLUSTER_PORT=6379
to:
REDIS_CLUSTER_CONNECTION_URL=rediss://<host>:6379
and updating the client initialisation to read the connection string rather than constructing the connection from parts.
That was it. The retry logic, the error handlers, the backoff — all of it was correct and all of it started working once the client was reconnecting to a valid endpoint rather than a stale one.
The Part That Got Missed
The host/port approach works fine when the cluster is stable. In development and early production, clusters are stable. The issue only surfaces when a failover or maintenance event occurs — which, in a managed cluster with automatic failover enabled, happens silently in the background and is supposed to be invisible to the application. The assumption was that it was invisible, because nothing obviously broke. In reality, Lambda invocations that happened to hit the reconnection window were failing or hanging, and the symptom was generic enough that Redis wasn’t the first suspect.
The connection string approach is the documented recommendation for managed Redis clusters precisely because it decouples the client’s reconnection logic from the cluster’s internal topology. The host/port approach is fine for standalone Redis instances where the node doesn’t change. It’s the wrong default for a cluster with failover.
What the Reconnect Strategy Looks Like Now
With the connection string in place, the existing reconnect infrastructure works as intended:
Retry strategy — exponential backoff starting at 1 second, doubling per attempt, capped at 10 seconds. Covers transient unavailability without hammering a cluster that’s mid-failover.
Error handler — cluster-level error listener calls disconnect() then connect(), giving the client a clean reconnection path rather than waiting for ioredis’s internal recovery.
Connection validation — callers check client.status === 'ready' before use. If not ready, a new connection is established, protected by an isConnecting flag to prevent concurrent initialisation races in high-concurrency Lambda environments.
Shared client lifecycle — the singleton is never torn down mid-operation. An earlier pattern of calling client.quit() on the shared instance after individual operations was removed; it was disconnecting the client for every subsequent caller in the same Lambda container, which produced its own class of confusing failures.
Lessons
The right primitives don’t automatically compose correctly. Retry logic, error handlers, and backoff are all necessary. They are not sufficient if the reconnection target itself is wrong.
Managed clusters with automatic failover require endpoint-aware reconnection. The cluster endpoint DNS record is stable even when the underlying nodes change. The connection string approach lets ioredis re-resolve that record on reconnect; the host/port approach does not. For ElastiCache or any other managed cluster, always use the connection string.
Silent failures in shared infrastructure are the hardest to attribute. The Redis client wasn’t throwing errors visible to callers — it was hanging on reconnection and eventually timing out. The symptom presented as slow or failed downstream operations, not as “Redis is not reconnecting.” Time to diagnosis was longer than it needed to be.
Don’t call quit() on a shared client. This one is obvious in retrospect.
Troubleshooting
Reconnection hangs after a cluster failover: Check whether the client is initialised with a connection string or with individual host/port properties. If the latter, switch to the connection string — this is the most likely cause.
client.status stuck in reconnecting indefinitely: The retry strategy has a finite attempt count. If all retries are exhausted without a successful connection, ioredis stops retrying and the client remains in a non-ready state. The error handler’s manual disconnect()/connect() cycle resets this. Verify the error handler is registered on the cluster client, not on an individual node.
Concurrent Lambda invocations initialising multiple clients: The isConnecting flag prevents this only if it’s module-level state shared across the invocation. In Lambda, module-level state persists within a container but not across containers. Multiple containers may each initialise their own client — this is expected and fine. The flag guards against concurrent initialisation within a single container.
TLS handshake failures on reconnect: ElastiCache clusters with in-transit encryption require tls: { rejectUnauthorized: false } in the Redis options, and the rediss:// scheme (not redis://) in the connection string. A connection that works initially but fails on reconnect is sometimes a TLS configuration issue surfacing only after the initial session expires.