Senior & Lead Engineer
Interview Mastery Hub

Why a Single Redis Master is Insufficient for Strict Distributed Locking

In a standard Master-Replica Redis setup, replication is asynchronous. If the master crashes immediately after granting a lock before replicating to replicas, the promoted replica can grant the exact same lock to a second client, causing catastrophic double-execution.

The Redlock Algorithm Workflow

1. Client initializes $N$ independent Redis master nodes (e.g. $N=5$).
2. Client gets current millisecond timestamp $T_1$.
3. Client attempts to acquire the lock sequentially across all $N$ instances using identical key, random unique value, and TTL (e.g. 10s) with a short timeout (e.g. 50ms) per node.
4. Client computes elapsed time $Elapsed = T_2 - T_1$.
5. Lock Acquisition Success: If the client acquired the lock on majority of nodes (>= 3 out of 5) AND $Elapsed < TTL$, the lock is valid for $ValidityTime = TTL - Elapsed$.
6. Failure & Rollback: If client failed to acquire on majority, it immediately sends an unlock script (Lua EVAL) to all instances, even those where it thought acquisition failed.

1. Cache Avalanche (Mass Simultaneous Expiration)

• Problem: Millions of keys expire at the same instant; massive traffic rushes to the DB.
• Solution: Add Random TTL Jitter:

  $ttl = 3600 + rand(0, 600); // 60 to 70 minutes randomized expiration
  

2. Cache Breakdown / Dog-Piling (Hot Key Expiry)

• Problem: A single viral product key expires; 20,000 concurrent requests miss simultaneously.
• Solution: Distributed Mutex Lock or Background Pre-warming / Stale-While-Revalidate.

3. Cache Penetration (Non-Existent Keys Attack)

• Problem: Malicious requests query non-existent IDs (id = -9999), bypassing cache and hitting DB on every request.
• Solution:
  1. Cache Empty Objects / Nulls with short TTL (e.g. 60 seconds).
  2. Bloom Filter: Use Redis Bloom filter to determine in $O(1)$ whether an ID could possibly exist before querying database.

Comparison Matrix

Sliding Window Lua Script in Redis

local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])

-- Remove timestamps older than current sliding window
redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
local current_count = redis.call('ZCARD', key)

if current_count < limit then
    redis.call('ZADD', key, now, now)
    redis.call('EXPIRE', key, window)
    return 1 -- Allowed
else
    return 0 -- Blocked (HTTP 429)
end

Architectural Blueprint

[Client Socket] ---> [Load Balancer / Nginx] ---> [WebSocket Gateway Cluster (Node/Go/Reverb)]
                                                          ^
                                                          | Redis Pub/Sub Cluster / Kafka
                                                          |
                                                 [Notification Service]

Core Architecture Components:

1. Stateful Connection Nodes: Gateway instances maintain persistent TCP socket connections and keep a local in-memory lookup map of user_id -> [socket_fd_1, socket_fd_2].
2. Channel Partitioning in Redis Pub/Sub: Broadcast messages are partitioned by tenant or topic channels (broadcast:global vs user:12345).
3. Backpressure & Drop Policies: If a client's TCP socket buffer is saturated, drop non-critical status updates to prevent memory exhaustion on the gateway node.

Choreography vs Orchestration Saga

Because 2-Phase Commit (2PC) blocks distributed databases, distributed systems use Saga Pattern:

• A sequence of local transactions where each step publishes an event.
• If any step fails, the Saga executes Compensating Transactions in reverse order (e.g. RefundPayment, RestoreInventory).

[Order Created] -> (Deduct Inventory) -> (Charge Stripe) -> [Invoice Failed]
                                                |
                                                v
                              [Trigger Compensating Action]
                                       Refund Stripe
                                     Restore Inventory
                                    Mark Order Cancelled

Idempotency Keys:

Each step checks if the order_id has already been processed to safely allow retries without duplicate charges.

The 3 States of Circuit Breaker

1. Closed: Normal state. Requests pass through. Tracks failure rate.
2. Open: If failure rate exceeds threshold (e.g. 50% over 10s), the circuit trips to OPEN. Requests fail immediately with a fallback response without calling the slow downstream service.
3. Half-Open: After a cooldown period (e.g. 30s), a limited trial number of requests are allowed through. If they succeed, circuit resets to CLOSED; if they fail, it returns to OPEN.

Key Architecture Decisions

1. Base62 Encoding: [a-z, A-Z, 0-9]. A 7-character string allows $62^7 \approx 3.5 \text{ trillion}$ unique URLs.
2. Distributed ID Generator (Snowflake / Redis Atomic Counters): Generate unique 64-bit auto-incrementing integer -> Convert to Base62.
3. Caching & CDN:
   • Return HTTP 302 (Temporary Redirect) if click tracking analytics are required on every hit.
   • Return HTTP 301 (Permanent Redirect) if latency is prioritized (browser caches redirect permanently).
   • Cache hot 20% of URLs in Redis cluster (80/20 Pareto principle).

CAP Theorem: Consistency, Availability, Partition Tolerance

In the presence of a Network Partition ($P$), a distributed system must choose between:

• $CP$ (Consistency): Reject writes to maintain data accuracy (e.g. Redis Cluster, MongoDB, HBase).
• $AP$ (Availability): Accept writes on both sides of partition; resolve conflicts later (e.g. DynamoDB, Cassandra).

PACELC Theorem (Extending CAP for Normal Operations)

If Partition ($P$): Choose between Availability ($A$) or Consistency ($C$). Else ($E$): Choose between Latency ($L$) or Consistency ($C$).

Exponential Backoff with Jitter

$retryDelay = min(3600, pow(2, $attemptCount) * 10 + rand(0, 5));

• Attempt 1: 20s
• Attempt 2: 40s
• Attempt 3: 80s
• Max Retries Exceeded: Route the failed payload to a Dead Letter Queue (DLQ) for automated alerting and manual developer replay.

Sharded Counters / Write-Back Buffering

1. Redis Atomic Increment:

   Redis::incr("video:views:{$videoId}");
   

2. Batch Flusher Daemon: Every 10 seconds, a background worker consumes the aggregated count from Redis via INCRBY and writes a single consolidated SQL update:

   UPDATE videos SET views = views + 5000 WHERE id = 5;
   

Reduces database write operations by 99.9%.

Consistent Hashing Ring

1. Maps both Cache Servers and Keys onto a 360-degree circular hash ring (e.g. 0 to $2^{32}-1$).
2. A key is assigned to the nearest server located clockwise on the ring.
3. When a server is added or removed, only $K/N$ keys need remapping, preventing total cache thrashing.
4. Virtual Nodes: Assigns multiple virtual points per physical server to ensure even distribution across nodes.

Protocol Comparison

Active-Active vs Active-Passive

• Active-Passive (Warm Standby): Primary region handles 100% traffic. Secondary region replicates data. DNS failover (Route 53) flips traffic if primary fails. RTO: 2-5 mins.
• Active-Active (Multi-Region): Both US-East and EU-Central serve local traffic. Requires globally distributed conflict-free replicated data types (CRDTs) or distributed database like CockroachDB / Spanner.

Distributed Context Propagation

1. The edge API gateway generates a unique trace_id and span_id.
2. Passed downstream across all HTTP/gRPC headers via W3C Trace Context (traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01).
3. Each microservice logs the trace_id and exports span metrics to OpenTelemetry collector / Jaeger for visual timeline waterfalls.

Mutual TLS (mTLS) with Service Mesh (Istio / Linkerd)

• Both client and server authenticate each other using X.509 cryptographic certificates rotated automatically by the service mesh control plane.
• All internal pod-to-pod network traffic is encrypted with TLS 1.3.
• Combined with short-lived JWT scoped tokens to verify service-level authorization.

Redis Sorted Sets ($O(\log N)$ Complexity)

# Add / update player score
ZADD leaderboard 4500 "user_9821"

# Get user current global rank (0-indexed)
ZREVRANK leaderboard "user_9821"

# Fetch top 10 players with scores
ZREVRANGE leaderboard 0 9 WITHSCORES

Internal SkipList data structure delivers sub-millisecond lookups across millions of elements.

Connection Pooling with ProxySQL / PgBouncer

• Without connection pooling: $400 \text{ pods} \times 20 \text{ connections} = 8,000 \text{ open DB connections}$, overwhelming database thread management.
• With Multiplexing Proxy: 8,000 application client connections multiplex over 50 persistent, high-speed connections to MySQL, keeping DB CPU in the sweet spot.

Caching Strategies

1. Cache-Aside (Lazy Loading): Application reads cache; on miss, reads DB and writes to cache. Standard for most web apps.
2. Write-Through: Application writes to cache, and cache synchronously writes to DB before returning. Consistency guaranteed, higher write latency.
3. Write-Back (Write-Behind): Application writes to cache immediately; cache asynchronously flushes batch writes to DB in the background. High write performance, small risk of data loss if cache crashes before sync.

HMAC-Signed URLs with Expiration & IP Binding

1. Backend generates an HMAC-signed URL containing: path, expires_timestamp, and optional client_ip.
2. Cloudflare / CloudFront edge worker validates the signature in $O(1)$ at the edge node:

   https://cdn.example.com/video/master.m3u8?token=a8f9c1...&exp=1772189400
   

3. If token expired or tampered, edge rejects with HTTP 403 without touching origin servers.

Change Data Capture (CDC) via Debezium & Kafka

Instead of dual-writing in application code (which risks desynchronization if search fails):

1. Database writes to transaction WAL (Binlog).
2. Debezium CDC connector tails the Binlog and streams row change events to Kafka.
3. An indexing consumer batch writes updates into Elasticsearch / Meilisearch. Guarantees 100% data consistency with sub-second synchronization latency.