Defeating Cache Stampedes with Distributed Locking at UCBOne

When building the Backend-Driven UI (BDUI) engine for UCBOne, I encountered a massive infrastructural challenge. The engine dictates the entire user experience—rendering dynamic JSON schemas for iOS, Android, and Web clients.

Because millions of users request these layouts simultaneously (e.g., during a marketing push or a sudden outage), caching is mandatory. But what happens when the cache expires under peak load?

You get a Cache Stampede (or “Thundering Herd”).

graph TD
    Reqs(["🔴 5,000 Concurrent Requests"])
    Reqs --> Lock{{Redis ADD / SETNX Lock}}

    Lock -->|"✅ 1 Django worker acquires"| DB[(PostgreSQL)]
    DB -->|Rebuild JSON Schema| Cache[(Redis Cache)]
    Cache --> R1["✓ Return response to Winner"]

    Lock -->|"❌ 4,999 workers fail"| Wait["⏱ Spin-Wait 50ms"]
    Wait --> Check{{Cache Ready?}}
    Check -->|"Yes - Cache Hit"| R2["✓ Return response from Cache"]
    Check -->|"No - retry (< 10s)"| Wait
    Wait -.->|"Max wait (10s) reached"| DB

The Thundering Herd Problem

Imagine a scenario where the cache for the home screen layout expires. At that exact millisecond, 5,000 concurrent mobile apps request the layout.

Because the cache is empty, all 5,000 requests bypass the cache and hit the PostgreSQL database simultaneously to rebuild the complex JSON schema. The database CPU spikes to 100%, connections exhaust, and the entire platform crashes.

The Solution: Distributed Locking (Mutex)

To solve this, I engineered a distributed locking mechanism using Redis SETNX (via Django’s cache.add() method).

When a cache miss occurs, the system doesn’t immediately query the database. Instead, it attempts to acquire a distributed lock for that specific resource before executing the expensive computation.

1. Worker A acquires the lock via add("_lock:component:card:v1", 1, timeout=30). It queries the database, rebuilds the heavy JSON payload, and writes it back to the cache.
2. Workers B through Z fail to acquire the lock. Instead of hitting the database, they enter a non-blocking spin-wait loop. They pause for 50ms, then check the cache again (up to a maximum wait time of 10 seconds).
3. Once Worker A populates the cache, Workers B-Z immediately fetch the data from memory and return it to the users.

Here’s a simplified look at the locking logic from my CacheService:

def _acquire_stampede_lock(self, cache_key: str) -> bool:
    """Try to acquire a distributed lock via Redis SETNX."""
    lock_key = f"_lock:{cache_key}"
    try:
        # cache.add() maps to SETNX in Redis. Returns True if set, False if exists.
        return self._cache.add(lock_key, 1, timeout=self._STAMPEDE_LOCK_TTL)
    except Exception:
        # If Redis is unreachable, fail open so the request can proceed
        return True

By implementing this lock, I converted 5,000 concurrent database queries into exactly 1 database query and 4,999 memory reads, entirely eliminating database saturation during high-traffic spikes.

Surviving Transience: The Circuit Breaker Pattern

Distributed locks are great, but they rely heavily on Redis. What happens if the Redis cluster itself experiences a transient network partition? If the Django application waits for Redis to respond, the entire API layer hangs, causing a cascading failure across the microservices ecosystem.

To prevent this, I wrapped all Redis interactions in an in-memory Circuit Breaker. To protect against Redis being unreachable, I stored the breaker state purely in the Django process memory using threading.Lock(), rather than relying on an external store.

• Closed State: The system communicates with Redis normally.
• Tripped (Open) State: The Circuit Breaker monitors consecutive Redis connection errors. If the failure threshold (e.g., 5 errors) is breached, the circuit trips open. All subsequent requests instantly short-circuit the Redis network call and gracefully fall back to executing database queries or returning safe default schemas.
• Half-Open Recovery: After a recovery timeout (e.g., 30 seconds), the circuit allows a single probe request through. Once Redis is confirmed healthy, the circuit closes, and normal high-throughput caching resumes.

Here is a look at the core in-memory state transition logic:

@property
def is_available(self) -> bool:
    """Check if the circuit is closed or ready for a half-open probe."""
    with self._lock:
        if self._state == self.STATE_CLOSED:
            return True
            
        if self._state == self.STATE_OPEN:
            # If recovery timeout elapsed, transition to half-open
            if time.monotonic() - self._last_failure_time >= self._recovery_timeout:
                self._state = self.STATE_HALF_OPEN
                return True
            return False
            
        # STATE_HALF_OPEN: allow exactly one probe request through
        return True 

By combining distributed locking with aggressive in-memory circuit breaking, I engineered an orchestration layer capable of surviving both massive traffic spikes and severe infrastructure degradation without ever dropping a user session.