A warm-up script writes ten thousand cache keys in one second, each with a one-hour TTL.
One hour later, all ten thousand expire in the same second. Every request misses. The database, sized for cached traffic, takes the full uncached load at once.
The incident happens an hour after a deploy that looked completely clean, which is what makes it hard to attribute.
Python
ttl = base_ttl + random.randint(0, base_ttl // 10)
Ten percent of randomness spreads the expiry over six minutes instead of one second. It is one line, and it is the cheapest reliability improvement available to anyone running a cache.
One useful email a month
Design system patterns, front-end techniques and notes from the learn platform. No promotions, no digest of other people’s links.