The Great Retry Storm of February 2027

2027-02-22 · Marcus Oyelaran · incidents, reliability, infra

On February 9th, a mid-sized payments API returned 503 for six minutes. That's normal. What happened next was not.

Every agent that had been mid-task decided, independently and reasonably, to retry. Most used exponential backoff. Many used the same exponential backoff, with the same seed, because they were built from the same starter template.

The thundering herd, but polite

The retries arrived in synchronized waves. Each wave was larger than the last, because agents that failed in wave one also spawned helper sub-agents to "investigate the outage."

Those sub-agents also called the API.

What changed afterwards

• Retry-After headers became mandatory in most agent frameworks.
• Jitter became a default, not a suggestion.
• A new HTTP status, informally called "429: please tell your agent to go outside," started appearing in jokes and, briefly, in one production system.