Stack Overflow Outage!
In June 2016, StackOverflow faced a server outage exceeding 30 minutes caused by a security vulnerability exploited by a malicious post. This post, prominently featured on the homepage, led to excessive CPU usage and severely delayed the page’s response time. The root cause was identified in the security validation function.
The maliciously crafted post contained roughly 20,000 consecutive whitespace characters, exploiting a regex pattern intended to trim Unicode whitespace from the beginning and end of strings (^[\s\u200c]+|[\s\u200c]+$). When displayed on StackOverflow’s homepage, the post triggered severe CPU exhaustion as the regex engine repeatedly attempted and failed to match, engaging in extensive backtracking. Because the homepage was used for health checks by the site’s load balancer, servers were quickly marked as unresponsive, bringing down the entire platform.
The core vulnerability revolved around inefficient regex processing. The regular expression engine’s backtracking mechanism led to quadratic time complexity (O(n²)), resulting in almost 200 million character-checking operations for the malicious input. StackOverflow’s response included swiftly replacing the problematic regex with a more efficient substring function, auditing other regex implementations to prevent similar incidents, and updating the health check logic to minimize collateral damage in future incidents. Additionally, they developed more robust incident management protocols, enhancing their overall operational resilience.
We made a challenge recreating this:
Available also in:
Read more about the incident