Beginning at approximately 11:30 PM EST, we experienced a widespread network outage that caused a large number of IP addresses within our network to be inaccessible from outside the network. I'm happy to report the issue has been resolved as of 1:45 AM EST. All services should now be back online and running as normal.
The problem was determined to be an error with our anti-DDoS equipment routing packets towards its uplinks rather than its downlinks, causing a synchronization issue between anti-DDoS nodes, and this caused multiple protocols on a large number of IP addresses that relied on these nodes to be inaccessible for the duration of the incident. Our teams resolved the issue by manually changing the routing for all the protocols individually, and all affected services should now be online as normal.
We apologize for the inconvenience, and thank you greatly for your patience in this matter.
Kedd, Március 28, 2017
