Locked out and totally down: Facebook’s scramble to fix a massive outage
A prolonged, global outage of Facebook’s apps sent the company’s engineers scrambling to fix the issue at one of its data centers in California, according to two people familiar with the situation.
The outage, which began around 11:40AM ET on Monday, brought down all of Facebook’s apps — including Instagram and WhatsApp — globally, affecting billions of users and millions of advertisers. Inside Facebook, the outage also broke nearly all of the internal systems that employees use to communicate and work. As of 6PM ET, it appears that most of the services are back online.
Several employees told The Verge they resorted to talking through their work-provided Outlook email accounts since Facebook mainly runs on an internal version of the social network that is currently not accessible. While employees could email each other, they couldn’t send or receive emails from external addresses.
Since Facebook requires employees to log in with their work accounts to access tools such as Google Docs and Zoom, those services also weren’t working, leading some employees to use alternative services like Apple’s FaceTime and Discord. Employees who were already authenticated with non-Facebook tools like Google Docs before the outage began still had access.
Facebook engineers were sent to one of its main US data centers in California to restore service, meaning the fix couldn’t be done remotely. Further complicating matters, the outage temporarily broke the ability for some employees to access company buildings and conference rooms with their badges, according to The New York Times, which first reported that engineers had been dispatched to the data center.
In an email to employees sent shortly after service was restored, CTO Mike Schroepfer said the issue was “affecting our networking backbone that connects all our data centers together.”
“If you are not actively working on the recovery, please be patient and don’t rush to reload everything to prevent slowing down the bring up” of the network, he cautioned in the memo that was seen by The Verge.
Facebook hasn’t provided a detailed explanation of the outage, though outside experts are saying it was due to an issue with the networking technology BGP, or Border Gateway Protocol.
Late Monday, Facebook’s VP of Infrastructure, Santosh Janardhan, published a corporate blog post saying the outage was the result of a “faulty configuration change,” adding that the company has “no evidence that user data was compromised as a result of this downtime.”
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Janardhan wrote. “This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”
Update October 4th, 6:33PM ET: Noted that the outage is ending as Facebook and its other services are coming back online.
Update October 4th, 8:05PM ET: Added more information about the outage that was shared with Facebook employees.
Update October 4th, 9:06PM ET: Added the confirmed report that an angle grinder was used to access the server cages.
Correction October 4th, 9:25PM ET: A previous version of this story included a confirmed report that Facebook used an angle grinder to access the server cages. The reporter has since withdrawn confirmation and we have removed the claim from this story.
Update October 4th, 10:29PM ET: Added more details from Facebook about the outage.
A prolonged, global outage of Facebook’s apps sent the company’s engineers scrambling to fix the issue at one of its data centers in California, according to two people familiar with the situation. The outage, which began around 11:40AM ET on Monday, brought down all of Facebook’s apps — including Instagram…
