Tech. Science. Culture.
We may earn a commission from links on this page

Twitter's Servers Reportedly Went Down During Extreme Heat Wave

The data center outage in Sacramento means the company operated in a tenuous "non-redundant" state for at least 5 days, according to internal communications.

We may earn a commission from links on this page.
Stock photo of servers
Large data centers contain thousands of servers that all need to be kept cool to run.
Photo: Gorodenkoff (Shutterstock)

Last Monday, Sacramento reached a scalding high of 113 degrees Fahrenheit, smashing the previous daily record of 108. The same day, a Twitter data center located in the California state capital failed, according to a report from CNN that cites a leaked internal company email.

“On September 5th, Twitter experienced the loss of its Sacramento (SMF) datacenter region due to extreme weather. The unprecedented event resulted in the total shutdown of physical equipment in SMF,” Carrie Fernandez, the social media company’s vice president of engineering, wrote in an internal message to Twitter engineers on Friday, according to CNN.

The server center outage hasn’t impacted site users, instead it just made Twitter more vulnerable, by taking its back-up data storage offline. But the failure highlights an increasingly worrying vulnerability of tech companies’ in the climate change era. Servers need to stay cool to run properly, but maintaining their temperatures is a costly challenge, especially when they’re located in places like California’s Central Valley.

Advertisement

Other tech giants have also recently experienced similar outages. During a heatwave in the United Kingdom earlier this year, both Oracle and Google servers went down. And large scale crypto mining operations in Texas had to reduce operations in July to account for heatwaves and a stressed power grid.

Last week’s heatwave was one of the most extreme ever for the West. It brought triple-digit temperatures to almost all of California, and shattered nearly 1,000 heat records. The intense weather taxed the electric grid, and the state only narrowly avoided blackouts (mostly thanks to text blasts that got people to reduce their energy usage).

Advertisement

But while the electricity stayed on, Twitter’s servers didn’t. The outage put Twitter in a “non-redundant state,” Fernandez wrote in the staff memo, as reported by CNN.

The company has data centers in multiple locations around the country including Atlanta, Georgia and Portland, Oregon, according to the news outlet. Those centers store duplicate data by design, to avoid data loss and total site outages. But as of Friday, Fernandez warned, “if we lose one of those remaining data centers, we many not be able to serve traffic to all of Twitter’s users.”

Advertisement

In his whistleblower complaint, Peiter “Mudge” Zatko specifically referenced the vulnerability and “insufficient data center redundancy,” of Twitter’s system. From CNN:

“Even a temporary but overlapping outage of a small number of data centers would likely result in the service [Twitter] going offline for weeks, months, or permanently,” according to Zatko’s whistleblower disclosure. (Twitter has criticized Zatko and broadly defended itself against the allegations, saying the disclosure paints a “false narrative” of the company.)

Advertisement

Since the Sacramento failure occurred on Monday and the staff memo was sent on Friday, there’ve been at least five days of server disruption. It’s unclear if the data center has been brought back online in the interim, or if the the outage is ongoing.

Twitter declined to respond to any of Gizmodo’s questions regarding their data centers. Instead, in an email to Gizmodo, a company spokesperson wrote, “There have been no disruptions impacting the ability for people to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue working to provide a seamless Twitter experience.”