Facebook's Data Center Humidity "Challenge" Was Indoor Rain

Illustration for article titled Facebook's Data Center Humidity "Challenge" Was Indoor Rain

In 2011 Facebook reported that their first data center in Prineville, OR had a high humidity issue. Probably not the best condition for servers, sure. But it turns out that wonky temperature controls were actually causing condensation in the data center. Like indoor rain. Like it was literally raining in the server room.


The cloud on cloud action was reported by The Register, which quoted Facebook's VP of Engineering, Jay Parikh as saying:

I got a call, 'Jay, there's a cloud in the data center.' 'What do you mean, outside?' 'No, inside'."

Though it's not impossible to interpret what was actually going on from Facebook's 2011 explanation ("cold aisle supply temperature exceeding 80°F and relative humidity exceeding 95%") they definitely left a lot of the drama out. As air circulated and cooled, it took on so much moisture that it was forming a cloud and raining down on the servers, which automatically shut down or just shorted out.

Parikh says Facebook's data centers have changed their protocols so such an incident will never reoccur. But just in case, all servers are now additionally equipped with "a rubber raincoat" around their power supplies. [The Register]

Image credit: Shutterstock/Milkovasa



In the 80's I witnessed an actual waterfall in a data center. Overhead water pipe burst due to a really poorly thought out plan to test for leaks. It collected in the drop ceiling until a tile broke. It then filled a raised floor with an inch or more of water. As I stared at the submerged power and data cables in the drop floor, it was the first and last time I uttered the words "not my job".