I mean raining in a data center is a pretty good reason to have some warning lights, although I'm not sure how that situation happens
this post was submitted on 08 Feb 2024
192 points (97.1% liked)
Very Real Tech Pics
1294 readers
1 users here now
There's no place like 127.0.0.1
founded 1 year ago
MODERATORS
When evaporated data condenses and the air gets super saturated.
Duh
Happened at a DC I worked at, but a little before my time. Someone thought it would be a bright idea to use the design of some 1970s Californian architectural prodigy in Canada. When winter hit, the humid air condensed on the inside of the uninsulated metal roof and it rained in the attic, eventually leaking out across the entire computer room ceiling. They had pictures that showed 6mm poly stretched out and taped above all the mainframes and DASD while water rained down from above, running off the sides of the plastic sheeting and pooling on the raised floor. They also used to throw wild alcohol-fueled Christmas parties, and climbed up on top of the racks so they could play games jumping from row to row. What a time to be employed!
"The roof's gone"
"How can you tell?"
"On the account it is not there anymore, and I can feel the rain"
"Oh, ok..."
although I’m not sure how that situation happens
Both the warning lights and the rain can be easily explained by some brain dead suit buying the wrong kind of fire suppression system because sprinklers using water are much cheaper than a system using a gaseous agent.
Not sure how some red light 🚨 would help much.
If you’re cold, they’re cold.
If you're in a data center and you are not cold, something went wrong and someone is about to get fired.
Unless you’re in the cold aisle.
I was on a tour of a modular DC that consisted of modules of rack filled shipping containers. Definitely wasn't cold inside the containers.
Modular is a whole different game. They are never as efficient as fixed infrastructure.
That's not rain. Someone is just downloading something from the cloud.
It's all fun and games until the halon gets triggered
First day working on a data center:
"You see that red light next to the door? If that turns on and you hear a horn, you have 45 seconds to leave the room or you will die. Also, if it goes off, you must not for any reason whatsoever try to enter the data center, actually just leave the whole building at that point."
Scariest caveat to a security brief. Funny story, that data center actually burned down while I worked there. No, I didn't do it, I wasn't even on the building at the time. But I was part of the cleanup crew. I can still feel the smell of burnt plastic if I try real hard to remember.
The fuck happened?
An investigation concluded that a power line next to a vertical ventilation overheated, melted the wires and created a short circuit electrical arc. The power properly went off but a flammable outer lining in the ducts caught fire. The insulation correctly slowed down the fire, but the plastic on the wires caught fire. The wires went up into the data center and carried the flames inside through the ducts. All in all, the fire preventions worked to some extent but still several million dollars in servers got ruined and rendered inoperative. A whole rack burned until the halon triggered and mostly snuffed the fire, but the furniture in an office below, adjacent to the duct lines burned into a crisp further damaging the data center with smoke and heat.
It was the most surreal week on that job, taking all the equipment to the dining hall building next door, tearing everything apart, cleaning and testing every component to separate what was damaged beyond repair and what could still work at least as contingency. The funny thing is that the backup servers where in an office next to the one That burned down so the service was off for good. They got lucky and the tape archive was properly sealed from the fire, it was inside the data center. But we still had to verify each of the tens of thousands of tapes for data corruption.
I guess the moral of the story is, off site backup means more than 20 meters away.
Add: oh I forgot to mention. The reason you will certainly die in a data center fire is not just the oxygen displacing gas. The doors are designed to create an airtight magnetical seal and when the fire suppression triggers it locks the doors permanently. It needs power to break the seal and open. There's a big red emergency button next to the door, but in case of a fire it just buys you an extra 30 seconds to scram out of there before the system rearms the gas release. Miss the window and you're trapped inside in an oxygen depleted environment. Firefighters had to tore the entire door frame off the concrete to access the data center.
Oh yeah, Halon will fuck you right up.
If anyone happens to end up in a datacenter and the alarm goes off, leave the building.... Just go right outside. Do not stop and collect tools, do not pass go. Do not collect $200.
There are many centers that don't use Halon anymore and it's "safe" to stick around, but heck, if that alarm goes off, I'd still get out. Nothing I'm in there to do is more important than my continued survival. If it's actually safe, I'll get delayed by what? An hour? Two? Maybe a day? Who cares.
$200? Back in my days it was 4000. ;)
Do they have ELSRDs available?
Water cooling 👍
Only hackers on steroids can make it rain inside a data centre.
cloud computing
Wait is that the red for the hacker alert or the red because the place is overheating?
New Dark Web Diaries podcast cover image looking lit