How a Mop Led Me to the Cloud
I remember how things were before "the cloud." Not too bad, but not so great either.
It was about ten years ago, and my IT team had a pretty OK setup. Our killer onsite data center gave quick, reliable data access to the locals, while our remote people worked in through VPN. We had concentrators on both sides of the country and a combination of MPLS and leased lines to tie all of our sites together, so they could get back to the data center at HQ. It wasn't ideal, but it worked. As long as we stayed on top of things at all times.
Back then, I was one of those guys who'd never dream of giving up complete control. Then something happened that changed the way I felt about moving to the cloud.
It started off as no big deal. At 6:00 a.m., my pager went off. Nothing to worry about: just a bad disk on one of the big disk shelves. Monitoring had gotten a little overzealous in this instance, but it was still worth some attention and sys admin had it covered. Except he didn't.
A half-hour later, disk failure occurred in a server on the same row in a different rack. This little problem wasn't so little. After all, this was a mail server. The disk was mirrored, but another outage would take mail down until we were able to restore things from an onsite backup.
Still fixable. No reason to panic. Sys admin to the rescue and we're back in business. But it was very curious. Two disk failures in separate machines within only an hour? Something was up. I jumped in my car and cruised on down to the office. No reason to get a ticket. Things were under control.
But when I arrived, it was clear that wasn't the case. We were in full meltdown mode. Dozens of servers were in the low-90 degree range. One of the giant air conditioning units had failed, and the machines on that side of the data center were overheating rapidly.
My first thought was to fix the problem. My second was to notify employees to prepare for potential outages. Within minutes, however, I was doing the only thing I really could do: making frantic calls to local AC repair guys and wondering how the heck I ended up in the business of AC maintenance.
I had backups for my backups, but apparently we needed all four - not just three - AC units to keep our spinning, whirring, blowing, glowing data center cool. I was no longer an IT guy. I was facilities guy, and it all happened within the course of an hour.
Ultimately, when they ripped open the AC units, they found about 50 cubic feet had frozen solid. A Freon leak had filled the chassis with ice, engulfing its innards with frozen water that was just waiting to get all over everything.
After plenty of chipping and melting, there was water everywhere. It was all hands on the dumbest looking deck: towels, mops and lots of wet knees. As I shoved a mop back and forth, I began thinking hard about the benefits of cloud hosting.
While this incident did get us on the path to the cloud, it wasn’t just to avoid future mop-up duty. Really, my move to the cloud was more about saving time. After all, any time things go sideways, it’s always the IT guys who had to leave the helpdesk unmanned to push the mops - proverbially and literally.