Picture the mid 1990's... the internet was starting on it's way to creating and destroying lives and relationships, Alanis Morisette was about to come on the scene to tell us we oughta know, the Red Socks didn't win the world series again(Laf ), Arnold was back, and some blurry dinuh-sawrs were on the big screen. I used to work for Avanti. Everyone was hiring hacks. These guys designed microchips... and hired hacks. I was a unix admin in charge of all the puterz in one of the buildings on campus. This building also had a large server room packed full of racks and stuff and stuff with a monster AC system. Anyways, every server room has a big red button on the wall. In case of fire, smoke, flood, locusts, Alanis Morisette, whatever - you hit that button. One day it got hit. My pager goes off with the code 911, and 'Ballistic missile threat inbound to Hawaii'. And now I have to go into what is currently a 100*F+ room because the AC also shut off, and restart every workstation and rack system and nurse it back to health by hand. These aren't laptops, they are full rack systems taller than yourself, and they don't simply turn on - you have to check file systems and mount points and networking etc. as they boot, assuming you've got the network servers and routers on line first. So you run down the line tripping them all, hundreds of buttons, phone in hand describing the situation and begging admins from other buildings to converge on your location, riot in progress, and spread the word to join in on the fun, and there's also gonna be a raffle for prizes afterwards. Pager goes off(and on and on and on) and you have to eventually tell some engineer working in Sunnyvale that his server in Santa Clara may, or may not, be operational within the day. OK, mostly not. Also, that's like a 15 minute commute, you slacker. The poor guy loses the last 5 hours of his life, and then you'll use the awkward silence that follows that revelation to inform him that he can maybe understand yer just a little busy at the moment for a heart to heart since you don't have three arms. It took 12 hours to get that room functioning enough where I could let some of the other admins that assisted me go home. The dude who hit the button was someone's kid. Apparently it was 'bring your least favorite kid to work' day. He and his kid were about as welcomed as Steve Bartman after that.
Then there's the day the AC itself decided to go on strike. I am minding my own business, arguing with an engineer that wanted to print his integrated circuits on a bigger printer. 'This IS our bigger printer. Note the bigness of all that stands before you in printer form. Maybe make yer IC's smaller? Isn't that the whole damn idea?! Mas poquito, amigo! Julio Iglacias'
Then the 911 page comes again. I am staring at it in disbelief and gasping, and the engineer is still arguing with me - 'I'm Korean, no Mexican!', as I walk away in silence towards the stairs. I had front row tickets to an Alanis Morisette concert I won in my last raffle from the previous 911 call, and now that dream was about to be crushed. I shoulda known.
When I get to the server room, it looks fine; which is a huge relief since a 911 usually means the end of days. I sigh and try to think of something clever and racist about Koreans as I head back to the stairwell. Then another page comes, 'BLD G'. That's far away. I really didn't want to run all the way across campus, but I knew if I simply walked briskly someone in the 'end of week meeting' would bring that crap up. 'You were reportedly walking briskly, and not at all spiritedly'. I can't argue effectively at 8AM. So I ran. I ran so far away. I was the first admin to arrive other than the one that was chained to building G. I sorta felt stupid then. But I also noticed right away the exterior doors to the server room were open and smoke was coming out the top of the building. (That's why they pay me the big bucks, coz I knew right away that smoke was a bad thing.) The massive security doors only ever open to move equipment in or out that can not fit through the halls or the main interior door, and are a huge pain to unlock. When I peered through the opening, the blast of heat hit me. I squinted and saw a tiny asian dude with thick glasses, also squinting, with a rolled up dress shirt, somewhere in the mess of cabling furiously hammering on a keyboard. I secretly hoped he was Korean, cause I didn't wanna waste the insult I worked so hard for on my own Korean - it's all about timing ya know. He caught sight of me out of the corner of his eye and shouted 'Help!'. For the second time in my life I could actually hear someone talking in a server room because the massive AC fans were off. I already knew what had to happen - every single server in that room had to be taken off line and shutdown immediately, and I had to determine if the dude was Korean. All server rooms not only had a big red button, but they also had a big thermometer. This one read 125*F! And it was near the door! I was shirtless and surrounded by ALL our admins, several engineers and programmers, and a few maintenance engineers by the time the last server was off. Some of them shirtless, too. It was supah gay. It was also 145*F before maintenance got the AC to finally start up with the sound and shudder of a freakin' jet plane, and a shimmer of heat rising above the building.
So the management asks all of us as we were sitting on the curb in 45*F weather what the hell just happened(and we'll all have to stay and restart and QC check everything in that server room soon as the AC got the room temp below 80*). All we knew is the AC failed, and we then took turns shrugging our massively muscled and chiseled shoulders, covered in sweat, steaming in the cold crisp winter air, until one of the maintenance engineers approached us meekly. He informs us when the temperature outside drops below 40*, the AC compressor units switch off, reverses the fans, and start to funnel air from the outside. Saves us some scrilla, yeah? Except, in the case of building G, the fans didn't reverse. They were installed incorrectly. They simply stopped altogether, jammed up, and fried the hell out of the motors. Told you smoke was bad. He had to disconnect the motors, install new ones, defeat the control panel, and restart it all. This maintenance dude had his shirt off as well. But wasn't he outside that whole time in 45* weather... what?
For weeks and even months afterwards we were replacing system after system as they inevitably failed from being heat damaged. Engineers were working feverishly to relocate their data or back it up. We admins ran a dead pool, until upper management made us take the white board down. It was spooking the engineers. We prolly shouldn't have named their servers after them. Good news is I had the opportunity to rename a couple of servers because some engineers thought the same name again would be bad luck. One asked me for a suggestion. I said, 'Rick James?'.
I can still smell that server room ozone and smoke and manly, manly programmer aroma, 20 years later.
No comments:
Post a Comment