Friday, August 12, 2011

An adventure.

So yesterday, was like any other day at work. It started normal, I was basically going through the paces. I was walking down a flight of stairs, then it happened. The power spiked for a second, then complete power loss for 35 to 45 seconds. Now that doesn't seem like a long time, but when this building is home to multi million dollar corporations, that is a very long time. So I walk out of the stairwell, I am so lucky I know this building well, and can find my way around. The lights had not even came back on until I was outside. I am seeing the poof of black smoke bellowing out of the backup generators firing up. Ok good, I am thinking we will return to full power really soon. Nope that was wishful thinking on my part. So I start making my calls I'm supposed to, so I can let people know what has happened, and the status of the situation. My first phone call, I get, (jokingly) "Do you know the Seahawks are playing?"  I said, (jokingly) "Do you know there's a power failure?"  I get a response of, "Oh, wow, ok, really, holy shit!" I said, "Well , the backup generators are running, but we only have half power" he says, "ok, I called someone, and they are on the way" I say, "Ok, thank you" I run into the building and there's alarms going off, to say the generators are running. I have a person stop my in the lobby, and say, "My wife's stuck in the elevator!" Holy shit! I'm thinking, I can't do anything without the proper authorization, but how am I supposed to tell this guy that? I call back my first call, and tell him about the person who was stuck in the elevator. He says, "He's on his way, and he knows what to do, just wait for him" I say, "ok".

So finally, the guy shows up. I tell him everything that's going on. He says, "I just need to hit the bypass switch, and the person will be out of the elevator" He hits the switch and I walk towards the elevator, and the person gets out of the elevator. Good, finally. I walk back to the guy, and say, "They're out of the elevator" He says, "Good, now all we have to do, is wait for the generator to switch off, and wait for the power to come back online" At this point we were running half power, and half generator. Some things were working, others were not. The countdown began for the generators to switch off and full power to be restored. as of 10 pm that night, full power was still not restored and the generators were still running.

The aftermath:
So 1 of the company's had their own generator and it was running along side the buildings, so they turned out to fair pretty good considering. Some of the other company's though, did not fair very well. Today I heard about 2 major server rooms being fried. 1 company had to have a part shipped up from Oregon, and they didn't have their servers until that part arrived. Another company's server room had a fail safe device that failed. They had to reroute traffic from their servers to others. Then on top of all that, the cooling towers for the whole building went down, that means there is no A/C anywhere in the building. Damn! What a crazy mess of an adventure this was.
Till next time.

No comments:

Post a Comment