Unplanned day off from work today.

It’s amazing the havoc a brownout can cause. Just before noon on Monday our building had a brownout that lasted all of a second—the lights never made it completely off before they were back on again—but it was enough to stop all work since then. We have just over 100 workstations, but each workstation is literally composed of multiple computers such that the total number of machines is multiplied by five. The server room is of a similar composition. We managed to get all but a couple of the workstations back up and running in short order, but the massive database that coordinates everything on the servers was badly corrupted. They’ve been working on rebuilding it ever since. They sent the first shift home and have canceled every shift since then. I went in yesterday on the off-chance we’d be back up and running in time for afternoon shift, but it didn’t happen so they told me to plan on staying home today. Which I am. And possibly tomorrow too.

The thing that amazes me is that this could have been prevented with a decent UPS system in place, but they don’t have one. At the very least I’d expect them to have one in the server room, but the company doesn’t want to spend the money. We could have been back up and running within 20 minutes of the original brownout had we had a UPS in the server room as the database corruption has been the source of our downtime. Instead we’ve lost two and a half days worth of work.

Sometimes I just don’t understand how upper management in even the brightest companies think.

5 thoughts on “Unplanned day off from work today.

  1. Sometimes it’s boneheadedness. Sometimes it’s the deliberate decision that the TCO of a UPS outweighs the potential damage due to a power failure. Sometimes such a decision is the wrong decision.

  2. I’ve yet to meet a manager with two brain cells to rub together.  No foresight, no hindsight. 

    Management is not a case of the cream rising to the top.

  3. The lab I worked for had a wonderful UPS system. First a big bank of batteries (looked like car batteries) took out every little dip and then the diesel generator kicked in when needed. Usually we didn’t even know the switch had occurred. In the summer time the P&L company paid us large chunks of money for the ability to tell us to go to generator at peak load. The system paid for itself.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.