BACKUP GENERATOR FAILURE
Considering how valuable backup systems such as UPS and generators are, it’s amazing just how forgettable they can be.
They are systems that you hope to never need, much like insurance, you purchase it then put your documents away and never look at them again until there’s an accident. And it’s the same with data centre systems. Just because they’re expensive doesn’t cancel out human nature.
Unfortunately, this was the case, that after the power was restored, on this one occasion, the switch was not fully recharged and was not checked, so that when another power cut occurred, the generator did not start, leaving their data centre to go dark. Their hubris was their downfall.
Generators are not built to run 24/7, they are a backup system for power outages which don’t often occur, so for most of its life a generator will be sat unused, waiting to kick into action. But when a generator is sat for a long time, the fuel gets contaminated and sludge will develop by a process called Oil Fouling. If not cleaned every 6 months or so, this “dirty” oil can easily wreck the generators mechanical parts, resulting in a very expensive fix. Alongside this are so called “diesel bugs”, microbes inside your fuel tank that will multiply in stagnant fuel, rapidly producing sediment as they sink to the bottom of the tank and die. If this sludge is drawn through into the generator it can damage the engine and cause engine failure right at the moment you need it most.
