Friday, March 09, 2007
Environmentally friendly power cable fault??!!
It took three months for me to discover a serious intermittent fault on a cable supplying power to our Linux mail server. The really strange thing was that it never once was off when we opened on a weekday morning, never turned off during a work day, but turned off (without exception again) every single night, and most sundays during the day! The first inkling was when somebody noted that emails weren't working when they came in on a Sunday. When I checked the log graphs of the server it became clear that it was turning off most nights and during the weekend. The "inexact" nature of the timing, and the "it turning off when nobody was expected to need it" factors seemed to point to human interference. Not so. Power fluctuations seemed to trigger on/off switching of the cable supplying the UPS. During the day, there was more on than off and the UPS remained charged. At low activity times it was more off than on, and when the UPS ran out of charge it turned off the server. Every morning (6:30) some kind of activity triggered the power back on and the server would reboot and stay on.