PDA

View Full Version : Unexpected down time this morning (3/15/06)



Steve
03-15-2006, 02:53 PM
Hi folks,

The website and forums were down this morning for several hours. Things seem to be back but we are currently experiencing random forum errors. If you get an error while browsing the forum, just hit refresh and it should come back. We are working on solving that problem now.

For the technically curious, we run our web servers from a private hosting facility with backup and redundant power systems, generators, constant disk backups, armed guards, etc., and they usually prevent things like this from happening. :) This morning there was a cascading power failure that wiped out everything... each system that failed added too much stress to the next backup system in the chain until there were none left. Fortunately our hosting parners have a staff of smart people and they had everything back in about 3 hours. Since this is the first time we've been down for that long, I'm not too upset about it. Sorry for the inconvenience.

-Steve

Steve
03-15-2006, 04:26 PM
Shortly after I posted my first update, the forums started spitting out errors continuously. Apparently we had some database corruption when the power died. We have deleted the corrupt stuff and restored it. Everything seems to be back to normal now, but if you notice anything weird (error messages, missing posts, etc) please let me know.

-Steve