Prevent Server Outages
Posted: Mon Oct 13, 2008 2:30 pmTwill wrote:Update #3:
We're going to probably have to go back through some backups, dig out just off-topics and the lost usergroups and restore them individually.
This probably wont happen until Tuesday (it's thanksgiving weekend here in Canada and so our families seem to want to spend time with us for some odd and unknown reason), so hang tight until then, off-topics and your clan forums will be coming back as soon as we can get them there.
We've heard conflicting things from the data center - we've been variously told it's the memory controller, then the hard drive something or other and then finally "sorry if the chiller failure caused you any problems"...which is what took out the database 3 months ago.
I'll post more when I know more.
Twill
Ensure there is a battery backup for the RAID controller write caching. Raid controller cards typically have 128-512mb of ram onboard, and require a battery to keep the data in the ram in case of problems. If the database is in the process of a write, and it fails, with the battery the data will get written to the drives when the system comes back up. If the battery is dead or not there, then when the system comes back online, the data will be lost, and this can cause serious and random database corruption.
EDIT: The battery is a little battery inside the server screwed to the raid controller card...it's not a UPS for power for the entire server etc...