[15-Jun-2008] Downtime today

[bedub1]

I'll be following this up with the hosting company and with myself to see how we can better handle such mishaps in the future.

You don't want to "handle such mishaps" cause that is Disaster Recovery. You want to avoid them all together, which is called Disaster Prevention. There is software that can help you maintain multiple locations of identical information. There are other options to distribute the load between two remote locations, or to set a 2nd location as a live-state failover in case the first location goes down.

I typically find that websites such as this require 100% uptime, and "restoring from a backup" doesn't allow for 100% uptime. It means it can never go down. Disaster prevention.

[Jackofalltrades]

Thanks for explaining what happened Lack, speaks well of the accountablity of the site managers. The extension of the game round in hours was a very pleasant surprise to see the ability within the system on a global scale. Many thanks for helping to continue providing a quality site. Kudos

[t-o-m]

when will this thread be removed from all forums?
it wouldnt be moved to announcements would it? i thought it was staying out of there so it wouldnt put a downer on the newest update (medals and such)

in the GD forum?

the abandonned part of the tournie's forum (no one goes there )

or just deleted alltogether?

[alcazar]

I was glad the site was down...got some time off from CC

Still love the site!

[the_lion]

its ok lack, we still love you

f*ck that... this is shit. Turns were missed because of this "down time." No one thought to run some update statements that would give those who lost turns due to downtime the ability to take their turns. Unless you count whatever that shit is that some folks appeared to have 36 hours to take their turns. What a bunch of fucking amatures.

[t-o-m]

its ok lack, we still love you

f*ck that... this is shit. Turns were missed because of this "down time." No one thought to run some update statements that would give those who lost turns due to downtime the ability to take their turns. Unless you count whatever that shit is that some folks appeared to have 36 hours to take their turns. What a bunch of fucking amatures.

has your period passed now lion?

good now enough complaining, not lack's fault

[wpg27offsuit]

its ok lack, we still love you

f*ck that... this is shit. Turns were missed because of this "down time." No one thought to run some update statements that would give those who lost turns due to downtime the ability to take their turns. Unless you count whatever that shit is that some folks appeared to have 36 hours to take their turns. What a bunch of fucking amatures.

has your period passed now lion?

good now enough complaining, not lack's fault

Not Lacks fault - unfortunately in the computer world - it is easy to blame a failed fan, a power surge, a failed generator,fail this fail that. I agree - it is no humans fault - however, we all know (or at least those of us in the business) know that computers fail. Power fails, cpu's fail, fans fail, hard drives fail - everything fails eventually.

There are COUNTLESS ways to monitor the site, and the databases, where the instant there was a failure an email could have been sent - say to someones cell phone (SMS even) - and they would have been notified. I do not fault Lack for having a life outside of CC, and being out of town. I do fault no automatic notification process in the event of a failure. Depending on a hosting company to notify you when there is a problem - is WRONG - they offer a server - the LAST THING on their mind is notifying their customers that there is a problem.. Their goal is to fix it as quickly and as quietly as possible. If their customers never find out "whew" that was a close one.

I am not upset - I just think this should be a lesson to those wonderful people who own and maintain this site to have something other than the hosting company notify them when there is a problem.... YES - you will get erroneous SMS / emails and messages when the system "burps". but in time you will learn what to react on, and what to ignore.

Down for 12 hours - tsk tsk tsk - How many paying users? Auto Fail over should definitely be in place at this time.

[General_Clark]

I'm currently playing four games. I only missed one turn in one of the games. It's quite alright though. Even if I could have played that round or not, I still may get waxed in the end.

Thanks for letting everyone know what happened. I was thinking it was something on my end.

[simor]

good now enough complaining, not lack's fault

Then whose?? Who is running this site? Santa? are you ppl for REAL

There are ways to prevent situations like that, someone already mentioned it before. I'm not expecting to have CC hosted in multiple places around the globe to withstand one of the data centers being torn down due to an earthquake but if backups is all there is against disk errors then honestly... its a joke. This is the quality you get with those $3 per month hosting services. Oh wait, 12*$3=$36, it's even priced almost the same
You know, it's all about maximizing profits and as long as most ppl are clapping lets keep charging them while selling them the same low quality service. But all it takes is to employ some redundancy, set up some RAID (for those not familiar: you have multiple (2+) exactly same HDDs running at the same time, when one fails the others take over immediately without ANY system downtime) but hey, its another $500 so why bother when 90% of "customers" will praise us for bringing them the site back online after every pathetic crash.

Not Lacks fault - unfortunately in the computer world - it is easy to blame a failed fan, a power surge, a failed generator,fail this fail that. I agree - it is no humans fault - however, we all know (or at least those of us in the business) know that computers fail. Power fails, cpu's fail, fans fail, hard drives fail - everything fails eventually.

Exactly, and that's why data centers have backup power lines, local generators, multiple cable connections everything is ready for a failure so when it comes the customers aren't affected and they get the quality they deserve... they PAY for! Imagine a data center without any of these basic things doubled - NOBODY would want to give them their money... well, except a bunch of idiots. Any resemblance of customers is unintentional

Cheers

[bedub1]

RAID won't protect against data corruption or file-system errors. A virus will eat data off a raid just as fast as off a single drive. The AC not being hooked to the Generator, causing overheating and crashing of systems is a bad part on the hosting companies part. Overheating CPU's and hard drives, causing errors, crashing the database has the potential to corrupt the entire database and OS. Sometimes the hardware gets soo hot...that the capacitors will blow or pop.....rendering the motherboard trash. If the system has experienced this situation, i highly recommend pulling the case off and physically examining the server. Do you know if they vacuum the server and fans etc to keep airflow up and dirt out?

[t-o-m]

STOP BLAMING LACK

[Frop]

The 7 cents per day for who knows how many premium customers should be sufficient to 'buy' redundancy, whether it be database replication, automatic rollbacks or w/e. We shouldn't have to suffer more than a few minutes downtime tops (and even that's disgraceful). I don't know where all the premium money is going right now, but it's certainly not being spent on maintaining redundancy at the so called datacenter. I would be ashamed.

[Twill]

Simor, you're clearly upset, and going to keep pounding it home to us. We run our servers in one of the best data centers available right now. It's not cheap and it's not something that we skimp on - our servers, and their maintenance, are CC's single largest cost.

You're right about RAID to a certain extent though, and for the record, the DB server actually runs on 6 HDDs, 2 in Raid1 and 4 in Raid10 - but when the data center cooks the entire room your servers are in, well, RAID doesn't help all that much.

Exactly, and that's why data centers have backup power lines, local generators, multiple cable connections everything is ready for a failure so when it comes the customers aren't affected and they get the quality they deserve... they PAY for! Imagine a data center without any of these basic things doubled - NOBODY would want to give them their money... well, except a bunch of idiots. Any resemblance of customers is unintentional

It was the switching to the backup generators that caused the problem

Our servers are run by Rackspace. Feel free to check them out if you want to know what kind of backups they have.

Bedub, yes, disaster prevention is something that is essential and it's something that CC does have, but within the limits of the budget. Maintaining a global remote failover, especially for a highly active DB such as this, is not something that's cheap nor easy - you know that, you've done sys admin before. If you want us to jack the price up to $50 a year, sure, we'd be happy to add in the extra dedicated servers and extra DBA staffing we'd need for this...but I'm pretty sure nobody wants to pay that

Lack is, however, actively working with our data center to set up a more robust system both for prevention and recovery.

MOBAJOBG - the site may be lagging as we run DB fixes and restore game logs etc. It's temporary, but you may notice some slowdowns today and tomorrow. They shouldn't be major though, so if things come to a grinding halt, let us know!

Speaking of Game Logs, most game logs should now be restored and available if they were missing before.
We may have a couple that could not initially be fixed or restored for one reason or another, but Lack is going to poke at them a little more and see if he can't find them in the mess of data that we ended up with. As with all corrupted data, we'll do our best, but some will probably be lost for good.

We'll keep you posted if anything changes.

Twill

[t-o-m]

this was always a funny game because of the downtime:
Game 2655703

[bedub1]

Bedub, yes, disaster prevention is something that is essential and it's something that CC does have, but within the limits of the budget. Maintaining a global remote failover, especially for a highly active DB such as this, is not something that's cheap nor easy - you know that, you've done sys admin before. If you want us to jack the price up to $50 a year, sure, we'd be happy to add in the extra dedicated servers and extra DBA staffing we'd need for this...but I'm pretty sure nobody wants to pay that

Thats my single biggest problem. I know what people need, they know what they need, I can set it up, but who the hell wants to pay for it???

And I bet everybody's server went down when the server room hit 100+ degrees....This isn't just a problem for CC...but for all of rackspace's clients. I think rackspace should have duplicate/failover facilities to protect against entire data centers going down due to earthquakes power outages etc....CC shouldn't pay for it...rackspace should!

I've dealt with Cell Phone Sites in Las Vegas NV in the 120 degree heat...when the AC goes out....those things get hot FAST. They have backup electric fans to circulate air from outside...but the input air is still 120 degrees...which doesn't help cool down a server thats designed to run at 110 degrees....But helps prevent it from hitting 200+... Still a good idea to inspect the equipment though...

[Twill]

this was always a funny game because of the downtime:
Game 2655703

Wow.

Just spoke to Lack and he's seen a one or two of those. Fix is in the works to get people their points.

[simor]

Of course I am upset. I'm paying for something and you handled this really badly. If you can't eliminate situations like that (believe me I can udnerstand it and learn to live with it) then at least make sure you fix them properly. The fact that the site was up at some random time (and no, it wasn't 5 min after it went down) and rounds were missed and then that pointless +12h this is just lame way out.

I remember during previous downtimes you wrote "Rounds will be lost, we are sorry" bla bla bla. Well, don't be sorry. LEARN! Next time your drives boil or a meteor flies through the air vent and burns your RAM and you have to restore the site from backup - PLEASE restore it properly! How hard is it to stop rounds from being missed, then adding 24h to all games and then letting us in. That would actually deserve a clap from everyone.

Oh, and please be as big of a pain in ass to rackspace as I am to you. The fact they can't switch to generators properly and you call them one of them best... ahh no comments

[t-o-m]

this was always a funny game because of the downtime:
Game 2655703

Wow.

Just spoke to Lack and he's seen a one or two of those. Fix is in the works to get people their points.

yeh i ummm....lost 500 points due to downtime, can you restore them?

[Optimus Prime]

Of course I am upset. I'm paying for something and you handled this really badly. If you can't eliminate situations like that (believe me I can udnerstand it and learn to live with it) then at least make sure you fix them properly. The fact that the site was up at some random time (and no, it wasn't 5 min after it went down) and rounds were missed and then that pointless +12h this is just lame way out.

I remember during previous downtimes you wrote "Rounds will be lost, we are sorry" bla bla bla. Well, don't be sorry. LEARN! Next time your drives boil or a meteor flies through the air vent and burns your RAM and you have to restore the site from backup - PLEASE restore it properly! How hard is it to stop rounds from being missed, then adding 24h to all games and then letting us in. That would actually deserve a clap from everyone.

Oh, and please be as big of a pain in ass to rackspace as I am to you. The fact they can't switch to generators properly and you call them one of them best... ahh no comments

Last time I checked, you think they handle everything badly here, simor. You come across as one of those people who won't ever be happy with what they do unless they happen to randomly do it your way sometime. Really, missing a turn or two, NOT A BIG DEAL. Move on.

[t-o-m]

missing a turn or two, NOT A BIG DEAL

yeh, i mean look at this guy:
http://www.conquerclub.com/player.php?mode=find&submit=Search&player1=benjij&sort=D
he doesnt mind if he misses the odd turn or two...
(p.s he's banned from CC for delibarate deadbeating)

pps - simor - really? does it matter all that much that you must have a tantrum? do you want you 3.5cents back?
actually - i was my 3.5 cents back! ino - lets start an uproar and demand 3.5cents!
(lets not)
so what are you complaining about?

[Optimus Prime]

missing a turn or two, NOT A BIG DEAL

yeh, i mean look at this guy:
http://www.conquerclub.com/player.php?mode=find&submit=Search&player1=benjij&sort=D
he doesnt mind if he misses the odd turn or two...
(p.s he's banned from CC for delibarate deadbeating)

pps - simor - really? does it matter all that much that you must have a tantrum? do you want you 3.5cents back?
actually - i was my 3.5 cents back! ino - lets start an uproar and demand 3.5cents!
(lets not)
so what are you complaining about?

Deliberate deadbeating is an entire different situation than missing 1 turn in a game due to some server problems, t-o-m. You know that.

[tzor]

I think several people need to step away from the keyboard and remember ...

• This is only a game ...
• A game where some people pay a fixed amount per year to play ...
• A game where data updates happen on a minute by minute basis ...

And you are talking about maintaining multiple data centers and being able to hot switch on a moment's notice ... are you daft?

I should know this sort of stuff because I work for a place where real people pay real money for real 24/7 financial systems where failure could mean the loss of real money and real corporate profits, not just a few lousy points on one of your games which could be easily made up by you vastly experienced players anyway. That costs, but these people are paying real money, not just $25 per year, but thousands per month!

I've also worked for online game companies where flat fees made general sallaries difficult, and barely paid to keep the hardware up and running. It's not easy getting good serivce these days, and it's not cheap. But you can only get the level you can afford.

That being said I'm sure they are going through the "post-mortums" and will probably find a number of ways to improve the system, but the fact of the matter is that this was handled well and in an exceptional manner. The fact that the backup was only an hour old was a godsend.

[a.sub]

THIS IS TERRIBLE I WANT MY MONEY BACK og wait im a freemium
jk guys, so kay we still think ur amazing, its not like it was ur fault and even if it was most websites hav crashes. keep up the good work.
as for me not being a premium i will be once i get a job (its hard finding a job at 16)

[Twill]

Guys, stop attacking simor, he's allowed to vent

And on more recent news, Lack has tracked down a lot of the games that didn't register wins and is working on them now.

We'll let you know when they are fixed and you are flooded with points

Props to Rackspace as well, they are doing good followup with Lack...good support, even if their backup power needs work

Twill

[t-o-m]

Guys, stop attacking simor, he's allowed to vent

he attacked you (and lack) we attack him...is attacking not legal anymore?
twill can we move this thread to flame wars?

and gdgd about lost games and such