[10-Jul-2008] Downtime yesterday

[bedub1]

Do the servers have a temp sensor? Set them to automatically shut themselves down if the CPU exceeds 70 degrees C and if the ambiant air/motherboard temp exceeds 60 or so degrees C. Then the next time this happens, CC will be down, but you won't experience any corruption in the database and it *should* be quick and easy to bring everything back up.

Disaster PREVENTION - not Disaster RECOVERY

[lackattack]

Do the servers have a temp sensor? Set them to automatically shut themselves down if the CPU exceeds 70 degrees C and if the ambiant air/motherboard temp exceeds 60 or so degrees C. Then the next time this happens, CC will be down, but you won't experience any corruption in the database and it *should* be quick and easy to bring everything back up.

Their techie explained to me that this hardware-level auto-shutdown is being done and is part of the problem. The database engine doesn't get a chance to close properly when the power gets cut suddenly and that messes up the data (but only data stored in MySQL InnoDB tables).

I'm looking into ways to prevent sudden power loss from corrupting this data.

[Twill]

@ Lack
damn 4:30am?

I was monitoring it til 1:30am when I gave up...this is why I leave these things to you

@ Everyone
We will be looking at other data centers, but as I'm sure you're aware, moving's not an easy nor a inexpensive prospect.

We will also, of course, be following up with the Data Center to figure out this whole "it's gettin' hot in hear, so shut off all your sites" thing and if they intend to actually fix it.

Considering Rackspace's generally very good history, this is surprising to say the least.

As always, sorry for the disruptions.

Twill

[bedub1]

Do the servers have a temp sensor? Set them to automatically shut themselves down if the CPU exceeds 70 degrees C and if the ambiant air/motherboard temp exceeds 60 or so degrees C. Then the next time this happens, CC will be down, but you won't experience any corruption in the database and it *should* be quick and easy to bring everything back up.

Their techie explained to me that this hardware-level auto-shutdown is being done and is part of the problem. The database engine doesn't get a chance to close properly when the power gets cut suddenly and that messes up the data (but only data stored in MySQL InnoDB tables).

I'm looking into ways to prevent sudden power loss from corrupting this data.

The software I've seen, will send a command to the OS to shut down. The OS then sends a command to the database engine that "hey...the computer is turning off.....you need to shut your shit down". I know there are scripts that can be written to manually close the database etc. When my power gets lost, my server runs till 50% power left...then issues a command to itself, and all my other workstations, to power off. Vista/XP just shuts the system down properly....just like you hit start- shut down - but sometimes the script might need to be modified to send commands to the database prior to sending commands to the OS. I know it's possible...and an ounce of prevention can keep away serious disaster....

[bedub1]

@ Lack
damn 4:30am?

I was monitoring it til 1:30am when I gave up...this is why I leave these things to you

@ Everyone
We will be looking at other data centers, but as I'm sure you're aware, moving's not an easy nor a inexpensive prospect.

We will also, of course, be following up with the Data Center to figure out this whole "it's gettin' hot in hear, so shut off all your sites" thing and if they intend to actually fix it.

Considering Rackspace's generally very good history, this is surprising to say the least.

As always, sorry for the disruptions.

Twill

Sorry to double post.

Cell sites have massive batteries in them to keep them powered up when cut off from AC power. In Vegas, the AirConditioning will also die when you loose your AC power. Thus...there are thermostatically controlled DC fans to keep air circulating and prevent the destruction of the equipment due to overheating. It's not a long term solution...once you notice the power is down (automatic alarms send e-mails and text messages etc) then you need to get a generator onsite to power it back up, charge the batteries, and run the Air Conditioner.

[Caleb the Cruel]

Well I'm in the complainer's boat. I'm no computer techie, but I know that in 2008 this is unacceptable. There must be easy ways to prevent such a disaster, such as switching to a more reliable server like others have stated. We pay to play, not to say "Ah man, CC is down again!". I've paid, let me play when I want.

[KLOBBER]

As always, sorry for the disruptions.

Twill

It would be nice if you would get as sick of "always" saying that as we are of "always" hearing it, and take immediate practical steps to prevent any future disruptions, BEFORE they occur. (Among professionals, prevention of shabby service issues before the fact is called a "win-win" situation).

It is highly unprofessional to be constantly waiting in a state of impotence for the next disruption, fixing it only after the fact, and finding yourself in the embarrassing position of "always" having to apologize.

An ounce of prevention is worth a pound of cure.

You can use our premium money to implement any and all necessary steps, if you like. Just do it soon!

[Frop]

Bullshit yet again, that's obviously the crappiest excuse ever. If they had decently configured UPSs all the systems would have at least 10-15 minutes to shut down properly (a 'soft' shutdown initiated by said UPS). On top of that you even forced me to agree with KLOBBER (out of all people) for a change.

As always, sorry for the disruptions.

Twill

It would be nice if you would get as sick of "always" saying that as we are of "always" hearing it, and take immediate practical steps to prevent any future disruptions, BEFORE they occur. It is highly unprofessional to be constantly waiting in a state of impotence for the next disruption, fixing it only after the fact, and finding yourself in the embarrassing position of "always" having to apologize.

[Markomuncho]

Man this screwed me over big time!!
I in a torrie where am playing 22 games at same time, I missed a go in near every one of the games so behind with cards and been hit with full sets, I go no chance of a come back now.
how come you only add 10 hours???
I can only log on once a day, you ecpect me to get up in the night too see if you fixed it yet.

Not a happy chappy

[brandoncfi]

I though the downtime was great...it forced me to mow the lawn out of bordom

[ksslemp]

What's a SERVER?

[cena-rules]

As long as it happens whilst Im asleep or at work Im not bothered either way

[Gozar]

how come you only add 10 hours???
I can only log on once a day, you ecpect me to get up in the night too see if you fixed it yet.

Not a happy chappy

This person makes a good point. Your opening page even states that I can take my turn "with my morning cup of coffee". But if the site is down in the morning, there is no more coffee until tomorrow.

So how about a 24 hour add on after a server disruption?

[saaimen]

sounds like a plan

[KLOBBER]

A better plan would be having no server interruptions, wouldn't it?

[Incandenza]

I'm no computer techie, but I know that in 2008 this is unacceptable. There must be easy ways to prevent such a disaster, such as switching to a more reliable server like others have stated. We pay to play, not to say "Ah man, CC is down again!". I've paid, let me play when I want.

I'm not sure what planet you and the other complainers live on, but on Earth, computers are occasionally unreliable. Utter, 24/7/365 reliability, like you might see on espn.com or cnn.com or fark (which itself goes down every so often), costs great heaping gobs of money and still doesn't guarantee anything.

Expecting any website with a small handful of paid employees and a limited revenue stream to perform flawlessly, always, is lunacy. Besides, in the last 12 months, CC has been down for, what, 24 hours total? Maybe 36? You guys are getting up in arms about that? Seriously? You don't feel like you're getting your 25 bucks worth?

[gloryordeath]

I had two games I had to redo and did worse in them, I would expect no less in my luck. But thanks for getting it up Lack

[Mr_Adams]

no free speed games this time? darn

[Caleb the Cruel]

I'm no computer techie, but I know that in 2008 this is unacceptable. There must be easy ways to prevent such a disaster, such as switching to a more reliable server like others have stated. We pay to play, not to say "Ah man, CC is down again!". I've paid, let me play when I want.

I'm not sure what planet you and the other complainers live on, but on Earth, computers are occasionally unreliable. Utter, 24/7/365 reliability, like you might see on espn.com or cnn.com or fark (which itself goes down every so often), costs great heaping gobs of money and still doesn't guarantee anything.

Expecting any website with a small handful of paid employees and a limited revenue stream to perform flawlessly, always, is lunacy. Besides, in the last 12 months, CC has been down for, what, 24 hours total? Maybe 36? You guys are getting up in arms about that? Seriously? You don't feel like you're getting your 25 bucks worth?

Perfection is not possible, I know that. However improvement is possible and should be expected! Downtimes are happening more frequently rather than less frequently which is simply not right. And no, I do not feel I have gotten what I paid for. I paid to play as many games as I want, when I want. Not just when the server decides to have a good day.

[yeti_c]

Considering Rackspace's generally very good history, this is surprising to say the least.

Note to others - Rackspace supported YouTube - until Google bought them out...

C.

[MOBAJOBG]

Conquer Club was down for 10 hours

The cause was similar to the mishap on June 15:

At approximately 5:00 P.M. CDT, our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment. We began to bring affected customers back online as temperatures stabilized at approximately 7:30pm CDT.

Our database server got too hot and shut down, corrupting the data. When they got the server back online the database engine wouldn't even run. Our last backup was 13 hours old, and it would have sucked to roll back the games and user accounts so far back in time. Fortunately the hosting company's database expert was able to fully recover all the data. As I requested he called me at 4:17 am to tell me it's ready so that I can add 10 hours to all games before putting the website back online.

So there you have it.

My apologies for the downtime, at least your games shouldn't be affected this time (except for speed games, please submit a support ticket if your speed game was ruined).

I'm going back to sleep now...

Well, I've to disagree with that statement because I definitely remember that I've taken over S.America successfully but it is now showing me that I've ran out of time without even bothered to deploy my 8 armies.
http://www.conquerclub.com/game.php?game=2806331
2008-07-09 22:21:44 - Incrementing game to round 2
2008-07-09 22:22:35 - MOBAJOBG receives 3 armies for holding Africa
2008-07-09 22:22:35 - MOBAJOBG receives 5 armies for 16 territories
2008-07-09 23:22:35 - MOBAJOBG ran out of time
I'm disappointed as I did experience superb dice which gave me the S.America continent.

I'm clearly at the mercy of my opponent since both Oceania & S.America except for 1 territory each are within red's grasp.

Okay, I've got an even better than superb dice on Round 3 so I'm well into the lead now ...looks like everything is definitely under my control.

[bedub1]

I forgot to mention: Thanks for staying up all night Lack to fix the server when you got the chance at 4am....I appreciate your dedication....i just wish you didn't have to do it. I still think you should be sipping margaritas on a beach someplace....

[Mr_Adams]

we all aplaude Lack's hard work. If I had a credit card, I'd buy membership just as a way of saying "good job Lack"

[Twill]

A better plan would be having no server interruptions, wouldn't it?

Now I wonder why we didn't think of that plan...

Despite what some people seem to think, we don't actually like down time either and we are constantly trying to improve the level of service that we can offer.

We're working with the Data center, checking about moving servers to other centers with more stable power and cooling systems and seeing what we can do on the software level to better manage power cuts and data corruption.

If that fails, and Rackspace cannot get this data center under control, or move us to one which is more reliable, we are also prepared to explore alternative options, hosts and platforms that might offer more flexibility and stability.

Rackspace has treated us well thus far, and has helped us through scaling problems, difficult technical issues and has generally been very responsive to problems - even at 4am. There are a lot of reasons for us to trust that they are working on the problems as they say they are. But as I said, we will be exploring other options if they persist.

Twill

[Blitzaholic]

Conquer Club was down for 10 hours

The cause was similar to the mishap on June 15:

At approximately 5:00 P.M. CDT, our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment. We began to bring affected customers back online as temperatures stabilized at approximately 7:30pm CDT.

Our database server got too hot and shut down, corrupting the data. When they got the server back online the database engine wouldn't even run. Our last backup was 13 hours old, and it would have sucked to roll back the games and user accounts so far back in time. Fortunately the hosting company's database expert was able to fully recover all the data. As I requested he called me at 4:17 am to tell me it's ready so that I can add 10 hours to all games before putting the website back online.

So there you have it.

My apologies for the downtime, at least your games shouldn't be affected this time (except for speed games, please submit a support ticket if your speed game was ruined).

I'm going back to sleep now...

awesome, ty lack