[10-Jul-2008] Downtime yesterday

[lackattack]

Conquer Club was down for 10 hours

The cause was similar to the mishap on June 15:

At approximately 5:00 P.M. CDT, our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment. We began to bring affected customers back online as temperatures stabilized at approximately 7:30pm CDT.

Our database server got too hot and shut down, corrupting the data. When they got the server back online the database engine wouldn't even run. Our last backup was 13 hours old, and it would have sucked to roll back the games and user accounts so far back in time. Fortunately the hosting company's database expert was able to fully recover all the data. As I requested he called me at 4:17 am to tell me it's ready so that I can add 10 hours to all games before putting the website back online.

So there you have it.

My apologies for the downtime, at least your games shouldn't be affected this time (except for speed games, please submit a support ticket if your speed game was ruined).

I'm going back to sleep now...

[vikprez]

I missed CC today...but it happens. Keep up the good work

[richardgarr]

night lack, glad it all worked out.

[InsomniaRed]

Sleep well.

Glad the games weren't affected.

Can I send in a support ticket for being bored out of my mind all day?

[Incandenza]

All evening, I envisioned you hunched over a keyboard, stoked to the gills on coffee and red bull and cigarettes and cheap gin, muttering to yourself as you endeavored to get your pride and joy back up n' running.

Thanks for the added time. And well done, sir.

[xeno]

These things happen. It's cool.

[edwinissweet]

saweet

[yeti_c]

Good work Lack - I hope the hosting company is compensating you for their inadequate air con system.

C.

[Tense]

Good work Lack - I hope the hosting company is compensating you for their inadequate air con system.

C.

I certainly hope so aswell.

/Tense

[JoshyBoy]

Good work Lack - I hope the hosting company is compensating you for their inadequate air con system.

C.

I certainly hope so aswell.

/Tense

Ditto... fortunately i was not affected by the downtime

[danodukebb]

yes, fortuently for me i got off work at 4 so i could do what i needed to do, and also i did not sleep the night before so i passed out at 6 and woke up now!

[Iffy Boatrace]

I guess you're already considering which provider to switch to. Twice in quick sucession is not good.

[snufkin]

it did of course affect those who were making their turns while the server went down (couldnt finish forting or get a card)
you should move the servers to alaska!

[MOBAJOBG]

Conquer Club was down for 10 hours

The cause was similar to the mishap on June 15:

At approximately 5:00 P.M. CDT, our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment. We began to bring affected customers back online as temperatures stabilized at approximately 7:30pm CDT.

Our database server got too hot and shut down, corrupting the data. When they got the server back online the database engine wouldn't even run. Our last backup was 13 hours old, and it would have sucked to roll back the games and user accounts so far back in time. Fortunately the hosting company's database expert was able to fully recover all the data. As I requested he called me at 4:17 am to tell me it's ready so that I can add 10 hours to all games before putting the website back online.

So there you have it.

My apologies for the downtime, at least your games shouldn't be affected this time (except for speed games, please submit a support ticket if your speed game was ruined).

I'm going back to sleep now...

Well, I've to disagree with that statement because I definitely remember that I've taken over S.America successfully but it is now showing me that I've ran out of time without even bothered to deploy my 8 armies.
http://www.conquerclub.com/game.php?game=2806331
2008-07-09 22:21:44 - Incrementing game to round 2
2008-07-09 22:22:35 - MOBAJOBG receives 3 armies for holding Africa
2008-07-09 22:22:35 - MOBAJOBG receives 5 armies for 16 territories
2008-07-09 23:22:35 - MOBAJOBG ran out of time
I'm disappointed as I did experience superb dice which gave me the S.America continent.

I'm clearly at the mercy of my opponent since both Oceania & S.America except for 1 territory each are within red's grasp.

[mibi]

I guess you're already considering which provider to switch to. Twice in quick sucession is not good.

amen. You'd think for what Rackspace charges, they would have a better operation or partners.

[GrimReaper.]

things happen great job on geting the site back Up though.

[KLOBBER]

Dear Lack,

I'm sure that everyone understands that this is not your fault.

However, this happens so often that I'm thinking maybe it's about time to look into switching over to a more reliable server. There must be more reliable servers out there, somewhere, as this kind of thing doesn't happen so often to other websites. Also, due to the time-sensitivity of your data (our data), this problem is more serious here than it would be on other websites.

Surely I'm not the only one who can understand this. Right?

[t-o-m]

ah well, i was just about to sleep anyway

[JACKAZZTJM]

night lack, glad it all worked out.

sorry but that was kinda gay

[Optimus Prime]

Dear Lack,

I'm sure that everyone understands that this is not your fault.

However, this happens so often that I'm thinking maybe it's about time to look into switching over to a more reliable server. There must be more reliable servers out there, somewhere, as this kind of thing doesn't happen so often to other websites. Also, due to the time-sensitivity of your data (our data), this problem is more serious here than it would be on other websites.

Surely I'm not the only one who can understand this. Right?

It's happened only a handful of times in the entire time I've been here. Yes, it happened twice this month, but as you said yourself, it was entirely not CC's fault. I'm sure that Lack will take care of us just fine.

[cairnswk]

Thanks for the heads-up lackattack...but seriously i am in the camp of thought that perhaps U should start investigating moving the site to more reliable providers. Twice in one month for a similar situation is most unheard of these days, and "shit happens" is not necessarily a good excuse. As expressed, I'm sure you'll look after after us all though. Hope u don't have this challenge to contend with again soon.

[gdeangel]

Well, I'm going to buck the love-in trend here. This site interruption does not come as a surprise because the site has noticeably been overloaded at many times in the past few days and weeks, even though there may not have been a complete collapse. The periodic "hiccups" on this site, where stuff times out repeatedly, may go without official announcements or action until the server overheats, and of course servers can overheat because of exogenous forces like problems with the public utility, but I can't help think that there is a little too much shoe string and chewing gum for what you are trying to pull off here.

Now before you pan me because I'm freemium, one way to deal with this is to limit freemium to 1 game, and offer some revenue generating option that looks like the current freemium, perhaps with a few other bells and whistles. And to mitigate the effects of peak load to do massive build game auto attacks (i.e., make auto-attack have a maximum read limit of 50-100 dice lines) and rework the architechture of freestyle games to be hosted on separate servers or more distributed client side load.

I haven't worked on this type of thing for years, but clearly there are problems going on under the hood. The stuff I'm talking about might not be relevant any more, I don't know, but pretending that the site usually hums along fine is just not facing reality. Is there any continuous QA, or only when things break?

[chipv]

Well, I'm going to buck the love-in trend here. This site interruption does not come as a surprise because the site has noticeably been overloaded at many times in the past few days and weeks, even though there may not have been a complete collapse. The periodic "hiccups" on this site, where stuff times out repeatedly, may go without official announcements or action until the server overheats, and of course servers can overheat because of exogenous forces like problems with the public utility, but I can't help think that there is a little too much shoe string and chewing gum for what you are trying to pull off here.

Now before you pan me because I'm freemium, one way to deal with this is to limit freemium to 1 game, and offer some revenue generating option that looks like the current freemium, perhaps with a few other bells and whistles. And to mitigate the effects of peak load to do massive build game auto attacks (i.e., make auto-attack have a maximum read limit of 50-100 dice lines) and rework the architechture of freestyle games to be hosted on separate servers or more distributed client side load.

I haven't worked on this type of thing for years, but clearly there are problems going on under the hood. The stuff I'm talking about might not be relevant any more, I don't know, but pretending that the site usually hums along fine is just not facing reality. Is there any continuous QA, or only when things break?

This has got nothing to do with the outages -

our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment.

Reducing load is necessary but are not going to prevent these scenarios.

[happy2seeyou]

Funny thing.... I was on another site and was telling a few people how they should join CC because it's a fun site... When they went to the site to look around.. NOTHING. ouch

[cleveridea]

Not a great testimony for the LAMP stack, specifically the M.