Export thread

What the hell was all that?!?

#1

Dave

Dave

Seems we had a DB crash. Our search index kept crashing and the db was repairing it over and over. So I contacted the host and we attempted to fix it, but it just wasn't happening. So we reverted back to the backup, truncated the table and am reindexing the search to rebuild the db table.

So we're back and all is well.


#2

Gared

Gared

Oh, so that's what was going on early this morning PDT; when it kept telling me that it couldn't post my responses due to a server error, but kept posting them anyway?


#3

Dave

Dave

Oh, so that's what was going on early this morning PDT; when it kept telling me that it couldn't post my responses due to a server error, but kept posting them anyway?
Yup. The search index was corrupted and was being rebuilt over and over until the server said, "Fuck it!" and just crashed instead. I had a hard time getting into it to repair and when I tried it hung on me. I had to have the host kill the process. Once that was done I emptied the table so that the db was working, rolled back the files to last night's backup, and we were up. Search will be wonky until we finish rebuilding the cache, but it's nearly done.[DOUBLEPOST=1364485591][/DOUBLEPOST]Good morning!


#4

Ravenpoe

Ravenpoe

Oh God... Where did we go?

Were we dead?


#5

Dave

Dave

Oh God... Where did we go?

Were we dead?
As close as we've ever been! Worst case scenario is we would have lost everything from last night until this morning. We have daily backups of the DB and site so we will never lose more than 24 hours at the most.


#6

ThatNickGuy

ThatNickGuy

Oh God... Where did we go?

Were we dead?
Purgatory.

Don't worry. We'll all meet up in the church later for a reunion.


#7

Dave

Dave

I should bring the site down Friday and then fix it Sunday.


#8

ThatNickGuy

ThatNickGuy

B-b-but...then we'd have to wait to discuss the new Doctor Who!

You can't do that to us, Dave. You just can't!


#9

Hylian

Hylian

I should bring the site down Friday and then fix it Sunday.


#10

GasBandit

GasBandit

I noticed the problems, but I don't seem to be missing anything that I've noticed yet.


#11

ThatNickGuy

ThatNickGuy

I noticed the problems, but I don't seem to be missing anything that I've noticed yet.
I unhooked your bra.

...wait, hang on.


#12

PatrThom

PatrThom

Was on vacation the past week, didn't notice anything at all.

--Patrick


#13

jwhouk

jwhouk

Might not be a good idea to take anything down, not with the largest DDoS attack in 'net history going on as we speak.


#14

Dave

Dave

Might not be a good idea to take anything down, not with the largest DDoS attack in 'net history going on as we speak.
I seriously thought that's what happened, but then I started getting the SQL errors while still being able to hit the server. So it wasn't a connectivity issue. When I went to the database directly, the table was hung and not moving. Which is a problem. Or was. Hopefully was.


#15

Bubble181

Bubble181

The ONE time it's calm at work and I have time to visit HF during my lunch hour, and I get "we're upgrading, check back later" for the whole time. Pfuh!
Also, bunch of meanies giving Dave the 10 different thingies on his post so I can't like it without disturbing the one-of-each order...*twitch*


Top