Export thread

Yesterday's Halforums Clusterfuck.

#1

Dave

Dave

So I go to a meeting yesterday and it's boring as hell. I get back and the site is down. Completely down and the URL does not resolve at all. "Uh oh, " I say to myself. "What the hell is this?"

So I email the host and open a ticket, receiving the following:

Hi David,
It is...We've email a couple of time. The drive has been struggling the last few week and is in need of more space.

We've cleaned everything we known we can clean but are not the one to prune farther.
This is from our monitoring:

VPS726 (HalForums): Disk Percent Used ( / )(SNMP Custom) Down ESCALATION (96 % (Value) is above the error limit of 95 %)

For about $10 a month we can add and attentional 10 gb of disk space that will work wonders in preventing these outages.
My first order of business was to see what space I could save. Ame is going to shit and it's a damned good thing I'm looking at XenForo because I had to remove the test site to free up space. Ame put a lot of work into that and it's just gone - but I didn't have much choice as I needed to clear up space.

Speaking of space, my email to the host back was not very nice and ended with this:

But I still have concerns that I'm paying for 45 GB RAID space and my stats say I am only using about 19 GB and am told I'm over.

Filesystem - /dev/sda1
Size - 18G
Used - 12G
Avail - 4.7G
Use% - 72%
Mounted on - /
So why they said we were over limit has been a bone of contention for a number of months, but the fact that we got pulled down chapped my ass.

The response I got back was:

David, we did not, nor would we take your site down...The server is struggling and has been rebooted 2x since my 17:17 email.

The 45 GB, I don't know about and will defiantly check.

The only thing I know this minute is that for the past few weeks your server has had space issues and required constant monitoring.

Let me check the sales agreement and find out just what is going on.

btw at 72 % your users will be happier too.
So I sent back a nicer message saying basically I don't care what's happening but we need to be back up. They couldn't do it remotely and had to send someone on-site to reboot & swap out drives. We finally came back up.

In the end this may end up being a good thing. The last email I got was:

I Just took a look and Yes you should have a 45gb allocation. I check when I reboot. We may have to stop the drive to correct the space allocation. That can be done later
So the $155 a month I'm paying is finally going to be buying what we're supposed to be getting. This means that we'll be able to have our test bed without taking shit from the host.

In the end (otherwise known as tl;dr), I'm sorry for the downtime and we'll have to be down 15-20 minutes (when we work out the time) so they can set us up on the 45 GB that we have been supposed to have for months.

I'll give fair warning.


#2

Dave

Dave

Quick note - the site will be going down tonight at 10 PM CST to do the upgrade.


#3



Matt²

I was too busy watching Night Court reruns yesterday, I didn't notice.


#4

Krisken

Krisken

Good to know the problem is being fixed without further cost, Dave. It is irritating to find out that they have been charging for 45 gigs but for some reason had you on 20 gigs.

They're all like "Durr, your space is drying up."
and you're all like "Read the fucking contract!"
and they are all like "Durr. Oops."

Give 'em hell, Dave.


#5

drifter

drifter

The day I decide I need to take a break from Minecraft, and I couldn't access Halforums. Hate youuuuu (the hoster, not Dave)


#6



Disconnected

So are you getting 'credit' on the fact you paid for 45 all those months?


#7

Dave

Dave

Good question. And after tonight when it gets fixed I'll be sure to ask.


#8

Hylian

Hylian

Sounds like I missed all the fun last night


#9

KCWM

KCWM

They tried to Kurtz us.


#10

ThatNickGuy

ThatNickGuy

WE'RE ALL GONNA DIE!!


#11

strawman

strawman

You're paying $155 per month for 45GB and this -insert expletive here- level of service?

I'd have them over a barrel for their incompetence, especially given the high rate you're paying. Even great hosts make occasional mistakes, but the one idiot wasn't even willing/able to check what type of account you had before responding?

I'd carefully look over their SLA and request a refund for previous months of overpayment, as well as a refund for the downtime which was obviously their fault. I'd also make it clear that you lost X hours of professional website development and that you need them to retrieve just those files from a recent backup (at that rate I hope they are doing full daily backups for you!).

Are you sure they are a good host? Seems like you're overpaying for terrible service. If they've been "monitoring" this "space problem" for the "past few weeks" but never notified you that there was even a problem, then indicate that your account is lower capacity than you're paying for - honestly, they are simply lying through their teeth and trying to cover up some internal problem. It wouldn't surprise me if they we reselling VPS hosting from some other service, and not paying enough attention to their logs. The "reboot" is likely them moving your instance to another server since you are one of the few customers that actually uses the resources you're paying for.

Oh well, them's the breaks I suppose. Don't be a nice guy about this - if their processes and administration capability are broken, you need to know sooner rather than later.

Also, make regular offline backups of your important data. Compressed it should be very small, and a once a week backup of all of your and Ame's work, as well as the databases will give you significant peace of mind. It'll also free you from thinking that you are trapped at a given host - you can have the site set back up in a few hours on another service means you can tell them, "My server is down. I've verified that it's not my fault. You'll lose my $150+ per month if you can't fix it faster than I can spin it up on another host."


#12

North_Ranger

North_Ranger

Thanks for the update, Dave. I was quite worried something was amiss... Glad to know we're back on track again.


#13

GasBandit

GasBandit

Hmm. I think I'll make text file copies of all my blog posts, just in case.


#14

MindDetective

MindDetective

I kept checking Twitter and didn't see an update!


#15



Matt²

WE'RE ALL GONNA DIE!!
They tried to Kurtz us.
AAAAAAAAAHHHHHHHHHHHHH!!!!!!!!!!!!!!


#16

Dave

Dave

I kept checking Twitter and didn't see an update!
Not until I noticed it was happening - which was a couple hours into it.


#17

MindDetective

MindDetective

I feared it might have been something like that.


#18

SpecialKO

SpecialKO

Agreed with the Puffster. Kick some ass, Dave. Gently, in small increments at a time, if necessary, but someone's ass needs to get kicked and have either refunds or comped service pouring out their mouths.


#19

Dave

Dave



Fuck yeah. That's more like it!

Let the ass-kicking commence.


Top