[Ipg-smz] When MSPs Let You Down

[Tom Henderson]

Fellow Guilders,

I do volunteer work for a community radio station. It has a modest 
pledge support, and support from CPB. It's volunteer-staffed, save three 
full time employees & two part timers. It also has a very popular 
website. Oddly, crawlers for Google and podcasters and a strange gaggle 
of info-gleaners make up the majority of its traffic. There have been 
six IT regimes that have managed the site since its inception in 2009; 
all were local volunteers that had ongoing clues about what the site 
should be. It went from static pages through Drupal to where it is 
today-- Wordpress with a now-unsupported CMS theme originating in Cairo. 
The one in Egypt.

The site had Dreamhost, a pioneer in hosting methods, as its original 
host. Over the years, barnacles and rust and cruft overtook the site. 
The local community likes having a historical record of its content, as 
the local birdcage liner had a crappy paywalled site largely bereft of 
meaningful content. The radio station's site is constantly referenced 
for its historical record, and its allegiance with the town's pioneering 
Community Access Television Service content. Good journalism.

The station records all of this, it's on-air content, live music 
recordings, and news/public affairs content in MP3s within the Wordpress 
theme/CMS. It has grown to 200GB in size. Underneath Wordpress is MySQL 
(or equivalent) whose database size grew to nearly a gigabyte over the 
years. This is a pretty thick site.

The hosting was on a shared plan. Didn't cost much. About four years 
ago, it had sufficient activity that Dreamhost would arbitrarily take it 
offline as it was using too much CPU on the shared host where it lived. 
Dreamhost didn't have a hosting plan that could take both the size and 
the CPU activity the site generated for a rational price. It caused 
increasing disruption as Dreamhost took it offline. I was asked to take 
a look and see what I could fix. That long journey was finished over the 
past weekend.

I installed WordFence as a plugin, so that I could stanch site crawlers 
but also the myriad hackers whose automated bots would try to login and 
take over the website. Hundreds of hack attempts every day came in from 
across the planet. Over a few weeks, I also found crawlers with no 
identity vacuuming the content, which if you'll remember, is extensive 
and ultimately added to CPU loads, thus causing Dreamhost to flip the 
F-U switch until things quieted down. Then, gratuitously, they'd turn 
the site back on. Eventually, I blocked all but North American 
countries, banning the rest of the planet, as this was a community radio 
station, and not Voice of America. A few volunteers growled at this, but 
hark-- the station stayed below the ceiling on CPU load, and the admin 
activity for this volunteer, me, went back to a minimum.

Eventually the content size grew too large, and Dreamhost didn't offer 
any rational plans to accommodate the size/load combo. I tried a 
migration to Name.Com, who turned out to be bunglers, fools, idiots, and 
posers. I have the docs to back up this assertion. Eventually, I made my 
way to HostGator. Both Dreamhost and Name.com do NOT have 24/7 support, 
but HostGator does. Indeed their Indian (Bangalore) engineers who work 
on the late shift (late evening NYC/EDT) are blackbelts. The first 
migration attempt failed, and so over last weekend, I attempted another 
(the plan is VERY cost effective) doing a manual site migration.

Like all other MSPs, HostGator prefers that you let them do it, but 
because that act is free (as in beer), it's on their schedule, and 
they'll flip the switch when they're good and ready. This policy is 
rough for production websites. Therefore, I did a manual migration 
because: I can. HostGator left out a piece of totally-needed docs as 
regards configuration that was eventually revealed. The 200GB migration 
took about four hours to transfer (an amazing speed, really), then more 
time for configuration before I transferred DNS from Dreamhost to 
HostGator, this completing the transfer for users. The switch was 
flipped. The site worked for a few minutes because of the unrevealed 
documentation.

Dreamhost screwed up in the midst of this. Their MySQL admin program 
(PHPMyAdmin for those in the know) was an ancient version that coughed 
blood when exporting just 132MB of the huge crusty tables. Eventually, I 
had to CLI export the site's database, and manually transfer and import 
the old database into HostGator. This is where HostGator failed, as they 
would have two key settings that required changes before the site would 
work for users. After banging on HostGator support and even their 
Twitter feed, an admin woke up, flipped the switches, and sent a 
nastygramme to me and the station's manager about waking him up. Oh, and 
he had the title of Webmaster III. Hostgator does not give you root 
access to the database, and so admins have to do root-auth'd tasks. The 
newly revealed switches that needed to be flipped (and an unrevealed 
auth rule) turned the site on like a holiday tree. Nonetheless, the site 
has been stable for 48hrs as I write this, and was down for only 12hrs.

There are a few more assets to transfer, but they're trivial. 
HostGator's front-line support was superior to Dreamhost and Name.com. I 
shall not grace their firewalls again.

Much of the world surrounds activities that can be described as small 
business, and the station is small. It's not an NPR affiliate, rather, 
totally independent. The bar of having an effective voice on the Internet.

The net sum is this: Some MSPs will go the extra mile, even if they're 
not perfect. Sometimes it really is best to let Darwin be your guide.

Tom

-- 
Tom Henderson
ExtremeLabs, Inc.
+1 317 250 4646
Twitter: @extremelabs
Skype: extremelabsinc