[Ipg-smz] When MSPs Let You Down

Wed May 29 15:39:21 UTC 2019

Great read!

On Wed, May 29, 2019 at 8:02 AM Tom Henderson <thenderson at extremelabs.com>
wrote:

> Fellow Guilders,
>
> I do volunteer work for a community radio station. It has a modest
> pledge support, and support from CPB. It's volunteer-staffed, save three
> full time employees & two part timers. It also has a very popular
> website. Oddly, crawlers for Google and podcasters and a strange gaggle
> of info-gleaners make up the majority of its traffic. There have been
> six IT regimes that have managed the site since its inception in 2009;
> all were local volunteers that had ongoing clues about what the site
> should be. It went from static pages through Drupal to where it is
> today-- Wordpress with a now-unsupported CMS theme originating in Cairo.
> The one in Egypt.
>
> The site had Dreamhost, a pioneer in hosting methods, as its original
> host. Over the years, barnacles and rust and cruft overtook the site.
> The local community likes having a historical record of its content, as
> the local birdcage liner had a crappy paywalled site largely bereft of
> meaningful content. The radio station's site is constantly referenced
> for its historical record, and its allegiance with the town's pioneering
> Community Access Television Service content. Good journalism.
>
> The station records all of this, it's on-air content, live music
> recordings, and news/public affairs content in MP3s within the Wordpress
> theme/CMS. It has grown to 200GB in size. Underneath Wordpress is MySQL
> (or equivalent) whose database size grew to nearly a gigabyte over the
> years. This is a pretty thick site.
>
> The hosting was on a shared plan. Didn't cost much. About four years
> ago, it had sufficient activity that Dreamhost would arbitrarily take it
> offline as it was using too much CPU on the shared host where it lived.
> Dreamhost didn't have a hosting plan that could take both the size and
> the CPU activity the site generated for a rational price. It caused
> increasing disruption as Dreamhost took it offline. I was asked to take
> a look and see what I could fix. That long journey was finished over the
> past weekend.
>
> I installed WordFence as a plugin, so that I could stanch site crawlers
> but also the myriad hackers whose automated bots would try to login and
> take over the website. Hundreds of hack attempts every day came in from
> across the planet. Over a few weeks, I also found crawlers with no
> identity vacuuming the content, which if you'll remember, is extensive
> and ultimately added to CPU loads, thus causing Dreamhost to flip the
> F-U switch until things quieted down. Then, gratuitously, they'd turn
> the site back on. Eventually, I blocked all but North American
> countries, banning the rest of the planet, as this was a community radio
> station, and not Voice of America. A few volunteers growled at this, but
> hark-- the station stayed below the ceiling on CPU load, and the admin
> activity for this volunteer, me, went back to a minimum.
>
> Eventually the content size grew too large, and Dreamhost didn't offer
> any rational plans to accommodate the size/load combo. I tried a
> migration to Name.Com, who turned out to be bunglers, fools, idiots, and
> posers. I have the docs to back up this assertion. Eventually, I made my
> way to HostGator. Both Dreamhost and Name.com do NOT have 24/7 support,
> but HostGator does. Indeed their Indian (Bangalore) engineers who work
> on the late shift (late evening NYC/EDT) are blackbelts. The first
> migration attempt failed, and so over last weekend, I attempted another
> (the plan is VERY cost effective) doing a manual site migration.
>
> Like all other MSPs, HostGator prefers that you let them do it, but
> because that act is free (as in beer), it's on their schedule, and
> they'll flip the switch when they're good and ready. This policy is
> rough for production websites. Therefore, I did a manual migration
> because: I can. HostGator left out a piece of totally-needed docs as
> regards configuration that was eventually revealed. The 200GB migration
> took about four hours to transfer (an amazing speed, really), then more
> time for configuration before I transferred DNS from Dreamhost to
> HostGator, this completing the transfer for users. The switch was
> flipped. The site worked for a few minutes because of the unrevealed
> documentation.
>
> Dreamhost screwed up in the midst of this. Their MySQL admin program
> (PHPMyAdmin for those in the know) was an ancient version that coughed
> blood when exporting just 132MB of the huge crusty tables. Eventually, I
> had to CLI export the site's database, and manually transfer and import
> the old database into HostGator. This is where HostGator failed, as they
> would have two key settings that required changes before the site would
> work for users. After banging on HostGator support and even their
> Twitter feed, an admin woke up, flipped the switches, and sent a
> nastygramme to me and the station's manager about waking him up. Oh, and
> he had the title of Webmaster III. Hostgator does not give you root
> access to the database, and so admins have to do root-auth'd tasks. The
> newly revealed switches that needed to be flipped (and an unrevealed
> auth rule) turned the site on like a holiday tree. Nonetheless, the site
> has been stable for 48hrs as I write this, and was down for only 12hrs.
>
> There are a few more assets to transfer, but they're trivial.
> HostGator's front-line support was superior to Dreamhost and Name.com. I
> shall not grace their firewalls again.
>
> Much of the world surrounds activities that can be described as small
> business, and the station is small. It's not an NPR affiliate, rather,
> totally independent. The bar of having an effective voice on the Internet.
>
> The net sum is this: Some MSPs will go the extra mile, even if they're
> not perfect. Sometimes it really is best to let Darwin be your guide.
>
> Tom
>
> --
> Tom Henderson
> ExtremeLabs, Inc.
> +1 317 250 4646
> Twitter: @extremelabs
> Skype: extremelabsinc
>
>
> --
> Ipg-smz mailing list
> Ipg-smz at netpress.org
> http://netpress.org/mailman/listinfo/ipg-smz_netpress.org
>
-- 

Mac McCarthy

Get my* "Wine Tasting 101" - Kindle Edition only $1.99*
*http://amzn.to/13GWHPI <http://amzn.to/13GWHPI>*
*Amazon Prime* members! *Borrow this book* *FREE*!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://netpress.org/pipermail/ipg-smz_netpress.org/attachments/20190529/e47c16ff/attachment.html>