Server Go Boom

So last thursday I find the server that houses this site, my domain, my pictures, is unavailable. The server actually belongs to a friend who runs a business on it. I have helped him over the years keep the machine and the site up and running. It’s been housed on the same box since at least 2002. Prior to that I ran it out of my house starting in 2000. Most of my email flowed through this domain. I’ve had a website of some sort since that time and still get hits on some older, educational material. There are some other non public bits that are useful to me as well. And even though I go through bursts of activity followed by long stretches of public silence, I really didn’t want to just let it go.

So I found myself scrambling first to get my friend’s site back up since he makes some part of his livlihood off of the machine. Working with the hosting vendor, it appears to have been a hardware problem. The server shut down and started spontaneously rebooting. It wouldn’t come back up.

What makes this particularly ironic is that my day job consists largely of running the backup and disaster recovery infrastructure for a major corporation. So it would have been wildly embarassing if I had not kept reasonably current backups of my own site. I did, of course. (phew!) But this event definitely pushed me to make some changes I had long been considering.

The first was related to email. Even if I let the website dissapear, I couldn’t get rid of the scottharney.com domain that I had been using since 2000. We had always run our own mail server on this machine. Back years ago, this was no big deal: slap qmail or postfix on a box (boo sendmail), point your MX records at it, and stand up a pop3 listener and away you went. It grew into a beast over the years. The commercial side used generic addresses like “sales@” and “help@” to support its customers. They received tons of spam. so we had to implement spamassassin with bayes learning server wide. I layered in tmda on certain addresses. Eventually their was qmailadmin, squirrelmail, IMAP, SSL, and an MDA listener on port 587.

Still, 95% or more of SMTP traffic was spam. As it turns out, google hosts mail at Google Apps. You point your DNS MX records at them and they will host your domain. At no cost. It includes POP3, IMAP (over SSL of course), mail transfer on port 587, and of course web-based gmail. gmail does a fine job of spam filtering. so I scrambled to point domains at it. Pretty quickly I added google calendar sync and a gmail client to my work blackberry. It’s really feature rich and I should have done it a LONG time ago.

The other thing I did was start putting my pictures on Flickr instead of locally. I still like album, but flickr is just plain easier and more feature rich.

Then I finally got around to getting wordpress back up. It was actually pretty easy, just copy back over all the website parts that were their before, create an empty database, and then import my sql backup. At some point, I may go ahead and pay the piper and move all this to slicehost, but for now, I’m settled again.

So, just a little reminder to take freqent backups and think really hard about how you would recover in the event of a worst case, server go belly up, kind of disaster. It can happen to you.


Scott Harney

   (GPG key)
<>

Resume


An online copy of my resume (PDF)

Photo Album


My current pictures via Flickr.
Older family pictures.

    Wedding


    I got married on 9/4/2004. So click for details, already.

    Old stuff


    Links and writings from older versions of this site
    Old stuff
    Oldest stuff

    Free DNS