Lessons learned … again

I re-learned a few old lessons last week.

  1. 1&1 suck like it’s going out of fashion
  2. The only backup is a full disk image backup
  3. I am not a Linux sysadmin

And finally a lesson for 1&1 .. good customer service is about exceeding expectations not covering your arse*.

1. I had a budget root server with 1&1 internet for about 5 years. Apparently about a month ago 1&1 emailed me stating that they were going to physically move my server, and that they would not be responsible for any data loss during that time. I say apparently because I don’t remember it. It’s quite likely I thought it was a marketing email and trashed it without a second look, or maybe it got junked… who knows.

Anyway,  they switched my server off, moved it, and switched it back on. The hard disk which was running for 5 years basically had a heart attack and died. I was told by a 1&1 call centre chap that “this is expected”, and of course they don’t back it up for you.

2.  Fortunately I was backing up my databases using a home-made combination of mysqldump and [the fabulous] rsync, but it was only my personal server for tinkerings such as this blog, so I wasn’t paying for a full mirror. ( Error ). Not only was there an absolute tonne of software, and highly custom configurations, but I had totally forgotten to backup the file uploads on this blog. ( s%&t! ). So I’m really sorry for the 404s, especially on my most popular post, where the demo files are currently AWOL; I shall work on reinstating these, because half my traffic goes to this article.

This blog is now running on a small VPS at Slicehost. and for $5 per month I keep a daily, weekly and adhoc mirror of the complete image. Marvelous :)

 3. I am a web developer, not a Linux sysadmin, but the lines blur … a lot. If you’re a back end developer working for a fairly small company, you will have to do a lot of this stuff. It actually makes you a much better developer, but installing a complete operating system is not just a massive pain in the arse*, but there are lots of issues you’re probably not qualified to tackle. (Security, performance, maintenance, etc.. ). The magic of yum got me a working LAMP server within about 10 minutes from a base Fedora 10 install, but I spent my weekend configuring Postfix, and Dovecot and generally ensuring the system was in reasonable shape. As usual, it took lots of Googling, forum browsing, and chatting to the excellent Slicehost support people – This is not what I do for a living and I’m not particularly good at it either. I build web apps, and I need to remember that.

* UK Glossary: arse = ass