I knew that off-site storage was necessary.  I have space on remote servers in secure locations… one of these days, I’m going to figure out a good way to, uh… hey, I gotta go pick up the pizza!

But, it’s worse than that.  I actually have a mirrored-disk, network-attached storage (NAS) box specifically for backups.  I have external drives specifically for making local backups, too.  So, short of my office burning down, I should be covered, no?  Uh… no.

Turns out that my backup software to the NAS box had been failing silently because of some change in the client configuration.  Moreover, I’d procrastinated on replacing the batteries on my UPSs– they’d been pulled out for that reason (see where this is going?  no surge protection?) Now, mix that with my preference to use “logical volumes” for my storage because of their flexibility in sizing and storage, and toss in a completely unusual (but nonetheless real) power surge/outage.

What you get is a garfed logical volume (i.e., the physical device with the filesystem information on it survived but was irretrievably scrambled…) and the local backup external disk gets completely fried at the same time.  No network backup.  No local backup.  And the data itself has vanished.  What did I lose?  Well, my company books and tax records, for one thing.  (The actual happy ending is that because of my paranoia level, I had actually imaged my drives not too long ago to yet ANOTHER drive that I kept unattached.  I was able to reconstruct most of my data, but it took hours of work and validation.  Who knows what miscellaneous data I’ll notice is missing at some point.)

As W. famously said, “There’s an old saying in Tennessee — I know it’s in Texas, probably in Tennessee… that says, fool me once, shame on… shame on you. Fool me… you can’t get fooled again.”

So true, Mr. President.  So true.

Well, I can’t get fooled again.  So, no more procrastinating.  First order of business was to order myself a shiny, new UPS.  I went with the CyberPower 1500AVR.  At less than $200, and with the juice to run my (excessive) pile of gear for at least a half hour, it’s easy and inexpensive insurance against some putz putting his backhoe into the power main.  Still, they’ll think of something that I haven’t.

So, it’s on to my data.  Firstly, I forced myself back into a rational storage model.  Being your typical ADHD type, I have a tendency to keep EVERYTHING and keep it wherever it happens to land.  Forever.  Don’t know what it is?  Better just leave it.  You never know.  Right?

No, no, no… that’s not OK.  What I need to do is to keep my documents in, say, a DOCUMENTS folder.  Branch out by use (not… say… document type or date).  OK, so, ‘business’ and ‘personal’.  Business can have ‘clients’ and ‘financial’… and so on.  Everything goes into one of those folders.  EVERYTHING.  Is is a photo?  OK, it goes into personal/photos.  Or a separate photos folder.  Not both.  That file that I can’t identify?  My system to create an “unidentified” folder with dated sub-folders.  Everything I find that I can’t identify goes in those.  If, after a month or two, I still don’t know what it is and haven’t needed it… it goes.

Great, so, now I’ve got all my data in one place on my system… that is… my desktop system… which… doesn’t include my virtual systems… or my MacBook… oh boy.

Enter on-line storage and synchronization.  After some checking around, I went with Dropbox.  Other good options include: mozy, Jungledisk, Ubuntu One, Amazon S3, and many others.  Dropbox appealed to me because of the drop-dead ease of installation, the cross-platform clients, and what seems like a pretty reasonable price for all that (about $10/mo. for 50GB of storage as of this writing).

Basically, Dropbox puts, what looks like, another folder on your system.  In this case, though, if you drop a file in there, it’s copied up to their storage and synced to any systems that you also have connected to your account.  Drop your resume into that folder on your Linux box and edit it on your Mac, output the PSD to the Dropbox folder, and open it up on your… iPod Touch?  Virtual XP instance?  Anywhere you can get to a browser?

I’ve always had a tendency to roll my own solution for this kind of thing, but the technology has advanced to the point that there is no way that I can justify the kind of effort it would take to get something like that righter than they have for that kind of money.  $120/year?  That’s one internal hard drive.  Dropbox installed and was doing its thing in about 5 minutes– on three systems.

So, we’ve got reliable, filtered power and offsite storage of critical data.  The last thing I needed to address was local backups.  Given the inexpensive drives available these days, it only makes sense to me to keep very complete backups locally so that I can do a complete recovery if the problem is a fried disk.  So, I had that pretty well covered already with FSArchiver for complete images and Back In Time for scheduled backups to the NAS box (a D-Link DNS-323).

Maybe next time some super-freaky, never-happen-in-a-million-years thing happens, I won’t have to spend the day engaged in self-loathing and fear.  I’ll just restore my stuff and keep on keepin’ on.  In the meantime, my next project is to write a little script to make sure that my backups are actually taking place.  If they’re not, I need to make something to pester me about it.

I’ll probably do that tomorrow… d’oh!

In reality, you could use MySQL for years and really know little or nothing about these database engines.  I did.  After all, if you’re setting up a database to manage, say, your office equipment in a small business, you’ll maybe have a couple of hundred rows?  You may only access the database once a week.  Who cares what the engine is?  MySQL will default to MyISAM (usually), and that will work fine.  You’ll never notice, nor care.

However, when you begin to approach almost any kind of consistency, performance, or resource constraint, you’ll want to think about which engine you’re using.  This, in turn, will affect your backup process.

The 10,000 foot comparison of MyISAM and InnoDB

I always find that it’s nice to look at these things in a table:

* you technically can do a files-based backup with InnoDB (in fact, I do), but it’s a bit more involved.  See below for a discussion.

** ibbackup is a commercial product sold by the developers of the innodb engine, now a subsidiary of, gulp, Oracle.  The price? According to their site: “…please contact innodb_sales_ww at oracle.com…” i.e., “how much ya got?”

Let’s hit these in order:

ACID: atomicity, consistency, uh… isohedron, um… danger?  Whatever.  It’s an acronym that means that the engine has mechanisms in place to ensure that “transactions” are performed reliably.  The classic example is the bank transfer.  You need to know that if you take the money out of account A, it gets into account B.  If not, the whole thing is called off.  MyISAM does not implement these mechanisms.  InnoDB does.

Foreign Keys (FK): Similarly to ACID, FK assures that your data makes sense in relation to your model.  For example, if you’re looking at an 8-track tape collection (I’m super old), you can’t have a recording without the recording’s artist.  So, before the recording table gets “They Only Come Out at Night” inserted, the artist table must contain Edgar Winter.  If you delete Edgar from the artist table, you’d better be sure that “TOCOAN” is also out of the recording table.  So, unlike ACID, which enforces proper operation of the database, FK enforces proper/logical USE of the database.  MyISAM does not support FK.  InnoDB does.

Full-text Searches: In short, if you have a large text-type field, MyISAM supports special functions to index that field so that it can be searched against keywords with natural language and keyword expansion extensions if desired.  InnoDB doesn’t.

Locking: MyISAM does not support row-level locking.  That means that if you want to lock a field to prevent it from changing during an operation, you need to lock the entire table that it’s in.  That means all write operations stop until you’re done.  In a heavy load situation, that’s, as they say, “bad”.  This is one of the main reasons that InnoDB whoops MyISAM in its ability to support concurrent access, particularly at heavy loads.

Memory Use: All those fancy features of InnoDB aren’t free.  One cost is additional memory use.  For instance, InnoDB maintains “clustered” indexes and includes the primary key in any secondary indexes.  This speeds data access and improves CPU and IO efficiency, but at a cost of storing all that information in memory.  MyISAM stores  data and indexes in separate files.  Less efficient, but easier on the RAM.

Speed Winner: Oh boy, I hate to even get into this one, but most of the benchmarks I’ve seen show InnoDB to win speed contests in most cases.  This becomes particularly true at high load levels.  This makes a lot of sense if you consider the concurrency and efficiency characteristics.  Or, if you have some anecdotal evidence to the contrary, let’s just agree that you’re right, and I’m wrong.  (But I’m not.)

BACKUP!: Finally.  OK, so the big difference is that it’s not quite as easy to just copy the InnoDB files and expect anything good to happen.  With MyISAM, you can.  This means, according to the MySQL peeps, that you also cannot use the mysqlhotcopy utility with InnoDB since it uses the file mechanism to make the backup. You can, however, use the mysqldump utility with either.  Both are easier to do with a logical volume snap or replicated slave server to shutdown during the backup.  I also included InnoBase’s ibbackup program in case you’ve got some money burning a hole in your pocket.  Given the available options, I’m not sure I see the point in such a thing, but if you need some of the features it provides and don’t want to roll your own, it might be worth looking into.

So, that brings us back to my solution.  Given that I personally prefer to default my storage to InnoDB unless there’s a reason not to, I went with the replication model– create a slave server and calmly shut that down, make the backup, and restart the slave.  The thing I like about this approach is that it requires a minimum of fuss, causes no downtime, provides a running backup server, and does nothing that precludes me from using whatever type of storage engine I choose or mixing them together.

Two Pronged Approach to Backups:

Firstly, I turn on binary logging on the slave.  Then, when I shut the database down, I copy the relevant files (ibdata, .ibd, ib_logfile, .frm, and the my.cnf) to the backup staging area.  After that I can start the database back up, and it’ll sync itself back to the master.  There’s definitely some exposure to Murphy’s Law in there (e.g., the master takes on more data while the slave is shut down and then crashes before it syncs) but the probability of that is pretty minuscule.

Then, on a less frequent basis, I take a mysqldump of the database.  It takes a lot longer, but it’s got the advantages of not being binary, so it’s readable and less likely to become completely corrupted.  It’s sort of a backup-backup in case I need to fall back from a corrupt binary backup.

OK, maybe that’s enough on that topic.  Can hardly wait to see what I write about next…

I’m currently involved in moving an application from SQL Server (2000) to MySQL.  The reasoning isn’t terribly relevant, but primarily it’s an open source v. closed source issue.  This (aging) J2EE app required a proprietary bridge between the application server (Java on Linux) and the database (SQL Server on Windows 2000 Server).  That odd adapter has caused us all stress over the years.  Being proprietary, we didn’t know exactly how it worked nor whether it would be around tomorrow.

The cost was also a consideration.  Updating the server to the current version would likely cost us thousands of dollars just for the RDBMS itself.  We’d likely move to the latest OS and upgrade the hardware, too.  It got very pricey, very quickly.

We came to the conclusion that a better solution would be to migrate the whole mess over to MySQL on a Linux server.  I am intimately familiar with MySQL and knew that for very little dough we could provide a fast data store with replication for fail-over, and we’d eliminate that vaguely disconcerting feeling that relying on closed source proprietary software creates.

A More DIY Model of Disaster Recovery

However, one of the nice things that MSSQL does provide is a backup mechanism from within that database manager itself.  MySQL does not.  I needed to evaluate my options given the situation and choose the most reliable, simplest-to-implement backup solution that wouldn’t cost additional money nor require disruptive, time-consuming reconfiguration of existing systems.

It appears to me that there are three common methods for backing up a simple MySQL database:

1. mysqldump
   Shut down the database, dump out a text file with all of the SQL statements required to recreate the database, and restart.  Then, back up the file like any other file.
2. LVM snapshots
   Lock the tables, create a logical volume snapshot of the data partition, and unlock the tables.  Then back up the snapshot and remove it when completed.
3. Replication
   Create a slave database server that replicates the target server, and simply stop the slave, back up the data, and restart the slave.

Mysqldump has the advantage of being very simple to setup and script.  Also, the backup is a text file, and so a ding to the file isn’t likely to destroy it completely.  With a binary backup, a data ding could render the whole thing useless.  Still, it requires a fair amount downtime, which is bad, and the backup files can be very large, which just makes backups and restores take that much longer.

LVM snapshots sound great.  There’s very little downtime (just enough to create the snapshot), and the backup is the actual datafile.  It’s likely to be smaller and restore takes essentially no time since the data file can just be dropped back into a running server and used.  Still, snapshots only work when the filesystem is built on logical volumes (another good topic to discuss, I think).  We were looking at some older systems that used standard physical partitions, and I was not enthusiastic about shutting them down, backing everything up, and reinstalling on LVM just to accomplish this.

So, I settled on replication.  MySQL provides a very slick ability to create a master-slave replication arrangement wherein one server maintains an identical copy of the master database (god willing).  There’s just a bit of setup, but with this arrangement, there’s no downtime for the master, and the slave can be backed up in any convenient way without concern for the time involved.  Also, should something befall the master, it is quite likely that the application server can just be pointed to the slave and carry on as though nothing happened.

As it happened, we had a machine in our colocation cabinet just sucking power, doing nothing.  I set up a replication slave on it in just a few minutes.  Then, I wrote a quick script to shut it down, run the backup, and restart the slave.  I added that to the cron table, and we had our backups licked.

Now, obviously, I’m not talking about a very complex setup here.  This is an application that is critical to the business, but not what I consider heavily used.  Should it ever graduate to the level of something like a load-balanced cluster, I’ll have to revisit this topic.  In the meantime, we’re securely covered with a backup process that doesn’t require downtime and provides the side-benefit of a spinning fail-over server as an even faster disaster recovery path.

A Condescending Lecture on Backups

Backup, backup, backup… it’s so often overlooked until it’s too late.  Do yourself a favor, spend a little money and time and back up more than you think you need to.  When the Bad Thing happens, a good backup strategy can mean the difference between a routine mechanical replacement and restore (everyone’s happy!) and an uncertain scrounging of data or total loss (ain’t nobody happy).  Make it a priority.

And like most things, the best way to start is to just start.  Now.  Don’t worry if it’s not perfect.  Just take a disk image if that’s all you can do.  Then, consider what would happen if various nasty scenarios occurred.  Then think about how you might recover.  What if the machine’s stolen (it happens)?  What if the machine bursts into flames (that’s happened to me)?  What if an earthquake crushes the colo (probably not, but what if)?

Classify your data by criticality and back it up accordingly.  I recently nearly lost my company books when a power surge took out my active drive and the routine backup drive.  I was lazy and calculated the probability that both of those drives would fail at the same time and decided without really thinking about it that I’d get around to good backups… later… sometime… Luckily for me, I’d had at least the forethought to take an image of my drives and store them offline.  I’d forgotten that I’d done that and spent the better part of a day engaged in self-loathing.  Now, I regularly take some time to consider how I might improve my backups– net-based, portable disks swapped offsite, RAIDed online storage, or maybe all of the above.  It’s done wonders for my comfort factor to know that if my backups turn out to be insufficient, it’s likely that I’ll be more worried about the zombies than my business records.

Oh, happy Halloween.