Spammers, call yourselves buzz marketers if you like, but there is a spot in hell with your name on it.

Anyway, I’m sure there were some genuine comments in there.  I just don’t have the time to trawl through them.  I’ll keep looking for a better way to manage that.  Comments are my favorite part of blogging.  This is like having a dinner party, and people keep showing up at the door dressed like the UPS man to say ‘Hi, your house very nice.  I appreciating it all time, dude. Buying V1aGr4 very cheap!!!’  Slam!

Now… on to technojunk!

Due to unforeseen circumstances, I have been thrust back into the world of PHP for a current engagement.  Without going into the pros and cons of PHP (truly, it has both), let’s just say that I’ll be using it.  That being the case, I’ve had to rather quickly decide on a development framework for the app… and boy howdy, are there a bunch of them these days.

Favorite slogan:
“The Zoop Framework: PHP development without the suck.”  Nice.

I dove in looking at some of the application framework comparisons (like this one on  Wikipedia) and came to the conclusion that a) there’s a LOT of them, and b) I was just going to have to pick one and get on with it.

So, I decided that I needed a few basic things to feel self-actualized:

• PHP5
• MVC
• Extensibility
• Object Relational Mapping (ORM)

This is old news, but PHP5 has a few notable advantages over PHP4 for my purposes:

• Faster.  Faster’s better.
• Passing by reference (PHP4 passed by value.  In a nutshell, that meant passing copies of things around instead of “the thing itself”.  That can be very, very confusing.
• Greatly improved Object Model:
  • Visibility (Public, Protected, Private access)
  • Class methods and properties (no instance needed to access these, nice for utilities, factories, whatnot)
  • Interfaces, Abstracts, Finals… Oh boy! (These are basic OOP constructs that PHP4 just lacked)

That implies to me that a framework that isn’t based on PHP5 doesn’t take advantage of those things, and therefore is made more for compatibility than performance and robustness.

Model-View-Controller (MVC) is the cool kids’ way of structuring a web application so that there is separation in the code between those pieces that represent the “business model” of the thing (e.g., accounts, users, ratings, postings) from the presentation level (i.e., what you see what you use it).

Implementation-wise, the controller and view layer are separate, but in practice, they are pretty tightly coupled– you pretty much need to know in the controller what views you have and what they do exactly.  The really important separation is between that stuff and the model.  Your application design works so much better if you think of the underlying structure completely without regard for how you present it. Again, like PHP5, I don’t know anyone who doesn’t do this at this point.

On the other hand, there are more elaborate frameworks (e.g., Drupal) that are more like content management systems than web application frameworks purely.  I wasn’t interested in buying into the learning curve associated with such a system, and I also wanted a rather spare skeleton on which to build our system so that we understood how every piece worked and weren’t locked into another “product’s” development process.

Extensibility means that I want to be able to add modules to do various things without having to perform surgery on the framework itself.  For the most part, the basic frameworks allow this, but… never assume or you make an ASS out of U and ME.

Object relational mapping (ORM) is… well, it’s different things to different people, but… simply put, it provides a layer of abstraction between your objects and SQL so you don’t have to sprinkle nasty SQL snippets all over the place.  Also, the thing I was keen on was a way to describe relations (e.g., “belongs to”,”has many”) so (a oversimplified example) that I could just say:

thing1->thing2->name

instead of having to do something like

thing1->thing2_id
Thing2(thing2_id)->name

in a case where Thing1s have a Thing2 owning them.

Finally, the client had lots of experience with CodeIgniter.  That led me in that direction, but it fell a little short in that it’s not PHP5-based and doesn’t come with ORM (though, you can add it).  Turns out that there’s a nifty offshoot of that project called Kohana that does, indeed, meet all of my desired traits.

So, that’s what I’m doing.  Why Yii?  Why not PHP on Trax (get it?  Ruby on Rails in PHP?) I’ll tell you why.  I need to get some work done.  It seems to me that one could spend months evaluating the pros and cons of these things and get absolutely no work done.  My advice on most things: just pick one and go.  You can generally make any of them do anything you want to.  Once you’ve looked at the basics, you quickly digress into hair-splitting and procrastination, and I need NO help with that.

My experience so far?  I’m pretty happy with it.  It allows for nesting of templates so you can specify (ala Rails’ yield) an application layout into which you can insert a section template in which you can insert a page template… etc.  Performance is good.  PHP rewards the user for tolerating a lack of structure with a lot of flexibility, so there’s much less time spent trying to figure out type mismatches or other obscure minutiae that Java tends towards.

If I had my druthers, I’d have gone Grails for this project, but given the aforementioned constraints, that didn’t work out.  I think Kohana will provide a very nice alternative without requiring much from the system administrators, again unlike Java, which can creep dependencies into the server environment if you’re not careful.

I’ll report back as we progress on any interesting lessons that come up.  In the meantime, so far, so good.

And then I was handed the corporate-issue Windows XP laptop. Hm.

I mention this only because I know that, at some point, I had similarly strong opinions on the choices people make about their computing platform. I may have even thrown out the off-handed jibe at others for those choices. To them, I apologize.

I’ve made it no secret that I don’t care for Windows– at least up to, and including, Vista. I’ve yet to indulge myself in Windows 7, which gets good reports, in general. Having paid for and experienced a lot of suffering at the hands of such winners as Windows ME and Windows Vista, I think it’s fair to say that my opinion is hard won.

That said, Linux is so flexible and open as to barely operate at times. I’ve documented some good times that I had when the decision was made by some member of the open-source anarchic syndicalist commune that two video cards of different manufacture shall not be supported– or my logs filled with indecipherable memory abstraction errors or multipath disk naming voodoo that defied translation.

Then there’s the Mac. Remember all those colorful generation-one iMacs making little flower shapes on the commercials? Let us summarize that experience with the phrase “frowny face monitor with Xs in its eyes.”

Right now, I run Linux on my desktop with Windows in VirtualBox, and I tote around a MacBook. I am pretty happy. They all do what I need in the appropriate context.

Is one better than the others? I don’t know. For what? Linux video editing is a joke. Mac’s open/closed schizophrenia and high cost are off-putting. And I can’t tell if Windows, itself, is spyware or just all the stuff that mysteriously installs itself at every turn.

On the other hand, my Linux machine allows me very complete control of about any conceivable option and is customized to my liking as a DesktopServerHybrid just how I need it. The MacBook has never crashed and is light, very well-built, and functional, particularly for multi-media applications. Lastly, there’s very few, if any, applications that I can’t get to run on Windows if all else fails.

My point is this: you like your Windows? Good for you. You don’t like Linux? Fair enough. Macs are for emo blogging wannabes? Maybe. Does any of it make any difference whatsoever to anyone? Not really.

Mocking another’s computer is as pathetic as mocking their sexual orientation or height. The more sure you are that I’m wrong, the more you likely need to take a Stuart Smalley look in the mirror and let yourself know that it’s OK for us all to be different. We’re good enough, we’re smart enough, and doggonit, people like us.  Yes, even people who use Macs.

Anyway, one of the characteristics of this “disorder” is the inability to throw junk away.  Instead, I *used to* keep it.  It could be broken (as in “garbage”) or semi-functional (as in “garbage”).  Broken stuff would get put away for the day that I’d need the parts or fix it (as in “when hell freezes over”).  Semi-functional stuff would stay out to be used once, frustrate me, and then collect dust until the next time I consider throwing it out.  Use once, frustrate, collect dust.  And repeat.

A variation on this theme is stuff that is completely functional but doesn’t meet my needs or perform at the level I’d like.  I’d had just such a situation with my laser printer and fax machine/scanner.  While they worked OK, they were both starting to lose their quality mojo.  Also, they were separate components and required an additional box to enable them to be put on my wireless network.  The last little annoyance was that the printer did not do full-duplex.  Often, I’d print out long documents, and to save paper, I’d manually divide the job and flip the sheets over.  Eesh.

Well, it’s that time of year when I think “time to spend some capital!”  I looked around my office, and there they were… taking up all kinds of plugs and counter space.  The laser printer was producing gray, stripey documents and having a hard time feeding single sheets.  The faxes would only go through half the time, and scans were so slow as to be… well, I have things to do.

So, I ordered myself one of these: The Brother MFC-8890DW

Now, I don’t print an awful lot nor scan that often.  Still, I only paid roughly $400 for this thing.  That’s about what I paid for my laser printer some years ago.  The short story is that, so far, I am greatly pleased.

The MFC-8890DW is a fairly large piece of kit (as my friend, Roy, likes to say now that he’s British), but it takes up less counter space than my old fax did.  Set up took just a few minutes, really, and the thing is on my wireless network (with WPA2 encryption).  It prints duplex (i.e., both sides), it prints fast (they claim 32 PPM), it scans up to 2400dpi, and scans to Samba shares, hosts, email, USB stick, and lord knows where else.

They seem to be getting better at balancing cheap, plastic construction with the impression of durability because it seems quite solid.

Adding this printer to my linux machines required… well, very little.  I said “new printer”, and Ubuntu found it and installed the drivers just like that.  Given the history of struggle between linux users and about any peripheral (without some geekfest driver compilation), that was a beautiful thing. PCs and Macs were equally easy.

So, there you go, an uncompensated recommendation of the MFC-8890DW.  While I’ve only had it for a short time (I’ll report back if something nasty happens), and I haven’t been able to configure some of the interesting fax to and from email features simply because of lack of time or need for them, I can say that this thing appears to be a good purchase.  It saves me time, space, and the frustration of having to wonder if my prints or faxes are going to happen any time soon or with the quality that I’d like.  I think that’s worth $400.

I knew that off-site storage was necessary.  I have space on remote servers in secure locations… one of these days, I’m going to figure out a good way to, uh… hey, I gotta go pick up the pizza!

But, it’s worse than that.  I actually have a mirrored-disk, network-attached storage (NAS) box specifically for backups.  I have external drives specifically for making local backups, too.  So, short of my office burning down, I should be covered, no?  Uh… no.

Turns out that my backup software to the NAS box had been failing silently because of some change in the client configuration.  Moreover, I’d procrastinated on replacing the batteries on my UPSs– they’d been pulled out for that reason (see where this is going?  no surge protection?) Now, mix that with my preference to use “logical volumes” for my storage because of their flexibility in sizing and storage, and toss in a completely unusual (but nonetheless real) power surge/outage.

What you get is a garfed logical volume (i.e., the physical device with the filesystem information on it survived but was irretrievably scrambled…) and the local backup external disk gets completely fried at the same time.  No network backup.  No local backup.  And the data itself has vanished.  What did I lose?  Well, my company books and tax records, for one thing.  (The actual happy ending is that because of my paranoia level, I had actually imaged my drives not too long ago to yet ANOTHER drive that I kept unattached.  I was able to reconstruct most of my data, but it took hours of work and validation.  Who knows what miscellaneous data I’ll notice is missing at some point.)

As W. famously said, “There’s an old saying in Tennessee — I know it’s in Texas, probably in Tennessee… that says, fool me once, shame on… shame on you. Fool me… you can’t get fooled again.”

So true, Mr. President.  So true.

Well, I can’t get fooled again.  So, no more procrastinating.  First order of business was to order myself a shiny, new UPS.  I went with the CyberPower 1500AVR.  At less than $200, and with the juice to run my (excessive) pile of gear for at least a half hour, it’s easy and inexpensive insurance against some putz putting his backhoe into the power main.  Still, they’ll think of something that I haven’t.

So, it’s on to my data.  Firstly, I forced myself back into a rational storage model.  Being your typical ADHD type, I have a tendency to keep EVERYTHING and keep it wherever it happens to land.  Forever.  Don’t know what it is?  Better just leave it.  You never know.  Right?

No, no, no… that’s not OK.  What I need to do is to keep my documents in, say, a DOCUMENTS folder.  Branch out by use (not… say… document type or date).  OK, so, ‘business’ and ‘personal’.  Business can have ‘clients’ and ‘financial’… and so on.  Everything goes into one of those folders.  EVERYTHING.  Is is a photo?  OK, it goes into personal/photos.  Or a separate photos folder.  Not both.  That file that I can’t identify?  My system to create an “unidentified” folder with dated sub-folders.  Everything I find that I can’t identify goes in those.  If, after a month or two, I still don’t know what it is and haven’t needed it… it goes.

Great, so, now I’ve got all my data in one place on my system… that is… my desktop system… which… doesn’t include my virtual systems… or my MacBook… oh boy.

Enter on-line storage and synchronization.  After some checking around, I went with Dropbox.  Other good options include: mozy, Jungledisk, Ubuntu One, Amazon S3, and many others.  Dropbox appealed to me because of the drop-dead ease of installation, the cross-platform clients, and what seems like a pretty reasonable price for all that (about $10/mo. for 50GB of storage as of this writing).

Basically, Dropbox puts, what looks like, another folder on your system.  In this case, though, if you drop a file in there, it’s copied up to their storage and synced to any systems that you also have connected to your account.  Drop your resume into that folder on your Linux box and edit it on your Mac, output the PSD to the Dropbox folder, and open it up on your… iPod Touch?  Virtual XP instance?  Anywhere you can get to a browser?

I’ve always had a tendency to roll my own solution for this kind of thing, but the technology has advanced to the point that there is no way that I can justify the kind of effort it would take to get something like that righter than they have for that kind of money.  $120/year?  That’s one internal hard drive.  Dropbox installed and was doing its thing in about 5 minutes– on three systems.

So, we’ve got reliable, filtered power and offsite storage of critical data.  The last thing I needed to address was local backups.  Given the inexpensive drives available these days, it only makes sense to me to keep very complete backups locally so that I can do a complete recovery if the problem is a fried disk.  So, I had that pretty well covered already with FSArchiver for complete images and Back In Time for scheduled backups to the NAS box (a D-Link DNS-323).

Maybe next time some super-freaky, never-happen-in-a-million-years thing happens, I won’t have to spend the day engaged in self-loathing and fear.  I’ll just restore my stuff and keep on keepin’ on.  In the meantime, my next project is to write a little script to make sure that my backups are actually taking place.  If they’re not, I need to make something to pester me about it.

I’ll probably do that tomorrow… d’oh!

Having said that, I also know that I can’t run everything that I’d like to run in Linux. And as much as I love my MacBook, I don’t think I care to buy all my software over again for the Mac, thank you very much.

No, for better or worse, I’m a Linux guy. Note that I am NOT recommending that ANYONE in their right mind switch to Linux for their desktop. I have a propeller on my beanie the size of a helicopter blade. If you’re starting out and can afford it, go Mac. If you can’t afford that, go Windows 7. If you’re a total geek who sits in his (or her) underwear until three in the morning trying to get a cron job to be able to get a custom kernel to build so you can hook up your old radio-controlled light-switch-remote to your machine… Linux is for you. Now, go take a shower and go to bed.

So, what’s a guy to do? I have clients who run Fedora, and I run Ubuntu. I have apps that *only* run on Windows, and the Linux “equivalent” is… let’s say, lacking. (I’m talking to you, GnuCash).

Welcome to virtualization. With virtualization, you can run nearly any (I haven’t tried hacking OSX, though I hear it’s possible) OS on nearly any other OS. It amounts to a PC within a PC. You install the virtualization software (e.g., Sun’s VirtualBox, VMWare, xen, kvm), create a virtual disk, and boot this virtual machine into your installation media. Just like that (in some cases) you’ve got a window with Windows on your desktop. No dual-booting. No second machine. Sweet.

Well, kind of. VMWare is probably the granddaddy of commercial virtualization solutions, and it can be pricey. It’s also, apparently, not the best performer on the block. The open source solutions, specifically kvm, are… well, again, if you’ve got a lot of crumbs in your beard and a Jolt-cola-gut, you might have the patience for all the obtuse, esoteric incantations needed to get the thing set up. (I can just hear him saying “well, if you can’t handle the command line… and I do NOT have crumbs in my beard…”)

Anyway, enter VirtualBox. The VirtualBox project (now owned by Sun… or somethinged by Sun… who knows…) is a super-easy-to-install virtualization solution that will even work if your processor doesn’t have the virtualization extensions (not well, but it’ll work).

I’ve installed VBox on Linux, Windows, and Mac, and it has worked great each time. Performance is, I think, quite remarkable given the situation. The interface, after installing the guest utilities that come on an ISO disk image, can be resized or run fullscreen, and except for a couple things I’ll get to, you nearly can’t tell it’s not running on the “bare metal”.

It’s not perfect, of course (but nothing is). For one thing, the relationship between things like virtual processor allocation (you can assign as many as you have to the guest OS) and performance is unintuitive. In my case, assigning two (after having run on one) caused a significant performance degradation in Windows. There are issues with APIC extensions having the same problem.

Also, I see very strange performance problems with some applications wherein they bring the guest OS (and the host, if you’re not careful) to a standstill. I can’t elaborate too much, because I haven’t seen much of a pattern, but for some reason, Dreamweaver, for instance, can barely operate if I view “Design/Code” mode, but “Design” by itself is fine, and “Code” by itself is fine.

Still, my instances of Fedora and BSDixes run great, and since the UI is much less of a concern, they can be run in console mode for the most part.

My latest experiment, having successfully converted my physical XP installation into a VirtualBox guest (called P2V, “physical to virtual” conversion) is to go the other way. I’m going to configure a client’s mail server on a virtual instance and then burn that image to a hard drive to install into the currently active hardware. If all goes well, we’ll skip all the iffy upgrades that we’d need to do otherwise, and only need to be down for the time required to burn the image and reboot.

So, if you’re a Windows person wanting to learn Linux, a Linux person needing to run Windows apps, or a Mac user wanting either… check out VirtualBox. There’s an open source version (OSE) and a personal use and evaluation version (PUEL). I recommend the PUEL version unless you have a good reason to go with the OSE. There are a few things, such as USB support, that the PUEL provides that the OSE does not.  In either case, though, it’s easy enough that you needn’t be a total geek to get it running.  And just because I am one doesn’t mean I don’t appreciate that.

I have a client that has a PPTP-based VPN solution in place. While I prefer a more robust SSL or IPSec VPN– this isn’t about me. That’s what they have, and it works for them. I needed to make my tools work with that situation.

On my end, though, I run a Linux desktop with virtualized (ask me later) instances of client servers or development environments. In this case, I had a Windows XP guest system running, but I needed to be able to access my Linux system as well on their network. So, while Linux’s NetworkManager would happily make a connection to their relatively oldish VPN server device, I couldn’t make another from the XP client at the same time.

What’s more, their VPN server device was having no part of routing my network’s traffic. (Note: I am not specifying the parts involved here because I don’t want to start a ‘you should have done THIS!’ discussion. I’m very much a ‘get it working and move on’ person.)

OK, so, I had a couple of choices. I could go into their network and see if I could reroute all the traffic that looked like it came from me back through the VPN (I actually tried that, and it didn’t work, but it was no good anyway since it was too invasive). Worse, I could reconfigure their VPN device to route my traffic, but again, that’s their machine, and I wasn’t going to spend all day figuring out how to accomplish that– on a device that isn’t even made anymore.

That’s all a long build-up to what turned out to work, not require anything on their side, and take me all of about 5 minutes. The answer was to NAT the traffic from my guest OS to the Linux box. (NAT = network address translation. That’s when you wrap all the traffic from your network in your exposed single address, so that routers treat it like it came from that machine. That’s how you can have a bunch of computers in your house running on that one IP address that comes with your DSL.)

192.168.1.X = Linux box with actual VPN connection
10.0.0.131 = The assigned VPN address for that machine on the client network
192.168.1.Y = XP virtual instance
10.0.0.0/24 = client network

Step 1) Route the traffic from the XP client.

route add 10.0.0.0 mask 255.255.255.0 192.168.1.X

Step 2) Make sure the Linux box is routing traffic.

echo 1 > /proc/sys/net/ipv4/ip_forward

(set it permanently in /etc/sysctl.conf as net.ipv4.conf.default.forwarding=1)

Step 3) Setup iptables to NAT the traffic. (Connect to VPN first, of course.)

iptables -t nat -A POSTROUTING -d 10.0.0.0/24 -o ppp0 -j SNAT --to 10.0.0.131

What you see there is my adding a rule to iptables that tells it that the last thing to do with any packets going to my client’s network is to wrap them in the aforementioned outer envelope with a return address of the Linux box’s assigned IP address in the client’s network.

If all goes well, the receiving end recognizes that this is a NATed packet, unwraps it, does whatever with it, and responds to the source with another packet wrapped similarly in an envelope that delivers it to the Linux box, where it’s unwrapped and passed back to the originating XP machine.

And, for what it’s worth, it did go well. I was able to route both machines’ traffic over the same connection without having to noodle with the client’s internal routing at all.

“I love it when a plan comes together.” — Hannibal Smith, The A-Team

RDP, VNC, ABC…

Well, if you’re lucky, you can use something like the Windows Remote Desktop (RDP), which, generally speaking, can give pretty good results if you’ve got a nice fast connection with very low latency.  You might also have access to something like PCAnywhere, GoToMyPC, or some other proprietary solution.  Likewise, they’re able to be optimized to get pretty reasonable results.

If you’re unlucky, you’re managing a Linux box, and the best that you could hope for was VNC.  I’m not sure what I’m missing about VNC, but it has generally sucked.  I mean, it’s great that it works at all, but I’ve tried Tight, Ultra, Vine, etc., and they’re all laggy and clunky.  It’s one of those things where it’s just annoying enough that I throw up my hands and go all command line on their asses.

That’s fine too, but the truth is, my ADD likes to have twenty things going at once, and it’s just very nice to have a desktop that lets me run process, file, or resource monitors and terminals for one machine on one and all those things for another machine on a separate one.  VNC lets me do it, but the performance, particularly over a sub-optimal connection, is not spectacular– unusably so often.

And The Angels Descended from Heaven…

Some time ago, I read about NX.  This was one of those cases where I noticed the mention of something one day, and the next day it was everywhere.  I had no clue what it was, and, frankly, I was too set in my ways to check it out.  Finally, about a year or so ago, I decided to look into it.  It turned out that it was a remote access system that was something between VNC and remote X-Windows.  Finally, I pulled down NoMachine’s “Free Forever” Linux edition and installed it on my servers at the colo and the client on my Mac laptop.

Yoiks! Before too long, I was running a desktop over a long-distance net connection (on top of a VPN, to boot), and it performed like I was sitting in front of it– with the obvious exception of any kind of large graphics or video, there’s only so much you can do to compress graphics.  But my monitors and terminals reacted to my input almost instantaneously.  This was a huge step forward for me and anyone involved in the remote administration of Linux boxen.

I haven’t even touched the NX network facilities that allow for nodes to be defined, to distribute applications between those nodes, and then to nest them together on the client, but I use it constantly in the most basic remote desktop configuration. I hope to delve into those additional features to configure myself some aggregated remote desktops for different networks (assuming I understand those features correctly).

How It Works: Taking X Back to Its Roots

NX works by re-discovering the original nature of X-Windows.  X was developed as a network-based protocol.  It’s far less integrally tied to the operating system than, say, Windows.  The operating system runs the apps and outputs to an “X server”, a separate application responsible for rendering that output.  Usually, that X server is running on the same machine, and it appears to simply be a desktop environment no different than a Mac or Windows.  But, by changing the display variable, X can output to any arbitrary X server, whether it’s local or across the world.  The X server does the heavy lifting of drawing the output to whatever machine it’s on.

The problem is that X doesn’t do a very good job dealing with the world as it is.  It doesn’t handle compression of the, sometimes fat, data stream.  It doesn’t adapt to different bandwidths.  It isn’t very smart about “differential” analysis of the action on screen (i.e., only include in the data stream that which has changed on the screen and leave the rest.)

NX does all that and does it amazingly well.  I have to color-code my desktops so that I can tell them apart on my Linux workstation– my daily use desktop.  I rarely have that problem with RDP or VNC.  It’s pretty obvious that that’s what you’re looking at in those cases since you usually have to cut display parameters so far back that the display looks like a bad 1990 animated GIF.  With NX, the display looks nearly identical to my local display.  All without much impact at all on my local resources.

OK, Now I Start Whining

Of course, NX isn’t perfect.  I had some trouble getting my headless servers setup to work with it without causing a mad avalanche of installation to get Gnome or KDE working, but XFCE was quick to install and works great.  Like most things Whateverix, there’s a fairly high noodle factor to get it optimally set up, but thus far, on Fedora, Ubuntu, and Debian distros, it’s worked nearly flawlessly.  And that includes clients for Windows and Mac.

There aren’t too many things you can say that about.  I’ve become quite accustomed to things just not working quite right– most anyone who had the misfortune to install Ubuntu’s Jaunty release with, let’s say, an ATI graphics card can likely attest to that.  Mine installed to a black screen, required the installation of pre-release drivers from an unsupported repository, filled my logs with propeller-headed nonsense about BARs and MMIOs, and just generally was a turd.  But that’s another story…

My point is that NX, for me at least, has been one of those almost entirely pleasant, worthwhile experiences that has definitely made my work easier and more pleasant.  Imagine that.

Commercial Vs. Open Source

Now, there’s NX and there’s NX… ever a fan of open source, I tried to install FreeNX rather than the commercial version from NoMachine.  Sadly, it was immediately apparent that there’s some internal squabble about libraries (I’m running 64-bit Linux) or Something-Else-I-Don’t-Care-About, and it simply wouldn’t install correctly.

Faced with a perfectly functional (and free, as in beer) version from NoMachine, I wasn’t about to spend yet another day trying to decode the passive-aggressive oneupmanship that can be open source “support” (‘RTFM n00b!’).  Now, I should say that I think open source is the cat’s pajamas.  I contribute money when I can’t contribute time, but there is a tendency in the community to think that most users or low-level kernel driver engineers.  I’m not.  I’m just a big doofus who likes the stuff.  Honestly, I’m pretty far down the geek path, but one can only dedicate so much time to getting their tools to work.  Ya know?

Give It a Shot

So, if you’re someone who wants to remotely access a Linux box, and you’ve been dissatisfied with your options so far, give NX a whirl.  You’ll need to download the Client, Node, and Server packages for the server.  The client requires only that package.  Install those guys and (at least in my case) little or no server-side configuration was needed.  It just worked “right out of the box”.  Poke around and check out the options. They also claim to support Citrix, RDP, and VNC, though, again, I haven’t had the inclination nor time to check those out yet.

It’s secure (runs over SSH), relatively easy to setup (at least by Linux standards), and free for Linux.  Do it.  You can thank me later.

In reality, you could use MySQL for years and really know little or nothing about these database engines.  I did.  After all, if you’re setting up a database to manage, say, your office equipment in a small business, you’ll maybe have a couple of hundred rows?  You may only access the database once a week.  Who cares what the engine is?  MySQL will default to MyISAM (usually), and that will work fine.  You’ll never notice, nor care.

However, when you begin to approach almost any kind of consistency, performance, or resource constraint, you’ll want to think about which engine you’re using.  This, in turn, will affect your backup process.

The 10,000 foot comparison of MyISAM and InnoDB

I always find that it’s nice to look at these things in a table:

* you technically can do a files-based backup with InnoDB (in fact, I do), but it’s a bit more involved.  See below for a discussion.

** ibbackup is a commercial product sold by the developers of the innodb engine, now a subsidiary of, gulp, Oracle.  The price? According to their site: “…please contact innodb_sales_ww at oracle.com…” i.e., “how much ya got?”

Let’s hit these in order:

ACID: atomicity, consistency, uh… isohedron, um… danger?  Whatever.  It’s an acronym that means that the engine has mechanisms in place to ensure that “transactions” are performed reliably.  The classic example is the bank transfer.  You need to know that if you take the money out of account A, it gets into account B.  If not, the whole thing is called off.  MyISAM does not implement these mechanisms.  InnoDB does.

Foreign Keys (FK): Similarly to ACID, FK assures that your data makes sense in relation to your model.  For example, if you’re looking at an 8-track tape collection (I’m super old), you can’t have a recording without the recording’s artist.  So, before the recording table gets “They Only Come Out at Night” inserted, the artist table must contain Edgar Winter.  If you delete Edgar from the artist table, you’d better be sure that “TOCOAN” is also out of the recording table.  So, unlike ACID, which enforces proper operation of the database, FK enforces proper/logical USE of the database.  MyISAM does not support FK.  InnoDB does.

Full-text Searches: In short, if you have a large text-type field, MyISAM supports special functions to index that field so that it can be searched against keywords with natural language and keyword expansion extensions if desired.  InnoDB doesn’t.

Locking: MyISAM does not support row-level locking.  That means that if you want to lock a field to prevent it from changing during an operation, you need to lock the entire table that it’s in.  That means all write operations stop until you’re done.  In a heavy load situation, that’s, as they say, “bad”.  This is one of the main reasons that InnoDB whoops MyISAM in its ability to support concurrent access, particularly at heavy loads.

Memory Use: All those fancy features of InnoDB aren’t free.  One cost is additional memory use.  For instance, InnoDB maintains “clustered” indexes and includes the primary key in any secondary indexes.  This speeds data access and improves CPU and IO efficiency, but at a cost of storing all that information in memory.  MyISAM stores  data and indexes in separate files.  Less efficient, but easier on the RAM.

Speed Winner: Oh boy, I hate to even get into this one, but most of the benchmarks I’ve seen show InnoDB to win speed contests in most cases.  This becomes particularly true at high load levels.  This makes a lot of sense if you consider the concurrency and efficiency characteristics.  Or, if you have some anecdotal evidence to the contrary, let’s just agree that you’re right, and I’m wrong.  (But I’m not.)

BACKUP!: Finally.  OK, so the big difference is that it’s not quite as easy to just copy the InnoDB files and expect anything good to happen.  With MyISAM, you can.  This means, according to the MySQL peeps, that you also cannot use the mysqlhotcopy utility with InnoDB since it uses the file mechanism to make the backup. You can, however, use the mysqldump utility with either.  Both are easier to do with a logical volume snap or replicated slave server to shutdown during the backup.  I also included InnoBase’s ibbackup program in case you’ve got some money burning a hole in your pocket.  Given the available options, I’m not sure I see the point in such a thing, but if you need some of the features it provides and don’t want to roll your own, it might be worth looking into.

So, that brings us back to my solution.  Given that I personally prefer to default my storage to InnoDB unless there’s a reason not to, I went with the replication model– create a slave server and calmly shut that down, make the backup, and restart the slave.  The thing I like about this approach is that it requires a minimum of fuss, causes no downtime, provides a running backup server, and does nothing that precludes me from using whatever type of storage engine I choose or mixing them together.

Two Pronged Approach to Backups:

Firstly, I turn on binary logging on the slave.  Then, when I shut the database down, I copy the relevant files (ibdata, .ibd, ib_logfile, .frm, and the my.cnf) to the backup staging area.  After that I can start the database back up, and it’ll sync itself back to the master.  There’s definitely some exposure to Murphy’s Law in there (e.g., the master takes on more data while the slave is shut down and then crashes before it syncs) but the probability of that is pretty minuscule.

Then, on a less frequent basis, I take a mysqldump of the database.  It takes a lot longer, but it’s got the advantages of not being binary, so it’s readable and less likely to become completely corrupted.  It’s sort of a backup-backup in case I need to fall back from a corrupt binary backup.

OK, maybe that’s enough on that topic.  Can hardly wait to see what I write about next…

I’m currently involved in moving an application from SQL Server (2000) to MySQL.  The reasoning isn’t terribly relevant, but primarily it’s an open source v. closed source issue.  This (aging) J2EE app required a proprietary bridge between the application server (Java on Linux) and the database (SQL Server on Windows 2000 Server).  That odd adapter has caused us all stress over the years.  Being proprietary, we didn’t know exactly how it worked nor whether it would be around tomorrow.

The cost was also a consideration.  Updating the server to the current version would likely cost us thousands of dollars just for the RDBMS itself.  We’d likely move to the latest OS and upgrade the hardware, too.  It got very pricey, very quickly.

We came to the conclusion that a better solution would be to migrate the whole mess over to MySQL on a Linux server.  I am intimately familiar with MySQL and knew that for very little dough we could provide a fast data store with replication for fail-over, and we’d eliminate that vaguely disconcerting feeling that relying on closed source proprietary software creates.

A More DIY Model of Disaster Recovery

However, one of the nice things that MSSQL does provide is a backup mechanism from within that database manager itself.  MySQL does not.  I needed to evaluate my options given the situation and choose the most reliable, simplest-to-implement backup solution that wouldn’t cost additional money nor require disruptive, time-consuming reconfiguration of existing systems.

It appears to me that there are three common methods for backing up a simple MySQL database:

1. mysqldump
   Shut down the database, dump out a text file with all of the SQL statements required to recreate the database, and restart.  Then, back up the file like any other file.
2. LVM snapshots
   Lock the tables, create a logical volume snapshot of the data partition, and unlock the tables.  Then back up the snapshot and remove it when completed.
3. Replication
   Create a slave database server that replicates the target server, and simply stop the slave, back up the data, and restart the slave.

Mysqldump has the advantage of being very simple to setup and script.  Also, the backup is a text file, and so a ding to the file isn’t likely to destroy it completely.  With a binary backup, a data ding could render the whole thing useless.  Still, it requires a fair amount downtime, which is bad, and the backup files can be very large, which just makes backups and restores take that much longer.

LVM snapshots sound great.  There’s very little downtime (just enough to create the snapshot), and the backup is the actual datafile.  It’s likely to be smaller and restore takes essentially no time since the data file can just be dropped back into a running server and used.  Still, snapshots only work when the filesystem is built on logical volumes (another good topic to discuss, I think).  We were looking at some older systems that used standard physical partitions, and I was not enthusiastic about shutting them down, backing everything up, and reinstalling on LVM just to accomplish this.

So, I settled on replication.  MySQL provides a very slick ability to create a master-slave replication arrangement wherein one server maintains an identical copy of the master database (god willing).  There’s just a bit of setup, but with this arrangement, there’s no downtime for the master, and the slave can be backed up in any convenient way without concern for the time involved.  Also, should something befall the master, it is quite likely that the application server can just be pointed to the slave and carry on as though nothing happened.

As it happened, we had a machine in our colocation cabinet just sucking power, doing nothing.  I set up a replication slave on it in just a few minutes.  Then, I wrote a quick script to shut it down, run the backup, and restart the slave.  I added that to the cron table, and we had our backups licked.

Now, obviously, I’m not talking about a very complex setup here.  This is an application that is critical to the business, but not what I consider heavily used.  Should it ever graduate to the level of something like a load-balanced cluster, I’ll have to revisit this topic.  In the meantime, we’re securely covered with a backup process that doesn’t require downtime and provides the side-benefit of a spinning fail-over server as an even faster disaster recovery path.

A Condescending Lecture on Backups

Backup, backup, backup… it’s so often overlooked until it’s too late.  Do yourself a favor, spend a little money and time and back up more than you think you need to.  When the Bad Thing happens, a good backup strategy can mean the difference between a routine mechanical replacement and restore (everyone’s happy!) and an uncertain scrounging of data or total loss (ain’t nobody happy).  Make it a priority.

And like most things, the best way to start is to just start.  Now.  Don’t worry if it’s not perfect.  Just take a disk image if that’s all you can do.  Then, consider what would happen if various nasty scenarios occurred.  Then think about how you might recover.  What if the machine’s stolen (it happens)?  What if the machine bursts into flames (that’s happened to me)?  What if an earthquake crushes the colo (probably not, but what if)?

Classify your data by criticality and back it up accordingly.  I recently nearly lost my company books when a power surge took out my active drive and the routine backup drive.  I was lazy and calculated the probability that both of those drives would fail at the same time and decided without really thinking about it that I’d get around to good backups… later… sometime… Luckily for me, I’d had at least the forethought to take an image of my drives and store them offline.  I’d forgotten that I’d done that and spent the better part of a day engaged in self-loathing.  Now, I regularly take some time to consider how I might improve my backups– net-based, portable disks swapped offsite, RAIDed online storage, or maybe all of the above.  It’s done wonders for my comfort factor to know that if my backups turn out to be insufficient, it’s likely that I’ll be more worried about the zombies than my business records.

Oh, happy Halloween.

Not really.

I find that I get a lot of use out of my own notes (kept in Zim!) and the blogs of other propeller-heads.  I’ve stopped writing in my other blog for the time being for reasons I won’t go into, but I feel the need to write about… something.  There’s two things that I know much about: backyard farming and technology.  I’ve been a computer dork literally since programs were stored on rolls of paper, paper cards, and formatted graph paper looking stuff (RPG FTW!).

The other blog was about gardening.  This one’s about technojunk.

These days, I’m a consultant– typically engaged in some variation on the theme of technology and marketing.  Sometimes more technology.  Sometimes more marketing.  To be honest, I can be ambivalent about both.  The whole of idea of “progress” seems pretty ludicrous as we’re destroying the only planet we have and showing no signs of letting up, and it’s unusual, unfortunately, when marketing doesn’t mean selling people stuff they don’t need by stretching the truth to its legal limits.

But technology is in my blood, and I’ve resigned myself to that.  I been fiddling with computers in one form or another since I was about 13.  Maybe younger.  I’m 43 as I write this.  30 years.  Amazing.

As I mentioned, I get a lot of use out of the technical notes, reviews, howtos, and what-not of other netizens.  I figure it’s about time that I contribute something useful to that body of knowledge.  That’s what my really good new blog is about.

The principles are something like this:

• Nothing written here will be influenced by cash or prizes.  I’ll take cash and prizes, but it won’t influence what I say.  In fact, I’ll probably end up getting sued.  Ah well.
• I’m writing for people like myself in that I still believe that human consideration and decency is worth something, even if the person being addressed has not R’d T FM. (Read the f*cking manual.)  In that spirit, I will mercilessly purge comments or users who act like incorrigible asses.  Oh, except me.  My blog.  I get to act however I want.
• I hope also to write in a way that normal, non-complete-geeks might get something out of it.  Obviously, if the topic is logical volumes or stateful packet inspection, that’s kinda hard.  If it’s a new router, though, I should be able to tell a small business owner what I thought of it without trying to demonstrate my leet speek.
• I will not dumb down.  See above.  I think there’s a happy medium.  That’s usually what I shoot for: technical accuracy and literary clarity.
• I reserve the right to establish additional principles as history demonstrates their need.

There ya go.  Welcome to Rian’s Really Good Technoblog.  I hope you get something useful out of it.  Me too.