Home network rebuild, part two: Virtual Hosts

Having previously got my new fileserver/NAS box up and running I've now moved on to splitting out the roles of my previous monolithic server into several virtual servers.

Virtual Servers
The first server (not actually virtual) hosts the basic network infrastructure required to get everything else running. NFS for various filesystems (not least of all home dirs), AoE for network attached disk space, DNS, DHCP and tftp for booting clients, and NTP for network time. On top of that I'll probably set up networked syslog there too.

The second server is the phone system -- asterisk. It's going onto a virtual machine of its own because having functioning phones is important and I'll probably install it, configure it and then keep the updates to a bare minimum.

Then we get onto the two fun boxes. One is the main, stable server. It runs Debian lenny (stable) and hosts mail (exim and dovecot), apache, subversion, squid, network monitoring, music streaming, printing, and the like. The second is running Debian squeeze (though it'll probably become sid at some point) and is for more bleeding edge stuff.

The idea is that if the version on lenny isn't up to my needs instead of back-porting or installing individual packages from unstable as I used to, I can simply install it on the unstable box. It's also intended to be a bit more of a play area for ideas. If I actually find myself needing the unstable version of dovecot, say, I'll probably spin it off onto its own box.

There are a few remaining areas which are going to present a problem. One is MythTV. It requires hardware (the tuner cards), and often bleeding edge releases of various things. At the moment virtualizing it doesn't seem like a win so it will stay on the old server for the time being and eventually, perhaps, be shifted onto a spare Via M10000 board I have and powered up and down for recording with wake-on-lan packets. We'll see.

Implementation
The two physical servers I have are the new Atom 330 based NAS box and my old dual core Athlon 3800 box. Each has a couple of gigs of memory and gigabit ethernet connection. The Atom box won't support kvm but kqemu seems to run fine on it. In fact, I've found the version of kvm I'm running (kvm85 on linux-2.6.30 amd64) is actually slightly unstable and prone to crashing and locking up, so I'm using qemu (with kqemu) on both servers at the moment.

I've handled networking by using bridging and having experimented with approaches a bit I've gone with creating an LVM for the basic operating system for each host and presenting that to the VM as /dev/hda, but then any data partitions (e.g. mail stores, home directories) are separate LVMs accessed from within the host over AoE.

Having these partitions available to all hosts while they're up rather than having to attach them to the machine as virtual devices makes managing them easier, but despite playing with booting the whole virtual machine over AoE using pxegrub, I found the complexities of managing the vmlinuz and initrd images externally to the host running them (they have to be external so they can be served up by TFTP) to outweigh any benefits.

So far, and since switching to kqemu instead of kvm, they seem to be up and running and performing well enough for the tasks they have. The original server is being steadily stripped of its responsibilities and services and once I've built an asterisk virtual machine it will be taken down, stripped of its disks and rebuilt as a minimal physical host for running kvm/qemu images. Plus that inconvenient MythTV server, of course.


Future Plans
The main thing I'm planning to get around to at some point is sorting out live migration of virtual machines. I'd like the hosts to all sit on the low-powered Atom server for much of the time but as other machines come up, migrate them over. So for example, when I fire up my quad-core athlon box it'd be nice to have the two main servers migrate transparently onto it for increased speed. With wake-on-lan configured and an appropriate shutdown script to migrate them back this would mean I could have CPU power on demand without actually having to deal with host downtime.

A variation on this would be to have the various desktop machines around the house boot into a virtual-hosting shell and grab their (or any other) virtual machine from the network. I could then power off my desktops for much of the time without losing session state (including open network connections) and remote machine access.

Why?
None of this seems like an immediately obvious Big Win, and in truth a large part of the reason for doing it is for its own sake so that I have up to date, hands on experience of these things.

All that said, there are real advantages I'm already seeing at this early stage. The virtual machines are all hosted on LVM volumes. When I perform an upgrade I can first snapshot the volume and if it all goes wrong, roll back to the previously working state. There have been times when I've really wished I could have done that with a buggy upgrade.

Also, having the main parts of my network infrastructure on a virtual host means that I can effectively upgrade them, repartition them and generally carry out all the sorts of low-level maintenance which previously meant crawling about in the attic swearing at scratched CD-ROMs and buggy BIOSes from the comfort of a laptop in the garden.

For a home network, virtualization is perhaps not as exciting a prospect as in a large data centre and the advantages of scale are lost, but there are still some more mundane advantages that I'm looking forward to.