Background
Up until July we had a server in the attic with a terabyte of RAID 5 storage (Linux software RAID on SATA disks) which ran virtually everything except the routing tasks which were moved onto a OpenWRT-based Netgear WGT634u some time ago.
In July I needed to upgrade the disk space and took the opportunity to build a new server and explore the possibility of using ATA over Ethernet (AoE) and moving some of my services onto virtual hosts. The server I bought was a TranquilPC BBS2 server, which is a low power, Atom 330 based thing with three hot-swappable SATA drive bays and Gigabit Ethernet. Being an Atom it won't support things that require AMD-V or VT-x extentions (e.g. kvm) but it should be good for lightweight kqemu-based virtual machines. To this machine I added three 1TB Western Digital WD10EADS drives.
The BBS2 is a nice piece of kit originally intended for use as a Windows Home Server. It runs Linux very nicely though and I went about setting it up as a Debian server. The idea was that it'd have a 1GB RAID 1 root partition mirrored across two drives and the rest as a chunk of RAID 5 LVM space. If I had been bothered about performance I could have sliced it into a RAID 0, RAID 1 and a RAID 5 chunks for different purposes but it's not something I really care about at home.
BBS2 Installation
Debian pretty much installed out of the box, complete with my LVM and RAID requirements. I booted from a flash stick and everything worked fine until I rebooted, when it couldn't find a bootable device. I booted into rescue mode and accidentally installed a boot-block to the flash stick which at least got me up and running.
I burned several hours trying to get it to boot from the internal disks because grub is known to be slightly finnicky about the BIOS order of devices versus the operating system order, and I wondered if I'd mucked it up. But no, it seems that the problem is simply that the SII3124 hosted hot-swap disks aren't bootable. Damn.
For a while I booted it off the USB stick with a grub install but for longer term I got a compact flash to SATA adapter and a spare 128MB CF card stuck that in as a bootable disk. It seems to work and having yanked out each of the RAID disks in turn the box still boots. Success!
I set up 5GB LVM volumes for /usr and /var and a 100GB home partition which I installed as /srv/nfs/home. Most of the machines around the house already use autofs to mount /home from either /var/export/home on the local machine or over NFS. It seems /srv has become an official part of the FHS since I last looked so I'm switching from /var/export to /srv/nfs. Finally I also set up a /srv/backup for my rsync/hardlink based online backup system.
The idea for this box is that as a low-power server it will be on all the time and hosting the basic network services I need to boot anything else -- disk space (NFS and AoE), DHCP, DNS, tftp and NTP. Everything else can live on virtual machines hosted either on the BBS2 or, if necessary, on a more high-powered machine.
Initial tests seem to show that the disks manage real world performance (on top of RAID5 and LVM) of ~30MB/s, which though a bit disappointing is good enough for my purposes. For those interested, the hdparm -t results for various devices are:
Raw hard disk (/dev/sda): ~75MB/s
Raw RAID 1 (/dev/md0): ~70MB/s
Raw RAID 5 (/dev/md1): ~35MB/s
LVM/RAID 5 (/dev/array/test): ~28MB/s
I'm actually quite surprised by the loss in performance from raw disk to RAID 5, and that might be something I investigate at some point. I don't actually need the performance to be better but it bothers me.
Edit: The problem was that the RAID 5 array had lost a disk. It wasn't rebuilding but the performance still suffered, interestingly. Rebuilding occurred at ~30MB/s when I kicked it off and the performance this morning is ~85MB/s for both raw RAID 5 and LVM/RAID 5.
Disclaimer: I have nothing to do with TranquilPC. When I started thinking about this project I like something prebuilt for a change, particularly given I could shop based on whole-system power consumption that way. I did a quick search for low powered servers and TranquilPC came up as highly rated, quite cheap, small and independent and while not official supporting of Linux, at least very open-minded about it. So far I'm quite happy with my purchase.
