Back in 2009 I built a “slightly more than NAS” home server and documented that build on my old blog. I’ve migrated the posts to this blog, you can find them here, here, here, here and the last one in the series here.
The server survived the move from the UK to the US, even though the courier service I used did a good job of throwing the box around, to the extent that a couple of disks had fallen out of their tool less bays. Nevertheless, it continued soldiering on after I put the drives back in and replaced a couple of broken SATA cables and a dead network card that hadn’t survived being hit by a disk drive multiple times.
I was recently starting to worry about the state of the disks in the server. They were the still original disks I put in when I built the machine, and they were desktop drives that weren’t really built for 24×7 usage, but they coped admirably with running continuously for four years. One started showing a couple of worrying errors (command timeouts and the like), so it was past time to replace all of them before things got any worse.
After some research into affordable NAS disk drives, I ended up with five WD Red 2TB disks for the ZFS raid portion of the NAS and an Intel X25-M SSD drive for the system disk.
First order of the day was to replace the failing disk, so armed with these instructions, I scrubbed the array (no data errors after over four years, yay) and replaced the disk. This time I did use a GPT partition on the zfs drive instead of using the raw disk like I did when I originally built the system. That way it’s at least obvious that there is data on the disk, plus I can tweak the partition sizes to account for slight differences in size of the physical disks. You can replace a raw disk with a GPT partitioned one, you just have to tweak the command slightly:
zpool replace data ada1 /dev/ada1p1
Basically, tell zfs to replace the raw device (ada1) with the newly created GPT partition (/dev/adap1).
The blog post with the instructions on how to replace the disks mentioned that resilvering can take some time, and boy were they not kidding:
root@nermal:~ # zpool status pool: data state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Fri Dec 27 07:17:54 2013 131G scanned out of 3.26T at 121M/s, 7h32m to go 32.8G resilvered, 3.93% done
At that rate I ended up replacing one disk per day until I managed to replace all four disks. Not that big a deal if I have to do this once every four years.
Why all four disks after I said above that I bought five? I had forgotten that you can’t add a disk drive to a raidz array to extend its capacity – if you want more disks in your array, you have to create a new array instead. Ideally I should have moved all the data to a separate, external disk, recreated the array with all five disks and then moved the data back from the external disk. As the motherboard doesn’t have USB3 connectivity this would have taken way to long so came up with a different approach.
Basically, instead of creating a single big zfs partition on every drive, I created two. One was the same size as the partition on the old 1TB drive, the second one filled the rest of the disk. I also needed to make sure that the partitions were correctly aligned as these drives have 4K sectors and I wanted to make sure I didn’t suffer from performance degradation due to the partition offset being wrong. After some digging around the Internet, this is what the partitioning commands look like:
gpart create -s gpt ada1
gpart add -a 4k -s 953870MB -t freebsd-zfs -l disk1 ada1 gpart add -a 4k -t freebsd-zfs -l disk1_new ada1
I first used a different method to try and get the alignment “right” by configuring the first partition to start at 1M, but it turned out that that’s probably an outdated suggestion and one should simply use gpart’s “-a” parameter to set the alignment.
Once I’ve redone the layout for all four disks with the dual partitions as I showed above, I should be able to add the fifth disk to the array and create a second raidz that uses five disks and also 4k sector alignment (which requires playing with gnop).