Now that I have had a few days to form an opinion on ZFS, I will provide a more in-depth analysis, while still only scratching the surface. Before I begin, let me explain my build.
- AMD ASROCK E350M1-USB3 Mini-ITX Motherboard (4x SATA 3 ports, all on a single controller as it seems)
- 4GB DDR3 1333 RAM (1x4GB)
- 2x2TB HDDs (2 new and cheap, 1 external that is 6 months old that I pulled out of its case)
- 1x120GB 2.5″ HDD manufactured in June 2007, used as ext4 root
ZFS as a whole is amazing. Copy-on-write, clones, snapshots, compression, deduplication, NFS sharing, SMB sharing (not on Linux yet), and encryption (currently Solaris closed-source only) are some of its best features, to name a few. The features are simply astounding, which is what makes ZFS the volume manager of choice (I say volume manager because its package contains more than just a filesystem) for anyone who is interested in managing the way their data is stored. Personally, I have no real need for ZFS; I have plenty of space, copies of my data in two physical locations 700 miles apart, and infrequent hard copies of my data. The real motivator for trying ZFS is pure “sport,” if you will. Without further ado, here follows my notes on the whole setup. I began my setup with Pendor’s guide over at Github. Pendor wrote a Gentoo Linux overlay, made a custom LiveCD, and a nice and easy guide for installing ZFS. His overlay really helped me get up and running much faster than I otherwise would have been able to. I was able to shortcut some of the steps because I did not aim to have a ZFS root. Regardless, I thank Pendor for his excellent guide. The first troubles I ran into were, even after switching to the HEAD revision of the ZFSOnLinux project at Github, that I could not compile SPL (Solaris Porting Layer) under any kernel in the 3.0 line. There seem to be some incompatible functions that SPL relies on that have changed their interfaces. After moving to 2.6.39, the compile went as smooth as warm butter. From there, it has been smooth sailing. I have not encountered any bugs at all, but that is to be semi-expected since the ZFS code isn’t a rewrite of ZFS but an adaptation for Linux.
Creating the initial zpool is, not unlike every other command for ZFS, simple. It is a one-liner that will setup three disks in a RAID-Z1 (RAID-5) format:
zpool create rpool raidz sdb sdc sdd
Next, create the first filesystem and sharing it over NFS. This one will be used for my Time Machine backups over the network:
zfs create rpool/cleteTimeMachine
zfs set sharenfs=on rpool/cleteTimeMachine
There is one small problem with the previous command. Time Machine cannot be limited through its own user interface. Time Machine always consumes as much as it can grow to. To fix that, I used the following command:
zfs set quota=500g rpool/cleteTimeMachine
Time Machine doesn’t compress its backups, so I should have ZFS do this for me:
zfs set compression=gzip rpool/cleteTimeMachine
After setting up Time Machine, I created a few more filesystems. Namely, one to backup the NAS itself (the root drive isn’t a part of the RAID), one to backup my family’s computers (CrashPlan), and one for my files.
zfs create rpool/cleteFiles
zfs create rpool/crashPlan
zfs create rpool/cleteNASBackup
All properties are scope-aware, so you can set deduplication and compression at the pool level and revoke it for the filesystems that perform their own deduplication and/or compression:
zfs set dedup=on rpool
zfs set compression=gzip rpool
zfs set dedup=off rpool/cleteTimeMachine
zfs set dedup=off rpool/crashPlan
zfs set compression=off rpool/crashPlan
Creating snapshots is easy and so is destroying data:
zfs snapshot rpool/cleteTimeMachine@12345abc
rm -rf /rpool/cleteTimeMachine/*
zfs rollback rpool/cleteTimeMachine@12345abc
I really cannot stress how simple ZFS/ZVOL has been to use. It has really been a painless experience so far. Keeping tabs on your filesystems are just as easy. I aliased together a command that will print out information about the pool, its health, space usage, and compression and deduplication ratios.
cleteNAS ~ # zstatus NAME USED AVAIL REFER MOUNTPOINT rpool 559G 3.02T 38.6K /rpool rpool/cleteFiles 199G 3.02T 93.4G /rpool/cleteFiles rpool/cleteNASBackup 1.18G 98.8G 689M /rpool/cleteNASBackup rpool/cleteTimeMachine 253G 247G 252G /rpool/cleteTimeMachine rpool/crashPlan 103G 297G 103G /rpool/crashPlan NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT rpool 5.44T 831G 4.63T 14% 1.01x ONLINE - NAME PROPERTY VALUE SOURCE rpool/cleteFiles compressratio 1.00x - rpool/cleteNASBackup compressratio 3.14x - rpool/cleteTimeMachine compressratio 1.21x - rpool/crashPlan compressratio 1.00x - all pools are healthy
ZFS on Linux performance is lacking. I do not use ZFS-Fuse, but an actual, real port of ZFS that compiles kernel support, which is called ZFSOnLinux. While the kernel-enabled version of ZFS is much faster than ZFS-Fuse, it is not optimized for speed at all. I have not really tested read performance, but I have tested write performance and I have found it to be roughly 8-10MB/sec when deduplication and compression are on. Without either of those, it soars to 20+MB/sec. One thing to note is that I am testing over 802.11n, so these numbers are probably very inaccurate. Basically, my testing consisted of writing to the drive from two local computers and one remote computer at a time.
One other performance related item that I have noticed is that deduplication and gzip compression together will lower write speed to 8-10MB/sec instead of well over 25MB/sec with both disabled. They also use 60% or so of each processor. All of this is expected with software RAID and the low processing power of a 18W TDP processor.
NOTE: I have verified the speeds using local dd testing. These tests above are an accurate representation of performance.
I have recently turned off atimes (access times) in hopes that it will give a small bump in performance. Since this machine will be used for backups and file storage, I am not very concerned with speed. What I am concerned about is drives being able to sleep. So far, they haven’t spun down a single time, despite the fact that I have all drives set to a 5-minute spin-down. I believe this to be mostly due to the CrashPlan engine. There seems to be a bug with CrashPlan where it will keep the drives running, but I have yet to confirm this. If anyone has ideas about how to stop CrashPlan from keeping the drives up, I would greatly appreciate the help.
Edit: I am pleased to report that my drive performance has been improved greatly, most likely due to an update to ZFS or SPL updates. I have been keeping my ZFS on Linux and SPL at the master branch in order to try to get fixes as soon as they are released. The data I have is of little importance and can be rebuilt easily, so I do not mind doing that so much.
I ran some more tests last night and received the following performance with compression=off dedup=off. For a RAID-Z this is expected performance since ZFS on Linux is still in its early stages (the performance is still a low priority):
cleteNAS media # dd if=/dev/zero of=/rpool/media/output.img bs=8k count=1024k 1048576+0 records in 1048576+0 records out 8589934592 bytes (8.6 GB) copied, 150.951 s, 56.9 MB/s
With compression=gzip-1 and dedup=off (note any compression at all increases speed greatly when writing from /dev/zero):
cleteNAS media # dd if=/dev/zero of=/rpool/media/output.img bs=8k count=1024k 1048576+0 records in 1048576+0 records out 8589934592 bytes (8.6 GB) copied, 37.8502 s, 227 MB/s