NAS

Monday, Aug 28, 2017 · 1500 words · approx 7 mins to read

ATP 236 spent quite a bit of time covering Crashplan’s exit from the consumer unlimited backup market, leading into discussion about network attached storage devices and their utility. Listening to how the hosts use their Synology models made me think about my own setup.

My NAS is the backbone of my home network, so I thought I’d write down what I built and how I use it, just in case it’s interesting to anyone passing by. I’d long been cold on the need and utility of building my own NAS, instead having the idea that something from a specialist NAS device company like QNAP, Synology or Drobo was always going to be better than rolling my own.

Having tried a home user model from QNAP a few years ago (TS-212-E, total garbage), and then building an 8-bay Drobo for work (OK as a SMB share host, not so great for other uses), I was turned on to the combination of FreeNAS on server-grade PC hardware as the right solution for what I needed.

Note: some of the links below go to Amazon UK and have my company’s affiliate tag.

Hardware

I chose a HPE ProLiant MicroServer Gen8, 16GiB ECC DDR3, 3 x Western Digital Red 3TB “NAS” drives, and a pair of Corsair Voyager 16GB USB 3.0 flash drives to boot from. FreeNAS recommends 1GiB of RAM for every 1TB of storage as a good rule of thumb, and with 9TB of total storage for the system to manage (although not that much exposed to the network due to the redundancy scheme) and just two DIMM slots, 2 x 8GiB ECC DDR3 sticks were the obvious choice.

You can get the Gen8 MicroServer with a range of processors, including Intel Xeons, but the Celeron G1610T is the most popular configuration option you’ll find, especially if you want to pick up a Gen8 on the second hand market. The Intel Celeron G1610T is an Ivy Bridge-class processor with 2 cores at 2.3GHz, and crucially for my uses, ECC memory support, a low TDP (35W) and support for Intel’s virtualisation extensions (VT-x).

Even as I’ve asked my NAS to do more over the last few years the processor hasn’t been the bottleneck at all. I’d recommend that model over the more powerful Xeon-equipped ones for all but the most focused Gen8 MicroServer installations.

The boot media is important to note. When you install FreeNAS, it prefers to be booted from USB flash storage, leaving the hard disks in the system free to be used completely for whatever disk configuration you have in mind. Crucially, it supports preparing the boot environment on more than one USB flash drive. Take advantage of that: I’ve had a USB boot drive failure in the last 6 months, but it didn’t cause any downtime. I used FreeNAS’ ability to keep going on the remaining drive to replace it easily.

The hardware configuration has since expanded to include a single SSD for fast zero-redundancy storage (although annoyingly it only runs in SATA2 mode, not SATA3), and I’ve replaced the original pair of Voyager flash drives for the boot environment with a trio of fresh Corsair Vegas (with zero downtime!).

I’d recommend a minimum of 3 disks for storage and 2 flash drives for booting, to give you the ability to tolerate disk failures for both your main storage and your boot environment. While you can get fault tolerance with just 2 disks with mirroring, it’s recommended that because the chances are high of further disk failures if one dies, due to the increased load on the others when rebuilding the array, that you go with a minimum of 3 disks to be on the safe side.

The Gen8 has 4 drive bays, although they’re not hot swappable. Interestingly, and very useful in practice, the Gen8 also has an Ethernet-based management interface called iLO, that’s accessible as long as the Gen8 has power and a network cable plugged in to the management interface port. You can also do normal networking over that port as well, if you can only connect one cable.

The management interface exposes the inner workings of the Gen8 in a nice web interface, letting you manage the hardware while it’s running without having to muck about in the BIOS. And if you do need to access the BIOS, that’s exposed over the iLO too. There are even iOS and Android clients, so you can watch your Gen8 boot remotely and interact with it, all without having to connect a physical display and keyboard. That’s saved me a good amount time over the last couple of years.

Software

As mentioned, I use FreeNAS. When I first installed it back in September 2015 it was based on FreeBSD 9, but today it’s based on FreeBSD 11. That means it supports bhyve-based virtual machines as well as the usual FreeBSD features such as jails (with full network stack virtualisation if you need it) and ZFS.

Think of FreeNAS as a ZFS-based FreeBSD distribution with a nice management UI. After installation it gets right into the job of asking you what to do with any disks that it can find, sets those up as you see fit with ZFS as the filesystem, and then gives you a wide range of choice about what to do with it.

It’s that flexibility that gives FreeNAS its power. You could just use the storage services to expose some SMB shares and leave it at that, but if that’s all you wanted to do then something like a Synology might be a lower friction route.

The combination of ZFS and a full-featured FreeBSD userland, and the nice FreeNAS web-based UI to drive it, gives this kind of NAS the ability to do a whole bunch more than just serve up network storage.

Usage

Here’s what I have my FreeNAS system on the Gen8 do for me:

DNS and DHCP for my main VLAN, running inside a jail
Certificate collector for my LAN, running inside a jail
Offsite backup collector for my Hetzner-hosted servers, running inside a jail
Ubiquiti UniFi controller for my Wi-Fi network, running inside a jail
Plex server, running as a jailed plugin
Transmission BitTorrent downloader, running as a jailed plugin
Nextcloud instance, running inside a jail
Ubiquiti Network Management System, running inside an Ubuntu VM

While I initially imagined and built the server as simple network storage, you can see it’s taken on a lot more due the flexibility of the jails, jailed plugins, and the ability for it to run non-FreeBSD operating systems as virtual machines, all on top of the ZFS storage you configure.

Network infrastructure hosting is a good fit for something like a Gen8 and FreeNAS combo, given FreeBSD is rock solid on the networking front, very powerful, and easy to configure to get access to that power. I’ve written before about my Ubiquiti adventures. All of that is run from my FreeNAS installation on the Gen8.

If you run ZFS on a remote server, like I do with my business infrastructure, you can use it to send ZFS snapshots of that remote server to a receiving FreeNAS installation, for offsite backup. FreeNAS also has UI support for sending its own snapshots to a receiving ZFS system, for offsite backups of your NAS.

Conclusion

Total cost for the Gen8 system I built was dominated by the disks at £150 each. The Gen8 base system itself, with 4GiB of RAM, was just £125, which is a great deal for HPE ProLiant “enterprise” class hardware with ECC support, ease of tooling and access, and the iLO system. The RAM was around £170 for both sticks. Along with the USB flash drives for boot, the total cost was around £775.

With 4.3TB of usable space from 3 x 3TB disks in RAIDZ1, an equivalent 4TB usable space Synology with redundancy would cost you £430 or so today (DS216j model). So rolling your own is considerably more expensive, but you’re paying for increased flexibility (although Synology’s DSM gains ever more features as time goes by), the power of ZFS, and the general purpose of it being x86 and running FreeBSD.

To get something similar from a Synology, especially something x86-based with the power to run virtual machines, you’re looking at the DS916+ 4-bay with the ability to upgrade the RAM. You’ll outrun the cost of a Gen8, RAM and disks that way though.

So would I recommend looking at building your own NAS? Yes, as long as you plan for using some of the extra power and flexibility that would get you, and as long as you choose a solid little hardware platform like the Gen8. I can’t recommend that little computer highly enough, especially for today’s prices.