SuperMicro X8DTN+

It turns out to be possible to get used SuperMicro servers fairly easily locally in Santa Clara. The U4 variety are quite bulky to lift, so you need to be quite strong to handle one by yourself, and they're a little too noisy for home use (see below), at least if you don't want to replace all of the internal fans (which are designed to provide high static air pressures, to keep the drives in the system cool, so not a straightforward thing to replace with anything quieter while preserving similar cooling properties), but at least they're reliable, well designed, easy to manage remotely (via an added IPMI card, the F motherboards have the IPMI feature builtin), made of solid steel, and easy to work inside because there's plenty of room. Overall, quite a lovable design, especially if you've come to dislike small PC cases for servers. These SuperMicro 4U cases are just the opposite of "small": they're basically like really large PC cases, except made from steel rather than the typical flimsy materials that are often used in PC cases to keep their weight down.

The server I'm putting together here is a 4U 24-drive SuperMicro 846A-R900B, with a BPN-SAS-846A backplane, a X8DTN+ motherboard, 72GB ECC memory, dual E5520 Nehalem CPUs, dual gigabit ethernet, an IPMI card for remote management.

The BPN-SAS-846A backplane accepts both SATA and SAS hard drives, although it's tricky to find inexpensive used SAS drives, not sure why.

An example of an inexpensive used SAS drive is the Seagate 2TB ST32000445SS which can be had for around $25 these days in lots of ten drives. It's actually a SED, which is one acronym used to describe self-encrypting drives. See here on how to use the SED feature with hdparm with SATA drives. Since the Seagate 2TB ST32000445SS is a SAS drive and hdparm only works with SATA drives (see here), I'm not sure yet if there's any way to encrypt the disk encryption key using Linux. It seems it might only be possible to perform a secure erase of the Seagate 2TB ST32000445SS drives using proprietary software (see section "13.7 LSI MegaRAID SafeStore Encryption Services" of the "MegaRAID SAS Software User Guide" manual) and a different controller card with SafeStore software pre-installed, which wouldn't be of much use except for secure-erasing the drives prior to disposing of them, as we obviously want to use the M1015 card with Linux.

The Seagate 2TB ST32000445SS has a sector size of 512 bytes, so you have to use ashift=9 when creating the ZFS pool, which is a little old-fashioned compared to the modern "Advanced Format" drives which have 4k sector size and use ashift=12, but these Seagate 2TB ST32000445SS SAS drives seem reliable and fairly speedy and run at a temperature of less than around 40C even in the hot summertime. Since it's a SAS drive, remember that in smartmontools you have to use the -A flag to scmartctl read the drive's S.M.A.R.T. info. In Debian Jessie, the drives show up as /dev/disk/by-id/scsi-35000c500342222cb which is not their serial number (perhaps it's the drive's WWN), making things a little inconvenient when it comes to locating a specific drive.

I replaced the scary-noisy power supplies that came with the case (PWS-902-1R, my goodness, avoid these, they use 40W with server off, and make a sound like a jet engine!!) with dual platinum 920W PWS-920P-SQ power supplies.

The excellent support team at support@supermicro.com (who have been doing a phenomenal job answering my queries and are surprisingly kind enough to provide this sort of technical advice to home users), provided me with the information that the PWS-920P-SQ power supplies were compatible with my 846A-R900B case and its power distribution module (albiet not as a supported configuration).

In the PWS-920P-SQ model number of the power supply, the "SQ" part apparently means "Super Quiet", which they really are, and they pull quite a bit less power than the PWS-902-1R (about 4W when the server is off, 80W each while the server is idle with four drives). I'm running the wall power through two APC BackUPS Pro 1000 units. Based on apcupsd if seems that I need to run one of the UPS units in network mode. I need to look into this aspect some more, so as to get apcupsd into a configuration such that if either path of the redundant power supply chain fails, the system could shut itself down safely, there's more hints on this kind of setup here: running two separate instances of apcupsd.

The total cost of the system (without hard drives) ended up being around $800, and $250 of that went toward replacing the power supplies with quieter ones. Perhaps that could be done more cheaply by using an ordinary ATX power supply, if you're prepared to sacrifice the power supply redundancy aspect, but for now I've decided to avoid doing that, so as to at least keep the system in a configuration where it is still using proper server-grade components.

When running idle, the system uses around 170w with four drives installed, and the internal fans are running at 3500RPM so provide quite a bit of sound even at the "Energy Saving/ES" setting in the BIOS, but at least it's an expandable setup. The memory can be reasonably expanded by adding more 4GB sticks in the future (the motherboard can take up to 18 of them). It's handy to have so many hard drive slots, they can be used to do things like making backups, as the drives are hot-pluggable.

To get the drives in the backplane working, I needed to add an M1015 card cross-flashed to Initiator Target (IT) mode with the P16 firmware, which turned out to be quite an adventure, described here.

To connect the drives to the M1015 controller card I also needed to hunt down the proper Mini SAS to Mini SAS SFF-8087 "iPass" connectors (with sideband SGPIO, whatever that is) which turned out to also be quite tricky, I resorted to reading the Molex data sheet to find the magical part number: 79576-2104.

These elusive 79576-2104 cables are apparently what's needed to connect the M1015 cards to the backplane, they're basically like four SATA cables "rolled into one", and the one meter length connectors are an eye-popping $16 each. I can't believe they even consider them to be "enterprise" connectors: they seem very flimsy compared to proper SCSI connectors, but I guess this is how things are in the SAS / SATA world -- they need to keep the costs down however they can.

With this kind of server, since the motherboard and CPU both implement VT-d properly, you can do fun things like running vmware's ESXi and "pass through" the M1015 disk controller card to the guest virtual machine, which is kinda neat, giving your guest virtual machine direct access to the drives in the backplane. It seems a little risky to depend on virtualization for any data that really matters, though: the freeNAS forums have lots of threads warning against it, and I tend to agree. Best to avoid the temptation...

IPMI client

Earlier, in the "olden days" of Java, before all the security fixes, it used to be possible to run SuperMicro's IPMI client from a web browser: it was implemented as a Java applet which loaded automatically when you connected to your server's IPMI card. As you can imagine, this Java applet-based approach, even on the best of days, was quite unreliable.

These days, this Java applet-based approach has become quite a nightmare to even get working, because Java has had many security fixes and you'll likely find that your operating system's default Java configuration disallows SuperMicro's IPMI client tool's applets from even being loaded, never mind risking running them in your browser.

So instead of attempting to get SuperMicro's browser-based applets to work, it's best to give up (while you still have your sanity), and instead spend your time more wisely on attempting to get their standalone Java tools downloaded and set up on your favorite client operating systems. I haven't made much progress on the Microsoft Windows 7 yet, but so far I've been successful in getting SuperMicro's standalone IPMI client running on both Ubuntu Linux and Apple OSX, see below for details.

You'd think that SuperMicro would make it straightforward to find the correct standalone client-side software for their IPMI card, but unfortunately there seem to be quite a few different downloads and names for the client software. I'm still not entirely sure which one is the most recent, or why SuperMicro uses two different names for their standalone client tool.

The SuperMicro IPMI client tool that works for me on both Linux (Ubuntu 15.04) and Apple OSX (Yosemite 10.10.4) is the Java based bundle here.

It allows auto-discovery on the LAN of your IPMI card, authentication to your IPMI card (username and password default to ADMIN in uppwecase, you should use a physically separate network and assume that security on this IPMI channel is "basically non-existent"), keyboard and mouse graphical remote control (slowish but usable) of the server if you click on the KVM Console tab, remote media provisioning to the server, and remote power control of the server, remote viewing of the sensors via the Sensors tab, etc., all in one tabbed window. This IPMI client tool's interface feels very old-fashioned, but at least it works. The IPMI client tool sometimes loses its connection to the server, and to get it back I need to quit from the IPMI client tool and start it up again, then log back in again to restore the graphical access, which is quite tedious. 

wget ftp://ftp.supermicro.com/utility/IPMIView/Linux/IPMIView_V2.10.2_bundleJRE_Linux_x64_20150203.tar.gz
tar zxvf IPMIView_V2.10.2_bundleJRE_Linux_x64_20150203.tar.gz
cd IPMIView_V2.10.2_bundleJRE_Linux_x64_20150203
java -jar IPMIView20.jar

There's also an other older tool called SMCIPMITool the download links are Linux x64 version, Apple OSX version (pure Java) and Microsoft Windows 7 (untested), using which you can run the "kvm access" component like this: 

java -jar SMCIPMITool.jar 192.168.2.153 ADMIN ADMIN kvm

Note that from Apple OSX and Ubuntu Linux, you can use the native ipmitool program which allows remote power on and sensor readings, e.g. as follows (it'll prompt for the password, by default it's ADMIN): 

ipmitool -U ADMIN -H your_ipmi_card_ip_address power status
ipmitool -U ADMIN -H your_ipmi_card_ip_address power on
ipmitool -U ADMIN -H your_ipmi_card_ip_address power off

Introduction to ZFS

Aaron's guide to ZFS is a good place to start. Also, his article about parchive is interesting.

For this SuperMicro server, I'm using three-way mirroring. The zpool is of size 16TB and consists of eight three-drive vdevs. If a single controller fails, only one group of eight drives would fail out of the 24 drives, so each vdev would still have one mirror drive of redundancy remaining. Admittedly, on the surface, using three-way mirroring seems wasteful of both power and storage, as only a third of the raw storage is available, but the eight additional drives don't use much power, and having a proper "enterprise-level" of redundancy with a simple configuration which also provides very high performance is a requirement for this system. The use of a three-way mirror also allows for the technique of "splitting the mirror" to create an immediate off-site backup (resilvering the replacement drives only takes a few hours).

The steps to setting up the zpool are here.

Looking for something physically smaller than a SuperMicro server?

While trying to figure out how to put together a low-cost NAS server for running ZFS, I came across iX system's "FreeNAS mini" which seems to be a very nicely balanced design in a neat small size.

There's a good review of it here: apparently it idles at around 43w with four hard drives installed, pretty frugal!

There is some discussion of its components here (the motherboard is an AsRock C2750D4I, so a large chunk of the "FreeNAS mini" system's expense comes from this and the memory). There is a thread here about stability problems with this AsRock C2750D4I motherboard, though, so it may have issues if you want to run Ubuntu or Debian on it rather than using it only for FreeNAS.