Review: my experiences with storage devices

[esh]

First off a bit of background. About 10 years ago we had no backup policy or method. At all. Since we started on a 286 in 1990 with 20MB hard drive and a huge stack of floppy disks this is somewhat amazing. But it's true. In about 2000 we moved up to a monthly CD-R backup. However by this point we had 500MB of 'critical' files and as most of you will know this isn't far off the CD-R limit of 650/700MB. So this didn't last long; people didn't want multi-CD backups.

DVD-R/RW backups lasted for a couple years. This gave us 4GB per disc, which we only sailed past a couple years ago now. But it was still grueling; it was manual, and hence only got done a few times a year, and people were starting to get mixed up which was the 'newer' version as people had local copies of files that they took on the road with them and edited/made new ones and....argh.

After that, hard drives were the most economical. I had a rather bad experience with longevity of CDR/DVDR discs -- don't trust them beyond 3 years for the RW versions in my experience, don't trust ANY of them beyond 5 years. So we got a few external drives starting at 80GB and worked our way up to 250GB drives using cheap £30 enclosures that connected via FireWire/USB.

Okay so last year we had piles of these. Like eight of them. Plus since they were almost TOO convenient people started using them as main drives, so sometimes the backup was no longer a backup and people forgot they weren't backed up. Plus, what was on which drive? Nightmare.

So, the requirements:

1) I want centralised storage connected to our server
2) It must be internally and externally accessible
3) It must be redundant (ie. one drive can fail and it will still be hunkydory)
4) It must be large enough that people can shove stuff on there without thinking "is there enough room?"
5) It must be 'fast enough' (where fast enough is 100Mbit internally and 1Mbit externally)
6) It must be able to automate backup

The clear solution was a so called 'DAS array' or a direct attached storage array. This is basically a self-managing box of disks. The cheaper solutions are literally just a box of disks that plug in to your system, but I wanted something a bit smarter that would deal with failed disks itself. It would also be kind of nice that the box could be plugged into ANY system should the server explode itself somehow. This satisfies most of the points below, including being centralised (attached to our server), and if it is on a known computer then I can easily code backup scripts. The only things to check are the space and speed.

So I tested two units, one was a Data Robotics 'Drobo' (the basic 4-slot model, £250) and thanks to a friend, I also got to play with a QNAP TS-859 (£900, but the 4-slot TS-419 is a slightly healthier £340). These were both filled with Western Digital Caviar Green drives, which are high capacity and spin down to 5400 rpm when not in heavy use to save power. The Drobo was filled with 4x 1TB of these and the QNAP had 8x 2TB. The complete Drobo package was around the £500 mark whereas the QNAP was £2,000 (obviously this comes down to a more manageable £700 for the 4-slot version with the same drives as the Drobo).

An important difference to make immediately is that the QNAP can be dropped straight into your network as a NAS (network attached storage) or connected via ethernet, USB, or eSATA to your server as a DAS. The Drobo supports Firewire and USB.

The QNAP uses striped RAID-5. This spreads parity data across the drives so any single drive can fail, and you only lose around ~1 drive worth of space from your storage -- with RAID-1 which is direct mirroring you obviously lose half your total space.

The Drobo uses some advanced proprietary method of RAID. This has the same redundancy of RAID-5 but is actually limited at the filesystem level. It currently supports NTFS (Windows) and HFS+ (Mac). The QNAP also has ext3/ext4 out of the box for Linux guys.

So the first headache with the Drobo is how do I get it onto the server? The server is Linux, and it can't officially use ext3 and HFS+ is read-only in Linux. NTFS will not support file permissions. Hunting around, I find that there *is* unofficial support for ext3 on Linux, but I didn't find many people using a Drobo on Linux with more than 2TB, which is some sort of default limit, meaning Linux split it into multiple volumes of 2TB each despite it actually just being one large one at the hardware level. Urgh. I want it as one massive volume else it defeats the purpose. What happens when one of these 2TB volumes fills up? Obviously the drive isn't actually full. Will Linux throw a hissy fit? I'm not sure I wanted to find out.

Further to this you can reformat and set up things with a larger than 2TB volume, but it is terribly unsupported and with horror stories on the internet abound of corruption using 4TB+ volumes on Linux I decided I didn't want to risk that method at all. This is for reliability above all else. I believe it is here that the weakness for Data Robotics' method for doing RAID at the filesystem level becomes apparent: if you do something that is slightly out of its scope you are inviting silent data corruption until the volume becomes unusable. In the traditional RAID method, as with the QNAP, the redundancy is at a lower level, and hence you can stick any damn filesystem on there that you care for. The RAID method does not mind. That said, I should mention that the Drobo unit can do all sorts of fancy smart healing and even automatic defragmentation because it knows about the filesystem on it.

So... does it work? Yes, it does. Both of them are wonderfully simple, though the Drobo is even *more* simple. Literally with the Drobo you slot the drives in and plug in the USB cable and you are all go in Windows. Just like that. You don't even need drivers.

The QNAP has a gigantic stack of options which will make the power user very happy. The QNAP is effectively its own little computer, and it does in fact run Linux. You can configure the QNAP to run FTP servers, SAMBA servers (that's Windows filesharing to you), or even BitTorrent by itself should you fancy. If I didn't have a server already it would be perfect. Since I do, it's not really needed.

The method of setup can be done as follows:
1) QNAP. This can be plugged directly into the network, but I need external access. This means ssh access (it is supported on Windows and Linux and is encrypted and supports compression). You can't have multi-user SSH on the QNAP without replacing the entire ssh daemon. What I do is use BOTH gigabit ports on the QNAP: one goes into the router so SAMBA can be shared internally and the other goes directly into a gigabit card on the server which can then mount the shares directly when a user logs in (using pam_mount.so in Linux) and hence ssh access works externally.
2) Drobo. We don't use macs here so NTFS is the only supported multi-TB solution, so I mount the Drobo on the server via ISCSI (this forwards the entire drive over a network rather than the filesystem, so I don't hit the 2TB problem) and then have a virtual Windows server (under VMWare server) that connects via the ISCSI initiator. This works just fine despite it sounding rather complex. No doubt it's not super efficient: you are using the CPU to shift the data from the Drobo to ISCSI and then again from ISCSI to Windows. For ssh access you then shift that data from Windows back to the Linux host!

For a direct speed comparison of the raw hardware I plugged in both devices directly into my 3.6GHz PC. The Drobo achieved 15-23MB/s depending on the type of operation. This is obviously more than sufficient to saturate a 100Mbit network. The QNAP can push 80MB/s in a read operation over gigabit! This is obviously far much faster primarily because it has 8 drives rather than 4, and most likely the RAID operation working at a lower level is somewhat less intensive. Still, one can assume it could manage 30-40MB/s with 4 drives instead of 8.

Both manage to saturate the 100Mbit network here even when connected to the server. The only situation where I couldn't do this was going from the Drobo->ISCSI->Windows->Linux ssh. This only topped out at 8.5MB/s because of all the CPU use going on in converting the ISCSI signals and then the ssh encryption on top. Still, since we can only get 100KB/s up here, that isn't a problem since ssh is designed for external use.

Both units are quiet. It is somewhat easier to see what's going on with the Drobo: four huge rectangular LEDs show which drives are healthy, and an array of blue LEDs along the bottom show how much space is being used (each one represents 10%). The magnetic front and simple clip levers also make it a trivial for anyone of any age to plug in drives should that be necessary. Another non-obvious bonus for the Drobo: you can stack in drives of any size, mix and match as you want. Traditional RAID systems such as on the QNAP need drives of the same size or you will waste space.

Before I summarise, I should mention there is an addon for the Drobo (called the Drobo Share) which is a £130 unit that can duplicate many of the features of the QNAP, including direct connection to the network. It brings the price up to a similar level as the 4-slot QNAP. You can also put various fileservers and apps on it.

So which is best for who?

If you're a Windows user who just wants a massive stack of storage that you know is safe, and you really don't care or want to worry about configuration, there is really no competition to the Drobo. You really do plug it in and go, and the drives just slide in. The entry level price is also less than the QNAP and you can mix and match any spare SATA drives you happen to have lying around. You don't have to use all 4 bays either. You can start with 2 drives and move up on the fly.

If you're a Linux user, or a power user who likes the idea of sharing your files out to any system be it Mac or Windows, and you don't have a server currently, the QNAP is the way to to go. A very nice web interface and very good data rates.

Simply put, the Drobo is for the consumer or small business, and the QNAP is for prosumer and enterprise. But the cost of ownership for the Drobo is far less a barrier to entry on the DAS ladder, and so that's what we've started with. However, I have no doubt that we'll move up to a QNAP next!

If you have any comments questions about storage related to this or other things, give a shout and I will try and answer.

[Rik]

That's a really helpful post, thanks, esh.

[Simon]

Indeed. 

[cavillas]

I use DVD's 

[Rik]

I hope you test them from time to time, Alf.

[cavillas]

Yes, they bend quite nicely to put in brown enevelopes.

[Rik]

 

[esh]

In 20 years we'll all be using vinyl for backups, mark my words.

Just don't restore your data on the wrong speed turntable.


Edit: actually this reminds me of a situation I heard of recently where someone found an old VMS machine with a 10MB drive in it and wanted to get the data off. That's pretty hard since the drive isn't ATA and you won't find old controllers for new PCs. The conclusion was if there was a serial cable use that, otherwise.... modulate the hard drive data through the pc speaker and then record it with a soundcard and demodulate it again. Now *that* is what you call a backup mechanism!

[Rik]

Now that is ingenious...

[Technical Ben]

Could use one of these to speed up access times...

[Rik]

Why didn't they have those when I needed them.