Product Review: OWC ThunderBay 4 (SSD vs. Disk)

[Please read my disclosure statement on product reviews.]

[ Updated with comments from Timothy Standing, the programmer of SoftRAID, July 21, 2014. I am grateful for his comments. ]

After I wrote my initial review of the OWC ThunderBay 4 Thunderbolt 2 RAID, the folks at OWC sent me some SSD drives to showcase the speeds this unit is capable of.

So, because I’d already written my initial review (I like the unit), I decided to focus exclusively on performance in this review and look at how the different ways we configure the RAID can make a difference on performance.

In this article, I’ll look at both SSD and standard hard disks – and, along the way, I discovered some big surprises.

Website: www.macsales.com

Unit tested: OWC ThunderBay IV (4-bay JBOD)

MSRP: $949 (12 TB spinning media) — $1,299 (1 TB SSD)

TECHNICAL NOTES

I’m doing all my testing on a late 2013 iMac. Speeds will improve somewhat if you are using a new MacPro and will be reduced somewhat on older systems. However, the speed relationships as you compare different RAID levels and settings will remain the same.

UPDATE – July 21, 2014

I just learned that my new iMac still only has Thunderbolt 1 ports on it. Currently, only the Retina MacBook Pro and new Mac Pro support Thunderbolt 2.

NOTE: My initial review of the OWC ThunderBay unit was done on a 2012 iMac. You can read my first review here to compare the performance between these two iMacs.

The drives are all 3 TB Toshiba drives, provided with the unit by OWC. (I will refer to these as “spinning media,” in contrast to the SSD (Solid State Drives) that I’ll be testing later in this review.

Speeds are measured using Blackmagic Design’s Disk Speed Test. All drives are empty.

NOTE: As drives fill with data, they slow down. As drives get within 20% of full, they slow down significantly. These results should be considered “best case.”

I’m using OS X 10.9.4, the current version of Disk Utility and the B15 release of SoftRAID.

NOTE: There is no indication on the OWC website, but the RAID unit offered for sale is the JBOD version. If you need to create a RAID 5, which requires SoftRAID, please talk to a sales rep at OWC before purchasing the unit to make sure you are purchasing the right version.

UPDATE – TIM STANDING

You are right that the files at the end of a disk read and write much more slowly than at the beginning. We recommend that video editors use only the first 30-50% of each disk for there performance critical work. SoftRAID has the ability to resize a volume, even one with files on it. This allows users to take their existing RAID 5 volume and shrink it down so that they are no longer using the slower part of each disk.

Larry asks: Can we partition the array into a “fast” partition for active editing and a “slow” partition for general storage?

Tim replies: Yes, when you create a volume which uses less than the entire disk, SoftRAID uses the fastest part first. So you can just take a set of disks, make a first volume with 30 – 50% of the Max size (the fast volume) and then go and create a volume with the rest. In my testing, if a volume at the beginning of the disk gives you 500 MB/sec a volume at the end will give you around 330 MB/sec. The real slowdown happens in the last 30%. The first 30% is pretty much the same speed.

ADDITIONAL TECHNICAL BACKGROUND

It isn’t intuitive, but hard disks and RAIDs can be optimized for different purposes. For instance, video requires very large files, audio requires medium-sized files, while databases send small bursts of data into the master database.

A technical setting called “stripe size” configures the hard disk, or RAID, for a specific purpose. While there are a number of options, here are the three recommended choices from SoftRAID.

Most of the time, the default stripe size of 128 KB will be fine – especially if you are working with a variety of files. For reasons I’ll illustrate shortly, I’m testing these RAIDs using the default stripe size of 128 KB.

The SoftRAID program, which is available at extra cost as part of the OWC ThunderBay RAID, provides single-menu access to optimizing the drive for specific purposes.

NOTE: While RAIDs are the best choice for video production, many times, configuring the individual drives in a RAID as separate drives(called a “JBOD”) is often better for audio production; especially when you are creating music using sampling. While JBODs don’t have any data redundancy, they will have much faster performance when you need to access lots of small files all at once; which is what is required when playing back a multitrack music project.

UPDATE – TIM STANDING

You write that audio files require lots of simultaneous i/o to many small files. If [an editor] can accept slower write performance, then a RAID 5 volume would be great for audio files. RAID 5 volumes have awesome read performance even for simultaneous i/o to lots of small files (the read performance for a 4 disk RAID 5 volume will be very close to that of a 3 disk stripe (RAID 0) volume). This is true even if the volume is degraded (i.e. one of the disks is missing).

While writing to many small files simultaneously to a RAID 5 volume is much slower than that of large files, it is often faster than writing to a single disk. When duplicating a folder of 100,000 files, each 128 KB in size, a RAID 5 volume is actually 1.5 times as fast as a non-RAID volume.

Audio editors [should also consider] a RAID 1+0 volume with a larger stripe unit size. If you are trying to read many small files at once, a RAID volume is much better than a non-RAID volume as long as the read size is smaller than the stripe unit size. So if you’re streaming 8 audio files and reading 64 kb blocks from each file, then a stripe unit size larger than 64 kb would be best.

If the program is smart enough to buffer larger chunks of audio, then a RAID 5 volume would give you the best performance as it would burst the data into the program’s cache almost twice as fast as a RAID 1+0 volume.

With RAID 4 and RAID 5 volumes, the stripe unit size is set at 16 kb. It was an architectural decision I made. I think it is the right one.

DEFINITIONS

Data Redundancy. The ability to recover your data in the event a drive crashes or dies. While all drives can (and should) be backed up, data redundancy allows you to recover data from a hardware failure.

BIG NOTE: Data redundancy does not allow you to recover a file that you deleted. Backups or Time Machine provide that capability.

JBOD. “Just a Bunch of Drives.” This configures all the drives in a RAID as separate drives, each of which appears on the desktop. JBODs do not have data redundancy, but do provide the fastest i/o when accessing multiple files.

RAID 0. A RAID configured for fastest performance, but without any data redundancy.

RAID 1. A RAID configured for maximum data redundancy, but very slow performance.

RAID 5. A RAID configured for fast performance with data redundancy such that if one drive dies, all your data is safe.

RAID 6. A RAID configured for fast performance, though not as fast as RAID 5, with data redundancy such that if two drives die at the same time, all your data is safe.

My general belief is that RAID 5 is the best balance between data redundancy and performance. Today, I get to test to see if that theory is correct.

NOTE: Here’s an article that explains RAID levels in more detail.

JBOD – A COLLECTION OF SINGLE DRIVES

There are two ways to format these OWC units – using Apple’s Disk Utility or the OWC SoftRAID application. In this article, I tested both.

Configuring a RAID as a JBOD turns all drives – in this case four – into individual hard disks. The benefit to buying a RAID when all you want is a JBOD is that all the drives are all in single unit, which decreases clutter on your desk and only requires one power and Thunderbolt connection.

However, with a JBOD, each drive is independent, which means that all drives perform at about the same speed; and no drive has any inherent data redundancy.

When I format the drives using Apple Disk Utility (in the Utilities folder), these are the results.

When I format the drives using SoftRAID, these are the results – about 2% slower. While not a big deal, this indicates that if you want to use the ThunderBay as a JBOD, there’s no big performance advantage to spending the extra money for SoftRAID.

RAID 0 – FLAT OUT SPEED

However, most of us buy RAIDS to get faster performance than we can get from a single drive. RAID 0 combines the speed of all the drives to yield MUCH faster performance. In fact, this is the fastest speed we can get; but, there is no data redundancy. Lose one drive in the RAID, and you’ve lost everything.

NOTE: As always, data loss can be prevented with good backups stored on a separate device.

Here are the results when formatting the unit as a RAID 0 using Apple Disk Utility. Whoa! This is, essentially, the speed of one hard disk multiplied by the number of drives in the unit.

Let’s reformat the RAID using SoftRAID and measure the performance. For this test, I set the stripe size to 1 MB. Hmm…. not as good as when using the Apple Disk Utility. In fact, 7.5% slower Writes and 2.5% slower Reads.

NOTE: Deleting or reformatting a RAID volume destroys all the data stored on that RAID. So only do this when there is no data on the RAID.

Here I reformatted the unit as RAID 0 with a stripe size of 128 KB. Changing the stripe size improved performance by 8%.  Setting the stripe size to 128 KB provides performance roughly the same as Apple’s Disk Utility. I would have expected better performance with a larger stripe size; interesting.

RAID 1

A RAID 1 is a fully-mirrored drive, where each drive is a mirror of the other. RAID 1 is the best option for DIT work on set, where speed is less important that an absolute guarantee that you won’t lose data if a drive dies.

Here are the results of a RAID 1 created by the Apple Disk Utility. These are in-line with the results I expect. A RAID 1 should have the speed and storage of a single drive, even though the data is being written, for security, to multiple drives at the same time.

Here are the results of a RAID 1 created by SoftRAID. Again, this is in-line with what I would expect, though the write speeds using SoftRAID are not as fast as Apple’s Disk Utility.

RAID 5

Apple’s Disk Utility only creates RAIDs 0 and 1. To create a RAID 5, we need to use SoftRAID. (There are a variety of other RAID formats, but, for video editing, the only other RAID format I would consider is RAID 6, which SoftRAID does not support.)

When we add data redundancy by configuring the system to a RAID 5, we gain security, but we lose the speed of one hard disk along with the storage capacity of one hard disk. This is due to using the equivalent of one drive to store the parity data, which is used to recover our data in the event of a disk crash. For this reason, if speed is your goal, RAID 0 is the best choice. For general editing, though, I recommend a RAID 5.

Here are the performance results of a RAID 5 setting. Two things I find interesting:

1. Write speeds are slower than Read speeds. This is not what I would expect.
2. Write speeds are about 10% slower than I would expect in a RAID 5 with drives of this speed and quantity.

INITIAL THOUGHTS

If video editing is your goal, these speeds are more than sufficient to support up to 25 streams of ProRes 422 video. This is PLENTY fast for virtually all editing, except stereoscopic 3D, uncompressed 4K, or high-bit-depth RAW formats.

Second, performance varies depending upon how we configure the RAID. So, let’s test some SSD drives.

SOLID-STATE DRIVES

SSD (Solid-state Drives) drives (I know, the second “drives” is redundant) generally provide big speed improvements, but with a huge trade-off in storage capacity and price.

When using traditional hard disks, this four-drive unit can hold up to 20 TB of data. When using SSD drives, it only holds 4 TB. Also, I would not recommend SSD drives for a RAID 1 configuration. Since a RAID 1 is not designed for speed, spinning media would be a better – and much more economical – choice.

OWC provided me with four Mercury Extreme Pro 6G drive (240 GB each) SSDs for my tests. Installing the drives so they can take the place of a traditional hard disk involves mounting them in a NewerTech AdaptaDrive.

The process is not hard, though it is persnickety. We need to place four screws to hold the SSD drive into the SSD drive carrier, then install the carrier into the drive sled for the RAID. Eight screws per drive. The process isn’t hard and took me about half an hour. (Except for the third drive, which I installed upside-down. Fortunately, the RAID won’t let you install a drive unless it is position correctly. I should also mention that I’m grateful OWC sends extra screws with the unit…)

NOTE: It is fashionable, in these minimalist days, to feel that we should only install a screw or two to save time. Folks, this is YOUR data we are talking about. Use all the screws.

SECOND NOTE: Pay attention to how the carriers are mounted in the sleds. If the sled does not slide all the way into the RAID unit, it is because you mounted the carrier upside-down in the sled.

UPDATE

A reader pointed out that additional sleds are available at the OWC website for about $20.

Once installed, the SSD drives need to be formatted. I used Apple’s Disk Utility for this and formatted them as HFS+; however, I discovered that I should have used SoftRAID. It wasn’t hard to change the formatting, but it wasted a chunk of time.

ERROR MESSAGE

After installing the SSDs, I opened SoftRAID, which asked if I wanted to update the Mac OS to support trimming to non-Apple devices. Since trimming does not affect performance, but IS a good idea for overall SSD longevity, I said NO.

At which point, the SoftRAID system crashed with a meaningless error message.

So, if you are using this unit with SSDs, be SURE to say YES when it asks if you want to support Trimming for non-Apple SSD drives.

UPDATE – TIM STANDING

Larry, thanks for the detailed description. This allowed me to find and fix this bug.

Creating the SSD RAID volume takes longer than creating a spinning media volume. After creating the RAID, it took about five minutes for SoftRAID to create the file system so the RAID would appear on the desktop. There is a thermometer which indicates progress, which you can see here.

NOTE: The default stripe size of 64 KB is probably not the best choice for video. I changed it to 128 KB; which also matched the stripe size of the spinning media tests above.

Now that the SSD drives are installed, formatted and the RAID is mounted, let’s take a look at the speeds we can achieve.

UPDATE – TIM STANDING

The reason for this delay is that the Mac OS X pauses all i/o to the volume until TRIM is complete on the SSDs. You can also see this in action if you create a 100GB file, drag it to the Trash, erase the Trash and then start copying a new file to the volume. It will take 10 – 20 seconds before the copy starts. You are waiting on the TRIM command. Either way, you get the benefit of freshly erased flash storage for your SSD to use. This is mainly going to affect performance when your SSD gets closer to being full.

SSD – RAID 0

The first thing I notice is that the SSDs are dead quiet. The fan noise is barely perceptible. However, the four drives in total provide slightly less than 1 TB in storage. Clearly, there’s a big trade-off between storage and performance; not to mention price.

Using SoftRAID, I created a RAID 0 (though I could have used Disk Utility as well).

However, on my late-2013 iMac, the SSD speeds are not what I was expecting. The Write speeds were 3% slower than using a RAID 0 with spinning media. Read speeds, though, were 11.5% faster. Still, on a RAID 0, I was expecting speeds closer to 1 GB/second.

NOTE: Video editing performance is more dependent upon Read speed than Write speed.

Configuring this as a RAID 5, we see write speeds 5% slower than when using spinning media, while Read speeds are about 12% faster.

Hmmm… the speed increases are not at the level I was expecting. (The OWC website claims speeds in the mid-800 MB/second range.)

This tells me that if you are planning to this unit while with an iMac or MacBook Pro, you would be far wiser to buy spinning media, not SSDs.

UPDATE – TIM STANDING

The results from the SSDs are not what I expect. I have seen performance of up to 1.4 GB/sec read – 1.2 GB/sec writes on a MacBook Pro using 2 Thunderbolt 2 ports to 6 SSDs. I have seen performance of up to 3.3 GB/sec reads – 1.4 GB/sec writes on a Mac Pro using 6 SSDs. Your results show that there is obviously some performance tuning I have to do.

I won’t release a driver until we have many billions of i/os of testing (we have been testing this extensively since last September). We wore out 6 disks during our testing of RAID 4, RAID 5 and RAID 1+0 volumes (and had a substantially higher electric bill). Our standard test is 3 Macs, each running 4 or 8 disks for 48 – 72 hours. This is usually over 500,000,000 i/os per Mac and about 120 TB or transfers. Our test is a multithreaded read/write verify type of test. Towards the end, it was taking 2 – 3 days to reproduce the bugs we were chasing. Oh yes, and we did 2,000 cable pulls on RAID volumes before we officially passed the driver on a RAID level.

Since I have so many months of testing the driver, I don’t want to make any changes to it so close to our release date. I plan to go back into the driver code in a month and figure out why some of the performance numbers are not what I expect. I suspect that I am running out of buffers and just need to increase the number of these in order to see the faster performance on SSDs.

UPDATE – July 22, 2014

Larry O’Connor, the CEO of OWC, also provided a comment to this review. [When connecting the ThunderBay to a new Mac Pro,] “our ThunderBay 4 hits over 1300 MB/s with 4 SSDs inside our Thunderbolt 2 unit formatted as RAID 0 – this maxes the bandwidth Apple allocated [ to Thunderbolt] pretty much 🙂

EXTRA CREDIT

SoftRAID provides several advantages over using Apple Disk Utility:

• The ability to spot potential hard disk problems using SMART sensors, before a disk fails.
• The ability to prevent the accidental erasure of a RAID
• The ability verify disks are working correctly
• The ability to change the disk order in a RAID, without losing data
• The ability to verify parity data

However, as a software RAID, I suspect that SoftRAID won’t achieve the same speeds as a hardware RAID. On the other hand, when comparing the speeds of the SoftRAID in the OWC unit to the software RAID in a Drobo, the OWC FAR out-performs the Drobo.

Because Thunderbolt allows up to six devices to be connected at the same time on the same bus, it is possible to connect multiple ThunderBay RAIDS and create a RAID 0 or RAID 5. While possible, I’m reluctant to recommend this approach. For me, there is too much opportunity to things to become disconnected or power to be off on one unit, and not the other.

If I need more speed or storage, I would recommend buying a RAID that contains more drives.

SUMMARY

There were several surprises for me in this test:

• I was surprised that Disk Utility pretty evenly matched the performance of SoftRAID.
• I was surprised that SSD speeds were somewhat slower when compared to spinning media when accessed via an iMac (I would expect, but was not able to test, that a new MacPro would be faster due to it’s bus architecture)
• I was impressed with how quiet the unit was, even when the spinning media drives were running at full speed

Here are my recommendations:

• Given all the other products on this market, the principle benefit for buying the OWC ThunderBay IV is to configure it as a RAID 5. There are cheaper alternatives if all you need is RAID 0. However, RAID 5 will also require you to purchase the SoftRAID software, which is available from OWC.
• There is no big advantage to using SSD drives, with the possible, untested exception of a MacPro, especially given their cost and limited storage
• The speed of this four-drive unit is more than fast enough for most audio or video editing.
• SoftRAID is a valuable utility, and necessary to create RAID 5 systems, but its write speeds are lagging those of a hardware RAID. On the other hand, whether we are capturing, importing or rendering, write speeds using ANY video editor are always far slower than the read speeds we need for playback.
• When buying a RAID for video editing, Read speeds are more important than Write speeds.

Given all of these observations, I think OWC has a solid performer that is small, quiet and, once set-up, more than meets the needs of most video editors working in any video format smaller than 4K, or with uncompressed RAW media.

---

Bookmark the permalink.