Partition a RAID for Speed

A RAID (Redundant Array of Independent Drives) is designed to provide storage capacity with high-speed data transfer. And, if configured as a RAID 5 or RAID 6, protection in case one of the hard drives inside the RAID dies.

NOTE: Here’s an article on what RAID levels mean.

However, one of the facts of life about a RAID is that the fuller it gets, the slower it goes. As the RAID fills, you lose some of the performance you originally purchased the RAID for.

Unless… you know the secret tip to minimize this speed loss; and discussing that tip is what this article is all about.

NOTE: This speed tip applies to RAID composed of spinning hard disks. SSD RAIDs use an entirely different technology and the benefits of partitioning don’t apply.

BACKGROUND

There’s a trade-off when formatting multiple hard disks into a RAID. If you want protection in case a drive dies – called “data redundancy” – you need to reserve some of the total storage capacity for “parity data.” Parity data is information about your data that allows the RAID to safely rebuild the data from a bad drive when you insert a new drive.

For a RAID 5, this “reserve” is equal to the total storage capacity of one drive. For a RAID 6, this reserve is equal to the total storage capacity of two drives.

There’s also a speed hit. To make the math easier, assume that each drive in a RAID transfers data at 100 MB/second. In a RAID 0, the speed of the RAID is based on the sum of the data transfer speed of each drive. So, a four-drive RAID 0 would transfer data at 400 MB/second (4 x 100 MB/s). In a RAID 5, the speed of one drive is lost due to creating parity data. So, using our example, a four-drive RAID 5 would transfer data at 300 MB/second ( (4-1) x 100 MB/s). In a four-drive RAID 6 the speed of two drives is lost due to creating parity data. So, a four-drive RAID 6 would transfer data at 200 MB/second ( (4-2) x 100 MB/s).

Again, these are just examples to explain the math. Most hard drives today transfer data faster than 100 MB/second.

One more fact: the fuller a hard drive gets, the slower it goes. That’s because disks fill data from the outside (which is very fast) to the inside (which is much slower).

WHY PARTITION?

A partition divides a RAID into logical “clumps,” which act as though they are separate devices, while, in fact, they are just sections within a single RAID. The total storage of your RAID is divided between the two partitions. What I’ve learned is that when you create multiple partitions, the first partition always uses the fastest portion of every hard disk in the RAID.

While there is no essential limit to the number of partitions you can create on a RAID, there’s no practical benefit to creating more than two. That’s because the first partition always includes the fastest portion of the RAID, while the second partition includes the slower sections. Today’s operating systems provide a very robust folder system for organizing files. It is better to use folders than partitions if all you want to do is organize files.

In the old days, we would partition hard disks because hard disk storage capacity outpaced the ability of operating systems to save and access files stored on very large hard disks. However, the need today to partition individual hard disks is no longer necessary. In fact, partitioning individual drives is generally discouraged.

But, by creating a RAID with two partitions, you can create a fast partition that doesn’t degrade in speed as it fills up. While the second, slower partition can be used for data that needs protection in case a hard drive dies, but doesn’t need high-speed access.

WHAT PERFORMANCE CAN WE EXPECT?

Here’s an example of the speed benefits of partitioning, from a recent RAID that I tested. (Your results will vary, but the conclusions are the same.)

After partitioning the RAID, this is the speed of the fast partition. It is virtually identical to the speed of a single partition system. However, unlike the single partition, the RAID won’t slow down significantly as the partition fills up.

Here is the speed of the slower partition; about 20% slower! And this will slow down further as the RAID fills.

However, to put all this speed in perspective, a single stream of 720p ProRes 4444 is about 30 MB/second. ProRes 422 is about 18 MB/second. The speeds illustrated here would support more than 15 streams of ProRes 4444 or almost 30 streams of ProRes 422.

Partitioning a RAID won’t make it go any faster than if you didn’t partition it. What partitioning does is prevent the RAID from slowing down as the partition gets full.

CREATING A PARTITION

Partitioning a RAID varies by manufacturer and unit. There is no single way to safely partition a RAID. Some manufacturers use Apple’s Disk Utility for partitioning. Others provide their own utilities.

When creating a RAID partition, put between 50 – 75% of total accessible storage (the total storage available after the RAID is created) into the fast partition. Put the remainder into the slower partition.

NOTE: On the two editing systems where I’ve implemented this system, I used a ratio of 50/50 on one RAID and 66/33 on the other; with the fastest partition getting the majority of storage.

The best advice about how to partition a RAID is to contact your RAID manufacturer for instructions. Sometimes, you can partition a RAID without losing any data on the system. In other cases, you will need to copy the data to a different drive before partitioning.

SUMMARY

Partitioning isn’t difficult, but it is easiest to do when the RAID is empty. If all you are doing is editing one stream of video at a time, there’s no benefit to partitioning.

If, on the other hand, you are working with 2K or higher resolution video, multicam editing, RAW files or video files with massive bit depth, speed is everything.

Partitioning is safe, straight-forward and maximizes the speed of your RAID. Which is what you purchased the RAID for in the first place.

---

Bookmark the permalink.