I got an email from Tiziano this morning with an increasingly common question:
I’m looking forward [to] buying the new Mac Pro later this year! My question is about the external disk to use for editing all formats from DV to 1080p 50 f/s and in the future maybe 4k (I use Final Cut Pro X). I’m planning to buy one of LaCie with Thunderbolt connection. I read your article on RAID, Is it still necessary a Raid system with Thunderbolt? I’m looking more on speed than security, I do the back up in other storage disks every day. What do you suggest?
THE SHORT ANSWER
RAIDs are always faster and store more than single drive. If you can afford it, always buy a RAID to store media for video editing.
However, there’s more to the story than that, which requires a longer explanation.
A DEFINITION
Before we start, though, let’s clarify a term: “hard drive speed.” The “speed” of a hard drive could be defined as the speed at which is rotates. While a useful measure in some engineering environments, this isn’t really helpful in the real world.
NOTE: For technical reasons, the faster a hard drive spins, the less data it stores. 7200 RPM drives store more data in the same space than 10,000 RPM drives.
Rather, let’s define “hard disk speed” as the speed with which the hard drive sends data to and from the drive to the computer. This is more accurately called the “data transfer rate,” or “transfer rate.”
Recording information sent FROM the computer to the hard disk is called “writing.” Sending information from the hard disk TO the computer is called “reading.” While the data transfer rate of these two operations is similar, writing (recording) information on a hard disk is almost always faster than reading (playing) information. (This speed difference is in the neighborhood of 5%.)
THE LONGER ANSWER
The data transfer rate of a hard drive is, essentially, determined by three factors:
NOTE: Recently, drive manufacturers have started accelerating drives by integrating solid state drive (SSD) mechanisms into the drive hardware. This can accelerate operations where the same file is accessed more than once, but has limited effect when accessing multiple different files.
In the past, the speed of hard drives was limited by the speed of the protocol that connected the drive to the computer. For example, in practical terms, USB 2 limited hard drive speeds on Mac to around 18 MB/second. FireWire 800 limited hard drive speeds to around 85 MB/second. It was because of these differences in the protocol that video editors on a Mac were strongly encouraged to use FireWire 800 drives for all media storage and editing. By comparison, a hard drive attached internally to a MacPro (using a SATA protocol) could transfer data around 120MB/second.
NOTE: There are variations in drives and drive speeds. These numbers should be considered ranges, I’ve seen single drives both faster and slower than these numbers.
The reason speed is important is that a single stream of ProRes 422, which is the video format Final Cut Pro X uses for optimization and render files, is about 20 MB/second. (It’s actually a bit less, but the math is easier using a round number.)
So, if you are using a USB 2 drive, the drive’s transfer speed is too slow to even play one stream of ProRes 422.
When connecting a drive using FireWire 800, the drive will, generally, play about four streams of ProRes 422 before the FireWire connection is fully saturated.
Clearly, as multicam editing, video image sizes and frame rates increased, something faster was called for. Enter USB 3 and Thunderbolt.
IT’S IN THE PIPES
USB 3 and Thunderbolt are very high-speed connection protocols. Which means they are a “pipe,” like a pipe that carries water. Both of these pipes are really, really big – built to carry lots and lots of water all at once.
USB 3 is built to transfer data around 460 MB/second, while Thunderbolt is built to transfer data at about 1,100 MB/second! Suddenly, the speed restrictions we faced with FireWire 800 or USB 2, which were very narrow pipes by comparison, are gone. But – and this is a big BUT! – a pipe like USB 3 or Thunderbolt can only carry the data that’s fed into it.
If I connect a single hard drive to a computer using a Thunderbolt connection, the hard drive is connected using a very large pipe, but the total amount of data (water) that the pipe can carry is based, not on the size of the pipe, but how fast that single hard drive can transmit data. As we saw above, that limit is about 120 MB/second.
This means that if the data transfer rate of the pipe is bigger than the hard drive, the speed of the hard drive determines how much data gets sent down the pipe to the computer.
NOTE: Both USB 3 and Thunderbolt are excellent protocols. However, from what I’ve been told, Thunderbolt is more efficient at handling large media files. So, if your computer allows you the option, Thunderbolt devices should be chosen over USB 3. Also, USB 2 ports can not be converted to USB 3.
IT’S TIME FOR A RAID!
The way we fill the huge data transfer pipes provided by USB 3 and Thunderbolt is by combining multiple hard drives into a single unit called a “RAID” (which stands for “Redundant Array of Inexpensive Drives”).
If one hard drive transfers data at 120 MB/second, then combining two hard drives can provide double that: 240 MB/second. If we gang ten drives together, we can totally fill the immense size of a Thunderbolt pipe. But… it takes TEN drives to fill it. Not one.
NOTE: In reality, there are a variety of different flavors of RAIDs, called RAID levels, with differing levels of performance. While the analogy I’m building is essentially correct, reality provides a bit more variety. This article explains more.
DECIDING WHAT YOU NEED
So, let’s boil this down into something a bit simpler.
If you are only shooting and editing single camera narrative, a single hard drive connected via Thunderbolt will be a good choice. (While FireWire is supported on older Mac systems, it is not supported on newer systems. If your computer only supports FireWire 800, then that determines the type of hard drive you need to buy.)
However, if you are doing multicam, stereoscopic 3D, 2K, 4K or high frame rate video, a single drive, no matter how you connect it, will not be fast enough. You need a RAID that contains at least 2 drives.
To determine how many drives you need, take the number of video files you are playing at the same time and multiply that by 20 MB/second. (In other words, an eight camera multicam shoot would be 8 * 20 = 160 MB/second.) Assume that you get 100 MB/second per hard drive in your RAID (this allows for necessary overhead in the RAID). This means a 2 drive RAID will transfer about 200 MB/second of data.
For media with resolutions higher than 1080p HD (for example, 2K, 4K, or Ultra HD), you absolutely need a RAID with a minimum of 4 hard drives. More drives are ALWAYS better. As soon as you go beyond 2 drives, you need to look for a RAID 5. They are out there, but there are not a lot of vendor choices at the moment. Currently, I know of two: Promise Technology and Areca.
THREE WILD CARDS
There are three wild cards further clouding this mix: Drobo, RAID controller chips, and SSD drives.
Drobo makes drives that I like, but they are not fast. In order to get the speeds I was referring to earlier, the RAID needs to use a hardware controller chip. However, Drobo, in order to get both the expandability and flexibility that it is famous for, uses a software RAID controller. This software RAID controller is far slower than a hardware RAID controller. While I recommend using Drobo for expandable backup storage, their current performance is not sufficient for any task demanding high performance. This includes the new Drobo 5D and Drobo Mini.
The second wild card is those hardware RAID controller chips. For whatever reason, and I’ve heard LOTS of theories, RAID level 5 chips that also support Thunderbolt are very, very scarce. Promise Technology invented their own (which is a VERY expensive option), as has Areca with their CineRAID drives. Everyone else is waiting for a final chipset they can integrate into their units.
This wait for a certified RAID 5 controller chipset that supports Thunderbolt has taken two years and, I hope, ends really soon. Until it does, however, choices for RAID 5 systems will be very limited.
WHAT ABOUT SSD DRIVES?
SSD (Solid State Drives) are blazingly fast. However, they only achieve that speed when you are accessing the same files over and over. That performance advantage falls off when you are constantly accessing different files. Also, SSD drives can only store a fraction of what traditional hard drives can store.
The trend for the future is blending the speed of SSD drives with the vast storage of spinning media. Which means, over time, transfer speeds for both single hard drives and RAIDs will increase.
The Apple Fusion drive, available in some of the new iMacs, is a blending of SSD with traditional hard disks. I really like these for boot drives and applications. However, I prefer the vast storage and greater speed of RAID 5 systems for media storage and editing.
SUMMARY
With new protocols, like Thunderbolt, the speed of our storage depends upon the number of hard drives that are working together to feed the pipe. One hard drive transfers data at about 100 MB/sec. When you combine multiple hard drives together into a RAID, you can increase the total speed until you get all the speed you need.
ADDITIONAL READING
Here’s a collection of articles I’ve written that expands on the subject of hard drives.
43 Responses to Thoughts on Hard Drive Speed
Newer Comments →-
kenackr says:
October 14, 2013 at 9:28 am
-
Larry Jordan says:
October 14, 2013 at 10:08 am
-
SamiNole says:
August 30, 2014 at 10:11 am
-
SamiNole says:
August 30, 2014 at 10:17 am
-
LarryJ says:
August 30, 2014 at 10:26 am
-
Rick L says:
October 14, 2013 at 10:49 am
-
Larry Jordan says:
October 14, 2013 at 12:26 pm
-
Nate O says:
October 17, 2013 at 9:16 am
-
Larry Jordan says:
October 17, 2013 at 9:22 am
-
Patrick Donegan says:
October 26, 2013 at 12:29 am
-
Larry says:
October 26, 2013 at 10:03 am
-
Hirschkorn says:
November 6, 2013 at 5:14 pm
-
Bovine says:
January 20, 2014 at 3:44 pm
-
brad says:
January 8, 2016 at 12:01 pm
-
Larry says:
January 8, 2016 at 2:11 pm
-
Michael says:
February 5, 2014 at 2:53 pm
-
Larry Jordan says:
February 5, 2014 at 9:32 pm
-
Morten says:
October 2, 2020 at 4:29 am
-
Larry says:
October 2, 2020 at 8:15 am
-
Teresa says:
March 21, 2014 at 9:46 pm
-
Larry Jordan says:
March 21, 2014 at 10:07 pm
-
4K Storage: How Fast is Fast Enough? | The Daejeon Chronicles says:
April 24, 2014 at 6:24 am
-
Media Management Fundamentals says:
June 3, 2014 at 10:34 am
Newer Comments →Once again, excellent article! You truly have the gift to make things clearer.
While I do consider myself “techy” and do possess a Computer Science degree, I still get gobsmacked with many of the tech advances in today’s ever changing world.
I have known and been well aware of the fact that data transfer rate between the hard drive & the CPU has been the boat anchor dragging down overall system performance when it comes to video editing. The raid concept is one answer to increasing data transfer rate and the explanation given above is clear & concise. No Problem there.
I’ve been using 4 x WD 7200 rpm enterprise internal drives in my Mac Pro early 2009 4,1, but I’m only working with DV at present. No Problem there. I’m using 32 gig of all identical ram in the Mac Pro and upgraded the graphics card to EVGA’s Nvidia GTX680 classified GPU with 4 gig of ram. No Problem there.
I’m finally going to move, kicking & screaming, into the Hi Def world where file sizes are significantly larger in the next 6 months. This prompts me to wonder about using WD 10,000 rpm 1 TB Velociraptors in a raid configuration and now we come to the point where I’m gobsmacked.
Recognizing that my Mac Pro is built as a Sata II machine, but current drives are now made under the Sata III classification with a higher data transfer rate capability, I thought I’d need a Sata III pcie bus controller card to tap into the higher data transfer rate capability. When I posed the question to the big dogs on Apple forums, I was told,”no, you don’t need the Sata III controller” without much of an explanation.
In his comment to your other article on raid levels, Tim Jones notes that a controller card is needed to do a raid 5, but he quickly indicated that an external chassis would be needed.
Gobsmacked 1) Can I do a 4 drive Velociraptor internal raid level 5 in my Mac Pro? My thinking is to keep the drives as physically close to the pcie buss as possible because it’s the pipe line to the CPU.
Gobsmacked 2) Since the Mac Pro internal drive back plane is fed by the factory installed Sata II controller. Is this going to matter from a data transfer rate standpoint if I don’t or can’t upgrade the Sata controller to Sata III?
Gobsmacked 3) Tim Jones notes that a controller card will be needed (I agree with his assessment of Apple’s software raid & their controller card), and quickly states that an external chassis will be needed. How does this square with the a dive I recieved for the Apple forums that no card is needed?
If an internal raid level 5 is possible, my intention would be to move the current WD enterprise drives to an external housing for back up use.
Kenackr:
Thanks for writing and all your comments.
All three of your questions can be answered the same way: As I understand it, the bus inside the MacPro is SATA II, which means adding a SATA III card won’t get you anything faster using internal drives. Connecting an external chassis via PCIe WILL be faster because it by-passes the SATA protocol and connects directly to the backplane using PCIe. Mini-SAS would be the preferred protocol.
in other words, to get maximum speed, you need to move outside the MacPro chassis into an external chassis and connect it to the MacPro using Mini-SAS.
Once the new MacPro’s come out, Thunderbolt would be the preferred connection.
Larry
Cheers,
I know this discussion is about a year old but I’d like to throw in my 2 cents. I agree with Larry that RAID is the way to go. I had a 2008 MacPro and configured it with internal raid with the cables from maxupgrades. I essentially got RAID 5 with 3 x 3TB drives (total of 6TB usable) and it was all tucked inside. I also purchased external RAID bays both from SansDigital. The first gave me many problems STAY AWAY from the TR4M but, the MS4CT+ ROCKS! They have eSATA and USB 3.0 and FireWire. For me that’s great since my old MPB has firewire, my new iMac has USB3 and my older MacPro has an eSATA card. maxupgrades sells this http://www.maxupgrades.com/istore/index.cfm?fuseaction=product.display&product_ID=189&ParentCat=427 for the 2009 MacPro but I can’t speak to that.
A few things to note…2 x drives in a RAID 0 are still faster than 3 drives in a RAID 5. However, with the RAID 5 you can lose a drive (crash) and still have all your data. To get the speed of Raid 0 and the back up/redundancy of Raid 5 you need RAID 10.
Hope this helps.
Sam
Also forgot to mention, the RAID controller/card I used to get the faster speeds (SATA III) on the 2008 MacPro was a HIGHPOINT|RocketRAID 622 that I picked up for about $50. Now selling for around $35-40 after experimenting with the RocketRaid 2640X4 at much more.
SamiNole:
Thanks for sharing your thoughts.
Larry
Larry-
Question about SSD. If my SSD is internal to the computer and my entire video project and media will comfortably fit in the SSD, can I realize the full SSD speed? I presume that your comment about realizing SSD speeds for one file was referring to a hybrid SSD/hard drive setup.
Rick
Rick:
Probably not, but I have not done tests to prove this.
The boot drive is actively used by the OS and applications and all background processes, which take priority over any data playback. This means that data needs to wait until the OS is done with the disk before playback can occur. This tends to lead to choppy playback.
I’m not saying it won’t work, but you won’t come close to the full speed the SSD is capable of. This is the reason I still recommend external drives for media.
Larry
So will an external thunderbolt SSD be a good solution if you don’t want a RAID? I’m looking at Lacie’s 256GB drive to use soley as my stratch drive since my projects never balloon over that size, and if they do then I’ll just buy another and RAID it 😛 It would seam like an external SSD would be better than an external spinning drive. I was thinking if the SSD wouldn’t be better than I would get two thunderbolt HDD’s and RAID them….I’ve noticed software RAID’s are pretty dang fast and the last time I did a software RAID was only on USB 2.0 so I can only imagine how fast it would be now. I just came from a 2010 13″ MBP to a 2013 15″ rMBP 3 months ago and I am just loving these new I/O’s! USB 3.0 was great because all my externals supported it so I saw an immediate boost there and thunderbolt is pretty nice although so far its only used for my firewire drives, gigabit ethernet and my Intensity Shuttle. Really hoping there is more unique PCIe “stuff” to plug into my other thunderbolt port for not only usability but for play of course 😀 I guess my Intensity Shuttle will have to do for now.
Nate:
From a performance point of view, an SSD would be fine. It has very limited storage and I think you’ll find yourself filling it up FAR faster than you would expect.
But, as a place to store projects, an SSD would be fine.
Larry
“While FireWire is supported on older Mac systems”
I thought Firewire and USB 3 could travel on the ThunderBolt cable,
it would be interesting to find out if the newer Macs without FW connectors
actually support that protocol in a ThunderBolt Docking Station.
Patrick:
You are correct – FireWire is supported on Thunderbolt systems using the FireWire to Thunderbolt converter cable. However, the speed you get from a FireWire device when connected via Thunderbolt will be essentially the same as FireWire natively.
Larry
I need to expand my external storage (I’m currently using an 8 TB two-drive G-Tech RAID Thunderbolt unit) and I want something bigger and faster. The way I see it, my best options are two LaCie products: one SSD two-drive RAID unit and one HDD five-drive RAID unit (LaCie 5big): both external, both Thunderbolt. (Never had a LaCie product before and I’m a bit wary, as I’ve read quite a lot of complaints about their reliability, but G-Tech don’t have a comparable product and my two-drive RAID setup is showing its limits when editing three-camera multicams).
The better buy? The LaCie 5big: even if it’s 10% more expensive, it’s way spacier (10TB VS 1TB) and, incredibly, faster. It’s software-based RAID, so you’d only get RAID 0 and RAID 1 – but it’s significantly cheaper than a hardware-based RAID (RAID 5). Not as safe though. (I also have a FireWire 6TB RAID around, so I’ll use that for backup.)
“SSD (Solid State Drives) are blazingly fast. However, they only achieve that speed when you are accessing the same files over and over. That performance advantage falls off when you are constantly accessing different files..”
This is factually incorrect. It is in precisely this situation where SSDs outperform HDD, simply because a HDD has to seek to a sector on the disk by physically moving the read head. This is a mechanical operation, compared with an SSD that is just addressing memory effectively.
yup, this guy has no idea what he is talking about on this one……..SSD are faster that a traditional disk in every way…..A LOT FASTER
Brad:
You are, in general, correct. However, a single SSD does not match the speed of an eight-drive RAID. And how the SSD is connected to the computer affects speed as well. Also, using SSDs involve a trade-off between storage capacity and speed. If you are editing terabytes of media files, SSDs to store that much media will be prohibitively expensive.
However, if budget is not an object and your storage needs are fairly modest, then SSDs are unquestionably the fastest way to work.
Larry
Hi Larry,
Thank you for this helpful article.
You write: “However, from what I’ve been told, Thunderbolt is more efficient at handling large media files.”
Do you have more information about this?
Thank you in advance!
Michael
Michael:
From what I’ve been told, USB3 is designed to transfer short packets of data; for example, database records and word processing documents.
While Thunderbolt is designed to transfer larger packets of date; such as those found in audio and video files.
Larry
Do you know if this is still applicable today with Thunderbolt 3?
Morten:
The basics of this article are still correct. Thunderbolt 3, after removing overhead to support displays, transfers data around 2,750 MB/second. This is a VERY large pipe, indeed!
This means that the speed of your storage is not limited by the protocol (Thunderbolt 3) but by the devices feeding it data. Since this was written:
* Spinning media hard drives transfer data around 150 MB/sec
* SSD drives transfer data around 400 MB/sec
* NVMe drives transfer data around 2,500 MB/sec
So, the determinant of the speed of your storage is NOT how it is connected, but the devices that feed into it.
Larry
Hi Larry…
I have a new system with 500GB internal SSD and an external thunderbolt RAID for media files. Where should I set my scratch disks for render and auto-save files for best performance?
Teresa:
Use the SSD for the operating system and applications. Use the RAID for all media and scratch disks.
Larry
[…] in performance. A drawback of using the computer’s internal drive for editing is that, as Larry Jordan has […]
[…] Thoughts on hard drive speeds (encapsulates the hard drive basics all on one page) […]