I got an email from Tiziano this morning with an increasingly common question:
I’m looking forward [to] buying the new Mac Pro later this year! My question is about the external disk to use for editing all formats from DV to 1080p 50 f/s and in the future maybe 4k (I use Final Cut Pro X). I’m planning to buy one of LaCie with Thunderbolt connection. I read your article on RAID, Is it still necessary a Raid system with Thunderbolt? I’m looking more on speed than security, I do the back up in other storage disks every day. What do you suggest?
THE SHORT ANSWER
RAIDs are always faster and store more than single drive. If you can afford it, always buy a RAID to store media for video editing.
However, there’s more to the story than that, which requires a longer explanation.
A DEFINITION
Before we start, though, let’s clarify a term: “hard drive speed.” The “speed” of a hard drive could be defined as the speed at which is rotates. While a useful measure in some engineering environments, this isn’t really helpful in the real world.
NOTE: For technical reasons, the faster a hard drive spins, the less data it stores. 7200 RPM drives store more data in the same space than 10,000 RPM drives.
Rather, let’s define “hard disk speed” as the speed with which the hard drive sends data to and from the drive to the computer. This is more accurately called the “data transfer rate,” or “transfer rate.”
Recording information sent FROM the computer to the hard disk is called “writing.” Sending information from the hard disk TO the computer is called “reading.” While the data transfer rate of these two operations is similar, writing (recording) information on a hard disk is almost always faster than reading (playing) information. (This speed difference is in the neighborhood of 5%.)
THE LONGER ANSWER
The data transfer rate of a hard drive is, essentially, determined by three factors:
NOTE: Recently, drive manufacturers have started accelerating drives by integrating solid state drive (SSD) mechanisms into the drive hardware. This can accelerate operations where the same file is accessed more than once, but has limited effect when accessing multiple different files.
In the past, the speed of hard drives was limited by the speed of the protocol that connected the drive to the computer. For example, in practical terms, USB 2 limited hard drive speeds on Mac to around 18 MB/second. FireWire 800 limited hard drive speeds to around 85 MB/second. It was because of these differences in the protocol that video editors on a Mac were strongly encouraged to use FireWire 800 drives for all media storage and editing. By comparison, a hard drive attached internally to a MacPro (using a SATA protocol) could transfer data around 120MB/second.
NOTE: There are variations in drives and drive speeds. These numbers should be considered ranges, I’ve seen single drives both faster and slower than these numbers.
The reason speed is important is that a single stream of ProRes 422, which is the video format Final Cut Pro X uses for optimization and render files, is about 20 MB/second. (It’s actually a bit less, but the math is easier using a round number.)
So, if you are using a USB 2 drive, the drive’s transfer speed is too slow to even play one stream of ProRes 422.
When connecting a drive using FireWire 800, the drive will, generally, play about four streams of ProRes 422 before the FireWire connection is fully saturated.
Clearly, as multicam editing, video image sizes and frame rates increased, something faster was called for. Enter USB 3 and Thunderbolt.
IT’S IN THE PIPES
USB 3 and Thunderbolt are very high-speed connection protocols. Which means they are a “pipe,” like a pipe that carries water. Both of these pipes are really, really big – built to carry lots and lots of water all at once.
USB 3 is built to transfer data around 460 MB/second, while Thunderbolt is built to transfer data at about 1,100 MB/second! Suddenly, the speed restrictions we faced with FireWire 800 or USB 2, which were very narrow pipes by comparison, are gone. But – and this is a big BUT! – a pipe like USB 3 or Thunderbolt can only carry the data that’s fed into it.
If I connect a single hard drive to a computer using a Thunderbolt connection, the hard drive is connected using a very large pipe, but the total amount of data (water) that the pipe can carry is based, not on the size of the pipe, but how fast that single hard drive can transmit data. As we saw above, that limit is about 120 MB/second.
This means that if the data transfer rate of the pipe is bigger than the hard drive, the speed of the hard drive determines how much data gets sent down the pipe to the computer.
NOTE: Both USB 3 and Thunderbolt are excellent protocols. However, from what I’ve been told, Thunderbolt is more efficient at handling large media files. So, if your computer allows you the option, Thunderbolt devices should be chosen over USB 3. Also, USB 2 ports can not be converted to USB 3.
IT’S TIME FOR A RAID!
The way we fill the huge data transfer pipes provided by USB 3 and Thunderbolt is by combining multiple hard drives into a single unit called a “RAID” (which stands for “Redundant Array of Inexpensive Drives”).
If one hard drive transfers data at 120 MB/second, then combining two hard drives can provide double that: 240 MB/second. If we gang ten drives together, we can totally fill the immense size of a Thunderbolt pipe. But… it takes TEN drives to fill it. Not one.
NOTE: In reality, there are a variety of different flavors of RAIDs, called RAID levels, with differing levels of performance. While the analogy I’m building is essentially correct, reality provides a bit more variety. This article explains more.
DECIDING WHAT YOU NEED
So, let’s boil this down into something a bit simpler.
If you are only shooting and editing single camera narrative, a single hard drive connected via Thunderbolt will be a good choice. (While FireWire is supported on older Mac systems, it is not supported on newer systems. If your computer only supports FireWire 800, then that determines the type of hard drive you need to buy.)
However, if you are doing multicam, stereoscopic 3D, 2K, 4K or high frame rate video, a single drive, no matter how you connect it, will not be fast enough. You need a RAID that contains at least 2 drives.
To determine how many drives you need, take the number of video files you are playing at the same time and multiply that by 20 MB/second. (In other words, an eight camera multicam shoot would be 8 * 20 = 160 MB/second.) Assume that you get 100 MB/second per hard drive in your RAID (this allows for necessary overhead in the RAID). This means a 2 drive RAID will transfer about 200 MB/second of data.
For media with resolutions higher than 1080p HD (for example, 2K, 4K, or Ultra HD), you absolutely need a RAID with a minimum of 4 hard drives. More drives are ALWAYS better. As soon as you go beyond 2 drives, you need to look for a RAID 5. They are out there, but there are not a lot of vendor choices at the moment. Currently, I know of two: Promise Technology and Areca.
THREE WILD CARDS
There are three wild cards further clouding this mix: Drobo, RAID controller chips, and SSD drives.
Drobo makes drives that I like, but they are not fast. In order to get the speeds I was referring to earlier, the RAID needs to use a hardware controller chip. However, Drobo, in order to get both the expandability and flexibility that it is famous for, uses a software RAID controller. This software RAID controller is far slower than a hardware RAID controller. While I recommend using Drobo for expandable backup storage, their current performance is not sufficient for any task demanding high performance. This includes the new Drobo 5D and Drobo Mini.
The second wild card is those hardware RAID controller chips. For whatever reason, and I’ve heard LOTS of theories, RAID level 5 chips that also support Thunderbolt are very, very scarce. Promise Technology invented their own (which is a VERY expensive option), as has Areca with their CineRAID drives. Everyone else is waiting for a final chipset they can integrate into their units.
This wait for a certified RAID 5 controller chipset that supports Thunderbolt has taken two years and, I hope, ends really soon. Until it does, however, choices for RAID 5 systems will be very limited.
WHAT ABOUT SSD DRIVES?
SSD (Solid State Drives) are blazingly fast. However, they only achieve that speed when you are accessing the same files over and over. That performance advantage falls off when you are constantly accessing different files. Also, SSD drives can only store a fraction of what traditional hard drives can store.
The trend for the future is blending the speed of SSD drives with the vast storage of spinning media. Which means, over time, transfer speeds for both single hard drives and RAIDs will increase.
The Apple Fusion drive, available in some of the new iMacs, is a blending of SSD with traditional hard disks. I really like these for boot drives and applications. However, I prefer the vast storage and greater speed of RAID 5 systems for media storage and editing.
SUMMARY
With new protocols, like Thunderbolt, the speed of our storage depends upon the number of hard drives that are working together to feed the pipe. One hard drive transfers data at about 100 MB/sec. When you combine multiple hard drives together into a RAID, you can increase the total speed until you get all the speed you need.
ADDITIONAL READING
Here’s a collection of articles I’ve written that expands on the subject of hard drives.
43 Responses to Thoughts on Hard Drive Speed
← Older Comments-
Sulaiman Sibai says:
June 12, 2014 at 8:58 pm
-
LarryJ says:
June 13, 2014 at 12:36 am
-
binoy says:
June 27, 2014 at 10:19 am
-
Larry Jordan says:
June 27, 2014 at 10:25 am
-
John says:
July 9, 2014 at 8:19 am
-
LarryJ says:
July 9, 2014 at 8:30 am
-
John says:
July 9, 2014 at 9:04 am
-
LarryJ says:
August 30, 2014 at 10:24 am
-
Ivo says:
April 30, 2015 at 10:43 am
-
Eddiojp says:
August 27, 2014 at 11:28 am
-
Barry says:
August 28, 2015 at 7:51 am
-
Larry says:
August 28, 2015 at 9:18 am
-
Barry says:
August 28, 2015 at 9:31 am
-
Larry says:
August 28, 2015 at 10:15 am
-
Barry says:
August 28, 2015 at 10:47 am
-
Lorenzo says:
December 12, 2015 at 1:52 pm
-
Larry says:
December 12, 2015 at 2:21 pm
-
Ron K says:
January 8, 2016 at 12:44 pm
-
Larry says:
January 8, 2016 at 2:16 pm
-
Ron K says:
January 8, 2016 at 2:56 pm
← Older CommentsLarry
I’m experiencing major lag when editing 4k on MBP mid 2010 with 8gb-ram so have decided on upgrading the HD to SSD. As my MBP has no USB 3 or Thunderbolt. So my last upgrade is SSD.
purchased 250 GB SSD from crucial – shall I install OSX onto the SSD and run FCPX off it? I was advised that I should move the project I’m working on to the SSD for even faster results? Or should I keep the Library on an External HD, (FW 800).
Or is there a faster set up?
Thanking all.
Sulaiman:
You will probably get the best results by using the SSD as your boot drive, and storing media on an external hard drive.
The lag is either caused by a slow hard disk – somewhat likely – or your computer not able to edit the camera native format you are using. YOu might also try optimizing your media to see if that reduces the lag.
Larry
my laptop is a old one lenovo ideapad z575 its 500 GB HDD. Can i use ssd drive for this laptop.maximum how much gegabytes can i use for this.
Binoy:
Probably yes. BUT, you would be well advised to contact Lenovo before investing. The cost of the SSD drive may be more than simply purchasing a new computer with an SSD built in.
Also, in general, if your computer supports SSD drives, it will support any size SSD drive. Again, check with Lenovo to be sure.
larry
Hi Larry
Can you point me in the right direction… just changed up from 5DM2 to the GH4… so will be shooting 4K to edit on a 1080P Timeline in FCPX.Currently trying it on my internal Mac Drive and it is a little slow and dropping frames.
I had a brief chat with you at recent show in Manchester and you recommended I work not on my internal drive but on G Technology RAID… will I be ok with the G RAID Studio 2 Bay Thunderbolt 2 storage and use this as my disk to edit or do I need the G SPEED Studio 4 bay?
I think you recommended the 4 Bay in your explanation above but the G Technology website says the G RAID Studio 2 bay supports 4K video?
Thanks
John
John:
Editing a single-stream 4K file using the 2-drive unit should be fine. Multicam will be a problem.
Also, make SURE you have at least 8 GB of RAM – 4K is a memory hog.
Larry
Hi Larry…. thanks for reply…really helpful… one other question actually… I have option to use FCPX either on my slightly older iMac – the connection out to the RAID would be by Firewire 800 though… although this machine has 16Gb RAM
Alternatively I have a Mac Book Pro 15 inch laptop with 8Gb RAM but better Thunderbolt connector (NOT Thunderbolt 2)
Now I just read this on the web… 4K video (a video format that has four times the resolution as 1080p and is gaining popularity with video professionals) requires more bandwidth than the original Thunderbolt can accommodate. Thunderbolt 2 will be able to stream that 4K video and write it to disk at the same time.
Does this mean I definitely need a machine with Thunderbolt 2 out to a RAID to edit 4K data… : (
Thanks
John
John:
Sorry for the delay in replying.
Unless you are editing “uncompressed” or 16-bit RAW files in 4K, a Thunderbolt 1 connection will be fine for most 4K editing.
Keep in mind, to take advantage of a Thunderbolt 2 port, you would need to connect a RAID containing either eight SSD drives, or 16 spinning hard disks.
Larry
Hi Larry,
and thank you for sharing this useful article!
I’ve got a Macbook Pro (early 2011) / 8 GB SDRAM / Thunderbolt 1, and Final Cut Pro X. I shoot and edit Full HD in a single camera streaming, and I’ll eventually switch to 4k in – let’s say – two or three years.
I’ve been told about the benefits of RAID 5 in video editing, so I’m about to purchase a Promise Pegasus2 R4 8TB (4 x 2TB) Thunderbolt 2. I’m on a budget right now, so I can’t afford the R6 (6 bays) nor the R8 (8 bays). Nevertheless, in my current Full HD enviroment I’d be just dandy with the R4.
And here’s my questions.
As you wrote above referring to 4k, “to take advantage of a Thunderbolt 2 port, you would need to connect a RAID containing either eight SSD drives, or 16 spinning hard disks.” Does this mean that a Thunderbolt 2 RAID storage system with only 4 bays is kind of a dead end when it comes to 4k video? I’d really like to use this very device (hopefully in RAID 5) after the next upgrade to 4k hardware: any chance this is possible (at least for a single camera streaming)? Or should I keep going with my two hhds (#1 on-line for storage/editing, #2 off-line for overnight backup via Time Machine) and buy maybe the Promise Pegasus2 R8 just when I’ll actually start editing 4k? You know: this R4 is quite expensive for me (1400 € here in Italy: 1550 $, more or less) and I’d rather not to waste money on unnecessary features…
Thank you in advance for any advice you can provide!
Best regards.
Once again I learn something (well learned too) thanks to Larry Jordan. – Thank you very much
Very helpful article. So, a question:
I can buy 2 x 500GB EVO 850s for $500, giving me a 1TB storage. In a raid config, this would give me read/write speeds around 1000MB/s, and kale good use of thunderbolt.
I can buy a 1TB Seagate Barracuda for $60, but if I want a comparable 1000MB/s, I would need to 10 of them in a raid. 10x$60 = $600, more than the two ssds. Granted, I would have 10TB of storage, vs. 1TB. But if 1 TB is all i need to complete an edit (i have relatively small projects), the two sssds are the better solution and buy, correct?
Barry:
Essentially correct. The two SSDs don’t have data redundancy and your speeds won’t be quite that fast, but, overall, you are correct.
Larry
Thanks; so, even a raid system like OWC’s Thunderbay, with 4 bays, running 4 spinning discs (at ~100mb/s each) couldn’t edit 4k natively, nor would it comes close to maximizing what thunderbolt has to offer (but it could do a compressed 4k edit). What I don’t get is why put together and sell a system like that, with an emphasis on its thunderbolt capability, when it can’t even fill the thunderbolt “pipe”, as you describe it, to say nothing of thunderbolt2? I mean, what to be gained by buying it, as a thunderbolt device, if it can only read/write at 400mb/s, with your basis drives. I guess you could by 10,000rpm drives o maximize the speed, but then these start to get expensive, and you may as well buy ssds, as far as I can tell. I guess it comes down to the balance between storage needs and speed needs. But, as I say, my projects are relatively small, and I can by cheap storage to archive raw footage, if I want to. I think for me, if I want to try and truly edit/output in 4k, it is either 4ssd or 8-10 bay raid system. Anyway, I’m trying to wrap my head around the 4k world, and your articles and online lectures have been really helpful, so keep it up!
Barry:
First, very, very, very few projects need to fill the entire Thunderbolt pipe. In fact, many formats of 4K media can be easily edited on a system running at 500 MB/second.
Not 12-bit uncompressed 4K, but the number of cameras that shoot that format are very limited and very expensive.
Why Thunderbolt? Because it is faster than a single drive, faster than FireWire, faster than USB 2, at least as fast as USB 3 (depending upon drive configuration) and relatively inexpensive.
A good analogy is buying a car. If you plan to commute, a Ferrari won’t do you much more good than a Volkswagen. When we are editing, most of us need fast speeds with vast storage. A 4-drive RAID meets that goal.
For the ultimate editing performance, a 4-drive SSD will max out Thunderbolt 2, which has a maximum data transfer rate of about 1375 MB/second. (The 2.2 GB transfer rate is applicable only to monitors, not data.)
Larry
Yes, it is making sense now to me. Thanks again.
Barry
Dear Larry
I would like to ask you help about my system performance.
I use fcpx last version on a mac book pro late 2013 equipped with 16GB and 1TB SSD disk, running osx last release.
Fcpx is installed on the Mac and the projects and files are all on an external hdd Buffalo thunderbold at 5400rpm.
So, I mainly edit 1080p clips, but after 2 or 3 hours fcpx slows down.
I mean that it becomes annoying to work while all editing tools get more to respond.
Do you think that a raid would be the solution?
Or do I should check some setting to increase the performance?
Thank you in advance!
Lorenzo:
If your system slows down over time, a RAID probably won’t help. Continuously slow, or stopping and starting would be indicators for a hard disk that is too slow.
I suspect your operating system needs some attention. Read this article and see if this helps:
http://larryjordan.com/five-step-trouble-shoot/
Larry
Hi Larry,
I usually work with audio, but I’m starting to do my own music videos using FCPX. I’m about to invest into a 5k iMac 4.0Gb Quad and 256Gb Flash, and I would really love your advise on what I would need for external drive. I’m starting to shoot at 4k using my iPhone 6s, and I also use a canon t3i.
I’m not sure what would be the best and most affordable option for external drive for 4k. I have a 2Tb firewire 800 Lacie drive, but after reading in your article, I realize i need i might need a raid array. What would you recommend?
Thanks
Ron:
RAIDs are an excellent idea when mixing multi-track audio or video editing.
Companies to consider include:
* OWC (www.macsales.com)
* Promise Technology
* CalDigit
Be sure they support Thunderbolt 2 or later.
Larry
Hello again Larry
Thank you much for the reply and the info. I will definitely check those sites out.
Cheers