How to Configure Video Editing Storage for Maximum Speed or Capacity
[ Updated Nov. 16, 2023, to correct a few errors. ]
Last week, I got into a discussion with Loren Miller about how to configure various storage options for speed. Then, during this week’s webinar, Kathy asked about how to get faster storage without spending a lot of money.
This week’s article looks at both questions.
NOTE: At the end of this article you can download either a PDF or Numbers file so you can modify these numbers yourself.
EXECUTIVE SUMMARY
- The greater the capacity or the faster the performance, the more storage will cost.
- Storage performance needs are the same regardless of which NLE you use to edit.
- If you are only editing single stream HD media, the speed of a single hard disk (HDD) will be fine; however, it may not have the capacity you need.
- Capacity matters as the amount of media in a project, or the number of projects you need to archive, increases.
- Performance matters when you edit 4K media or larger frame sizes, or when editing multicam clips with more than a few cameras.
- These numbers assume both the NLE and media are stored locally. Cloud storage and editing has different requirements.
- [Updated] Storage speed affects render or export speeds; the faster the storage, the faster the render or export. (I’ve measured this up to 2 GB/s.)
- If you are editing media from a server, 1 Gbps Ethernet transfers data at about 110 MB/sec. 2.5 Gbps Ethernet transfers data about 250 MB/second. 10 Gbps Ethernet transfers data about 1 GB/second.
RECOMMENDATIONS
- If you are editing single-stream HD media, get a hard disk with the capacity you need. RAIDs are fine, but not required.
- If you are editing multicam projects with more than four cameras, get an SSD. SSDs will always outperform any HDD RAID for multicam editing.
- If you need speed with limited capacity at low cost, get a SATA SSD.
- If you need maximum speed with limited capacity at a medium cost, get an NVMe SSD.
- If you need reasonable speed with maximum capacity at medium-high cost, get a HDD (Hard Disk Drive) RAID, formatted as RAID 0. However, if a drive dies, you lose all your data.
- If you need maximum speed with as much storage as possible and data redundancy, get a NVMe RAID, formatted as RAID 1+0 or RAID 4, with as much capacity as you can afford.
- If you need lots of speed with the maximum capacity possible, plus data redundancy in case one drive dies, get at least a 4-drive HDD RAID formatted as RAID 5. More drives increase both speed and capacity.
THE DETAILS FOR SPINNING (HDD) MEDIA
Let’s start with spinning media. Hard drives transfer data between 100 – 225 MB/second, depending upon age, technology, manufacturer and amount of free space. For this table, I used an average of 175 MB/s.
There are three Thunderbolt protocols in current use:
- Thunderbolt 2 supports maximum data transfer rates of about 1,250 MB/second. (No longer shipped.)
- Thunderbolt 3 supports maximum data transfer rates of about 2,850 MB/second.
- Thunderbolt 4 has the same data transfer rate as Thunderbolt 3: 2,850 MB/second.
NOTE: If you download one of the spreadsheets at the end of this article, you can substitute your own numbers to see results more specific to you.
A RAID collects multiple hard drives so that the computer sees them as a single unit. In doing so, it combines both their speed and storage capacity. How a RAID is formatted (called the RAID “Level”) determines ultimate speed and capacity, as illustrated here.
- RAID 0: If one drive dies, all data is lost. Backups are essential.
- RAID 4: If one drive dies, all data can be restored when a new drive is inserted. (This format is optimized for SSDs.)
- RAID 5: If one drive dies, all data can be restored when a new drive is inserted. (This format is optimized for HDDs.)
- RAID 6: If two drives die at the same time, all data can be restored when new drives are inserted. (Optimized for HDDs and not generally used for SSDs.)
- RAID 1+0: This divides all drives into two equal groups. Each group is formatted as RAID 0, while the two groups are then formatted together as RAID 1 (mirrors). [UPDATE: While RAID 5 is faster for HDD RAIDs, RAID 1+0 is faster for SSD RAIDs.]
Given that RAID 1 simply mirrors two drives, it makes no sense to configure a RAID with more than two drives as RAID 1.
RAID 5 requires a minimum of 3 drives. RAID 6 and 1+0 require a minimum of four drives. RAID 1+0 also requires an even number of drives.
THE DETAILS FOR SSDS
NOTE: All SSDs are PCIe. The “PCIe SSDs” listed above should actually say “SATA SSD.”
SSDs are much, MUCH faster than spinning media. But they cost more and hold less. So, it’s a trade-off: blinding speed with limited space.
For technical reasons, SSD RAIDs work best when formatted as RAID Level 4. This allows you to lose one SSD, replace it, then safely restore all your data.
My recommendation, if you need the speed of SSDs is to either create a 4-SSD RAID 4 using SATA SSDs, or buy the largest capacity NVMe SSD you can afford. Both options would completely fill a Thunderbolt 3 or 4 pipe.
UPDATE NOTE: Stand-alone SATA SSDs are increasingly hard to find. Most are replaced by NVMe SSDs.
SUMMARY
Storage provides lots of options for video editors. So many, that trying to decide which works best for you can be really confusing. The key is to understand your workflow. There’s no reason to spend a fortune on high-speed storage if you are only editing HD material.
Performance becomes critical as frames sizes increase or you start editing multicam projects.
EXTRA CREDIT
- Here’s the Numbers spreadsheet I used. RIGHT-CLICK the link and select “Download File.” You’ll find it in your Downloads folder.
- Here’s an Excel version of my spreadsheet. RIGHT-CLICK the link to download the file.
- Here’s a PDF of that same Numbers spreadsheet. Use this if you don’t have Numbers. (Right-click the link to download the file.)
Bookmark the
permalink.
Larry Recommends:
NEW & Updated!
Edit smarter with Larry’s latest training, all available in our store.
Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!
JOIN NOW
Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.
17 Responses to How to Configure Video Editing Storage for Maximum Speed or Capacity
Excellent rundown, Larry; I only add users should also keep in mind (if they haven’t read the RAID article I wrote which Larry has kindly published on this site) the limitations of unpowered SSD’s if you want to safely archive them on the shelf for more than a month or so. It won’t always be that way, we all need speed, (and Larry’s speed chart jibes nearly identically with my current Thunderbolt 2 layout) but we also need archiving capability. At this moment in the evolution of storage, spinning drives win out on affordability, capacity, and persistence of data on the shelf.
As for RAID configs, I think you nailed it– RAID 5 looks very attractive– I just worry about the speed of restoration when a drive fails; I don’t like the idea of losing a day to a disk rebuild! And that inspires me to inspire you…
How about a report comparing times rebuilding a mirror RAID 10 to rebuilding a RAID 5?
Loren:
Rebuild times vary depending upon the size of the drives, the number of drives and the amount of data.
To properly test this would require a large variety of drive sizes and enclosures. Many more than I currently own.
Larry
Caution overall with the usual nas systems. the manufacturers state that “two” NVMEs can also be used, but unfortunately these can only be used for nas internal cache functions and not as fast ssd drives for external access.
the link aggregation (on 2gbit) given by many nas manufacturers is also critical. because here you only have 1gbit net.
Achim:
Thanks for your comments. I think NVMe is most local area net servers is overkill. PCIe will do just as fine and cost a fraction as much. I have a PCIe SSD accelerator in my server, which which makes initial contact and directory display much faster.
Larry
I completely agree with you. buying expensive nvme drives for nas systems is complete overkill. the problem is that the nas manufacturers advertise big with “we also have nvme drives”. in the end you fall for it, as a non-technical person. thought the info would fit in here just fine. this also applies to link aggregation. net remain there also only 1gbit. synology advertises here in particular quite strangely.
Achim:
I have a 10 Gb adapter on my synology that does deliver much faster speeds when transferring large files, but NOT for smaller files.
I also saw a speed increase with link aggregation, but you also need to have a faster switch to support the speed.
Larry
Just when I think I have a good understanding of my little corner of editing in FCP, a question out of nowhere makes me doubt my understanding. Are you talking about just media files in your article on video editing storage, or should FCP libraries also be stored with this media? I usually have the FCP library on a separate HD (but NOT on the internal drive of my MBP.) Thanks.
Sharon:
Smile… I did not mean to confuse you. A very good system is to do what you are doing:
* Project / library files on Drive A.
* Media files on Drive B.
Then, for fastest results, export finished files to Drive A.
Larry
Thanks. You are always good to answer questions.
I have found Raid 0 to give the best speed on a 64 TB enclosure. One Volume is Raid 0 and three volume Raid 5. So with the library and media on the Raid 0 volume I, get speeds of 1800mb/sec. When I run AJA speed test and look at the instantaneous speed graph the Raid 0 speed shows interruption of read speed that cause dropped frames when playing timeline movie. The Raid 0 gives me less dropped frames than the Raid 5. If I use the laptop SSD for the render files I do not get dropped frames as much as having render files on Raid 0.For some reason the read does not stay constant over the play time length of the timeline project. Its complicated because the read speed gets interrupted somehow. SSDs are the only storage that give sufficiently high read speeds that these interruption do not drop below the minimum required to prevent dropped frames.
My best performance with FCP is a separate MacOS Volume with only the applications installed when MacOS is created plus FCP X and Softraid apps. I then use the Raid 0 volume for Library and Media within Library and the Render files on SSD laptop storage. With this arrangement I do not get dropped frames with 1080p FCP X Project at 29.97 fps. I have MacBook Pro Metal Max with 64GB ram and 2 TB ssd storage.
Norris:
Thanks for your comments. RAID 0 will always be fastest, RAID 5 provides protection in case one drive dies. However, all hard disk systems sometimes need to allow time for repositioning heads, or cooling or other activities that interrupt smooth playback.
In an ideal world, SSDs are better for video editing. But they are expensive and don’t hold as much as spinning media. However, if you are editing HD material (1080p, for example), I’m surprised you are getting any dropped frames.
Larry
I was also surprised but the interruption of the read drops in speed sometimes to 150 mb/sec and that is not enough speed to transfer a frame to the FCP, so a frame is skipped. If the the interruptions occur with very high average speed the interruption does not drop low enough to cause a dropped frame. The best quality and original media settings are the view settings I use with the timeline playback. I don’t get a lot of dropped frames with the Raid 0 setting. Oh, and I use CCC to backup my Libraries. The dropped frames is a problem of the Storage Media type. I have been working with OWC Softraid people to find out why the read has three or four drops in speed over the test period of AJA system speed test. No solution yet!
Norris:
Thanks for the follow-up. Let me know what you find out from OWC.
Larry
I think you are mistaken when saying that Raid 10 sacrifices 50% speed. On a 4 disk array, raid 10 will net 4x read and 2x write versus 2x read and 1x write on raid 5. Simply put, raid penalty is higher on Raid 5, 6. Also, raid 4 is a poor solution as the 1 parity drive will bottleneck your i/o.
Simply put, Raid 10 if you want performance + redundancy but it comes at a cost. Raid 5 if you want redundancy + maximize storage. Raid 6 for even more redundancy.
Trent:
It doesn’t work that way. Speed is determined by the number of drives recording unique data, not the total number of drives. So, in a 4-drive array, RAID 0 records data on 4 unique drives. RAID 1+0 records data on 2 unique drives with the other two drives recording duplicate data.
Also, RAID 4 is designed for SSD RAIDs, where the data from the parity drive is ignored, minimizing the speed penalty (in theory). RAID 5 is designed for spinning hard drives, maximizing throughput.
Here’s an article that illustrates the differences in speed between different RAID levels:
https://larryjordan.com/articles/real-world-speed-results-for-different-raid-levels/
Larry
So it seems that the perfect world solution for a small video production company would have 2 solutions: A HDD based, Raid 5 or 6 server for archived footage and projects…
And either internal drive or an external 2 drive NVME storage (Raid 0) for current projects etc.
Neil:
Yes. More generally: SSD RAID for active projects. HDD RAID for archives. The specific configurations depend upon your situation.
Larry