In the Olde Days, when hard disks were connected to computers by FireWire, we could select storage based upon capacity and price. Why? Because, fast though the FireWire protocol seemed at the time, it was slower than the speed of the hard disks connected through it.
The “gating factor,” as I call it, that determined the speed of our storage was the FireWire protocol.
This wasn’t a big deal because we were editing standard-definition DV video at the time which easily fit into the speed of a FireWire 400 connection.
Fast-forward to today and our connection protocols are no longer the problem. USB 3.1 Gen 2, USB-C, and Thunderbolt are all blindingly fast protocols far exceeding the speed of any single spinning media hard drive or SSD connected to it.
Unlike working with DV, the shift to shooting high-resolution frame sizes, higher bit-depth, RAW and HDR media and faster frame rates, combined with ever-increasing shooting ratios, are all creating more and larger media files.
Picking the right storage is always a balance between what we want versus what we can afford. Most of us don’t have unlimited budgets. So, how do we determine the storage we need? This Q & A should help you better understand the basics of storage so you can ask more informed questions as you plan your next storage purchase.
Sure. This is confusing stuff.
While you CAN edit on a single internal drive, today’s media files demand more and more storage. It isn’t that the internal drive doesn’t “work,” its that it isn’t big enough. For this reason, even if you don’t buy an external drive today, you need to start budgeting for it.
It is amazing how fast you can fill a 1 TB SSD. If you plan to regularly edit media, external storage is in your future.
Also, it is easy to carry an external device from one computer to another. If all your data is stored inside your computer, moving data between systems is not easy.
In general, you want the operating system and all applications stored on the internal drive, with all media and project files stored on external storage. This means you can optimize the internal drive for speed, while optimizing the external drive for capacity.
On my editing system, the operating system and a full set of Apple and Adobe editing software takes 125 GB of storage. Currently, all three of my current editing systems have Fusion drives; one is 1 TB and the other two are 3 TB.
However, if I were to purchase a new system, I’d get an all-SSD system. Why? Because even a 3 TB Fusion drive does not hold enough for most video projects. Fusion drives are fast. SSDs are far, far faster and you’ll appreciate this performance for the OS and apps.
Smile… However much capacity you buy, it won’t be enough. (I’m currently a 1-man shop and I have 176 TB of online storage capacity at the moment; and most of it is full.)
You can save space by editing camera native media – such as H.264 or AVCHD – which tends to create smaller files. But those files can be difficult to edit on older systems, or for multicam editing, or when rendering effects or color grading.
For the purposes of planning storage, here’s what you need to think about:
Calculate all these factors to determine total storage, then double it. This additional storage covers backups, render files, work files and the fact that no director ever shoots less footage than they plan.
Here are two excellent white papers from Apple that provides approximate storage requirements for ProRes files in the first paper, and a description of ProRes RAW in the second. These are good guides to use for estimating storage needs, even if you plan to use other codecs.
NOTE: To convert Mb/s to MB/sec, divide Mb/s by 8.
Yes, but less than it used to. Thunderbolt 2, Thunderbolt 3, USB-C and USB 3.1 Gen 2 are all fast enough for editing up to and beyond 4K files.
You will need the greatest storage bandwidth for editing:
In general, the connection protocol is no longer a big issue. The number of drives you can access is a bigger issue. (See the next answer.)
Here’s the deal: The more hard disks you have connected to your system, the faster you can transfer data and the more storage capacity you have available.
There are four things you need to consider when choosing a RAID:
NOTE: Here’s an article that describes different RAID levels. I recommend you format spinning media as RAID-5 and SSD media as RAID-4.
Yes. More drives means your RAID can store more data and transfer data faster.
Here’s a quick calculator you can use for estimating
Multiply the speed of each drive times the number of drives in each system to determine the bandwidth. So, a four drive RAID with spinning media, connected via Thunderbolt 3, will transfer about 600 MB/second. That same RAID, using SSDs, would transfer data around 1.6 GB/second.
Before the yelling starts, these are rough figures to be used for estimating only. But they will get you in the ballpark so you can get a sense of how much you’ll need to spend and the kinds of questions to ask your storage vendor.
In an ideal world, SSD is the best choice. But it is very expensive and doesn’t hold nearly as much as spinning media. For myself, I like SSDs as my internal drive and spinning media for all my RAIDs.
Absolutely yes!
Editing standard-def DV media requires 13 MB/sec. Editing 4K ProRes RAW files requires 250 MB/sec or more. RAW files themselves can require more than 1 GB/second of bandwidth!
Every camera vendor publishes data rate specs for the codecs recorded by their camera. Review these to determine how much storage you’ll need. Pay attention to frame size, frame rate and bit depth; all of which affect bandwidth and storage requirements.
NOTE: To convert Mb/s to MB/second, divide Mb/s by 8.
Remember, you’ll need storage bandwidth equal to at least twice that of the codec. So, if the codec requires 125 MB/second, your storage should reliably supply 250 MB/second.
Yes and no.
Large media files stream from storage to your computer more easily (faster) than small files. However, project files, render files and other work files tend to be small chunks of video.
In the end, video editing is a balance of both small and large files, so it doesn’t make sense to optimize your system for one file or the other.
It is not completely true that you get what you pay for, but it’s close. There are only three drive manufacturers in the world: Western Digital, Seagate and Toshiba. All other storage vendors are simply repackaging a drive from one of these three companies.
As we are editing essentially irreplaceable media, I would never recommend buying the cheapest drive. On the other hand, for locally-attached storage, there’s not a compelling reason to buy the most expensive drive in the market.
Yes. Enterprise drives are designed to run reliably under heavy-load, such as video editing. They are designed to last a long time and not lose data.
For systems that you are using principally for storage – such as backups and archiving – you don’t need enterprise drives. But, for systems that you are putting under heavy load, with demanding bandwidth and reliably requirements, enterprise drives are worth the money.
Yes and no.
RPM speed determines “latency,” how long it takes a hard drive to access a file. For databases and other random file searches, RPM speed makes a difference in how fast the database responds.
For long media files, RPM speed doesn’t make a big difference, because the drive is starting at the beginning of the file and streaming it till the end.
For render files and other work files, faster RPM speed can help improve performance.
For me, there’s no reason for a video editing system to use drives that go faster than 7200 RPM; the performance improvement is not worth the extra cost. For direct storage, I use 7200 RPM, for my server, I use 5400 RPM because the speed of Ethernet is far slower than the seek speed of the server RAID.
Remember, there are only three drive manufacturers. Drives may be cheap, but MAKING a drive is not cheap at all.
Every company tries to make the best drives it can. Most drives work reliably for years. Some fail. I don’t think any company intentionally makes bad drives.
Still, there are some drives that you might want to avoid. Backblaze, a Cloud-based storage company, has been publishing drive failure statistics for several years. I find their reliability tables very helpful.
Here’s their latest hard drive reliability report.
Shared storage means different things to different people.
The easiest shared storage is where media files are shared on a server, which every team member can access, with all projects stored and accessed locally. Here, just about any server will work, provided it has sufficient bandwidth.
MUCH harder is when you need to store and access projects as well as media from the server, or where multiple team members need to edit the same project at the same time. Both Avid and Adobe support this multi-editor approach, but you need to do your homework to determine what storage supports this. Just because the hardware says “server,” does not mean it will work in this kind of shared environment.
Final Cut supports editing projects directly from the server, but the server requires special configuration. More technical people than me have figured out how to do this. I have not.
Here’s another key thing you need to know: Each computer will access the server using either Ethernet or Fiber Channel:
Ethernet runs over copper or fiber cabling. Fibre Channel requires fiber cabling.
NOTE: Here’s a white paper from ATTO that can explain these differences further.
For me, The Cloud is a great tool for collaboration, but its value in editing is still not there, simply because connection speeds to the Internet vary so widely.
I prefer to use the Cloud for general team collaboration, video review and client comments, and keep all editing local. This also decreases risks of piracy or the inability to access my project because of Internet bandwidth limitations.
Keeping files on hard disk or RAIDs for medium-periods of time – say 1-7 years – is fine, provided you power up and exercise that storage at least once every six months. This keeps all mechanical parts lubricated and helps make sure that data doesn’t get lost.
Based on the experts I’ve talked to, it is not a good idea to archive media using SSDs for any length of time longer than 6 months.
For longer periods, I recommend storing files on LTO tape. The tape drive costs a lot, but once you purchase the drive, the cost of each tape cartridge is far less than buying a similar-sized hard disk.
Smile… so, doing homework makes your head hurt? OK, just to draw a line in the sand, here’s what I recommend.
4K, non HDR editing:
Multicam editing (4 cameras or more):
NOTE: Copy data to the SSD RAID for files you are editing, to take advantage of the high-performance of the system. Store unused files on the spinning media RAID.
HDR, VR, or RAW editing:
TOP PERFORMANCE editing:
NOTE: Copy data to the SSD RAID for files you are editing to take advantage of the high-performance of the system. Store unused files on the spinning media RAID.
Yup.
WHAT DO I USE FOR MY SYSTEM?
On my favorite editing system, I have:
For writing and audio editing, the single drive is fine.
For video editing, I use the locally-attached RAIDs, with media backup on the server. I’ve done 2-person team editing with this system once, storing media on the server and all projects locally on each computer for editing, stored on the server for backup and to transfer between editors.
Editing projects – Premiere or Final Cut – are always stored locally for editing.
SUMMARY
Storage is complex, because there are so many options. Not just in storage technology, but in all the different media formats that we edit.
My goal in this article was not to recommend a specific system, but provide a framework of information you can use in deciding what storage is the best for your editing.
As always, I’m interested in your comments.
15 Responses to Q & A: How to Pick the Right Storage
As always, great information Larry and, in particular, great specs on requirements.
Still looking forward to the New mac pro next year, 2019. We’ll wee what the requirements are by then.
Larry, I could be wrong here but I think RAID 5 is falling out of favor. We’re beginning to hear about RAID 6 more and more as well as RAID 1+0.
RAID 1+0 is a pretty strong option. If you were to use either 4 or 6 or more spinning media you get some very strong features. With pairs of drives, i.e. 4, 6 or more pairs, the paired drives can be setup as RAID 1 which mirrors each individual pair.
Then you can RAID 0, for speed, the entire set of mirrored pairs. You end of with speed and data security with the mirrored pairs. 4, 6 or 8 drives will give you 2, 3 or 4 mirrored pairs, all containing identical data if you choose to go that route.
I’m not a hard drive guru so I welcome comments as to the accuracy of my statement.
Billy Walker
Billy:
RAIDS are a balance between performance, paranoia and budget. What a RAID 5 does is allow one hard drive in the disk array to die, yet keep all your data secure until you can replace that drive and the RAID rebuilds. In creating a RAID 5, you lose the performance boost of one drive and the storage capacity of one drive. So, a 4-drive RAID 5 has the storage capacity and performance of a 3-drive system.
RAID 6 says that you can lose two drives AT THE SAME TIME, and all your data is safe until you replace the drives and rebuild the array. In creating a RAID 6, you lose the performance and capacity of two drives. So, a 4-drive RAID 6 would have the capacity and performance of a 2-drive system.
Does a RAID 6 protect your data better than a RAID 5 in the case of drive failure? Yes. Is it worth losing that much storage and performance in an online editing system? I don’t think so. In video editing, performance is everything; and it is very, very, very unlikely that two drives will die at the same time.
A RAID 1+0 system requires two RAIDS, each with four drives. One RAID is a mirror of the other; this is the RAID 1 portion. Then, each RAID is striped as RAID 0. This means that no drives are reserved for data protection in each RAID, but because the first RAID is mirrored by the second RAID, in the event of disk failure, no data is lost. So, in a 4-drive RAID, you get the speed and capacity of a 4-drive RAID, but you have to purchase two RAIDs to achieve this. Basically, you are spending twice as much to get, at most a 20% boost in performance.
Remember, NO RAID provides backups to your data, instead, they provide greater performance and capacity than a single drive. You still need to find some way to safely backup your data. For me, RAID 5 plus backups is the right balance.
Larry
Thanks for this thorough and informative article, Larry!
In the Backblaze reports, there’s a fourth brand mentioned, HGST/Hitachi. You named only three, WD, Seagate, Toshiba. What is the story about that fourth brand, HGST? Aren’t they a manufacturer?
Ronald:
Hitachi is owned by Western Digital. They were acquired in 2012.
Larry
Ah, I see. Thanks!
I have a MacBook Pro mid 2013, 2 USB 3 inputs, and I think a thunderbolt input, 1Tb SSD system drive. Can you, and if so, how would you set up a raid to this laptop?
Vicki:
You can’t create a RAID from your internal drive. No single drive can be a RAID.
RAIDs are external units purchased from companies such as:
* OWC
* G-Technology
* Promise Technology
* LaCie
* CalDigit
* mLogic
and many others, which are attached to your computer via Thunderbolt.
Larry
Thanks very much Larry. I always enjoy your perspectives on these topics.
Hi Larry. Am I still good to go along with the recommendations here or have there been any updates since then?
I’m looking to RAID my iMac Retina 5K, 27-inch, Late 2015 | 3.2 GHz Quad-Core Intel Core i5 | 24 GB 1867 MHz DDR3, with Graphics AMD Radeon R9 M390 2 God Bless.
I Live in the UK
Eyitayo
Eyitayo:
This article is essentially current. NVMe SSDs are the latest speed demon, but, essentially, the advice remains the same.
Larry
Many thanks
Many thanks Larry
I have a M1 MacBook Pro with a 256GB SSD. I have a 2TB external portable HDD I’ve been editing videos on for fun, but am looking to upgrade. My videos sometimes will have 8-16 videos (these are virtual music performances) on screen at once, usually in a grid, but I’m getting more and more creative as I go along. I just recently got a camera capable of shooting high quality 4K HDR. What kind of external drive would you recommend for this kind of workload? I’m not a professional and can’t quite afford to buy most of your suggestions above.
Kyle:
You need an SSD, which, in smaller sizes, aren’t that expensive. Take a look at the Samsung T-5 or T-7. (The T-7 is about twice as fast.)
Depending upon your projects, you should be able to fit everything into a 2 TB system. But, check your media files to make sure they will fit.
Larry