In the Olde Days, when hard disks were connected to computers by FireWire, we could select storage based upon capacity and price. Why? Because, fast though the FireWire protocol seemed at the time, it was slower than the speed of the hard disks connected through it.
The “gating factor,” as I call it, that determined the speed of our storage was the FireWire protocol.
This wasn’t a big deal because we were editing standard-definition DV video at the time which easily fit into the speed of a FireWire 400 connection.
Fast-forward to today and our connection protocols are no longer the problem. USB 3.1 Gen 2, USB-C, and Thunderbolt are all blindingly fast protocols far exceeding the speed of any single spinning media hard drive or SSD connected to it.
Unlike working with DV, the shift to shooting high-resolution frame sizes, higher bit-depth, RAW and HDR media and faster frame rates, combined with ever-increasing shooting ratios, are all creating more and larger media files.
Picking the right storage is always a balance between what we want versus what we can afford. Most of us don’t have unlimited budgets. So, how do we determine the storage we need? This Q & A should help you better understand the basics of storage so you can ask more informed questions as you plan your next storage purchase.
Sure. This is confusing stuff.
While you CAN edit on a single internal drive, today’s media files demand more and more storage. It isn’t that the internal drive doesn’t “work,” its that it isn’t big enough. For this reason, even if you don’t buy an external drive today, you need to start budgeting for it.
It is amazing how fast you can fill a 1 TB SSD. If you plan to regularly edit media, external storage is in your future.
Also, it is easy to carry an external device from one computer to another. If all your data is stored inside your computer, moving data between systems is not easy.
In general, you want the operating system and all applications stored on the internal drive, with all media and project files stored on external storage. This means you can optimize the internal drive for speed, while optimizing the external drive for capacity.
On my editing system, the operating system and a full set of Apple and Adobe editing software takes 125 GB of storage. Currently, all three of my current editing systems have Fusion drives; one is 1 TB and the other two are 3 TB.
However, if I were to purchase a new system, I’d get an all-SSD system. Why? Because even a 3 TB Fusion drive does not hold enough for most video projects. Fusion drives are fast. SSDs are far, far faster and you’ll appreciate this performance for the OS and apps.
Smile… However much capacity you buy, it won’t be enough. (I’m currently a 1-man shop and I have 176 TB of online storage capacity at the moment; and most of it is full.)
You can save space by editing camera native media – such as H.264 or AVCHD – which tends to create smaller files. But those files can be difficult to edit on older systems, or for multicam editing, or when rendering effects or color grading.
For the purposes of planning storage, here’s what you need to think about:
Calculate all these factors to determine total storage, then double it. This additional storage covers backups, render files, work files and the fact that no director ever shoots less footage than they plan.
Here are two excellent white papers from Apple that provides approximate storage requirements for ProRes files in the first paper, and a description of ProRes RAW in the second. These are good guides to use for estimating storage needs, even if you plan to use other codecs.
NOTE: To convert Mb/s to MB/sec, divide Mb/s by 8.
Yes, but less than it used to. Thunderbolt 2, Thunderbolt 3, USB-C and USB 3.1 Gen 2 are all fast enough for editing up to and beyond 4K files.
You will need the greatest storage bandwidth for editing:
In general, the connection protocol is no longer a big issue. The number of drives you can access is a bigger issue. (See the next answer.)
Here’s the deal: The more hard disks you have connected to your system, the faster you can transfer data and the more storage capacity you have available.
There are four things you need to consider when choosing a RAID:
NOTE: Here’s an article that describes different RAID levels. I recommend you format spinning media as RAID-5 and SSD media as RAID-4.
Yes. More drives means your RAID can store more data and transfer data faster.
Here’s a quick calculator you can use for estimating
Multiply the speed of each drive times the number of drives in each system to determine the bandwidth. So, a four drive RAID with spinning media, connected via Thunderbolt 3, will transfer about 600 MB/second. That same RAID, using SSDs, would transfer data around 1.6 GB/second.
Before the yelling starts, these are rough figures to be used for estimating only. But they will get you in the ballpark so you can get a sense of how much you’ll need to spend and the kinds of questions to ask your storage vendor.
In an ideal world, SSD is the best choice. But it is very expensive and doesn’t hold nearly as much as spinning media. For myself, I like SSDs as my internal drive and spinning media for all my RAIDs.
Editing standard-def DV media requires 13 MB/sec. Editing 4K ProRes RAW files requires 250 MB/sec or more. RAW files themselves can require more than 1 GB/second of bandwidth!
Every camera vendor publishes data rate specs for the codecs recorded by their camera. Review these to determine how much storage you’ll need. Pay attention to frame size, frame rate and bit depth; all of which affect bandwidth and storage requirements.
NOTE: To convert Mb/s to MB/second, divide Mb/s by 8.
Remember, you’ll need storage bandwidth equal to at least twice that of the codec. So, if the codec requires 125 MB/second, your storage should reliably supply 250 MB/second.
Yes and no.
Large media files stream from storage to your computer more easily (faster) than small files. However, project files, render files and other work files tend to be small chunks of video.
In the end, video editing is a balance of both small and large files, so it doesn’t make sense to optimize your system for one file or the other.
It is not completely true that you get what you pay for, but it’s close. There are only three drive manufacturers in the world: Western Digital, Seagate and Toshiba. All other storage vendors are simply repackaging a drive from one of these three companies.
As we are editing essentially irreplaceable media, I would never recommend buying the cheapest drive. On the other hand, for locally-attached storage, there’s not a compelling reason to buy the most expensive drive in the market.
Yes. Enterprise drives are designed to run reliably under heavy-load, such as video editing. They are designed to last a long time and not lose data.
For systems that you are using principally for storage – such as backups and archiving – you don’t need enterprise drives. But, for systems that you are putting under heavy load, with demanding bandwidth and reliably requirements, enterprise drives are worth the money.
Yes and no.
RPM speed determines “latency,” how long it takes a hard drive to access a file. For databases and other random file searches, RPM speed makes a difference in how fast the database responds.
For long media files, RPM speed doesn’t make a big difference, because the drive is starting at the beginning of the file and streaming it till the end.
For render files and other work files, faster RPM speed can help improve performance.
For me, there’s no reason for a video editing system to use drives that go faster than 7200 RPM; the performance improvement is not worth the extra cost. For direct storage, I use 7200 RPM, for my server, I use 5400 RPM because the speed of Ethernet is far slower than the seek speed of the server RAID.
Remember, there are only three drive manufacturers. Drives may be cheap, but MAKING a drive is not cheap at all.
Every company tries to make the best drives it can. Most drives work reliably for years. Some fail. I don’t think any company intentionally makes bad drives.
Still, there are some drives that you might want to avoid. Backblaze, a Cloud-based storage company, has been publishing drive failure statistics for several years. I find their reliability tables very helpful.
Here’s their latest hard drive reliability report.
Shared storage means different things to different people.
The easiest shared storage is where media files are shared on a server, which every team member can access, with all projects stored and accessed locally. Here, just about any server will work, provided it has sufficient bandwidth.
MUCH harder is when you need to store and access projects as well as media from the server, or where multiple team members need to edit the same project at the same time. Both Avid and Adobe support this multi-editor approach, but you need to do your homework to determine what storage supports this. Just because the hardware says “server,” does not mean it will work in this kind of shared environment.
Final Cut supports editing projects directly from the server, but the server requires special configuration. More technical people than me have figured out how to do this. I have not.
Here’s another key thing you need to know: Each computer will access the server using either Ethernet or Fiber Channel:
Ethernet runs over copper or fiber cabling. Fibre Channel requires fiber cabling.
NOTE: Here’s a white paper from ATTO that can explain these differences further.
For me, The Cloud is a great tool for collaboration, but its value in editing is still not there, simply because connection speeds to the Internet vary so widely.
I prefer to use the Cloud for general team collaboration, video review and client comments, and keep all editing local. This also decreases risks of piracy or the inability to access my project because of Internet bandwidth limitations.
Keeping files on hard disk or RAIDs for medium-periods of time – say 1-7 years – is fine, provided you power up and exercise that storage at least once every six months. This keeps all mechanical parts lubricated and helps make sure that data doesn’t get lost.
Based on the experts I’ve talked to, it is not a good idea to archive media using SSDs for any length of time longer than 6 months.
For longer periods, I recommend storing files on LTO tape. The tape drive costs a lot, but once you purchase the drive, the cost of each tape cartridge is far less than buying a similar-sized hard disk.
Smile… so, doing homework makes your head hurt? OK, just to draw a line in the sand, here’s what I recommend.
4K, non HDR editing:
Multicam editing (4 cameras or more):
NOTE: Copy data to the SSD RAID for files you are editing, to take advantage of the high-performance of the system. Store unused files on the spinning media RAID.
HDR, VR, or RAW editing:
TOP PERFORMANCE editing:
NOTE: Copy data to the SSD RAID for files you are editing to take advantage of the high-performance of the system. Store unused files on the spinning media RAID.
WHAT DO I USE FOR MY SYSTEM?
On my favorite editing system, I have:
For writing and audio editing, the single drive is fine.
For video editing, I use the locally-attached RAIDs, with media backup on the server. I’ve done 2-person team editing with this system once, storing media on the server and all projects locally on each computer for editing, stored on the server for backup and to transfer between editors.
Editing projects – Premiere or Final Cut – are always stored locally for editing.
Storage is complex, because there are so many options. Not just in storage technology, but in all the different media formats that we edit.
My goal in this article was not to recommend a specific system, but provide a framework of information you can use in deciding what storage is the best for your editing.
As always, I’m interested in your comments.
Final Cut Pro X 10.4
Edit smarter with Larry’s brand-new webinars, all available in our store.