I, like many others, am looking forward to the release of Apple’s latest powerhouse: the iMac Pro. Promised to breathe fire and leave skid marks across every digital highway on the planet, what’s a geek not to like?
Except… well, as we move into the world of 4K, HDR, and 360° VR video, a fast computer is important but no longer the critical factor. Our storage is. And, as we increasingly move into a collaborative environment, our ability to share projects and media with team members has a direct effect on whether we can meet our budgets and deadlines.
Here’s the problem: Apple focuses on creating computers that are powerful and flexible, combined with its legendary ease of use. All of which I fully approve. But, to achieve that speed Apple is increasingly using internal SSD drives as their main storage.
SSD drives have many advantages. When compared to traditional spinning media, they are:
Today, most computers sport 250 GB – 1 TB internal SSD drives. The new iMac Pro can be expanded to 4 TB, though pricing hasn’t yet been announced. It is blindingly fast, but its storage isn’t big enough. Which means that we need to connect it to external storage – and that’s where the problem lies.
HERE’S THE CHALLENGE
For editing HD media, any computer released in the last six years will be fine, though older systems will have problems playing or compressing HEVC (H.265) media.
However, when compared to HD:
In other words, a storage system that works perfectly fine for HD will be overwhelmed in both capacity and bandwidth when moving into these new formats. Stepping up to 4K HDR files could generate media storage requirements that are sixteen TIMES what you are using now, with bandwidth requiring hundreds of megabytes per second.
How are we going to handle this?
Yes, we can compress this larger media by working with proxies but this is something you need to think about before you commit to that exciting new HDR project.
For example, when I create new training, I always work in Apple ProRes 4444 because it yields the highest image quality for screen captures. My recent Final Cut Pro X 10.3 training – which had 220 movies in it – generated almost 6 TB of data during production; FAR beyond what any currently affordable SSD can handle.
DEFINTIONS
Capacity. The amount of data your storage device can hold. Measured in TB (Terabytes).
Bandwidth. The speed that your storage device can transfer data between itself and your computer. Generally measured in mb/s (megabits per second), this number is more useful when converted to MB/s (MegaBytes Per Second) by dividing mbps by 8. Thus, 1000 mbps equals 125 MB/sec.
Protocol. How a storage device connects to your computer. Common options are: Ethernet, Thunderbolt 3, Thunderbolt 2, Thunderbolt 1, FibreChannel.
Here are some numbers you can use to estimate bandwidth from your storage:
NOTE: Here’s a link to an Apple White Paper that provides more detail on ProRes data rates.
As every manufacturer will tell you, there is lots of variability in these numbers, but these are good to use for estimates and planning.
NOTE: To get maximum benefit from any protocol, say Thunderbolt 3, both the computer and storage device must support the same protocol. Recently, I was testing the speed of Thunderbolt 2 devices, only to discover that my computer only supported Thunderbolt 1, which meant that the devices were running slower than they were designed. Or, another example, to get the maximum speed from WiFi, both your computer and WiFi receiver need to support the latest protocols.
I WORK ALONE
If you are a solo editor, then your goal should be to purchase the fastest RAID using a connection protocol that is supported by your computer:
The more devices (drives) a RAID contains, the faster it will go and the more it will store. While there are other protocols – iSCSI, FireWire, eSATA – Thunderbolt will have faster performance, lower costs and better market support.
When purchasing storage my advice is to buy as much storage capacity as you can afford. There is no such thing as “too much storage.”
I WORK IN A GROUP
When you need to share media between members of a local workgroup, things get more complex. And, when you need to share files with members outside the local workgroup, you’ll need professional help in figuring out the best way to set this up.
When I’m creating new training, for example, three of us work on it – all accessing shared storage. Since Thunderbolt is not a “shareable” protocol, we need to use either Ethernet or FibreChannel. Since wiring my office with fiber is way too expensive, I’m forced to use Ethernet.
When sharing files over a network, HD is relatively simple because the bandwidth needs are so small (comparatively). But problems arise as I move into larger format files.
More personal examples:
Last night, I needed to transfer 3.2 TB of data from local storage to a network server via WiFi. The process took 21 hours. Backing up that data via 1 Gb Ethernet, which I need to do every night, took seven hours.
1 Gb Ethernet is not fast enough for 4K HDR media. However, rewiring my office to support 10 Gb Ethernet – which WOULD be fast enough – requires:
NOTE: The new iMac Pro is announced to support 10 Gb Ethernet natively. It is the only Mac to do so.
As you can imagine, all this gear ain’t cheap. And wiring for fiber, though potentially faster, is even more expensive, as are the switches.
What I’m currently doing is tailoring the video formats I create and edit to match the speed of my storage. Which is an interesting thought: My computers are more than fast enough, but my storage is not – especially when I’m editing in a workgroup.
SUMMARY
Increasingly, projects that used to fit comfortably in a terabyte, are now expanding into dozens, even hundreds of terabytes.
The speed of our computers is more than fast enough – especially the newer ones with high-performance GPUs – to handle almost all these new media formats.
But storage bandwidth – and how we connect our gear – is not keeping up. If you plan to expand into all this new technology be sure to budget for a major expense in storage capacity, storage bandwidth and faster network infrastructure.
Otherwise, you’ll be all ready to edit – and waiting for your media to arrive.
27 Responses to The Future of Video Editing Rests on Storage Bandwidth
Newer Comments →-
paul liggitt says:
November 20, 2017 at 6:35 am
-
Larry says:
November 20, 2017 at 7:25 am
-
Lisa Holloway says:
November 20, 2017 at 6:59 am
-
Billy Walker says:
November 20, 2017 at 8:02 am
-
Larry says:
November 20, 2017 at 8:59 am
-
Hal Staniloff says:
November 20, 2017 at 9:14 am
-
Larry Jordan says:
November 20, 2017 at 11:28 am
-
Hal Staniloff says:
November 20, 2017 at 9:39 am
-
Harriman Nelson says:
November 20, 2017 at 10:46 am
-
Michael Bulbenko says:
November 20, 2017 at 10:53 am
-
Larry says:
November 20, 2017 at 11:32 am
-
chris says:
November 20, 2017 at 3:09 pm
-
Larry says:
November 20, 2017 at 3:35 pm
-
Gregory Shaw says:
November 20, 2017 at 3:28 pm
-
Jeff Lo says:
November 20, 2017 at 4:23 pm
-
Larry says:
November 20, 2017 at 6:46 pm
-
Jeff Lovett says:
November 21, 2017 at 4:24 am
-
Larry says:
November 21, 2017 at 7:26 am
-
Christopher Meurer says:
November 20, 2017 at 5:12 pm
-
Marc M. Batschkus says:
November 22, 2017 at 1:14 am
-
Larry says:
November 22, 2017 at 6:54 am
Newer Comments →Dear Larry, Timely article and filled with just enough digestible facts to be of great value, thanks. I recently changed jobs and am now editing on my own again. Looking at the 4k workflow speeds and storage needs along with GPU processors is a critical metric as i am gearing back up to be freelance. I need to update my website and creating new marketing materials as well. I don’t even have any biz cards. Maybe they are even passe’. Thanks again.
Paul:
Business cards are ALWAYS a good idea.
Larry
You really spoke to the situation I am in right now Larry. I have a late 17″ 2013 MacBook pro I have maxed out the ram and replace HD with SSD, but like you only have a thunderbolt 1 port, 800 port and USB 2s. I have an array of drives (12 TB) daisy chained together and a G Raid studio raid I use for back up cuz the port is only thunderbolt 1 and it also drives my second monitor. I have been limping alone for over a year now waiting on the new promised iMac. But I never thought about the investment in drives I’m going to have to make with the new computer! UG!!!! This is exhausting- Financially and brain-width-wise! Just when I think I am getting a handle on all of this they jump past me! Thanks again for the help and support. I’m grateful for YOU!
Larry, you mention you always work in Apple ProRes 4444. Can you explain how that is done in Premiere Pro CC 2018 please?
Billy:
If you are on Windows, ProRes is not a good choice. Apple is moving away from supporting ProRes on Windows.
On the Mac, Premiere can edit it easily, provided you have the Pro Video codecs installed from Apple. These can not be purchased separately, so the easiest way to get them is to purchase Compressor ($50) which then allows you to download the codecs.
After that, Premiere is all set.
Larry
I too am bewildered as to how all these new machines are going to be able to produce this incredible video without proper storage bandwidth. Your insights are right on the money as always; we need more bandwidth to our storage devices and more robust storage solutions to support all this video.
I too am interested as to how you encode to ProRes 4444 for your screen capture work. Knowing why would be great also. Are you using a specific tool for the capture? Quicktime screen capture doesn’t do it – or does it?
Thanks for all your hard work! ???✨
Hal:
I use Telestream ScreenFlow to capture the screen in Apple ProRes 4444. It is one of their output options.
However, there is a bug in High Sierra the messes with color. If you use ScreenFlow don’t upgrade – yet – to High Sierra.
I too am bewildered as to how all these new machines are going to be able to produce this incredible video without proper storage bandwidth. Your insights are right on the money as always; we need more bandwidth to our storage devices and more robust storage solutions to support all this video.
I too am interested as to how you encode to ProRes 4444 for your screen capture work. Knowing why would be great also. Are you using a specific tool for the capture? Quicktime screen capture doesn’t do it – or does it?
Thanks for all your hard work! ???✨
Timley article! I just upgraded the HD storage in one of my MacPro 5,1’s (which I use for A/V) and ran into an apparently not-so-uncommon problem: many newer 4TB & 6TB drives are only recognized on boot-up only–if you restart the machine, they don’t mount.
See this: https://discussions.apple.com/thread/7132585
I love the newer MacPros, but the form factor just doesn’t work as well for me in a pro environment: limited internal storage, no room for internal cards, harder to rack, etc.
But with what’s on the horizon, it seems we’re going to need so much storage space, it will ALL be external except for our boot disk. :/
Hi Larry,
most helpful, but I’m curious as to why you’ve made no mention at all of USB3.
Michael:
Mostly because I forgot about USB 3.1 Gen 2, as USB 3 is too slow. The naming convention for USB does not help us here.
For Macs, Thunderbolt is the best option. For PCs, USB 3.1 Gen 2 may be better.
Larry
The data transfer portion of Thunderbolt 3 is actually USB 3.1 Gen2…. leading to much confusion…. So I’m covered if I see either of those things.
Chris:
Correct.
Larry
Larry, when one surveys disk array products it becomes apparent that read/write speeds are not simple multiples of the number of drives. As you drift down the cost chain, the promised speeds drop, apparently throttled by less capable controller hardware (or?). I would love to see an article delving into what the premium arrays do to maximize throughput that cheaper arrays don’t. And maybe even where to look for that sweet spot of maximizing performance purchase dollars.
I am considering buying a Drobo 5DT to use with my iMac which has a Thunderbolt 2 port. I already have a G Raid connected to it which I use for storage but have maxed it out since we only shoot 4K footage now. Do you consider the Drobo robust and fast enough for editing 4K footage and do you have any hard drive preferences to fill it with (since Drobo says basically ANY drive will work)?
Jeff:
I find Drobos to be very expandable and very easy to use, but they’ve never been very fast – especially if you are direct connecting them to your computer. For HD editing, I’d say they would probably work. For 4K work, I don’t recommend them.
I do recommend them, though, for general office stuff and backup.
Larry
So what do you suggest? I don’t have time to load footage onto the internal SSD of my Mac and offload it when I am finished with a project- we do just too many projects at simultaneously for that. Is there a better alternative to the Drobo 5DT that is expandable, has Thunderbolt 2 and not horribly expensive?
Jeff:
When it comes to storage, Drobo is unique in its expandability. There are other systems that provide this, but they cost a whole lot more.
So, if expandability is your key requirement, Drobo is your best choice.
For video editing, however, expandability is far less important than capacity and speed. Drobo can offer the capacity, but not the speed. And, when it comes to price, the more capacity you need or the faster you want it to go, the more you need to spend.
Keep in mind that as you move into 4K, or VR, or HDR, you are working with massively large files. Which means you need to budget for much more storage. And, because these files need plenty of bandwidth, you’ll also need something fast.
For direct attached RAIDs, I’m currently a fan of OWC (www.macsales.com). Affordable, very fast and you can purchase them in a variety of capacities. But, “affordable” is in the eye of the beholder – no large storage is cheap.
For network attached storage (NAS), which are always RAIDs, I’m currently a fan of Synology (www.synology.com) and QNAP (www.qnap.com). I currently own single drives from G-Technology.com, RAIDs from OWC and Synology. I’ve worked with storage systems from probably ten other vendors – almost all of which were good.
For the work you plan to do, you’ll probably need to spend around $2,000. There are smaller systems that cost less and larger systems that cost more. This is not a quote, just a suggestion.
Larry
Coming at this as a Cinematographer we also take many deep sighs as resolution sizes increase, in turn eating up media cards and tripling the time it takes to hand over footage at the end of the day as we wait for cards to transfer to at least 2 drives for safety. While too long to go in depth here, I shoot a lot a RED footage and it’s getting ridiculous as we are forced to shoot full resolution on the newest cameras to keep our lenses true to their focal lengths. (RED crops the sensor when you lower resolution) So when possible I try and use native 4 or 5k REDs to avoid being forced to shoot at 6k or 8k. The images look just fine and the files sizes are so much easier to deal with all the way through the edit. I’m really hoping that the resolution game slows down like the 3D game did. After all a great story will captivate an audience regardless if it’s shot in HD or 8k. I think we all get tired of chasing our tails on tech challenges when we would all prefer to spend that time creating cool stuff. Thanks for staying on top of it for us Larry!
To add to the picture, LTO-8 is now available and creates even more choices for archive either traditionally after finishing production or as ingest archive to save production storage. How to easily chose the right LTO tape is detailed in this article plus infographic
http://www.archiware.com/blog/4-criteria-to-choose-the-right-lto-tape-build-an-archive-save-money-along-the-way/
Marc:
Thanks for posting your article, I thought the info graphic was excellent. I also had not realized that LTO-8 was already available.
Larry