The Future of Video Editing Rests on Storage Bandwidth

I, like many others, am looking forward to the release of Apple’s latest powerhouse: the iMac Pro. Promised to breathe fire and leave skid marks across every digital highway on the planet, what’s a geek not to like?

Except… well, as we move into the world of 4K, HDR, and 360° VR video, a fast computer is important but no longer the critical factor. Our storage is. And, as we increasingly move into a collaborative environment, our ability to share projects and media with team members has a direct effect on whether we can meet our budgets and deadlines.

Here’s the problem: Apple focuses on creating computers that are powerful and flexible, combined with its legendary ease of use. All of which I fully approve. But, to achieve that speed Apple is increasingly using internal SSD drives as their main storage.

SSD drives have many advantages. When compared to traditional spinning media, they are:

Today, most computers sport 250 GB – 1 TB internal SSD drives. The new iMac Pro can be expanded to 4 TB, though pricing hasn’t yet been announced. It is blindingly fast, but its storage isn’t big enough. Which means that we need to connect it to external storage – and that’s where the problem lies.

HERE’S THE CHALLENGE

For editing HD media, any computer released in the last six years will be fine, though older systems will have problems playing or compressing HEVC (H.265) media.

However, when compared to HD:

In other words, a storage system that works perfectly fine for HD will be overwhelmed in both capacity and bandwidth when moving into these new formats. Stepping up to 4K HDR files could generate media storage requirements that are sixteen TIMES what you are using now, with bandwidth requiring hundreds of megabytes per second.

How are we going to handle this?

Yes, we can compress this larger media by working with proxies but this is something you need to think about before you commit to that exciting new HDR project.

For example, when I create new training, I always work in Apple ProRes 4444 because it yields the highest image quality for screen captures. My recent Final Cut Pro X 10.3 training – which had 220 movies in it – generated almost 6 TB of data during production; FAR beyond what any currently affordable SSD can handle.

DEFINTIONS

Capacity. The amount of data your storage device can hold. Measured in TB (Terabytes).

Bandwidth. The speed that your storage device can transfer data between itself and your computer. Generally measured in mb/s (megabits per second), this number is more useful when converted to MB/s (MegaBytes Per Second) by dividing mbps by 8. Thus, 1000 mbps equals 125 MB/sec.

Protocol. How a storage device connects to your computer. Common options are: Ethernet, Thunderbolt 3, Thunderbolt 2, Thunderbolt 1, FibreChannel.

Here are some numbers you can use to estimate bandwidth from your storage:

NOTE: Here’s a link to an Apple White Paper that provides more detail on ProRes data rates.

As every manufacturer will tell you, there is lots of variability in these numbers, but these are good to use for estimates and planning.

NOTE: To get maximum benefit from any protocol, say Thunderbolt 3, both the computer and storage device must support the same protocol. Recently, I was testing the speed of Thunderbolt 2 devices, only to discover that my computer only supported Thunderbolt 1, which meant that the devices were running slower than they were designed.  Or, another example, to get the maximum speed from WiFi, both your computer and WiFi receiver need to support the latest protocols.

I WORK ALONE

If you are a solo editor, then your goal should be to purchase the fastest RAID using a connection protocol that is supported by your computer:

The more devices (drives) a RAID contains, the faster it will go and the more it will store. While there are other protocols –  iSCSI, FireWire, eSATA – Thunderbolt will have faster performance, lower costs and better market support.

When purchasing storage my advice is to buy as much storage capacity as you can afford. There is no such thing as “too much storage.”

I WORK IN A GROUP

When you need to share media between members of a local workgroup, things get more complex. And, when you need to share files with members outside the local workgroup, you’ll need professional help in figuring out the best way to set this up.

When I’m creating new training, for example, three of us work on it – all accessing shared storage. Since Thunderbolt is not a “shareable” protocol, we need to use either Ethernet or FibreChannel. Since wiring my office with fiber is way too expensive, I’m forced to use Ethernet.

When sharing files over a network, HD is relatively simple because the bandwidth needs are so small (comparatively). But problems arise as I move into larger format files.

More personal examples:

Last night, I needed to transfer 3.2 TB of data from local storage to a network server via WiFi. The process took 21 hours. Backing up that data via 1 Gb Ethernet, which I need to do every night, took seven hours.

1 Gb Ethernet is not fast enough for 4K HDR media. However, rewiring my office to support 10 Gb Ethernet – which WOULD be fast enough – requires:

NOTE: The new iMac Pro is announced to support 10 Gb Ethernet natively. It is the only Mac to do so.

As you can imagine, all this gear ain’t cheap. And wiring for fiber, though potentially faster, is even more expensive, as are the switches.

What I’m currently doing is tailoring the video formats I create and edit to match the speed of my storage. Which is an interesting thought: My computers are more than fast enough, but my storage is not – especially when I’m editing in a workgroup.

SUMMARY

Increasingly, projects that used to fit comfortably in a terabyte, are now expanding into dozens, even hundreds of terabytes.

The speed of our computers is more than fast enough – especially the newer ones with high-performance GPUs – to handle almost all these new media formats.

But storage bandwidth – and how we connect our gear – is not keeping up. If you plan to expand into all this new technology be sure to budget for a major expense in storage capacity, storage bandwidth and faster network infrastructure.

Otherwise, you’ll be all ready to edit – and waiting for your media to arrive.


23 Responses to The Future of Video Editing Rests on Storage Bandwidth

← Older Comments
  1. Shameer Mulji says:

    Hi Larry,

    Great article. What are your thoughts on using MS’ Surface Studio desktop as a content creation device? Do you see stylus / dial / touch UI paradigms replacing KB / Mouse for professional content creation?

    • Larry says:

      Shameer:

      Everyone has an opinion – and they are all different – on what’s the “best” user interface.

      While I expect each to find success, for editors that are focused on speed and throughput, i don’t think you’ll find anything faster than keyboard and mouse.

      Larry

← Older Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends

Final Cut Pro X 10.4

FCPX Complete

Edit smarter with Larry’s brand-new webinars, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.