The Future of Video Editing Rests on Storage Bandwidth

I, like many others, am looking forward to the release of Apple’s latest powerhouse: the iMac Pro. Promised to breathe fire and leave skid marks across every digital highway on the planet, what’s a geek not to like?

Except… well, as we move into the world of 4K, HDR, and 360° VR video, a fast computer is important but no longer the critical factor. Our storage is. And, as we increasingly move into a collaborative environment, our ability to share projects and media with team members has a direct effect on whether we can meet our budgets and deadlines.

Here’s the problem: Apple focuses on creating computers that are powerful and flexible, combined with its legendary ease of use. All of which I fully approve. But, to achieve that speed Apple is increasingly using internal SSD drives as their main storage.

SSD drives have many advantages. When compared to traditional spinning media, they are:

• Extremely fast
• Extremely rugged
• Extremely small
• Extremely quiet
• But, they don’t approach the capacity of traditional drives.

Today, most computers sport 250 GB – 1 TB internal SSD drives. The new iMac Pro can be expanded to 4 TB, though pricing hasn’t yet been announced. It is blindingly fast, but its storage isn’t big enough. Which means that we need to connect it to external storage – and that’s where the problem lies.

HERE’S THE CHALLENGE

For editing HD media, any computer released in the last six years will be fine, though older systems will have problems playing or compressing HEVC (H.265) media.

However, when compared to HD:

• 4K files are 4X bigger
• 360° VR files are 4X bigger
• HDR files are AT LEAST 4x bigger

In other words, a storage system that works perfectly fine for HD will be overwhelmed in both capacity and bandwidth when moving into these new formats. Stepping up to 4K HDR files could generate media storage requirements that are sixteen TIMES what you are using now, with bandwidth requiring hundreds of megabytes per second.

How are we going to handle this?

Yes, we can compress this larger media by working with proxies but this is something you need to think about before you commit to that exciting new HDR project.

For example, when I create new training, I always work in Apple ProRes 4444 because it yields the highest image quality for screen captures. My recent Final Cut Pro X 10.3 training – which had 220 movies in it – generated almost 6 TB of data during production; FAR beyond what any currently affordable SSD can handle.

DEFINTIONS

Capacity. The amount of data your storage device can hold. Measured in TB (Terabytes).

Bandwidth. The speed that your storage device can transfer data between itself and your computer. Generally measured in mb/s (megabits per second), this number is more useful when converted to MB/s (MegaBytes Per Second) by dividing mbps by 8. Thus, 1000 mbps equals 125 MB/sec.

Protocol. How a storage device connects to your computer. Common options are: Ethernet, Thunderbolt 3, Thunderbolt 2, Thunderbolt 1, FibreChannel.

Here are some numbers you can use to estimate bandwidth from your storage:

• For smoothest editing, your computer needs access to uninterrupted bandwidth equal to 2x the data rate requirements of the codec you edit. As examples, providing two streams of ProRes 422 1080/30 requires about 37 MB/sec. Two streams of ProRes 422 UHD/30 requires about 148 MB/sec, while two streams of ProRes 4444 6K/30 HDR requires about 796 MB/sec. RAW formats require far larger bandwidth.

NOTE: Here’s a link to an Apple White Paper that provides more detail on ProRes data rates.

• A single SSD drive: 400 MB/second
• A single spinning hard drive: 125 MB/second
• 1 Gb Ethernet (1000 Mbps): 125 MB/second max (average around 108 MB/s)
• 10 Gb Ethernet (10,000 Mbps): 1,250 MB/second max (average around 1 GB/s)
• Thunderbolt 1: About 800 MB/second.
• Thunderbolt 2: About 1,250 MB/second – the rest of the bandwidth is reserved for monitors – provided you have enough devices in your RAID to “fill the pipe.”
• Thunderbolt 3: About 2,200 MB/second – the rest of the bandwidth is reserved for monitors – provided you have enough devices in your RAID to “fill the pipe.”
• RAID 0: Multiply the bandwidth from the devices list above times the number of drives in it. So, a four-drive RAID 1 using spinning media would be (4) * 125 = 500 MB/second.
• RAID 5: Multiply the bandwidth from the devices list above times the number of drives in it minus 1. So, a four-drive RAID 5 using spinning media would be (4 – 1) * 125 = 375 MB/second.
• RAID 6: Multiply the bandwidth from the devices list above times the number of drives in it minus 2. So, a four-drive RAID 6 using spinning media would be (4 – 2) * 125 = 250 MB/second.

As every manufacturer will tell you, there is lots of variability in these numbers, but these are good to use for estimates and planning.

NOTE: To get maximum benefit from any protocol, say Thunderbolt 3, both the computer and storage device must support the same protocol. Recently, I was testing the speed of Thunderbolt 2 devices, only to discover that my computer only supported Thunderbolt 1, which meant that the devices were running slower than they were designed.  Or, another example, to get the maximum speed from WiFi, both your computer and WiFi receiver need to support the latest protocols.

I WORK ALONE

If you are a solo editor, then your goal should be to purchase the fastest RAID using a connection protocol that is supported by your computer:

• Thunderbolt 2 for older systems
• Thunderbolt 3 for newer systems

The more devices (drives) a RAID contains, the faster it will go and the more it will store. While there are other protocols –  iSCSI, FireWire, eSATA – Thunderbolt will have faster performance, lower costs and better market support.

When purchasing storage my advice is to buy as much storage capacity as you can afford. There is no such thing as “too much storage.”

I WORK IN A GROUP

When you need to share media between members of a local workgroup, things get more complex. And, when you need to share files with members outside the local workgroup, you’ll need professional help in figuring out the best way to set this up.

When I’m creating new training, for example, three of us work on it – all accessing shared storage. Since Thunderbolt is not a “shareable” protocol, we need to use either Ethernet or FibreChannel. Since wiring my office with fiber is way too expensive, I’m forced to use Ethernet.

When sharing files over a network, HD is relatively simple because the bandwidth needs are so small (comparatively). But problems arise as I move into larger format files.

More personal examples:

• WiFi speeds in most recent laptops, max out around 46 MB/second.
• Low-end switches, like those from NetGear that cost less than $100, limit throughput to about 50 MB/second.
• Higher-end 1 Gb Ethernet switches, like those from Cisco, limit throughput to about 125 MB/second.

Last night, I needed to transfer 3.2 TB of data from local storage to a network server via WiFi. The process took 21 hours. Backing up that data via 1 Gb Ethernet, which I need to do every night, took seven hours.

1 Gb Ethernet is not fast enough for 4K HDR media. However, rewiring my office to support 10 Gb Ethernet – which WOULD be fast enough – requires:

• Converters on the back of every Mac from Thunderbolt to 10 Gb Ethernet
• New cabling in the walls to support 10 Gb Ethernet speeds
• New 10 Gb, high-capacity switch
• A storage system that supports 10 Gb connectivity

NOTE: The new iMac Pro is announced to support 10 Gb Ethernet natively. It is the only Mac to do so.

As you can imagine, all this gear ain’t cheap. And wiring for fiber, though potentially faster, is even more expensive, as are the switches.

What I’m currently doing is tailoring the video formats I create and edit to match the speed of my storage. Which is an interesting thought: My computers are more than fast enough, but my storage is not – especially when I’m editing in a workgroup.

SUMMARY

Increasingly, projects that used to fit comfortably in a terabyte, are now expanding into dozens, even hundreds of terabytes.

The speed of our computers is more than fast enough – especially the newer ones with high-performance GPUs – to handle almost all these new media formats.

But storage bandwidth – and how we connect our gear – is not keeping up. If you plan to expand into all this new technology be sure to budget for a major expense in storage capacity, storage bandwidth and faster network infrastructure.

Otherwise, you’ll be all ready to edit – and waiting for your media to arrive.

---