Understanding Storage and Media Bandwidth [u]

[ This article was first published in November, 2008. Updated: July 2009, May 2010, and significantly revised in June 2017. ]

Now that the shift to digital media is complete and the on-going race to every higher resolutions continues to pick up speed, it is increasingly important that we understand the role codecs and storage plays in the overall performance of our editing systems.

There are three key concepts we need to keep in mind:

1. Storage requirements change as frame size, frame rate and codecs change.
2. Hard disks are not infinite in speed, nor unlimited in storage.
3. Dropped frame errors and stuttery performance are generally caused by a storage system that is too slow for the media you are editing.

STORAGE

Its seems obvious, but it is very easy to ignore. File sizes and bandwidth requirements increase as frame size, frame rate and bit depth increase. The table below illustrates this.

Also, as you’ll see, different codecs create files of significantly different sizes.

HARD DRIVE SPEEDS

Hard drive “bandwidth” is the speed at which a storage device transfers data between itself and the computer. We also call this a drive’s “data transfer rate.” This speed is dependent upon:

• How the storage system is connected to your computer
• The number of drives it contains
• How fast those drives spin
• How full those drives are

Here are several things to keep in mind:

1. In the past, protocols like FireWire and USB 2 were slower at transferring data than the hard disk. So, how we connected the drive made a difference in its speed. Today, protocols like Thunderbolt and USB 3.1 Gen 2 are far faster than a single hard drive. So the speed of our storage is based on the number of drives they contain more than how the drives are connected to the computer.
2. A single spinning media hard drive transfers data between 100 – 150 MB/second.
3. A single SSD drive transfers data around 400 MB/second.
4. RAIDs are devices that contain multiple drives; either spinning or SSD. The speed of a RAID is dependent upon how it is configured.
5. USB 3.1 Gen 1 is FAR slower than USB 3.1 Gen 2.
6. As frame sizes increase, file sizes get bigger and bandwidth needs to be faster.
7. As frame rates increase, file sizes get bigger and bandwidth needs to be faster.
8. As bit-depth increases, file sizes get bigger and bandwidth needs to be faster.
9. Most editing codes – ProRes, DNxHD, GoPro Cineform, R3D – create large files which are optimized for editing performance.
10. The more data you store on a hard drive the slower it goes. A drive is fastest when it is empty. When a drive is totally full, it neither plays back, nor records. I strongly recommend keeping about 20% free space on your drive.
11. FireWire 800 drives are fine for standard definition video and single-stream HD editing. They will not be fast enough for editing multicam, 4K, RAW or extended bit-depth files.

In general, RAID 0 is the fasted configuration, while RAID 5 is the best balance between performance and guarding against data loss due to a drive failure. To determine the approximate bandwidth of a RAID, multiply the number of drives by the average speed of a drive. For example, a RAID 5 containing four spinning media disks will transfer data around 600 MB/second. Using four SSD’s will transfer data around 1,200 MB/second.

NOTE: By the way, if terms like RAID 0 or RAID 5 confuse you, read this short explanation of how RAIDS are defined.

Ideal Data Transfer Rates By Storage Connection Type

Connection	Data Transfer Speed
ThunderBolt 3	About 3,000 MB / second
ThunderBolt 2	About 1,400 MB / second
USB 3.1 Gen 2	About 1,000 MB / second
10-Gig Ethernet	About 1,000 MB / second
USB 3.1 Gen 1	About 450 MB / second
1-Gig Ethernet	105 MB / second
FireWire 800	70 – 80 MB / second
Formats Too Slow to Use
USB 2.0	10 – 15 MB / second
FireWire 400	20 – 25 MB / second
iSCSI	75 – 95 MB / second

 

Keep in mind that these speeds define the size of the “pipe” that carries the data. The actual data speed is dependent upon how many drives are connected, how the drives are configured and how full the drives are.

With this as a background, let’s see how this applies to video editing.

VIDEO FORMAT DATA TRANSFER REQUIREMENTS

Here’s how to read the table below.

Format indicates the frame size, frame rate and codec.

Store One Hour indicates how much hard disk space it takes to store an hour of material in a particular format. This allows you to estimate what your total storage needs are based upon the amount of material shot. Remember, to add 20% for general storage overhead and free space.

Single Stream indicates how much data, on average per second, is required by a single stream of that format. (Measured in megabytes per second: MB/s.)

Editing indicates the approximate bandwidth needed for normal “single-stream” editing in that format.

Multicam indicates the approximate bandwidth needed for a 4-camera multicam edit in that format. (Using proxy files will decrease this number significantly.)

Here’s the key point: The table below helps you to match the required data rate of your video format with the speed of your storage, indicated in the table above.

Bandwidth by Video Format

Video Format	Store 1 Hour	Single Stream (MB / second)	Editing (MB / second)	Multicam (MB / second)
SD
DV NTSC / PAL	13 GB	3.75	7.5	15
DVCPRO-50	27 GB	7.5	15	30
Uncompressed 8-bit (Beta SP)	72 GB	20.2	41	85
Uncompressed 10-bit (DigiBeta)	96 GB	26.7	54	110
ProRes 422 (NTSC or PAL)	19.5 GB	5.25	12	24
ProRes 422 HQ	28.1 GB	7.8	16	32
HD
HDV (25 mbps) 60i	13 GB	3.75	7.5	15
AVCHD 1080p/30	Up to 10.8 GB	1.5 – 3.0	6.0	12
AVCCAM 1080p/30	Up to 10.8 GB	1.5 – 3.0	6.0	12
AVC-Intra 1080p/30	45 GB	12.5	25	50
XDCAM HD (50 mbps)	28 GB	7.75	16	32
XDCAM EX 1080p/30	19 GB	5.2	12	24
DVCPROHD 1080p/30	45 GB	12.5	25	50
ProRes 422 (Proxy) 1080p/30	20 GB	5.6	12	24
ProRes 422 1080p/30	66 GB	18.1	36	72
ProRes 422 (HQ) 1080p/30	99 GB	27.5	55	110
ProRes 4444 (no alpha) 1080p/30	148 GB	41.25	85	170
R3D 1080p/30 *	137 GB	38	76	150
HDCAM 1080p/60	834 GB	237	474	948
4K AND BEYOND
UHD/30 ProRes Proxy	82 GB	22.75	46	92
UHD/30 ProRes 422	265 GB	73.5	150	300
4K/24 R3D *	586 GB	163	326	650
6K/24 R3D *	660 GB	183	360	720
Sony RAW 4K *	520 GB	300	600	1200

 

NOTES

* R3D and Sony files have a huge number of variations based upon compression rate, frame size, frame rate, and the amount of the sensor used. Consult the websites for specific Sony and RED cameras to determine the exact data rates for the media format you plan to shoot.

All AVC video formats are converted (transcoded) to ProRes422 during ingest into Final Cut Pro X. So, while the AVC source video is small, the converted ProRes are much larger.

ProRes is a variable bit-rate encoder, so file sizes vary depending upon format, image size, and frame rate. The HD specs for ProRes are  taken from Apple’s ProRes white paper. File sizes decrease for 720p files, or slower frame rates.

I’ve discovered that understanding the relationship between hard disk speeds and video formats greatly simplifies storage budgeting and solving dropped frame problems.

UPDATE – Nov. 17, 2008

Rich Roddman sent me the following note:

You have R3D files listed as not needing to create Pro Res files as they are imported into Final Cut. That is not quite true, while you can just drop the proxy file in the timeline to view or edit, if you use the Log & Transfer tool to import the R3D files they will transcode them into Pro Res (HQ). The 4K files are converted to 2K in in width, 3K import at 3K and 2K stays at 2K. Much like P2, it is not a fast process even with an 8 core tower.

Rob Naim adds:

There is a great widget from a UK company called Digital Heaven than can be helpful when working out video storage issues that you’ve discussed in your current newsletter.
Have a look at it here: http://www.digital-heaven.co.uk/videospace/

It doesn’t do data rates but does do storage needed for length of video and its free.

Larry replies: Thanks for both your thoughts.

 

---

Bookmark the permalink.