Understanding Codecs – And Why They Are Important

I was reading a thread this morning on Linked In where a video editor was lamenting that DVDs and Blu-rays are rapidly dying, with nothing similar to take their place – except posting on-line and delivering media files on thumb drives. At which point, he asked the question: “What codecs deliver the best results for files placed on thumb drives?”

NOTE: I use the phrase “thumb drive” to mean a small, non-powered thingy that plugs into the USB port of a computer.

I’ve written a lot about codecs over the years because they are central to our ability to shoot, edit and distribute media. Just a few of these articles include:

• Choosing the Right Codec for Capturing SD Video (2004)
• Choosing The Best Video Codec (2006)
• Picking a Codec for HD (2007)
• The Quest for an Archiving Codec (2009)
• A Primer: I-frame vs. GOP Video Compression (2016)

But, our industry continues to evolve, so its time to revisit this topic. (Oh, and you’ll find the answer to this question in the Summary, at the end.)

DEFINITIONS

A Codec (Compressor/Decompressor) is a mathematical algorithm that converts reality (sound and light) into binary numbers that can be stored in a computer; that’s the “Compressor” part. Then, it converts all those binary numbers back into images and sound that we can see and hear; that’s the “Decompressor” part.

Codecs exist for still images, audio and video files. Popular still image codecs include:

• TIFF
• PNG
• JPG/JPEG
• GIF

Popular audio codecs include:

• WAV
• AIF/AIFF
• MP3
• AAC

Popular video codecs include:

• ProRes (in all its formats)
• DNxHD (in all its formats)
• CineForm (yup, in all its formats, too)
• R3D
• H.264/H.265

NOTE: Some formats, like QuickTime, MXF and MPEG-4, are not actual codecs, but containers that hold a variety of different codecs. For example, a QuickTime movie can hold a video file using the H.264 codec and an audio file using the AIF format.

In the past, I used to enjoy keeping track of how many audio and video codecs there were. But, frankly, I gave up when the number shot past 400. While the industry is not generating as many codecs as it was in the heyday of converting from SD to HD, not a month goes by that some manufacturer or developer is not announcing some new codec.

It is safe to say that there are “a lot!”

NOTE: As a sidelight, codecs are often divided into “lossy” and “lossless.” A lossless codec preserves all the original image quality so that when an image is restored it is indistinguishable from the original. RAW high-bit-depth files are examples of a mostly “lossless” codec. (I don’t think there is a perfectly lossless video codec; the files are just too big.)

A lossy codec “throws out” visual information as part of the compression process, which means that the compressed image does not have the original quality of the source. Virtually all video codecs are lossy to some degree, however the amount of loss varies by codec.

ONE MORE CORE CONCEPT

This one is really important: All media, in order to be stored on a digital device, must use a codec. Even more important, all media stored on a digital device must be compressed.

Some video, like that shot on an iPhone is significantly compressed. Other video, like that shot on a RED or high-end Arri camera is compressed, but not to the same degree.

The reason for all this compression is that video files are HUGE and engineers are always looking for ways to make them smaller without sacrificing too much quality. For example, a single 1080p frame in a theoretical “uncompressed” format can, depending upon bit depth, exceed 100 MB! This would require 3 GB PER SECOND to play back a 30 fps sequence!! As resolutions expand, these numbers only get worse.

Without compression, we could not edit video on our computers.

THE PROBLEM

The problem is that there is no free lunch. The perfect codec would provide “infinitely” high quality, “infinitely” small files with “infinitely” efficient editing performance.

Sigh… Ain’t gonna happen.

Instead, in the real world, we get to pick two, as the triangle above illustrates. We can choose codecs that create small files with reasonably high-quality, but they are not efficient to edit; requiring serious computer horsepower to edit. H.264 and H.265 are examples of this category.

Or, we can create very efficient files with great quality, but these files are not small. ProRes and DNxHD are examples here.

Or, we can create very small files that are efficient to edit, but the overall quality is poor. DV could be an example here.

There are four principal goals to consider when choosing a codec:

• The size of the compressed file
• The speed of compression
• The speed of decompression
• The quality of the final image

For instance, if you are posting a file to the Internet, the size of the file and the speed of decompression are more important than how long it takes to compress the file in the first place or the quality of the final image. That is not to say these last two are unimportant, just less important.

On the other hand, if you are streaming a live event, the speed of compression is most important, because if you can’t compress faster than real-time, no one will be able to watch the event.

As a third example, for a network television program, the speed of decompression and the quality of the final image are of paramount importance.

WHAT’S CHANGED

In the past, back in the days of SD (standard-definition video), we would shoot, edit and distribute our media using a single codec. At the professional level, that codec would be DigiBetacam. Or, one level down, DV.

For all its problems with low resolution, interlacing and converting between three different frame rates; shooting standard def video was a walk in the park compared to the mess we find ourselves in today.

What has evolved today is what I call the “Three Codec Workflow.” We shoot one codec, then transcode (convert) it to a second codec for editing, then transcode it to a third codec for distribution. That middle codec is called an “intermediate” or “mezzanine” codec that exists solely to provide a high-quality, very efficient video format that is optimized for editing.

For example, an iPhone shoots H.264 in an MPEG-4 container. That gets converted to ProRes 422 for editing in Final Cut Pro X, then uploaded to YouTube/Vimeo/Facebook as a high-bit-rate H.264 in an QuickTime container.

Or, an Arri Alexa shoots using the ARRIRAW codec. This is transcoded to GoPro Cineform for editing in Premiere, then transcoded to a DCP for distribution to a digital cinema projector in a theater.

These are only two examples, there are dozens and dozens of variations. The point is that we need to use different codecs for different parts of the production and post-production process.

ADDING TO THE CONFUSION

In the past, we could count on codecs running on both Mac and Windows systems. Those days seem to be ending.

Apple has never made ProRes writeable (recordable) on Windows, though it will play easily. Also, recently, Apple has said that it will discontinue support for QuickTime on Windows.

This lack of QuickTime support is huge, because many video editors depend upon it for their own editing; Adobe Premiere comes instantly to mind.

GoPro Cineform and DNxHD are both solid codec alternatives for Windows, but the future lack of QuickTime support does not, yet, have an immediate solution. I’m expecting to hear more about this at the annual NAB Show later this month.

MAKING THINGS STILL MORE COMPLEX

At one level, the move to higher resolutions such as 4K or higher, is really no different than shooting and editing HD – except that we need MUCH more storage, with much faster bandwidth.

But, the move to full support for Rec. 2020 (sometimes called HDR), is more than resolution. It includes:

• Higher image resolutions
• Greater color saturation
• Greater variation in brightness

And to achieve these three goals requires video files that use 10-bit-depth codecs or greater throughout the entire production/post/distribution process. (Currently, the highest available bit-depth is 16-bits per channel (color).)

Shooting high bit-depth video is now commonplace. Most current cameras now offer a recording option that supports 10-bit video in some form.

Editing high bit-depth video is also straightforward. All ProRes formats support 10-bit video, with the two 4444 variants supporting 12-bit. Some, but not all DNxHD formats support 10-bit video, the rest only use 8-bit. GoPro Cineform supports 10-bit, with the RGB variants supporting 12-bit. So, while we need to pick the right codec, editing in higher bit-depth ranges is possible.

The problem is that many, in fact, most popular video codecs for distribution only support 8-bit images. This is one of the big benefits of the VERY-slow-to-roll-out H.265 codec; it supports 10-bit video or greater more easily than the current implementation of H.264.

With distribution, we need to carefully pick a codec that supports the bit-depth we need. And this varies by distribution outlet. There is no current standard codec that works for most situations.

A QUICK THOUGHT ON AUDIO CODECS

Fortunately, audio codecs are much easier to work with. AIF/AIFF and WAV files are two different containers for the same audio data. Working with either one during production and post is an excellent choice as both are considered high-quality, “uncompressed” formats.

Most video cameras and audio gear record 48 kHz sample rates at 16-bith depth. This is a good choice for recording on set.

High-end audio production will often work with 92 kHz sample rates, or higher, for the same reason that video producers shoot high-resolution 4K or 6K video and downsample to HD for editing: It gives them more data to work with when creating effects and mixes.

When compressing, AAC is a better choice than MP3. And sample rates of either 44.1 kHz or 48kHz at 16-bit depth will provide audio quality that exceeds normal human hearing.

NOTE: For distribution, I use 44.1 kHz for audio-only files and 48 kHz sample rates for audio which gets synced to video; both at 16-bit-depth.

SUMMARY

Codecs are like the engine to car. We need it to get anywhere, but most of us would rather think about something else. The problem is that, currently, NOT thinking about the codecs we are using can slow our editing, degrade our images or make our final edit undeliverable.

Making matters even more complex is that there is no single “best practices” codec. But, here are some thoughts.

• The best time to think about codecs is before you start production. Then, TEST your entire workflow before committing to production.
• Don’t assume you can use the same codecs on Windows that you use on Macs.
• Don’t assume all codecs are the same. Just as you pick different vehicles for different tasks, so, also, you pick codecs because they are best suited for that portion of a project.
• If you are shooting H.264/MPEG-4 files, transcoding to a more efficient codec for editing – ProRes, Cineform, DNxHD – makes sense, even though it requires more storage. Don’t let your storage force you to pick a specific codec, that is never a good choice.
• If you are shooting high bit-depth video, you need to make SURE that every codec you use during production and post will support at least 10-bit depth to retain the quality you originally shot.
• If you are not shooting with HDR in mind, H.264 is a fine distribution codec. If HDR is something you need to support, you need to research which codecs your distribution outlet will support.

And the answer to the thumb drive question at the top? Well, it depends. As long as you aren’t interested in supporting some form of HDR, compress the video using the H.264 codec, the audio as AAC, and store them both in an MPEG-4 container. Those formats will play on just about everything.

NOTE: Here’s a video that explains basic compression concepts.

However, it will surprise no one when I say that this will probably change by next year. Sigh…

---

Bookmark the permalink.