Why H.264 is a Bad Codec for Green-Screen Keys

Whether you use Apple Final Cut Pro, Adobe Premiere Pro, DaVinci Resolve or some other video editing software, green-screen keys are a mainstay of video production. (These are also called “chroma-keys” because they use color, “chroma,” or “blue-screen keys” because they use a blue background color rather than green.)

The problem is that, when it comes to keying (removing a color background), not all video codecs deliver the same quality – and the differences will drive you nuts. Why? Because more color is better.

All digital video images are composed of pixels arranged in a rectangular grid called a bitmap. Each pixel has one-and-only-one color, which also includes one-and-only-one grayscale value. Traditionally, the green channel is used to determine gray-scale values.

Here’s an example bitmap.

Here’s the same bitmap, but displaying just grayscale values. As you can see, each pixel has one-and-only-one color and grayscale value.

So far, so good.

WHAT IS COLOR?

Color values for digital video are a set of three numbers assigned to each pixel. These numbers represent the amount of Red (R), Green (G) and Blue (B) in that pixel (RGB). These three colors are called a pixel’s “color channels.” Each pixel has a unique set of color values. They might be the same from one pixel to the next, but they don’t need to be the same.

In the screen shot above, the color values for the  color highlighted by the red arrow has the color values contained in the lower green rectangle: Red = 0, Green = 128, and Blue = 255.

• For 8-bit color, such as DV and H.264, these color values range from 0 – 255.
• For 10-bit color, such as ProRes and some versions of HEVC, these color values range from 0 – 1023.
• For 12-bit color, such as ProRes 4444, these color values range from 0 to 4095.
• However, for video editing, all these color values are converted into a 16-bit uncompressed RGB color space where each pixel has 65,536 possible color values per channel.

The higher the bit rate, the more subtle the color variations between settings. While these are a LOT of numbers, the basics are easy to understand: the color of each pixel is defined by exactly three numbers.

WHAT IS A GREEN-SCREEN KEY?

Green-screen keying makes selected colors transparent. In “the olde days,” video used blue backgrounds while film used green. Those technical limitations are mostly gone and we’ve standardized on green because too many weather-people’s eyes were transformed into weather maps on live TV.

NOTE: Here’s a good Wikipedia article on the history of chroma-key.

Green is a good background color because none of us have green eyes of that intensity and few of us wear colors in that shade of green.

However, for chroma-key to work, the software needs to clearly recognize which pixels are green and which are not. For a pure green background that’s easy. The trick is accurately finding the edges.

For example, here’s the color value for the green background. Clearly, this is green.

Look at this yellow. Clearly, the color is yellow, but it’s composed of a lot of red and a lot of green. Is this a color in, say, a fabric, or an edge with lots of green spill? Should this color be kept, reduced or removed?

One more example, there’s more green in this pixel than any other color, but the color is not green. Is this blue, or more green spill from the set?

You see the problem. Chroma-key software needs to make some very tricky decisions: Which is the green in the background? Which is green spill from the set? What do you do if a color contains lots of green, but isn’t actually green? Should the green be totally removed, reduced, or retained.

Removing the background is easy. Determining the correct edges between background and foreground is really, really hard. Precisely identifying the color of each pixel is essential.

BUT, UM, WHAT’S THIS GOT TO DO WITH H.264??

Not to worry, I haven’t forgotten.

In an ideal world, every pixel would contain all three color values. While it is still tricky to determine the edges between foreground and background, at least the software would have as much color data to work with as possible.

The image above illustrates a chroma-subsampling called “4:4:4” – which I’ll define shortly. This is the color sampling used by ProRes 4444, a 12-bit codec which, as we know, generates huge files.

The easiest way to make a video file smaller is to remove color values. Every camera on the planet shooting Rec. 709 does this. How? By grouping pixels into two pixel blocks, then removing one red and one blue value from each block. This immediately reduces color data by 50%.

This means that for every two pixels, each pixel has a unique green value – because that’s where luminance values come from – while two pixels share the same red value and the same blue value. This is called 4:2:2 chroma-subsampling.

Unlike 4:4:4 color sampling, where color is defined down to the pixel, with 4:2:2 the closest we can get to defining an edge is two pixels. Why? Because the same color value is shared between two horizontal pixels.

Since the goal of H.264 is to create the smallest file size, even more color is removed. Now, pixels are grouped into four pixel blocks. Each pixel retains its unique green channel value. But a single red and blue value is now shared by all four pixels in the block. This reduces color data by 50% again! We now have only 25% of the original color data.

This is called 4:2:0 chroma-subsampling. If you simply look at the image, it looks fine. But if you are chroma-keying and trying to find accurate edges, the closest you can get is a four-pixel block.

CAN’T WE ADD COLOR BACK IN?

No. Once the color is lost, all we can do is guess at what the actual color is. This is, in fact, what the chroma-key software is doing. It is guessing what the original color might have been. Sometimes, these guesses are close to the truth. But not always.

This is especially true with fine details like hair. Because the detail is so fine and the color data is shared across a block of four pixels, the end result is flickering strands of hair, or edges so blurry that the entire image goes soft.

Color, once lost during production, can’t be restored in post. The best any software can do at that point is guess what the color might be.

SIDE NOTE: WHERE DO THESE NUMBERS COME FROM?

From the four-pixel block. Given a block of four pixels:

• 4:4:4 has four red, four green and four blue color values. Pixel colors are accurate to the individual pixel.
• 4:2:2 has four green, two red and two blue color values. Pixel colors are accurate to a two-pixel block.
• 4:2:0 has four green, one red and one blue color value. Pixel colors are accurate to a four-pixel block.

SUMMARY

Clearly, if you don’t have the budget – or the client doesn’t require high-quality – H.264 is cheap and easy. But it isn’t good.

Yes, all keying software uses luminance values to help figure out what’s an edge and what isn’t, but, as this example makes clear, the more color data you provide, the more accurate (i.e. cleaner)  your keying results will be.

Higher keying quality requires shooting formats that support more color per pixel, such as ProRes 422. (There are other codecs that support 4:2:2 as well – check your camera specs.) The trade-off is larger files and, in some cases, more expensive cameras.

---

Bookmark the permalink.