Why H.264 is a Bad Codec for Green-Screen Keys

Posted on by Larry

Whether you use Apple Final Cut Pro, Adobe Premiere Pro, DaVinci Resolve or some other video editing software, green-screen keys are a mainstay of video production. (These are also called “chroma-keys” because they use color, “chroma,” or “blue-screen keys” because they use a blue background color rather than green.)

The problem is that, when it comes to keying (removing a color background), not all video codecs deliver the same quality – and the differences will drive you nuts. Why? Because more color is better.

All digital video images are composed of pixels arranged in a rectangular grid called a bitmap. Each pixel has one-and-only-one color, which also includes one-and-only-one grayscale value. Traditionally, the green channel is used to determine gray-scale values.

Here’s an example bitmap.

Here’s the same bitmap, but displaying just grayscale values. As you can see, each pixel has one-and-only-one color and grayscale value.

So far, so good.


Color values for digital video are a set of three numbers assigned to each pixel. These numbers represent the amount of Red (R), Green (G) and Blue (B) in that pixel (RGB). These three colors are called a pixel’s “color channels.” Each pixel has a unique set of color values. They might be the same from one pixel to the next, but they don’t need to be the same.

In the screen shot above, the color values for the  color highlighted by the red arrow has the color values contained in the lower green rectangle: Red = 0, Green = 128, and Blue = 255.

The higher the bit rate, the more subtle the color variations between settings. While these are a LOT of numbers, the basics are easy to understand: the color of each pixel is defined by exactly three numbers.


Green-screen keying makes selected colors transparent. In “the olde days,” video used blue backgrounds while film used green. Those technical limitations are mostly gone and we’ve standardized on green because too many weather-people’s eyes were transformed into weather maps on live TV.

NOTE: Here’s a good Wikipedia article on the history of chroma-key.

Green is a good background color because none of us have green eyes of that intensity and few of us wear colors in that shade of green.

However, for chroma-key to work, the software needs to clearly recognize which pixels are green and which are not. For a pure green background that’s easy. The trick is accurately finding the edges.

For example, here’s the color value for the green background. Clearly, this is green.

Look at this yellow. Clearly, the color is yellow, but it’s composed of a lot of red and a lot of green. Is this a color in, say, a fabric, or an edge with lots of green spill? Should this color be kept, reduced or removed?

One more example, there’s more green in this pixel than any other color, but the color is not green. Is this blue, or more green spill from the set?

You see the problem. Chroma-key software needs to make some very tricky decisions: Which is the green in the background? Which is green spill from the set? What do you do if a color contains lots of green, but isn’t actually green? Should the green be totally removed, reduced, or retained.

Removing the background is easy. Determining the correct edges between background and foreground is really, really hard. Precisely identifying the color of each pixel is essential.


Not to worry, I haven’t forgotten.

In an ideal world, every pixel would contain all three color values. While it is still tricky to determine the edges between foreground and background, at least the software would have as much color data to work with as possible.

The image above illustrates a chroma-subsampling called “4:4:4” – which I’ll define shortly. This is the color sampling used by ProRes 4444, a 12-bit codec which, as we know, generates huge files.

The easiest way to make a video file smaller is to remove color values. Every camera on the planet shooting Rec. 709 does this. How? By grouping pixels into two pixel blocks, then removing one red and one blue value from each block. This immediately reduces color data by 50%.

This means that for every two pixels, each pixel has a unique green value – because that’s where luminance values come from – while two pixels share the same red value and the same blue value. This is called 4:2:2 chroma-subsampling.

Unlike 4:4:4 color sampling, where color is defined down to the pixel, with 4:2:2 the closest we can get to defining an edge is two pixels. Why? Because the same color value is shared between two horizontal pixels.

Since the goal of H.264 is to create the smallest file size, even more color is removed. Now, pixels are grouped into four pixel blocks. Each pixel retains its unique green channel value. But a single red and blue value is now shared by all four pixels in the block. This reduces color data by 50% again! We now have only 25% of the original color data.

This is called 4:2:0 chroma-subsampling. If you simply look at the image, it looks fine. But if you are chroma-keying and trying to find accurate edges, the closest you can get is a four-pixel block.


No. Once the color is lost, all we can do is guess at what the actual color is. This is, in fact, what the chroma-key software is doing. It is guessing what the original color might have been. Sometimes, these guesses are close to the truth. But not always.

This is especially true with fine details like hair. Because the detail is so fine and the color data is shared across a block of four pixels, the end result is flickering strands of hair, or edges so blurry that the entire image goes soft.

Color, once lost during production, can’t be restored in post. The best any software can do at that point is guess what the color might be.


From the four-pixel block. Given a block of four pixels:


Clearly, if you don’t have the budget – or the client doesn’t require high-quality – H.264 is cheap and easy. But it isn’t good.

Yes, all keying software uses luminance values to help figure out what’s an edge and what isn’t, but, as this example makes clear, the more color data you provide, the more accurate (i.e. cleaner)  your keying results will be.

Higher keying quality requires shooting formats that support more color per pixel, such as ProRes 422. (There are other codecs that support 4:2:2 as well – check your camera specs.) The trade-off is larger files and, in some cases, more expensive cameras.

Bookmark the permalink.

8 Responses to Why H.264 is a Bad Codec for Green-Screen Keys

  1. Mike Janowski says:

    This is an excellent explanation of color sampling.

    However, I’m tempted to say “What were you thinking? EVERYONE knows h264 (and indeed, all 4:2:0 sampled media) works terribly for CK. I’ve had to tell clients as much several times. So at least now I have the numbers to back me up.

    Of greater interest to me are your media management problems with Productions.

    I’ve been using Productions almost exclusively for over 3 years now, and though yes, I know there are some ‘got yas’ that you have to be careful of, I have found it to be a godsend; not the least of which is it eliminates multiple clip refs while working on episodes within a series, which used to torment me.

    Could you do an article describing those issues? I could send you my group’s experiences at ABC and see if there’s something you, or I, am missing

    • Larry says:


      Smile… the decision to record using H.264 was made before I got involved with the project. I suspect they chose this because they wanted small files, without thinking about the editing.

      I would be VERY interested in your write-up of how you are using productions! My problem was that there were elements in each episode that were the same. I kept tripping over the “only-one-clip” limitation. But, I was under deadline and didn’t have time to figure out an entirely different workflow.

      Please write something and share it with us.


  2. Rich Montgomery says:

    Sometimes you have no choice on this. My only available cameras shoot in a compressed, 8-bit codec. Usually, GS is not an issue as I work to get the lighting correct. However, shooting a cooking series (how to bake using the metric system) there was lots of GLASS to key through–UGH. I tried every trick to correct the various issues. Lesson learned though. I just wrote to say I am glad to hear the pro of pros and someone I learned immensely from had this same issue. Fortunately, the issues were minimal enough for the punters not to notice, and my “clients” are people I work with every day, not those who paid me. 🙂

    • Larry says:


      Thanks for writing! Sometimes YOU may know better, but if the client goes off and shoots something before you even get involved. Well, as you discovered, you live with what you’re given.

      It is in hopes of educating future clients that I wrote this article.

      And thanks for your very kind words.


  3. Kris says:

    Great explanation! Thank you, Larry.

  4. mark suszko says:

    I think in the not too far future, we’ll be beyond chroma key technology altogether. A combination of AI and machine vision already shows some ability to pull keys without a colored background, and the range- finding system in iPhones is being used to create synthetic variable depth of focus in post. What I think of as “z-“ keying” will pick out your actor from any background in real time, based on just distance to the camera, and store that in the encoding of the image format for easy access in live keying or in post.

    • Larry says:


      An interesting idea. What shooting with depth means, I think, is that we could isolate any object in the frame – foreground or background – and do something with it by accessing the depth data recorded with the image; without resorting to green-screen technology. However, I think LED video walls will become more popular first – Z-technology requires significant hardware upgrades throughout the entire production process.


Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!


Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.