Measuring Video Compression Quality

I’ve been thinking a lot, recently, about video compression quality and, thanks to some initial research by Espen Ryksen of Vizart Media, I wanted to find if there was an easy-to-understand and replicate way of comparing a source file to a compressed file.

BACKGROUND

Before we look at the measurement process, I want to provide a quick refresher on video compression.

The most important concept you need to understand about video compression is that the process of compressing a file ALWAYS removes data during compression. Always. This means that the more you compress a file to reduce it’s file size, the more data is removed.

Once removed, you can’t put this data back. This is the reason you don’t want to re-compress an already compressed file.

NOTE: A good analogy is pouring water from a 5 gallon bucket into a 1 quart pitcher. The essence of the water remains, but in the process of dumping the water from the bucket into the pitcher, you lost 4.75 gallons of water. (Um, for you metric people, there are 4 quarts in one gallon.)

This means that there will ALWAYS be differences between a compressed image and the source. The key is to make those differences as invisible as possible.

Another important note is that different movies, codecs and bit rates yield different results. My goal in this article is NOT to compare software, settings or codecs, but to show you how you can compare results for yourself to see which looks the best for your files.

Compression results can and do vary wildly depending upon software, settings, bit rates and the movie you are compressing.

DETERMINING VIDEO QUALITY

The video quality of a compressed image is dependent upon five settings:

• Compression codec (i.e. H.264)
• Compressed frame size
• Compressed frame rate
• The amount of movement between frames
• The data rate (bit rate) set for the compressed file

As these vary, your results will vary.

THE SETUP

What Espen did was to match the compressed image with the source image using the Difference blend mode in Motion. What Difference does is show the differences between two frames. This was a brilliant idea.

But, the problem with it is that in order to see the amount of changes in Motion, you need to start messing with either Gamma or Level settings, which changes the blend mode and yields inaccurate results.

So, I took the same concept, moved the files to Final Cut Pro X which means that I can measure the differences using the video scopes. This is much better and easier, plus it allows even inexperienced editors to see the amount of “damage” inflicted on their images using compression.

BIG NOTE

This process can be identically replicated in Premiere Pro CC – using its scopes and the Difference blend mode. I’m using FCP because I find the scopes in Final Cut easier to read than Premiere.

NOTE: You can’t use this technique in Final Cut Pro 7, because FCP 7 does not have a Difference blend mode.

Also, while I’m comparing files compressed in Compressor, this technique works the same for files compressed in any compression program:

• Adobe Media Encoder
• MPEG Streamclip
• Telestream Episode
• Sorensen Squeeze

This technique is a good way to test different compression technology and see which one works the best for your projects.

TEST MOVIE

Here’s the test movie; its a file I downloaded from Pond5. These two girls are spinning swiftly on a carousel (merry-go-round, if you prefer). I chose this frame because it is after the movement begins and, at 1:17 in, the playhead is not parked on any I-frames, which would make the compression look better than it actually was.

What makes this a hard image to compress well is:

• The amount of movement between frames due to the blowing hair and spinning background
• The hair texture and detail
• The subtle color gradations in their skin

The master file from Pond5 was 1920 x 1080 encoded using the PhotoJPEG codec. I converted this to 1080p ProRes 422 shortly after I downloaded the file last year.

For this test, I created four versions – all down-rezed to 1280 x 720:

• ProRes 422 – this became my new master file for this test
• QuickTime H.264 2-pass with a data rate of 2,000 kbps
• QuickTime H.264 1-pass with a data rate of 2,000 kbps
• QuickTime H.264 2-pass with a data rate of 200 kbps

ProRes 422 is the standard optimized video format for Final Cut Pro X and an excellent codec for material originating on a camera. (ProRes 4444 is a better codec for material originating on the computer.)

VIDEO SCOPES

The video scopes were set to Waveform, Luma, which emphasizes differences in edges and gray scale. I found these settings show more artifacts than displaying chroma.

TEST PROCEDURE

I created a new project and stacked the clips as you see here. I duplicated the source clip on the top layer to allow us to verify that the comparison system works properly.

(Click to see a larger image.)

Here’s the source clip, with the scope, so you can see levels for the source clip.

I then disabled all clips except the bottom layer and one other layer. (In this example, only the top and bottom layers are active.) There were only two layers active for each test: the compressed clip and the source clip.

Then, after selecting each layer, I opened the Inspector and changed Composite > Blend Mode to Difference. The Difference blend mode shows the differences between two clips in both gray scale and color.

RESULTS

(Click to see a larger image.)

When we compare the two source clips, notice that the scope is dead flat. This indicates that there are no differences between the two clips. This is exactly what we would expect, since its the same clip on both layers.

I did this test to make sure the idea of using the Difference blend mode would actually work. It does.

Now, let’s take a look at three difference compressed versions of each clip. The more the scope varies from a flat line, the more differences are in each clip.

(Click to see a larger image.)

This compares the 2-pass, 2000 kbps QuickTime version against the source file. Notice that there is now “noise” in the scope, which illustrates the differences between the source and the compressed version. However, the images are not so divergent that we can actually see them on the scope.

Is to compress the file such that differences are as small as possible, while still keeping file sizes small.

NOTE: It is important not to change video levels on one clip. While this does make the differences visible in the Viewer, you are actually ADDING differences by changing video settings. So, we can’t do that, it distorts the results such that they are no longer meaningful.

(Click to see a larger image.)

This compares the source clip with the 1-pass 2,000 kbps version. The image quality, as measured here, is similar to the 2-pass version, and the file sizes for these two five second clips are, essentially, identical.

Noise spikes up to 25%, but the bulk of the difference is less than 15%

(Click to see a larger image.)

This example compares the 2-pass 200 kbps version. Now we are seeing significant differences in the Viewer as well as the scope. The scope is showing difference spikes up to 50%.

The bigger the spikes, the more the differences in the image.

(Click to see a larger image.)

As you can see, this version has severe artifacting and image degradation caused by my purposely compressing this clip with a very low bit rate setting.

ADDITIONAL ANALYSIS

After I wrote this article, I had a small brainstorm. While we can’t adjust levels in Final Cut, because that would artificially alter the results of the Difference blend mode into meaninglessness, we CAN export a still frame from the blended clip and pull that into Photoshop for analysis.

So, I did.

NOTE: When exporting, save the clip as a TIFF or PNG to retain image quality – I used TIFF. Don’t use JPEG.

(Click to see a larger image.)

Here, for instance, is the 2000 kbps clip imported into Photoshop with highlights and mid-tones adjusted to emphasize the differences in the two frames. Here, the principal difference is visible in the edges of the two girls. It seems that compression changed the edge detail.

(Click to see a larger image.)

Look at the difference, using the same Levels setting, when comparing the clip which has a bit rate set way too low. Not only are edges damaged, but huge blocks in the image, specifically skin detail, is totally changed. I’m willing to give up a little edge detail, but not subtle gradations in color. This clearly points out that the compression settings are flawed.

NOTE: Remember, a perfect match yields a totally black clip.

Clearly, this two-step process of blending the images in Final Cut, exporting a single frame then analyzing it in Photoshop allows us to see very precisely what changed between the source file and the compressed file.

SUMMARY

My goal in this article is to give you a tool you can use to help improve the quality of your video compression by allowing you to easily determine the amount of differences between clips.

The basic rule of compression is that if it looks good, it is good. However, for those needing more empirical ways to measure results, this is a good place to start. Try these tests yourself on your own compressed files and see what you lose – and gain – as you change your settings.

As always, I’m interested in your comments.

---

Bookmark the permalink.