Pick The Right Video Archive Format

Posted on by Larry

[This article was first published in the July, 2011, issue of
Larry’s Final Cut Pro Newsletter. Click here to subscribe.]

 

Cheryl Foster sent me an email recently asking about archiving. She asked:

I would like to keep a .mov of the finished video (or sequence) for archiving. Obviously “none” compression is huge, H.264 is what TV stations find acceptable for TV spots, but I really would like a better quality. [What should I use?]

Larry replies: In the past, I’ve recommended PhotoJPEG for progressive video, and its cousin, MotionJPEG for interlaced video. But, I decided to do some research and share what I learned.

My son, Paul, is the digital archivist at the IMF (International Monetary Fund) in Washington, DC. As such, he’s responsible for archiving tens of millions of emails, digital documents, paper documents, and all the huge variety of formats they represent.

So, we had a conversation and here’s what he told me.

When archivists are trying to decide what video format to use for archiving, they are looking for something that meets three criteria:

While it is not necessary that the video format be an open standard, one of the reasons ProRes would not be considered is that it is developed and supported by only one company. This leaves it open to sudden termination such as the recent decision by Apple to sideline FCP 7 without appropriate conversion tools.

He asks, “what’s to prevent Apple from doing the same thing to ProRes?”

The format that was supposed to be the “next big thing” was MotionJPEG2000. He says this is a very nice codec that does a good job. However, currently, there are very few tools that encode it and even fewer that read it. This lack of migration support makes adoption difficult.

Instead, he says the current recommended archiving format is MPEG-2 in an MXF wrapper. A video conversion tool that my son recommends is FFmpeg — http://www.ffmpeg.org — its a lot geeky, but worth checking into.

However, since we are talking codecs and compression, I contacted Philip Hodgetts to learn what he recommends.

Philip replied:

I would concur with Larry’s suggestion of PhotoJPEG at 99% quality for archiving. MPEG-2 is way too lossy in my opinion, unless you’re at data rates of over 50 Mbit/sec, which is not possible with commonly available tools. Those tools also encode MPEG-2 [with chroma-subsampling of] 4:2:0, compromising color integrity in the future.

Right now there are more tools for ProRes in all its flavors than there are for high bitrate MPEG-2 at 4:2:2 profiles (High Profile (HP), 4:2:2 profile or MVP profile) and standard bitrate MPEG-2 is not an archive format.

For archiving, in order of preference I would say:

* PhotoJPEG in a MOV wrapper (or MXF but I’ve not seen that combination in the wild)

* ProRes 4:2:2 (regular or HQ );

* Uncompressed or none

* With any low bitrate MPEG-2 coming in close to the end.

Of course archiving the source media is also viable given that most source media is standards-based and will have decoding tools long into the future.

John Mozzer sent a comment in May of this year that is relevant to this discussion.

Here are some of my thoughts about choosing a video archiving format.

It seems to me, the answer should always depend on the original material, and the person asking this question in your April 2011 issue really doesn’t say. He only reveals using a Quicktime wrapper by mentioning the .mov extension.

However, the person starting the discussion published in your August 2009 issue mentions several legacy analog standard definition videotape formats (“1″ to Umatic to BetaSP”) and having transferred them to DigiBeta PAL. He explains the plan is to archive the media on a server. Why, then, have we not mentioned archiving as uncompressed SD?

Recently, an old friend entrusted me with the 3/4 inch U-matic videotape master of an 18 minute art video, which he co-produced during the 1970’s, because of my personal interest in its preservation. I took the videotape to a particular post production service (in order to take advantage of their Archangel digital restoration process, but that’s another story). When I explained that I ultimately wanted a “digital master file” as Apple ProRes 422, they suggested an uncompressed SD file in lieu of a ProRes file. I agreed, and am pleased with the resulting uncompressed 8-bit, 422, 720 x 486 video (in a Quicktime wrapper).

Upon researching uncompressed SD video, I found articles by Chris Pirazzi to be the most helpful (despite them being oriented towards programmers, which I am not.) Apparently, Pirazzi wrote the Quicktime uncompressed standard for Apple:

http://lurkertech.com/lg/video-systems/

http://developer.apple.com/quicktime/icefloe/dispatch019 (Written by Chris Pirazzi, Tim Cherna, and Peter Hoddie)

Are the data rates and file sizes a problem for people with a large library? If so, maybe technology is about to change that.

Larry adds: We have spent a lot of time talking about archiving hardware (and settling on LTO-5 as the hardware solution of choice). So, this is a good time to have a discussion about codecs, because it answers the question: “what video format are we archiving?”


Bookmark the permalink.

7 Responses to Pick The Right Video Archive Format

  1. […] will read Pick the right format | Refreshing hard disk storage | Quick note on […]

  2. mark says:

    Do you know how or if it’s possible to import an “mj2 file” (Motion jpeg 2000)directly into FCPX? without having to convert it beforehand?

  3. Jayson Moo-Young says:

    Started archiving with DNXHD LB for most of the stuff I have that were shot on Sony cams at 24 Mbps AVCHD, sometimes I wonder if I should just go XDCAM 50 since it’s also a standard that most stations accept. Any advice on this?

    • Larry says:

      Jayson:

      I would probably archive the format that I shot, rather than transcoding it. AVCHD is a very popular format that will probably be playable long into the future.

      However, when picking a transcoding codec, use one with a high-bit-rate to minimize compression artifacts. XDCAM 50 is also good, but given the choice, I’d stay with 24 mbps AVCHD.

      Larry

  4. Emily says:

    Hi Larry,
    In looking for advice on the best format to “archive” video footage in, I found this post. I’m curious if you have any insight or updates to add, given that it’s now 7 years later?

    • Larry says:

      Emily:

      Smile…. “the best format” is SUCH a tricky question. The answer depends upon what you value…

      * If you are looking to keep files small to maximize storage, using MPEG/4 H.264
      * If you are looking to maximize image quality (resulting in larger files), ProRes 422 for camera footage, ProRes 4444 for computer-generated footage, AVC-Ultra are all good options
      * If you are looking to archive edits, be sure to export and archive an XML file of your project
      * If you are on Windows, probably AVC-Ultra.

      The world of codecs is about to dramatically change and I’m trying to figure out what to advise. I strongly recommend you read this article to learn the latest news on this issue:

      https://larryjordan.com/blog/important-dont-lose-access-to-older-media/

      Larry

Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.