Pick The Right Video Archive Format

[This article was first published in the July, 2011, issue of
Larry’s Final Cut Pro Newsletter. Click here to subscribe.]

 

Cheryl Foster sent me an email recently asking about archiving. She asked:

I would like to keep a .mov of the finished video (or sequence) for archiving. Obviously “none” compression is huge, H.264 is what TV stations find acceptable for TV spots, but I really would like a better quality. [What should I use?]

Larry replies: In the past, I’ve recommended PhotoJPEG for progressive video, and its cousin, MotionJPEG for interlaced video. But, I decided to do some research and share what I learned.

My son, Paul, is the digital archivist at the IMF (International Monetary Fund) in Washington, DC. As such, he’s responsible for archiving tens of millions of emails, digital documents, paper documents, and all the huge variety of formats they represent.

So, we had a conversation and here’s what he told me.

When archivists are trying to decide what video format to use for archiving, they are looking for something that meets three criteria:

• The video format must be lossless
• The video format must be relatively well-adopted, that is, in wide use.
• The video format must have a wide variety of migration tools for file conversion.

While it is not necessary that the video format be an open standard, one of the reasons ProRes would not be considered is that it is developed and supported by only one company. This leaves it open to sudden termination such as the recent decision by Apple to sideline FCP 7 without appropriate conversion tools.

He asks, “what’s to prevent Apple from doing the same thing to ProRes?”

The format that was supposed to be the “next big thing” was MotionJPEG2000. He says this is a very nice codec that does a good job. However, currently, there are very few tools that encode it and even fewer that read it. This lack of migration support makes adoption difficult.

Instead, he says the current recommended archiving format is MPEG-2 in an MXF wrapper. A video conversion tool that my son recommends is FFmpeg — http://www.ffmpeg.org — its a lot geeky, but worth checking into.

However, since we are talking codecs and compression, I contacted Philip Hodgetts to learn what he recommends.

Philip replied:

I would concur with Larry’s suggestion of PhotoJPEG at 99% quality for archiving. MPEG-2 is way too lossy in my opinion, unless you’re at data rates of over 50 Mbit/sec, which is not possible with commonly available tools. Those tools also encode MPEG-2 [with chroma-subsampling of] 4:2:0, compromising color integrity in the future.

Right now there are more tools for ProRes in all its flavors than there are for high bitrate MPEG-2 at 4:2:2 profiles (High Profile (HP), 4:2:2 profile or MVP profile) and standard bitrate MPEG-2 is not an archive format.

For archiving, in order of preference I would say:

* PhotoJPEG in a MOV wrapper (or MXF but I’ve not seen that combination in the wild)

* ProRes 4:2:2 (regular or HQ );

* Uncompressed or none

* With any low bitrate MPEG-2 coming in close to the end.

Of course archiving the source media is also viable given that most source media is standards-based and will have decoding tools long into the future.

John Mozzer sent a comment in May of this year that is relevant to this discussion.

Here are some of my thoughts about choosing a video archiving format.

It seems to me, the answer should always depend on the original material, and the person asking this question in your April 2011 issue really doesn’t say. He only reveals using a Quicktime wrapper by mentioning the .mov extension.

However, the person starting the discussion published in your August 2009 issue mentions several legacy analog standard definition videotape formats (“1″ to Umatic to BetaSP”) and having transferred them to DigiBeta PAL. He explains the plan is to archive the media on a server. Why, then, have we not mentioned archiving as uncompressed SD?

Recently, an old friend entrusted me with the 3/4 inch U-matic videotape master of an 18 minute art video, which he co-produced during the 1970’s, because of my personal interest in its preservation. I took the videotape to a particular post production service (in order to take advantage of their Archangel digital restoration process, but that’s another story). When I explained that I ultimately wanted a “digital master file” as Apple ProRes 422, they suggested an uncompressed SD file in lieu of a ProRes file. I agreed, and am pleased with the resulting uncompressed 8-bit, 422, 720 x 486 video (in a Quicktime wrapper).

Upon researching uncompressed SD video, I found articles by Chris Pirazzi to be the most helpful (despite them being oriented towards programmers, which I am not.) Apparently, Pirazzi wrote the Quicktime uncompressed standard for Apple:

http://lurkertech.com/lg/video-systems/

http://developer.apple.com/quicktime/icefloe/dispatch019 (Written by Chris Pirazzi, Tim Cherna, and Peter Hoddie)

Are the data rates and file sizes a problem for people with a large library? If so, maybe technology is about to change that.

Larry adds: We have spent a lot of time talking about archiving hardware (and settling on LTO-5 as the hardware solution of choice). So, this is a good time to have a discussion about codecs, because it answers the question: “what video format are we archiving?”

---

Bookmark the permalink.