[ This article was first published in the July, 2006, issue of
Larry’s Final Cut Pro Newsletter. Click here to subscribe.
Updated 8/19/2006 and 5/8/07; April 2008; June 2008 ]
Philip Fass started me thinking about this again when he wrote:
Here’s a suggestion for the next newsletter: would you comment on the current viability of data tape as a backup/archiving medium? On message boards, some people say it’s outdated and recommend hard drives until Blu-Ray becomes affordable. Others think it’s an affordable, proven technology that’s good for people shooting tapeless. What do you think?
I’m trying to decide now, because I’ve built into a contract the option of recutting a video at a later date. So the material has to be kept safe.
Larry replies: Philip, I am so NOT a fan of archiving on hard disk I can’t see straight. The camera industry is not doing any of us a service by it’s headlong rush to tapeless acquisition without first thinking through how we are going to archive all this newly-shot media.
Last April at NAB, I felt like I was watching a number of camera manufacturers jump off the cliff of tapeless acquisition — hoping that someone would invent an archive parachute before they smashed on the rocks below. As of now, I haven’t seen anything I like.
The whole point of archiving is to be able to play something back at an undetermined point in the future. Not just saving it a for a week, or a month, or even a year, but five, ten, twenty years into the future.
At this point we have four options for archiving:
Let’s briefly look at each of these.
Where possible, this is your best option. Ideally, you should archive your camera master tapes.
Here’s an article that talks about the life-span of video tape: How Long Does Video Tape Last?
My first recommendation is to always use your camera master tapes for archiving.
DLT systems have been around for a long-time. Probably the best system is from Quantum (www.quantum.com) but it isn’t cheap — about $8,000. Still, the IT departments at major corporations have been using DLT for a long time. There’s an established market, standards, equipment, and media. All good things that you want in an archiving solution.
Quantum has an interesting Mac-based solution — called the SDLT 600A — that you can read about here: http://www.quantum.com/Products/TapeDrives/DLT/SDLT600/Index.aspx
Here’s my problem with archiving on hard disks — it’s your garage. Follow my reasoning.
Right! NONE of it. That’s because the technology industry builds equipment to be out-dated in about four years. They change a cable. They change a connector. They change a protocol. The device is too slow. Too small. Whatever. It doesn’t work.
Where would we be today if Gone with the Wind, or Casablanca, were stored on hard disks? Hard disks are essential for editing. They are great for short-term storage. They are miserable for archiving.
Just to make this discussion interesting, we don’t have a single optical disk format, we have five flavors to consider:
Again, let’s look at each of these.
These are fine for short pieces, provided you buy media from a brand-name vendor, burn it at a reasonable speed, say 16x, and store it like a fine photograph. Here’s an article that talks about life-span of optical media: The Life Span of DVDs
Blank media vendors that I currently like include: Apple, Verbatim, and MAM-A. Always buy premium quality mediaif you plan to archive for more than a year. Almost any brand is fine for storage less than a year.
The problem is that CDs only hold about 700 MB of data. This works out to about 3 minutes of DV video, or 26 seconds of DigiBeta.
You can’t play from the disc, it’s too slow, but you can save your final output to the disc — if it fits.
These are fine for archiving final output, again, provided you buy from a brand-name vendor, burn at a reasonable speed, say 2x, and store it like a fine photograph. (My personal belief is that burning slower is good. The KEY is to match your burn speed to the speed of the media. Don’t burn 8x media at 2x speed, or 2x media at 8x speed.)
Blank media vendors that I currently like include: Apple, Verbatim, and MAM-A. Always buy premium quality if you plan to archive for more than a year. Almost any brand is fine for storage less than a year.
Again, the problem is that these DVDs only hold about 20 minutes of HDV or DV video in it’s uncompressed, high-quality, most-editable state. If you are working with DigiBeta, your total storage is two-and-half-minutes. Again, fine for storing final output, but not good for storing dailies or original files.
Rimage told me at NAB they would have a Macintosh-based backup solution out this summer. I was just on their website and couldn’t find it. However, you can learn more here: http://www.qumu.com/products/rimage-disc-publishing.
The problem with this latest version of DVD burners is that machine-to-machine compatibility is still elusive. You can play the disc back on the system that created it, but getting it to play back on another system is very hit-or-miss.
Hit-or-miss are not words I like to hear when I am archiving something.
These burners will be short-term solutions as the mess between HD-DVD and Blu-Ray plays itself out.
Folks, I’m sorry. If someone sat down to plan a screwed-up product launch, they could not have done a better job than what we are witnessing between HD-DVD and Blu-Ray.
We have two incompatible formats, requiring two incompatible players, using two incompatible replication systems, all with delayed product ship dates. Half of Hollywood supports one format, half the other. Half the computer manufacturers support one, half the other and half are holding out for “something better.”
This is not an archive system, this is a soap-opera.
In two years, maybe, things will stabilize to the point where serious discussions can occur regarding what works and what doesn’t.
In the meantime, NEVER trust precious masters to a generation one product. Experiment — yes. Test — yes. Argue, debate, demand better — all yes. Archive — not on your life.
First, if possible, use your video-tape camera masters as your archive.
Second, if that isn’t possible, and you have the money, DLT is a proven technology. It isn’t sexy. It isn’t fast. It isn’t cheap. But it works, it has worked and it will continue to work – because the broader computer industry will continue to drive it. Which means we can piggy back on it for a while.
Third, if DLT is too expensive, and you don’t have camera masters, hard disks are your only option. However, treat them like gold and assume that drives die. Don’t put all your eggs in one basket.
But don’t think of it as archiving — think of it as treading water.
Philip Fass writes:
Larry, that was a GREAT article on archiving. Thanks!
Just an FYI: Sony makes one Firewire AIT-2 drive for Mac that’s only about $1200. The tapes are 80GB native. The few comments I’ve read on it seem pretty positive.
Here’s a description:
Larry replies: Philip, thanks for the heads-up. I haven’t heard anything about this deck, so its nice to learn of a less expensive solution.
– – –
Eric Siegmann comments on last month’s article on archiving:
Really though, I would hesitate to archive to DVD at all, at least for long term, and opt instead for AIT, DLT or other professional tape archive for data, and professional tape media digital clones (or safeties) for video. DVD media is easy to damage, there are many different formats so compatibility may still be an issue (go figure!), usually people burn and (try to) read them on the cheapest of no-name mechanisms, if you stick a label on it the disc can self-destruct over time, and most importantly their capacity is too small for most projects. Heck, you can get USB flash drives with 8GB.
BTW, I agree with your assessment of dual layer and Blu-Ray/HD-DVD: don’t even think about it! I think these formats are dead on arrival, but that’s another topic.
Larry replies: Thanks for your comments. The archiving situation, currently, is a complete mess. I’m looking forward to the situation improving in the future.
One of the key issues facing all of us is how to archive our projects and resulting media. Here are some thoughts from several readers on this subject.
Dave Blackham writes:
I was until recently the Head of Operations for Granada [Television in] Bristol, [England] who has operated FCP since V1.25. I built a 60TB SAN in Bristol then a 142TB SAN supporting both SD and HD for ITV in London. All promos and other inter-show material is now produced on Final Cut Studio in ITV.
My purpose in writing is at Granada we built a completely tapeless infrastructure post ingest and I saw your item regarding archiving. A solution that worked for us was using a Sony SIT1 drive locally attached to a G-5 which was in turn attached to the SAN. (The SAIT seems not to have drivers allowing a FireWire switch connection to become part of the SAN network, hence the G-5.)
We used Retrospect to manage the back-ups which worked very well as it also had the ability to run other Disk-to-Disk backups and other scripts overnight when needed. Also Retrospect can back up, say, a single line of commentary in the middle of a back up rather than rewrite the entire back up data.
The SAIT1 at 500Gb also has the ability to back up one HD 50 minute show, requiring about 400GB including handles. I always reckon 108 minutes of media per 50 minute show based on 5 second handles and about 750 edits per hour as a rough rule of thumb. The SSIAD tapes are not cheap but the drive is about £4500 which is a lot less than an SR deck and also preserves the show in its component layered form for possible re-versioning later.
With Apple purchasing Proximity I’m sure more elegant back up systems will be forthcoming but as Retrospect is application-agnostic this solution worked for us with large SAN infrastructures.
Larry replies: Dave, sorry to have missing you on my UK tour. Thanks for sharing your solutions.
Gerret Warner writes:
We purchased the HVX200 last fall and have now shot hours and hours of footage–all in DVCPro-50–and have been very happy with footage. But archiving is a challenge.
Our answer has been to do the following:
- Two Lacie Rugged 120 GB drives on the road because they are bus driven and can be used with my G4 powerbook anywhere. Our (4) 8 GB P2 cards get dumped to them through the day.
- Two OWC 500 GB FW 800/400 drives to archive all P2 8GB cards in duplicate (the drives are cheap, light enough to take on the road, and have worked fine). Each night in the motel, I copy from the LaCie’s to the larger OWC drives in duplicate. Then, once I’m home, footage is imported into FCP.
- After all that, you have 3 copies on drives: MXF files on each drive and Quicktime on the capture scratch drive.
It’s time consuming and sometimes anxious work trying to be sure you don’t blow away a card, but so far I’ve done that only once… in a too-long shooting day. They key has been to carefully catalog all footage as you go so you know you’ve got it on the drives.
The real challenge is long-term archiving. I’d love to hear recommendations on DLT, LTO or whatever else has been successful and dependable in the industry. There must be a lot of experience in the world of archiving critical data. I’m not sure I can wait for Blu-Ray data disks.
Larry replies: Gerret, long-term archiving is, for me, THE issue that needs to be solved in order for tapeless acquisition to really take off. While I am leaning toward an Exabyte tape-based technology, I haven’t seen anything that provides the key criteria necessary for archiving:
Archiving is the key technology, well, aside from whatever Apple is doing, that I’ll be looking for this year at NAB.
And I agree with you that Blu-Ray, or any other brand-new optical disc technology, is not to be trusted for long-term archiving until it has been aggressively tested in the market for a couple of years.
Nicholas Karfoot adds one more opinion:
In one of your training videos, I forget which one now, You where talking about the P2 system and the cons of backing up this sort of data. I just thought if you haven’t already seen it that Grass valley have teemed up with Iomega to produce a system called RevPro a 35GB storage system. Check it out here.
Larry replies: Nicholas, I had a chance to see this recently at the San Francisco Final Cut Pro User Group meeting. While it is small, portable, inexpensive, for me it doesn’t meet the needs of video editors.
First, it doesn’t hold enough. Second, it’s too new to know what the life-span of the media is. Third, it’s from the same folks that brought us SyQuest drives, Zip disks, and Jaz drives — all were EXCELLENT in providing removable storage; but I can’t access any of them today.
Don’t confuse removable media with long-term storage. Just because we can pop the disc out of our system today doesn’t mean we can put it back in five years from now.
Tony Liuzzi adds:
I was reading your newsletter about backing up. I use Mezzo for backing up FCP with an AIT drive.
Here’s the link to their website:
http://www.mezzotechnologies.com/M4.5/pages/product_info_pages/fcp_product_info/fcp_info1.html
Larry replies: Thanks to everyone for their ideas and suggestions.
UDPATE – April 2008
Steve MacDonald sent in this technique that he uses for archiving:
Lately, I’ve found an archiving which seems to work just fine for both standard
DV media as well as HDV media. Basically, what I do is create a burn folder in finder, drag all the media for a specific project from the capture scratch drive to this burn folder. (By the way, I do use your 2 drive method for all my FCP projects.)
Next, drag the project folder for this project from your HD drive to this burn folder. As you well know these are Alias files at this point. I rename these burn folders for clarity, one called ABC project folder, the other ABC media folder. Now, if you double click this folder you’ll see a small burn icon in the upper right, double click that and it will give you a total file size. My test for this method was a short 2 minute HDV project with a burn folder size of 1.09GB. In finder > file > burn disc. Once that burn is complete you can trash all the media files for that project from the scratch drive, then go in FCP and delete the project itself.
Re-mount the burn disc if you’ve ejected it. On your 2nd drive, > final cut pro documents > capture scratch, create a new folder and title it. Now open the burn disc and drag all the media files to this new capture scratch folder. Launch the FCP project from the ABC project folder on the burn disc. You’ll see all that media off-line, now you just re-link it to the new capture scratch folder and voila.
The beauty of this is you don’t have to re-batch capture your footage from tape and this media is on DVD for archiving. Of course, larger project would require multiple disc burns for your media. Sorry for the book length e-mail.
Larry replies: Your system works great as long as your media files are small. However, I STRONGLY recommend AGAINST using dual-layer DVD disks, as they are notoriously unreliable for playback on different machines. This means that you are limited to single-layer DVDs which only hold 4.3 GB of material when burned.
This works out to about 20 minutes of HDV material. So, if you had 50 hours of source footage, this would require 150 DVDs and more than two weeks to burn.
Which is not particularly practical.
UDPATE – June 2008
After reading my recent thoughts on archiving, Andrew Wilson sent this in:
The main reason I’m writing is that in the middle of all this weighing back and forth, a client of mine asked me if I could author a DVD of a video he produced less than 10 years ago. It was mastered on D2. After about a half hour on the phone, I found a place that could make a D2 dub for me. So my point is that it’s not just the computer industry that has these problems, but I think that it’s driven by the computer industry – that planned obsolesce, and need to upgrade to something better and faster and not compatible with the previous generation. It seems like the older something is, the more reliably it will be around for a long time – starting with film (as you pointed out on your show) to Betacam to finally HDV which I think is going to die out as a high-def tape-based format.
But it is true that the computer industry is much quicker to make things better and faster, just think, Betacam was around for quite awhile when Syquest drives first came out and AVID 9 gig drives were $2000.
Larry replies: Andrew, I agree. It’s nice to have the latest and greatest toy, but it is incredibly frustrating when you can’t access something you created as recently as ten years ago.
I’m hoping that if we keep complaining loud enough, a manufacturer somewhere will figure there’s a market here and find a way to meet it.
3 Responses to Commentary: In Search of a Backup Strategy
What tapes would you recommend to store DSLR footage on?
Ben:
At this point, our best archive options are LTO-5 or LTO-6. Look at companies like 1Beyond, Cache, and TOLIS Group.
Larry
[…] In Search of A Backup Strategy […]