I’ve written a lot about archiving our media and projects for the long-term. Last week, I asked readers to share their experiences with archiving by answering four questions:
Here are their responses.
Allynn Wilkinson
I thought I’d weigh in on your questions about long-term, archival storage. Unfortunately, my answers are not very encouraging; we just lost over 5 years of tapeless media files. To keep upbeat, I’ve named my current archival method: “Laziness and Dumb Luck.”
First my answers to your questions:
* How are you storing your media and projects now?
Multiple, stupid little cheap external hard drives and ccasionally SD cards.
* What have you used in the past?
A dedicated server of 20 terabytes backed-up nightly by our IT department.
* What works?
Nothing! But multiple, stupid little cheap external hard drives are at least under our control.
* What doesn’t work?
Dedicated servers managed by incompetent idiots.
* What advice would you give others?
Don’t trust your media to incompetent idiots (but how can you tell?)
Here’s my story…
As you might recall, I had the great misfortune to use the Final Cut Server (maintained by our IT department) for a couple of years. It was never great and often frustrating. Oh how I miss it! After Apple abandoned it we limped along for a year for or so but the inevitable crash finally came. Fortunately, the entire Library was backed-up and in a couple of weeks it had been rebuilt on to a dedicated server called “Lucy”. Lucy had 20 terabytes of space and was backed up nightly to a nameless back-end server.
For three years or more we chugged along putting all of our raw footage, final masters and occasional project files on Lucy trusting that it was being backed up every night. We had one drive failure two years ago that resulted in the loss of half a day of shooting because the drive failed before the back-up was complete (but after the SD cards had been cleaned). This made us a little less fanatical about cleaning SD cards until at least 24 hours after a shoot. Here the “laziness” factor enters into the equation.
Fast forward to May, just before Commencement. It’s the end of the academic year, all of the computers are full to overflowing with raw files and we need a few hundred gig to cover the weekend’s events. And… Lucy experiences a catastrophic failure. She’s down and we have to make due with stupid little cheap external hard drives. Life goes on but we don’t yet realize the scope of the problem.
IT gets Lucy back up and tells us that the recovery process has resulted in some files appearing in a “Lost and Found” folder. They suggest we designate one person to go through the files to identify them. I venture a look. There are thousands of orphaned files including several hundred named “000000.MTS”. Well that’s not going to help! Someone else does a spot check of files remaining in their original folders. Every single file looked at is corrupt. But it gets worse.
After a week or so, someone in IT decides to have a look at the nameless back-up server. You know, the one that’s been backing up Lucy every night. Well, let me just say that it did its job. It continued to back up Lucy every night and over wrote every single “good” back up with a corrupt file! So there you have it. Lucy is corrupt and, thanks to a bullet proof, dumb-as-a-rock, back-up policy, our back-up is corrupt. But it gets worse (and a little better)!
Someone not directly responsible for Lucy’s back-up finds a snap shot that was taken by a different backup server from early 2014. It’s hard to tell, but maybe we have some of our older footage. That’s a ray of hope. Still, we’ve definitely lost everything from early 2014 onwards (unless it is still on a local drive somewhere).
We have a meeting to assess the damage. One thing for sure that is lost is the footage from Commencement 2014. The Commencement address that year was given by an alum you may have heard of: Lin-Manuel Miranda. Just before “Hamilton” blew up into the biggest thing anyone had ever seen, he came to campus to rap the Commencement address! And we had it on film (a 3-camera shoot), we had it backed up, and we lost it!
The room falls into a very uneasy silence. We heartily admit that the vast majority of the stuff on Lucy was not needed. Indeed, 90% of it or more would probably never have to be accessed again, ever. But Lin-Manuel?! That was another story. A colleague breaks the silence and quietly asks me if I’ve cleaned the lab “recently”. No, I’ve been too busy (and lazy). She gets up and retrieves a battered envelope from the top of one of the computers where it has sat, abandoned for over over 12 months. It is the SD cards from the 2014 Commencement!
And that is why my current archival process can best be described as “Laziness and Dumb Luck”!
David Murray
I work on historical stuff so long term storage is very important to us.
* How are you storing your media and projects now?
I am using FCX. Once a project is finished (which include a master file), I copy the final project to a new library on a different hard drive. I consolidate it so I have all necessary files within this new library. This gives me a back-up of the project.
I also make a copy of ProRes 422 master file and put that onto yet another hard drive. This is my master back-up.
For important projects I also print them to HDV tape using our old-fashioned FC6.
* What have you used in the past?
Before I used to print everything to tape (first Betacam and then DVCAM and then HDV).
* What works?
I still shoot most of my stuff on HDV. It isn’t the greatest, but at least I have the original tapes as well as the original project and master file, a back-up project and a back-up master. If a disk dies, I can replace it without losing stuff. It means three hard drives but it’s the best I can come up with.
* What doesn’t work?
Trusting everything to one hard disk!
* What advice would you give others?
Back up, back up, back up. And never trust a disk.
Jim Essick
I am a one man band that needs to access footage for my clients over time (years) and my evolving storage is a constant challenge. I read up on this and most articles address parts of this challenge but few layout a strategy or plan that is easy to follow. I make reference to a flow chart or illustration to describe the workflow because I am a visual person and this way I can fill in the blanks on what hardware and practices I need to incorporate to insure my future media plans.
Currently I have many old single drives (the way I use to do it, one fills then buy another), then I bought dual drives with a mirror copy. The problem with dual drives was speed on multicam jobs and 4K footage. Next I bought a Promise 12 TB 5 drive raid, I liked it but it is now full, so I purchased a G-Raid XL Studio 24 TB 8 drive raid that is filling fast. What is next?
This topic is constantly operating in the background of my mind as how I can access and store my media long term. Here are the questions I ask myself:
Another question if I am doing this right, most of my footage is ProRes and I edit in ProRes, I create a file with original footage then create a project. Should I leave the original footage in its location or copy to the library? I believe I know the answer but how will that effect my archive strategy. I need to be careful of how and where my original footage and the library are located. The challenge is making sure your backups (drives/tape) are of size to contain both footage and libraries. So you have to manage footage and library sizes to match your backup. Flowchart would again be helpful.
I love digital media but what a challenge to store, I must say that stack of super 8mm film and my old DV tapes I have accumulated over my lifetime still play as long as my projector or player are operational.
Steve Sweitzer
* How are you storing your media and projects now?
Have everything on hard drives (JBOD) and then back up to the cloud with Crash Plan.
* What doesn’t work?
Even though I follow your advice and rebuild drive directories (Disk Warrior) annually, I occasionally have a drive fail. Then Crash Plan comes to the rescue. It takes forever but I’m able to recover my data.
Crash Plan is slow, it can take months to back up a 3 TB drive.
Paul Briton
I use several WD hard drives. Portable for working with files, and desktop ones for backup. I use a MacBook Pro SSD. I keep all media within the respective drives as HD’s are so cheap now.
As I have only been using FCPX for about three years I am keen to see what others recommend for long term storage.
Stu Aull
My “solution:”
Dave Bogie
* How are you storing your media and projects now?
On MacPro cylinders, we use Thunderbolt-to-optical boxes from Sonnet to push data through a fiber optic link to our data center located across town. The servers are configured using NFS. The bandwidth is great and the servers are managed and backed up by our IT department.
When we are finished with our FCP X library for any particular project, we carefully consolidate it and move it to off-line archive storage. Archives go to lesser, slower drives but the Libraries remain readily available. We usually copy Library back to the NFS from archive if we need to resurrect an older project.
* What have you used in the past?
Desktop boxes or internal spinning drives or SSDs.
* What works?
The NFS works. The operation required startup scripts in Terminal that locate the NFS servers and create a persistent IP address for the length of time our Macintoshes are up and running. Restarting the Macs runs the scripts. Umm, I may have stated that incorrectly; I really do not understand why or how the thing works.
*What doesn’t work?
We are photographers and video producers. So we have no confidence in our abilities to manage these storage resources ourselves. Our security requirements (Homeland Security, military-grade cyber security compliance) prevented us from acquiring a desktop (or under-desk) FCP-dedicated, NAS or SAN editing appliance. Getting our IT people to cooperate with us was not difficult once they understood our bandwidth, interface response time, and storage requirements.
* What advice would you give others?
Get your IT people to help. There will be somebody on their staff who just loves solving enterprise issues for Macintosh computers. However, you will want to bring with you definitive numbers that explain bandwidth requirements for HD video in terms IT people understand. At first, they didn’t believe us. Then you need to demonstrate what high definition video means in terms of storage capacity. At first, they didn’t believe us. Then you make the case that backing up your multi-terabytes is just as important as providing the servers and the connections. At first, they didn’t believe us.
Kevin Murphy
* How are you storing your media and projects now?
External hard drives (OWC and Lacie, but primarily on the OWC drives, at present)
* What have you used in the past?
Mac internal hard drive, with OWC and Lacie external drives.
* What works?
Using external drives right from the start of a video project. Due to memory size, I have separated active libraries from inactive libraries, each group residing on a separate hard drive.
* What doesn’t work?
Working on projects that are housed on my Mac’s internal drive. Memory issues pop up at the most inconvenient times.
* What advice would you give others?
Learn all that you can about active work, and storage possibilities, and use the best options you can afford.
Don Stafford
I have 2 long-term storage options, depending on content.
For feature films, I buy bare hard-drives, drop them into a dock, format for MacOS, then copy the raw footage, and the FCP X library to the hard drive. I then take the hard drive to my bank and put it into my safe deposit box. One project to one drive.
For short films, local projects, etc., I buy bare hard-drives, drop them into a dock, format for MacOS, then copy the raw footage, and the FCP X library to the hard drive. I then take the hard drive and it is kept in a fire proof file cabinet in my office. These drives may have multiple (unrelated) projects on them.
Jim Cunningham
I mostly do projects for large corporate clients. Projects can go on for a while with revisions and updates.
I primarily store on-going projects on several (client specific) RAID 5s. After about a year, I port to mLogic’s DLT w/ Pre-roll Post At the same time I also transfer a copy to a JBOD 4-5TB USB 3 drive, just in case the client (out of the blue — but it happens) needs to get to that project right away.
Craig Larson
This isn’t very scientific, but I hope it helps.
* How are you storing your media and projects now?
I archive my complete FCPX project libraries (media and all) on a general backup drive and a “client” project drive, so I have two separate backups in different locations. I add the archive date to the library name so I know they are both identical backups…
* What have you used in the past?
Have always used hard drives…
* What works?
Everything is together in one backup library folder. All media, clips, etc. I don’t have to hunt for anything. If I need to redo something, I just pull the library folder over to my working drive and go. I like have separate backups on two different drives.
* What doesn’t work?
Tedious process. Manual backups take time. Any subsequent changes to a project requires you to move the library over to your working drive, make the changes, and then repeat the entire archive process.
* What advice would you give others?
Find the best archive method that works for you. Also, it does no good to make multiple backup copies on different drives if you simply keep them all in the same drawer. That works great for drive failures, but not so much for fire or theft.
Jan Smith
* How are you storing your media and projects now?
I am using mDISCs, hard drives, and Blu-ray Discs, depending on the size and need to retain these over time.
* What have you used in the past?
I have used Iomega tapes (way back), floppy disks, CDs, and DVDs. I have been pleasantly surprised that even ordinary CDs and DVDs that are 10 or more years old are still readable, though for safekeeping I have copied them to mDISCs.
* What works?
Not being financially able to jump to a current tape system, I will continue to use mDISCs, hard drives (being careful to follow your advice on refreshing them periodically), and Blu-ray Discs (I make at least 2 copies). These work for now, and hopefully future systems (per Vint Cerf) will be able to read them. What worries me is that 4K and other hungry material are quickly making my present solutions inadequate.
* What doesn’t work?
Most of these will work, however clunky they might be. The reliability of the Iomega drives was awful, but that is ancient history. Small-capacity media will not be a solution.
* What advice would you give others?
Keep reading Larry’s news and other sources and keep up with technical developments. Affordable solutions always seem to happen sooner or later!
Richard Osso
* How are you storing your media and projects now?
I do lots of weddings, Dance Recitals, real estate videos and family projects. So I have many projects in front of me through each season. I have had a history of losing a couple of DRIVES and killing the files. Some, I never got back, others I backed up. I used SONY TAPE Video cameras for as long as I could to not have to deal with backups. I have moved into the all digital age….and my fears have been realized that I did lose one complete wedding, and a second file on another.
* What have you used in the past?
I have had a few different external hard drives over the years. But I have settled on LaCie.
* What works?
I don’t use drives bigger than 4 TB or smaller. Bigger they are, the more they FAIL with lots of files. Smaller drives means more drives eating up desk real estate.
* What doesn’t work?
Smaller drives….and RAID systems….cost of raids are too much…just get 2 drives and download the same files on both….cheaper.
* What advice would you give others?
I use 2 LaCie 4TB drives to just offload CARDS. So one drive backs up another. I did buy a Blu-Ray system to back up files, but that was too time consuming and a another workflow I realized I hated doing. So I just use 2 drives. When I need the files, I move the files to the EDIT DRIVE, edit the project and when it is done, delete all files. If need be, I make a MASTER DVD and keep that.
[Added in Update] Larry T. Brannin
In the past we would put only the final products (video) on a DVD data disc and we used CD Finder (My, wasn’t that a great program… love our German friends, Mr. and Mrs. Doerner, but I digress.) if we wanted that video to be sent somewhere years later, we could pull it up in CD Finder and using the assigned tag number, we knew right where it was on the shelf and we grabbed it. No problem Until files got bigger and bigger and, well, you know. So, of course, we moved to single-sided Blu-ray Disc. Then double-sided. But, the files kept getting …well, you know.
So, what do we do now? I have two local 8 TB Pegasus drives. I store my Archives and everything else important from FCP X on both (one is a backup). When a project is final, it is moved to another 12 TB Server which holds it for one year. After 12 months, if is not needed, (meaning…the guy from down the hall who walks in and says “You remember that such-and-such video you did last year, can I get you to re-edit that to a shorter version?” You surely have one of them, don’t you? Well, if he hasn’t come by now, then it, along with other files from the same time period, are all deleted after all files are backed up to tape. (By my IT guy who controls the MAC connected to the 12TB server) So, the originals are removed from both the two 8 TB drives and from the 12 TB drives and more space is made available. If needed after that time, I reinstall from tape onto the 12 TB (which is a network server) and from there back onto the two 8TB drives (which are local drives).
I keep track of what is on each tape by cataloging what I am about to archive (in Finder first) then, using Neofinder, (Remember those Germans? by now they’ve created Neofinder – never figured out what that meant) and Neofinder tells me what is on the tape and I go to my really big shelves and pull out that what is usually a 3-4 tape grouping to be used for backing stuff up. Have not lost anything in over 4 years now.
And my advice? Never erase anything until you have two backups. Period.
Larry adds: Thanks to everyone for sharing their techniques. Feel free to use the comments to add others.
As these comments illustrate, archiving is a mess. It is expensive, inconsistent, exceedingly labor-intensive and totally non-standardized. It does not surprise me that The Cloud does not factor into anyone’s archiving plans. Our files are too big and the upload data rates are far too slow.
My hope is that if we keep complaining, some day someone is going to listen and help us find an effective, affordable, long-term solution.
Like I said, this is my hope.
3 Responses to Readers Report: How I Archive [u]
I’m surprised there was not much mention of cloud storage options (except Crash Plan). Is transfer speed, file size or pricing the maim issues here?
John:
It is totally due to transfer speed. Even a truly fast uplink – say 50 mbps – only transfers data at about 6 MB/second. This is half the speed of the slowest thumb drive.
The Cloud is perfectly fine for small files, but totally inaccessible for the massive files that media creates.
Larry
For super-long-term, stick-it-there-forever storage we’ve been considering Amazon Glacier. It’s quite inexpensive ($0.01 per gig per month or less) but it is definitely “cold storage”! You can’t retrieve anything very quickly and, of course, it would take a long time to get it up there in the first place.