Clayton Moore sent me an email last week that triggered this essay:
“Larry, how long should a video production company keep copies of their edited footage? I’m in the process of sorting through this.”
This is a very easy question to ask, but very hard to answer.
First, let me say that there is no one, single, perfect answer. And I’m very interested in your thoughts – so please share them in the comments below.
That being said, I don’t think you should store materials for any client unless the client is paying you an ongoing fee for storage. I make it a habit to return all materials to the client after a project is complete, including clear instructions that they are responsible for archiving their materials.
I do this partly because I’ve yet to meet a client that is willing to pay a storage fee in perpetuity for materials that they may not use again. And, partly, because I don’t want the liability of replacing their materials in case of fire, theft or other damage. But the main reason I don’t store client materials is that I’m still struggling, like many others, to figure out the best way to store materials for the long term.
Let me start by discussing what I don’t recommend.
WHAT I DON’T RECOMMEND
I don’t recommend unplugging hard drives and storing them on the shelf. Or in protective cases. Or, well, anywhere.
Hard drives are designed to be powered up and spinning. I’ve had many, many reports from readers who stored hard drives, unplugged, on the shelf, only to lose data from them after a few years.
Hard drives are not archival systems.
For the same reason, I don’t recommend storing flash media from your camera on the shelf, either. First, because the actual life span of flash media is fairly short and, second, it does not respond well to excessive changes in temperature, humidity or dust.
Yes, I keep camera flash media where possible; though more often than not I reuse it. However, I never rely on that flash media for archiving.
NOTE: Recently, I attended the Creative Storage 2016 Conference and spoke to the folks at Micron about the longevity of Flash/SSD. You can read what I learned here.
THE BIGGER ISSUE
The bigger issue, of course, is that there are a limited number of archiving solutions for media professionals. Not that archiving media is any different from storing Excel files – except that our media files are so gosh darn BIG!
Many of my one-hour Webinar master files regularly exceed 100 GB in size. I record and master my webinars using ProRes 422 HQ or, if its a screen capture session, ProRes 4444. Both of these formats yield great images – and massive file sizes.
I’ve written in the past about LTO tape. For those that have an equipment budget, you can get a current-issue LTO drive and software for about $5,000; less than $4,000 if you get an earlier-generation system.
You can learn more about LTO here: www.lto.org
And here are two recent articles I’ve written about LTO archiving software:
But there are three other archiving options you might want to consider that may be more affordable:
MDISC
MDISC records data to devices that look a lot like CDs, DVDs, or Blu-ray Discs. The difference is that rather than using an organic substrate to burn data into, they use a stone-like carbon compound that, they say, lasts for 1,000 years. (Even if they are off by 50%, that’s still a long, long time.
Most modern burners support MDISC and the media, while more expensive that traditional burnable DVDs, is still quite affordable. For example, a DVD burner and pack of fifteen 25 GB discs is about $149; far less than an LTO system. Both discs and burners are sold on the MDISC website.
This is affordable enough that I’m looking to buy one in the next few weeks and test it. My biggest concern, though, is that the largest disc only holds 100 GB of data. While this is plenty big for most office and Photoshop work, many of my master video files exceed 100 GB in size. So, MDISC is best able to provide affordable storage for everything except media files.
My other concern is that DVD drives are slowly fading away. So, while the media may last forever, the drives may not be viable for much longer. Still, mDisc appears to be a better alternative than storing a hard drive on the shelf; provided you can live with the limited storage capacity.
Website: www.mdisc.com
SONY OPTICAL DISK ARCHIVE (ODA)
One of the benefits of shooting XDCAM is that media can be recorded on a Blu-ray Disc, rather than flash media, which means that the format automatically offers an archiving solution.
Sony took this one step further and bundled twelve Blu-ray discs into a package and called it ODA (Optical Disc Archive). Sony’s website states: “It’s the desktop solution that is ideal for long-term file protection and management, and an easy choice because each 12-disc cartridge functions as a single unified volume.” Sony rates its write-once/read-many (WORM) media at a 50 year life.
While the media is – mostly – affordable at $150 for 1.5 TB of storage, Sony priced the hardware far beyond the pocketbooks of its target audience, which they describe as: “A starter system ideal for the camera owner / operator or a small studio…” at $7,500.
I have a very, very, VERY hard time believing that the electronics inside this unit cost even 10% of the retail sales price. Sony needs to reevaluate its pricing, especially because LTO systems start at about half this price, while an LTO-7 tape costs about the same as an ODA cartridge yet stores 6 TB. And the system is not fast.
Website: pro.sony.com/bbsc/ssr/cat-datastorage/cat-opticaldiscarchive/product-ODSD77U/
THE CLOUD
I’ve discovered that the value of The Cloud as an archiving solution depends upon two things:
My friends in Australia brag about their 50 Mbps upload speeds; even here in the US, there are areas that are able to achieve these speeds or better. However, for me, in the technological backwaters of Los Angeles, I can only get a 1.5 Mbps uplink. That works out to about 200 KB / second! Stupidly slow.
Still, even a 50 Mbps uplink will take about 36 hours to upload 1 TB of data. How many of your projects only take 1 TB?
But the bigger issue, for me, is the likelihood that data I store in The Cloud is inherently insecure and subject to business decisions from web hosting services that I have no control over. Every day we read about massive hacks, data intrusions and lost files from companies that can afford the highest level security.
This is not a place I want to archive my irreplaceable media.
Yes, I use the web every day – but for distribution, not archiving. Anything I post to the web is backed up locally.
MY CURRENT METHOD
So, like many of you, I continue to wrestle with the issue of where do I store the stuff I need to keep; whether it is my own media or the clients.
Right now, I’m adding more RAIDs and storing all my files on multiple RAIDs, which I’ve duplicated and stored in two separate places. It isn’t the cheapest solution, nor do I think its the best. But its what I’ve got.
For a while, I was using an LTO-6 drive. But, when I was acquired, I lost access to the drive. So I have a collection of tapes, but no way to play them back.
There are three things I don’t have any control over:
As we wait for the elephants to stop dancing and resolve these archiving issues, here are the rules I follow:
I hope.
One Response to How Long Should We Keep This Stuff???
Back in 2011-2012 you posted a wonderful tutorial using the AppleScript DiskRefresh that will refresh a disk with two clicks. https://larryjordan.com/articles/a-free-utility-to-refresh-hard-drives/ I’ve been using that methodology ever since then, refreshing all my spinning HDs 3 times a year, in addition to cloning to a second HD. My backup needs are small compared to yours, but this low-cost make-do has (miraculously) worked for me.