This week I presented a webinar on how to protect your stuff using archiving. Why? Because hard disks are great for immediate storage, but inadequate for files that you want to keep for a long time.
Hard disks will reliably store data with the power off and stored on a shelf up to about five years. DVDs last about seven years. Flash drives last about eight years. (Source: mDisc). In other words, the tools we use to store media for immediate use are not the tools we want to use to store media for the long term.
The purpose of my webinar was to show how to use LTO (Linear Tape – Open Technology) tape for long-term archiving. (And even with LTO, the phrase “long-term archiving,” is somewhat misleading, as you’ll read shortly.)
NOTE: You can download my webinar here and I encourage you to do so.
One of the questions that came up during the session was “What about mDisc? Why not use them for archiving?” This is a great question that speaks to a much larger issue that I wanted to talk about today.
WHAT IS MDISC?
mDisc replaces the traditional organic substrate in either a DVD or Blu-ray Disc with an inorganic substance. Because of this simple change, media stored on an mDisc, according to the mDisc website, can last up to 1,000 years. Even allowing for excessive marketing hype and discounting it by 75%, this means mDiscs will last hundreds of years.
NOTE: Visit their website here.
I’ve had the chance to talk with the mDisc folks on several occasions and think they have a great idea and are working hard to share it with the world. My problem with mDisc isn’t their technology, but the technology industry.
TECHNOLOGY ACTS LIKE THE PAST DOESN’T EXIST
Technology, as an industry, acts like the past doesn’t exist. As we have seen far too many times in the past, whenever a new idea can do a task better, even if it is incompatible with what has gone before, the new idea wins; even if it is as simple as changing connectors.
My office is littered with the dead-ends of storage technology:
Or, if you need something more dramatic, look at the industry bombshell that exploded four and a half years ago with the release of Final Cut Pro X without the ability to update files from Final Cut Pro 7. The reverberations from that shock are still playing out in our industry today.
NOTE: In response to this firestorm, Apple worked with independent developers to create upgrade tools which are now widely available through the Mac App Store.
Or, if you want a media example, look at film over the last 100 years as we’ve struggled with:
Not to mention the myriad methods we’ve used to create special effects.
As media professionals, our job is to create media projects today, then preserve them for the future. The job of technology is to invent the future, without regard to what’s happened in the past.
These two missions are in conflict more often than not.
PRESERVING THE PAST FOR THE FUTURE
Which brings me back to mDisc. mDisc has what seems to be solid technology built around the hardware of the past – DVD and Blu-ray Disc. And both these hardware boxes are becoming increasingly difficult to find.
Both Apple and Adobe have discontinued their DVD and Blu-ray authoring programs. (The ability to create DVD or Blu-ray material inside Final Cut, Compressor, or Premiere is extremely limited.) Protestations from Sony not withstanding, Blu-ray Discs have never achieved the market success of DVDs.
To the best of my knowledge, no currently shipping computer (as opposed to used or refurbished units) ships with a DVD drive. If it wasn’t for XDCAM, Blu-ray Discs would be virtually invisible.
mDisc is limited, not by its technology, but by the tech industries infatuation with the simplicity of downloads at the expense of optical media. Anyone whose business relies on selling DVDs – the wedding video industry comes first to mind – knows that the optical media market isn’t dead. But it died long ago from the tech industry’s point-of-view.
Its my guess that within five years we won’t find any optical media hardware available in the market. At which point, what good is a disc that can last for 1,000 years, if we don’t have a way to play it back?
When it comes to tech, the past is an annoyance.
LTO ISN’T MUCH BETTER
For long-term archiving, I recommend LTO tape. It doesn’t have the longevity of mDisc; where tapes last only about 30 years. But it does have the advantage of a huge market that uses this gear on a daily basis; corporate America.
Every Fortune 2,000 business backs up their servers every day using LTO drives and tape. This multi-billion dollar market dwarfs the media industry and generates continuing demand for hardware and software that MUST, by design, be backward-compatible.
Every LTO tape cartridge from LTO-1 to LTO-7 is exactly the same size and shape. Every LTO drive reads tapes from two generations back. And the hardware road map looks several years out and is published for everyone to see.
There are still problems, though, caused by the onward rush of tech. LTO drives are updated about every 18 months and the newest drive, by design, only read two generations back and only write one generation back. (LTO-7 drives, which will ship around the beginning of the year, read LTO-5, LTO-6 and LTO-7 and write LTO-6 and LTO-7.)
This means that once you start archiving on LTO, you’ll need to plan to upgrade your hardware every 7-10 years and copy all your tapes in order to stay compatible.
LAST THOUGHTS
I’m not sure there’s a solution because media and technology have such divergent goals. It is a forlorn hope that the technology industry will slow down simply to allow us to preserve the past. It is equally forlorn for us to expect that the technology we use today to preserve the past will last very far into the future.
Never assume that your assets are safe. When it comes to preservation, follow the herd. They may not be right, nor make the “best” choice,” but they are big enough to influence what gets developed. And expect that you’ll be dubbing the material that you want to keep “forever,” from one hardware format to another every few years for the rest of your life.
As always, let me know what you think.
9 Responses to Preserving the Past is a Forlorn Task
Perhaps, it is time for the little guy to speak up and out. Technology would not thrive without support of the masses of little fellows. Where is the voice of the people who are actually the purse strings of technology? Do we sit back and allow the profit seeking, treasure chest filling Wall Street Lords and Rulers of Technology control us through their designs of obsoleting? It’s out of control when you have bought a product, found it defective or in need of repair only to find parts are not available or we repair by replacing with a newer product. So mDisc has come up with a solution. Then we the people, the users of technology, who hold the purse that supports technology should embrace the mDisc and let the industry know we have chosen and demand of them the products that support it. The archive world must take a stand and require industry to meet their demands rather than being led over perilous terrain like sheep by shepherds of greed. Don’t get me wrong. I appreciate where we have come. I have been apart of three quarters of a century of progress and technological advances. What I can hold in my hand today replaces tons of equipment required yesterday. I deal on a daily bases scanning old movie film. The cost of a 50′ roll of 8mm color film with processing in the fifties was $4.00. Today because some companies have seen the need and have provided the equipment I can take that vintage roll of film and copy individually every 4000 frames of its images to a digital file and charge not much more than the original film price. My customers would like it to last at least more than their lifetime. LTO tape, hard or thumb drive, the Cloud, DVD or Blu-ray disc, etc.? The original film with care can out last these. Enter mDisc – the only promise of permanence. Let’s stop being forlorn and stand up and use and support the mDisc and we will create a market that demands compatible equipment in years to come.
I think you make a very powerful point about how the tech industry chases “fashion” thinking in their development and support of these storage technologies. The problems won’t be so much finding blank optical media, as finding drives that will run it. The SSD trend continues to accelerate, pushing spinning drives to the side as time goes on.
There’s two tech trends that could be disruptive to the extrapolations you’ve laid out. One is H.265, the compression scheme that is supposed to supplant h.264, with roughly half the bandwidth of H.264 but as good or better quality. That leap in compression could make recordable optical media like DVD-R and BluRay relevant again, if only because they’d now have the capacity to store longer 2K and 4K programming. Kinda like helical scanning versus linear recording on tape did with videotape. But H.265 is mired in squabbles over who pays how much in royalties to use the tech. By the time that’s worked out, it could become irrelevant. Industry consortia are, as we speak, working on a replacement for ‘265 that would be royalty-free.
But another item on the horizon really thrills me. As with so many key technologies, this one’s a spin-off from the space program.
It’s the Square Kilometer Array (wiki or google it), a network of space telescopes being built over the next couple of years, that can see so deeply into space and time, it will take pictures of when God was a boy.
The significance to us in the media biz? The SKA, and other “big science” projects like the Human Brain Project, will each be generating about an EXABYTE of new data every day, all by themselves. That’s 1000 petabytes or 1 million terabytes or 1 billion gigabytes. Daily. An exabyte is essentially a disk image of everything on the internet today. Fresh, every day. Including the lolcat pictures and naughty bits. It’s a lot of stuff! Imagine generating that much content, every day. And finding a way to store it and access it.
The technology being developed to capture, manipulate, store, and transfer this firehose, nay, Niagra of data has yet to be INVENTED. But it is due to come online in 5 years. IBM heads the group developing this tech, and you know they are going to commercialize it as soon as it is ready. that’s when it changes everything about our business, and about how we live. When that commercilaization happens, it will make today’s fastest internet look like 1981 on a dial-up modem. The so-called “cloud” of today will become an ocean, with us swimming in the center of it. “Archival”, in terms of storage reliability and capacity, will no longer be a problem, ever.
We will live to see it.
But we have to have something today that will do the job of reliable, accessible storage, and remain readable, long enough to get to that “data ocean” era. Something to cover the next decade. I think it is going to be something to do with an evolution of solid state memory, perhaps using what’s called “spintronics”. Or it could be holographics, or quantum foam, I’m just speculating here. It may not need to last fifty years, because in ten or so, this discussion will have been made irrelevant.
Larry, You are absolutely on target with your observations regarding archiving and the march of technology! I remember a studio client of mine saying (in frustration) “what’s another new format when we’ve had 17 so far” — and that was back in the mid-90s. Continually advancing hardware, coupled with ever-changing software is incompatible with “preservation” as most archivists know it. While professionals have been aware of this problem for many years, consumers are just realizing the consequences of digital content acquisition as their smartphone and iPad apps become worthless when manufacturers update their operating systems and cloud services go out of business. If you can, please add a link to Vince Cerf’s recent comments on these matters.
HI Larry, you didn’t mention the relative costs for mDisk vs LTO. 25gb mDisks are 50 for $220 or $4.40 each, which comes out to $176 per terabyte if my math is correct. LTO 5 tapes are about $20/Tb. And while LTO is slow, at least you don’t have to change and label disks 40 times to record a Tb of data.
Of course the hardware/software cost to get started is another story. I was unable to find a new, off-the-shelf LTO 5 or 6 solution for under $3,500, and finally pieced together a used LTO 5 system for about $1500 (you can read the details about my search for cheap LTO here: http://www.paulkagawa.com/blog/).
$1500 is still a lot of money for something that just sits there most of the time. Maybe as the word gets out to individual users about the necessity of archiving, demand will rise and prices will fall.
Interesting discussion, Larry. Good points about mDisk–the read and write technology is the issue, not the medium.
I’ve been thinking about this a lot and have 16mm color films from the 1970s that are turning pink. Someone tells me they can be scanned through a special process and perhaps restored if there is any blue left in the image.
But then they would join the ever-modulating digital stream.
One of the oldest and most persistent archival mediums is the paper print used by the Library of Congress to archive B/W films from the early 1900s. Since they’re in analog form, the images can always be recaptured with whatever the current technology might be.
I think Technicolor films with 3 B/W color separations used to be a true archival medium but they gave it to China. I wonder if it’s in use there.
Analog images still might be the medium with the longest life. We have paper documents from 2,000 years ago, and illuminated manuscripts from 1,000 years back. Now we need a digital system for inscribing movie frames on paper.
Or we could just decide that movies are an ephemeral medium.
Robert:
I don’t think anyone would argue that film, news, even digital family movies should be ephemeral. However, that doesn’t mean we have a clue on how to archive them for the long term.
Regarding your films, I would STRONGLY argue in favor of transferring them to digital form before the color fades irretrievably. This also provides the greatest opportunity for restoration. Once they are in digital format, you can join the rest of us in figuring how to store them long-term.
Larry
Robert: 1970s Eastman print stock was notorious for fading. It’s why filmmakers of the time (such as Spielberg) would make RGB separations for archiving.
Scanning will give you the opportunity to adjust the remaining color on those prints in the digital realm.
I used Pro8mm in Burbank to scan about 40+ rolls of 8mm & super8 shot from 1974-1979 and I was AMAZED at what they were able to enhance on reels I thought were faded for good.
Good article!
Sony has an Optical Disc Archive that is pricey, but does not have the archival issues of LTO Tape. They have models that go up to 5.5tb, I think.
https://archive.org/details/sony-optical-disc-archive-manual-ods-d-55-u-ods-d-77-u-chinese/1%20Sony%20Optical%20Disc%20Archive%20Manual%20ODS-D55U%20%26%20ODS-D77U%20English
For those that poo-poo optical media?
I’m still using CD’s I bought in the 1980’s…and no end in sight with being able to use them.
I run one of the largest underground Archives in the world and am a big user of M-disc of all calibers. Recently the M-disc 4.7gb inkjet printable is nowhere to be found. I hope they are not discontinued it. If so, I will have to use BR-D M-disc for everything.
I also do archival testing of color imaging media for photography. The testing has expanded to include digital storage media. Nothing compares to the M-disc for digital preservation. When they come out with laser engraved quartz, then we will have another option.
Here is a rundown on some of the optical media tests…
https://daniel-d-teoli-jr-archival-collection-ii.home.blog/2019/10/23/nothing-can-compare-to-the-m-disk-when-it-comes-to-digital-archiving/
Daniel:
Thanks for sharing your thoughts – and links.
Larry