An Exercise in Frustration

It shouldn’t be this difficult.

NOTE: This blog grew out of my webinar last week on how to archive Final Cut Pro projects. If you are interested in archiving, click here to learn more.

The process of creating films and videos generates LOTS of data. From getting the footage shot, through the editing process, we are dealing with dozens, if not hundreds, of gigabytes of data.

And we solve this problem by buying ever more and ever larger hard disks and RAIDS. So far, a trifle expensive, but not really a problem.

However, once a project is done, we can’t just throw all these files away. The images and interviews are irreplaceable and have a value long after a project is complete.

We need to save these for the long-term, a process called “archiving.” And here, we are caught between a rock and a hard place.

In the camera industry’s head-long rush to convert to cameras that shoot tapeless video, they seem to have overlooked the fact that we need to save these video files after editing is complete.

I have spoken with executives at Sony, Canon, JVC, Panasonic, and the rest of the camera industry, not one offers any kind of long-term archiving solution.

NOTE: I define archiving as the ability to save all video project and source files for a period of 10 – 25 years. Backup is what we do to protect ourselves during a project from the risk of data loss.

One camera vendor told me, dismissively, that tapeless media should just be handled as part of an “IT workflow.” Which is great if you have an IT department and the budget to support it. But, I looked in my checkbook, and the ability to write a $100,000 check for an automated tape archive library is not possible this week.

For me, the long-term solution to archiving tapeless video is LTO-tape. This has the flexibility, storage, and longevity to meet all our archiving needs. But, while this seems obvious to me, manufacturers of LTO gear seem to be doing everything they can to avoid selling us the products we need.

HOW YOU CONNECT IS CRITICAL

Currently, most LTO-tape drives connect via FibreChannel, SAS, or mini-SAS. Which makes sense if you are selling to the corporate market, where every server supports plug-in cards, speed is everything, and budgets are always expressed in units of ten thousand dollars.

But that isn’t the reality of video editing.

More than 70% of all video editors work in companies of five employees or less. Generally, the video editor provides all their own tech support, without access to an IT department.

Additionally, I would guess that close to half of all video editing is done on a laptop with a FireWire-attached RAID or hard disk. Portability is more important to a producer shooting a documentary than sheer horsepower.

Or, take an independent filmmaker on a budget. They have a collection of multi-terabyte FireWire drives, containing all the elements of their film, scattered across their desktop while editing on an iMac to take advantage of the large screen. Saving money yet getting a large screen is more important than buying the fastest possible computer.

I don’t have any research here, but I would guess that two-thirds of video editors are working on a system that isn’t a MacPro. And every one of those editors is juggling terabytes of data and praying they can figure out a way to store it for the long-term.

NOTE: The newly announced Thunderbolt provides a very fast data bus that tape vendors can tap into that would give them access to every new Mac that Apple ships. However, it doesn’t solve the archiving problems of the millions of Mac users that already have gear.

SPEED ISN’T EVERYTHING

When backing up corporate servers, speed is important, because many of the files on a server change daily.

But this isn’t typical for video. Video and film shoot a lot of files in a very short period of time, then spend a while — sometimes a long while — editing them. In other words, once the initial backup is complete, the only files that change are relatively small project files, not the massive media files.

This means that a workflow that features easy-to-connect and easy-to-use is far superior to one that emphasizes blinding speed. We are making archives here to last for decades. Taking an extra few hours to back everything up is a non-issue.

My large media files don’t change that often. My small project files change daily. And I only need to archive my project once – when the entire project is complete.

A FireWire-attached LTO drive would not be a speed demon, but I’d bet you’d never be able to keep them in stock. Security of my assets is FAR more important than transfer speed.

THE SIMPLE TASK OF GETTING AN HP LTO-5 TAPE DRIVE TO WORK ON A MAC

Here’s a specific example.

HP is a leading manufacturer of LTO drives. About a month ago, I contacted them for a loaner unit to review so that I could use it to illustrate a webinar I just did on how to archive Final Cut Pro projects.

HP agreed and shipped me an HP StorageWorks Ultrium 3000 SAS LTO-5 tape drive. It uses an ATTO ExpressSAS H680 interface card, which, because it is a plug-in PCIe card, only works with a MacPro.

The drive arrived a month ago and I am STILL trying to figure out how to connect it to my MacPro!

NOTE: Installation on Windows and Linux is close to click-one-button easy. Not so on the Mac.

I teamed up with two software engineers to help me get this to work. The first took one look at the installation instructions and gave up.

The second spent many hours over the last four weeks trying to configure the system. Here’s what he learned:

1) The HP unit didn’t come with firmware compatible with the software
2) Updating the firmware is seriously non-trivial.
3) It’s loud. The drive is about two feet from his ear and measures 57dB in operation.

Here is a BRIEF summary of the steps necessary to install the HP StorageWorks Ultrium 3000 SAS unit on a MacPro. (Keep in mind this required a Unix-trained hardware engineer to accomplish — this is only a summary, the actual installation process took more than 30 steps and two weeks to figure out!)

1. Install the ATTO SAS card into PCIe slot #1 of the MacPro.

2. Go to Google and download MacFuse, which is special utility software that allows the drive to operate on a Mac)

3. Install MacFuse

4. Go to HP’s website — www.hp.com/go/ltfs — to download the latest drive firmware.
HINT: To get this to work, you need to know the version you need. To make this easier, HP doesn’t tell you which version that is.

5. Download and install ICUFramework.pkg

6. Download and install LinearTapeFileSystem.pkg

7. Format the tape cartridge – this is LOUD! 57 dB.
HINT: Formatting a tape can only be done from the Terminal window. You ARE familiar with entering Unix commands via Terminal, aren’t you?)

8. Update Firmware on your drive.
HINT: This involves a return to the Terminal.

9. Prior to installation move the downloaded firmware into the /VAR/ROOT/ folder.
HINT: You will be glad to know that this folder is invisible on the Mac.

10. Make yourself a SuperUser in order to install the software.
HINT: You should be familiar with how to become a root user.

And this is only the START! There are still about 20 steps left to go involving terminal commands, more downloads, accessing invisible folders, and, if I read the fine print correctly, sacrificing a small goat.

This is ridiculous!

THERE ARE OTHER SOLUTIONS

Yes, there are existing Mac solutions out there.

The Tolis Group — www.tolisgroup.com — has BRU Producer, a hardware/software system for archiving that runs on Mac, Windows, and Linux systems. I’ve spent the last 30 minutes on their website, I still have no idea what products I need or how they are priced. The purchase they product, they refer me to a dealer. When I go to a dealer’s site, for information or pricing, they refer me back to the Tolis Group site.

The Tolis Group has hardware bundles, software bundles, and add-ons, which, apparently, are neither hardware nor software, but assume you already have the hardware you need, whatever that is.

Cache-A — www.cache-a.com — is another company that makes archiving solutions for the Mac. They only have two products, which makes deciding a lot simpler. However, nowhere on their website do they mention a price.

OK, that means I need to buy it from a dealer. So, I went to four of the dealers listed on their website:

* AbleCineTech — doesn’t list Cache-A in their vendor list, and Cache-A can’t be found using a search
* Digital Video Group — doesn’t list the products they sell and their site is not searchable
* B & H Studios — Cache-A not listed, site not searchable
* TekServe — Cache-A can’t be found using a search

And these are not small dealers here. It seems I can only buy Cache-A through dealers and the dealers are too embarrassed to mention they carry the product!

Is selling archiving solutions REALLY that difficult??? Is this much secrecy about your pricing, your products, or your service necessary?

Apparently, archiving IS rocket science!

CONCLUSION

If you are an manufacturer that has an LTO-tape-based archiving solution that:

* Can be installed easily on the Mac OS
* Is willing to list the product and its price on their website
* Has an announced shipping date
* Has an announced price of between $2,000 and $4,000
* Can connect to Macs via Thunderbolt or Firewire
* Supports the LTO-5 standard, which makes file backup/restore VERY easy
* And includes archiving software that can be run by mere mortals and not IT gods…

Contact me. I will happily share the news of your product to the MILLIONS of video editors that are desperately searching for ways to safeguard their media.

Otherwise, we run the very real possibility that every movie we are watching today will slowly disappear over the next few years as our current archive solutions fail.

Larry

---