Organizing and Archiving Your Projects

[ This article was first published in the February, 2004, issue of
Larry’s Final Cut Pro newsletter — Click here to subscribe.
Updated 6/112004, 2/13/05, 11/16/08 and 5/31/10. ]

Your project is done. The client is happy and their check has cleared the bank. It’s time to trash that old project and get ready to tackle something new.

Except…

Flash forward six months. The client breathlessly calls up and says: “I just got the budget. I want to go back and reedit EVERYTHING!”

Great. The problem is that you trashed everything from your system about four months before she called. Now what do you do?

The process of storing project data so you can get access to it later is called archiving. And archiving a project is a lot easier if you take time to get it organized correctly in the first place.

That’s where this article can help; because organizing your project can be broken down into some fairly easy steps:

1. Organize your project BEFORE you start
2. Special issues in using tapeless media
3. Stay organized during your project
4. Know what to archive when the project is done
5. Avoid long-term storage problems

ORGANIZE BEFORE YOU START

There are lots of different ways to organize a project. This is my system. It works for me. And, if you don’t already have a system, this will work for you, too.

If you are serious about your editing, you’ll have at least two hard drives: one for the operating system and Final Cut Pro (called the “Boot Disk”) and a second hard disk to hold all your media.

As we quickly move into tapeless media, we need to add a third option: where to store the original source files that were first recorded to cards.

CRITICAL NOTE: Video files are gigantic. So big, in fact, that you can not successfully edit video using only one hard drive. You must have at least two — one solely to store media. This media drive can be either an internal or external drive. It can NOT be a partitioned drive. Also, USB drives are too slow on the Macintosh to store media. Make sure your drive connects via FireWire or SATA.

Here’s an article that discusses hard drive speeds and video formats.

On your boot disk, create a folder called, “FCP Projects.” This folder will hold all your non-timecode-based project files. I recommend storing this at the top level of my boot disk; don’t store this inside your Home directory.

Inside the FCP Projects folder, I create a folder for each new project that I create. Inside this project folder are all the elements I need for my project; except timecoded media. That goes onto the second drive.

IMPORTANT NOTE: As we move to larger and larger RAIDs and high-speed networks, the question arises where to store project data. In point of fact, you can store project data anywhere. I recommend keeping it organized in a project folder. Whether you store that on your boot drive, or on a second drive, or on a server, is your choice. In the past I’ve recommended the boot drive because it clarifies the difference between project data and media. On my current editing system I store project data on one drive, not the boot drive, and media on a fast RAID. Keeping these files separate is a good organizational tool. But they need not necessarily be stored on the boot drive.

On your second drive – the media drive – create a folder called, “Final Cut Pro Documents.” The name of this folder is VERY important, because Final Cut Pro is programmed to look for it. This folder will store all your timecode-based media — whether captured from tape or ingested from tapeless media.

Now, open Final Cut Pro, and go to Final Cut Pro -> System Settings and set the Scratch Disk to point to the new Final Cut Pro Documents folder you just created.

When FCP captures media, it automatically creates a new folder inside Final Cut Pro Documents > Capture Scratch named after your project file. This makes it VERY easy to find captured media on a project by project basis, because it’s all sorted by project inside a single, Final Cut Pro Documents, folder.

You can only have one Final Cut Pro Documents folder per hard drive. However, if I have a project that requires multiple hard drives to store, I create a Final Cut Pro Documents folder on each hard disk I plan to use for media, then point the Scratch disk (by clicking the Set button) to each Final Cut Pro Documents folder on each hard drive. Starting with Final Cut Pro 4 you can have up to 12 scratch disks.

If we go back to the Finder and open the Final Cut Pro Documents folder, we can see that FCP put at least three new folders inside:

• Audio Render Files
• Capture scratch
• Render Files

(You may also see another three files: AutoSave Vault, Thumbnail Cache folder, Waveform Cache folder, depending upon how your other System Settings are set.)

Best of all, I only need to create the Final Cut Pro Documents folder once. After that, the automatic filing of Final Cut keeps all the different media files for all my different projects in their own separate folders — all in one place.

NOTE: One of the most requested features for Final Cut is to have scratch disks be project-based, rather than computer system-based. However, that is not what Final Cut does. Instead, you set your scratch disks once, then never change them. All media is stored in folders names after your project inside your scratch disk.

SPECIAL ISSUES USING TAPELESS MEDIA

Tapeless media provides several new challenges to Final Cut Pro. First, and most scary, as soon as we record a card, we need to copy it to a hard disk, then erase the card. This means that we don’t have a media source – like a video tape – to return to if something bad happens to our media.

Next, many media formats need to be converted into a format that Final Cut can edit. Examples include: HDV, P2, and AVCHD.

For this reason, I create a new folder on one of my media drives titled: Source Media. If the amount of source media is small, I store it on one of my media drives. If I have a lot of source media, I’ll store this folder to its own separate drive.

Within the Source media folder, I’ll create a folder for each project. Then, within the project folder, I’ll create a folder for each card. I then copy the ENTIRE contents of the card into the folder.

Using Log & Transfer within Final Cut, I can open that folder and pull out the media I need. This way, I always have access to the source media files in case something goes wrong with the copies.

NOTE: I always store media to external drives. The reason is that it is easier to add external drives that internal drives. Also, if something happens to an external drive, or RAID, it is easy to send off the drive for repair without losing access to my entire computer.

STAY ORGANIZED DURING YOUR PROJECT

As soon as I get a new Final Cut project, I create a new folder in the FCP Projects folder on the boot drive. For this example, I’ll call my new opus: “My Project.” (For the obsessive amongst us, you would replace “My Project” with the ACTUAL name of your project.)

Here’s the drill: all timecode-based media gets captured and stored in the FCP Media folder, on the second hard drive. All other files — graphics, music, sound effects, memos, schedules, everything — goes into the FCP Projects > My Project folder on my boot drive. Playing music or sound effect files on the boot disk is generally not a big deal because the bandwidth they require is small compared to video.

Critical to the whole archiving process is that as soon as a production tape is shot, I give it a reel number and write it on the tape cassette. All my tapes have UNIQUE reel numbers. For my personal tapes, I simply number them. For professional projects, I give them a project name and reel number: for instance, “WC04_01,” where the project name is to the left of the underscore and the reel number is to the right.

Then, I am religious about making sure I add the correct reel number to every clip that gets logged or captured into Final Cut. And, remember, because of how we set the scratch disks, all captured media gets stored in the FCP Media folder on the second hard drive.

NOTE: When working with tapeless media, the name of the folder containing your source files becomes the Reel ID in the Log & Transfer window. Final Cut adds this automatically.

Another thing I am religious about is bringing every clip I plan to use into the Browser, rather than dragging it directly from the desktop to the Timeline. This means that every clip in my project shows up in the Browser.

Using this organizational system, ALL the elements of a project are in one of two places: the Media folder or the Project folder.

WHAT TO ARCHIVE WHEN THE PROJECT IS DONE

When the project is done and everything has been output and approved, you are ready to archive. The question is, “What?”

First, backup the complete contents of your project folder to CD or DVD. This protects everything that isn’t media.

Second, if you are using video tape, here’s my suggestion: Don’t archive media. That’s what your original camera source tapes are for. They are small and last for at least 20 years.

Note: Working with tapeless media presents us with archiving challenges that have not yet been solved. My current recommendation is to archive all tapeless media – and servers in general – on LTO Level-3 or Level-4 tape drives, made by companies such as Exebyte, HP, Dell, and Quantum.

---

As a side note, when I was working in broadcast with 1″ and 2″ tapes, I was taught to NEVER store tapes on their side. Always store them on edge. For the short-term, how they are stored won’t make any difference. Longer-term, however, storing tapes on edge decreases the effect of the earth’s magnetic field and any potential oxide shift due to gravitation pulling on the oxide of your tapes.

I’m sure that tape stock today is better than 20 years ago. And the size of tapes and strength of the signal also tend to minimize these adverse effects.

All that having been said, I still store all my tapes on edge and away from heat and light. Oh, and I make dubs of all my most important work every 20 years or so.

---

So, once you have archived your media, what are we going to do with the files on your hard disk? The answer is simple, we simply grab the project folder inside Final Cut Pro Documents >Capture Scratch on the second drive and trash it!

Why? Because if every clip you used is in the Browser, AND if you assigned every clip a reel number, you can easily redigitize your entire project by selecting the clips you need in the Browser and going to File -> Batch Capture.

Final Cut will ask you to insert the source tape reel, then merrily recapture all the missing media. Best of all, recapturing when you need the media means that you can take advantage of all the improvements in codecs and Final Cut that occurred between the time your project ended and when you need to recapture.

Render files are most easily managed using Tools > Render Manager. As long as you haven’t changed your scratch disk locations, FCP will track all render files for all your projects. Removing render files should ONLY be done at the end of a project.

So, in summary, backup your project folder to a CD or DVD. Trash media from your hard disk that is stored on video tape. Backup media that is stored tapeless. And use the Render Manager to delete render files.

Note: OK, one more caveat. If you somehow failed to get organized at the beginning of a project, your media files will be scattered all over your hard disks. Simply trashing a single folder won’t get rid of all your media, and may actually trash media from other projects that was stored in the wrong folder by mistake. In this case, a much better alternative is to use the Media Manager to “MOVE” all your media files from where ever they are to a single location — say a new empty folder on your emptiest hard drive. Then, once all your media has been gathered into one place, trash the folder. In all cases, however, there’s no need to keep the media from a project.)

---

Here’s a second side note. Some people may still want to archive their media on something other than tape. What should they use?

Well, a friend of mine uses removable hard disks. While really expensive, this has the advantage of keeping everything in a digital state. Just connect the disk and he’s ready to edit. Short-term this is a good solution. Long-term, it isn’t. There’s just no assurance that the same storage media will be around and able to be hooked up in ten or twenty years. I mean, look at all the old gear you’ve got cluttering your garage that used to be state of the art — and that stuff is probably far less than ten years old.

Others may want to archive to DVD. This is OK, as long as you understand the limitations. First, DON’T archive your media as an MPEG file. While your file sizes shrink, so also does your ability to edit them. In spite of the marketing hype, MPEG is a distribution format, NOT an editing format.

Second, remember that a one hour Beta SP tape holds about 60 GB of media. A DV tape holds 12 GB. A DVD that you can burn only holds 4.2 GB. So it will take three DVDs to store the contents of one DV tape, or 12 DVDs to hold one Beta SP tape.

Third, if you still feel compelled to store your media on a DVD be sure to create a DATA DVD. This keeps your media in a pristine digital state that is available for editing. Just copy your digital files from your hard disk to the DVD as they are, without additional compression.

(By the way, depending upon the speed of your computer, it will probably take longer to copy the files from your DVD to your hard disk than it would to simply redigitize them. NOT that I have an opinion, or anything.)

AVOID LONG-TERM STORAGE PROBLEMS

The more we move into tapeless media, the more challenging archiving becomes. Here are several other links that can help:

• In search of a backup strategy.
• Create a file-naming system that works.
• Video tape does not last forever.
• DVDs do not last forever.
• Hard disks unplugged and stored on shelves, slowly lose data.
• Restoring failing data on hard drives.
• Not all codecs are good for long-term storage.

This subject will continue to evolve. And I’ll keep updating this as I learn more.

UPDATE — 2/13/05

In other articles, I’ve mentioned Andreas Kiel, who’s the author of XML2Text. After reading last month’s issue, Andreas had some additional comments, especially about archiving media.

In connection with the ‘Archiving Media’ article, this utility [XML2Text] might be worth to mention. [Andreas is the author of XML2TXT.]

[Also,] I agree to the approach to leave or store video on tape – I also recommend it as the best and fastest way, even if sources came from different types (DV, DV50, SD etc).

The way described in your article assumes that all material stays in your hands and are on one of the editors own tapes forever. In the environments I’m working, this is only partially true. Many times people get tapes from the news gathering, archives or costumers etc. These clips are now on their HD. The original tape go back or in case of the news it is deleted already.

In former times I always put them back to tape and had some GUI scripting, which read the TCs. (That technique – though not elegant at all – worked fine for me and customers). With the nice XML features of FCP and XML2Text this can be done easier.

Drop all clips you want to lay off to video into a sequence and record to tape. Make sure the timeline timecode matches the tape (first clip’s TC in the sequence equals first clip’s TC on tape), export as XML and convert this with XML2Text to a ‘Batch List’ which reflects the new clip IOs and reel and keeps all relevant column info from the original.

This list can be archived with the project to retrieve clips later.

Additionally I always recommend to use batch lists of projects as they allow to build up a small clip/project/customer/tape database with FileMaker.

So, if we don’t need to archive the media, what gets archived? Everything in the FCP Projects > My Project folder on your boot disk.

Remember, this is the folder that stores your FCP Project files, your music, sound effects, graphics and every other non-timecode file related to this project. Because everything is all in one place it is a very simple matter to drag this folder to a blank CD or DVD (depending upon how much space you need) and, presto!, you’ve got a complete backup of everything you need to re-create your project at any time in the future.

No muss. No fuss. And no trauma trying to track everything down at the end of a project.

Very cool!

---

Bookmark the permalink.