UPDATE: How to Accurately Measure the Speed of Mac Storage

Recently, I wrote that accurately measuring the speed of storage is surprisingly difficult (link). Measurement tools from AJA Video Systems (AJA) and Blackmagic Design (BMD) substantially disagree.

Since I published that article – and sent copies to both BMD and AJA – I’ve heard nothing from Blackmagic as to the accuracy of their software. However, the folks at AJA took an active interest in figuring this out.

The problem with the AJA results was that both versions of their System Test software were presenting results which seriously understated storage speed. So much so, in fact, that the multicam streaming I was successfully editing in Final Cut or Premiere would have been impossible if the AJA results were accurate.

Something was broken.

WHAT I LEARNED EARLIER

AJA makes two different system measurement tools:

• AJA System Test Lite (free and available in the Mac App store)
• AJA System Test

After initial checking, AJA told me that there were bugs in AJA System Test Lite that led to inaccurate measurements. Last week, they posted a new version to the Mac App store, which I then tested. The short answer is that test results improved but there still seem to be problems.

For example:

• 5% speed differences between measuring storage speed as “one-time-only” versus continuous modes using System Test Lite.
• 13% speed differences between measuring storage speed as “one-time-only” versus continuous modes using System Test (Full).
• 18% (write) and 11% (read) speed differences between results in the Lite and Full versions.

Clearly, work still needed to be done. But, maybe the problem wasn’t the software itself?

WHAT I LEARNED RECENTLY

Last week, I spoke to AJA again and learned three valuable tips which affect the accuracy of their tools:

1. All AJA software exists to support AJA hardware, which is where the company makes its money. This perspective is perfectly understandable.
2. All AJA hardware uses uncompressed video internally. This means that any compressed format – this includes ProRes – is converted to uncompressed before AJA gear can do something with it.
3. Therefore, when using either AJA System Test or System Test Lite, accurately measuring the speed of storage depends upon the codec type you select for the test. Why? Because the test adds time to the results to compensate for the time their gear requires to transcode a video format, i.e. ProRes, into uncompressed.

For this reason, selecting a compressed video format will under-report the speed of storage. This makes sense if you are matching your storage to AJA gear, but not when you simply want a better understanding of how fast your storage hardware actually is.

So, I went back to retest my storage to see which differences remained between the two versions and the impact of changing the test file codec. 64 tests were involved.

MY LATEST TEST RESULTS

My testing system was a 16″ MacBook Pro with an M1 Pro SoC. Storage was an 8 TB Thunderblade SSD RAID from OWC; connected via ThunderBolt 3/4.

The consistent increase in speed becomes even more obvious when we look at these same results as a chart. (To create the chart numbers, I averaged the reported speeds of the Lite and Full versions.)

Key findings when switching the Codec Type to RGB from ProRes:

• The larger the bit depth, the faster the file transfer. This reflects, I suspect, less test overhead dedicated to processing the file. These numbers start to approach the speeds necessary to support very large multicam stream counts.
• There is less than 1% speed differences between the Lite and Full version when testing using One-Time-Only or Continuous. This is a big improvement.
• Lower standard deviation numbers mean the results between tests were more consistent.

NOTE: I did see deviation between the two software in write speeds for both 12- and 16-bit media; variations up to 10%. But the average data rate numbers seemed very consistent overall. (My test results are linked below.)

Using 16-bit RGB test media yielded the fastest and most consistent results, with almost no difference between the Lite and Full versions of System Test.

WHAT SHOULD WE DO?

Simply change the test settings to get more consistent answers. For example, when using AJA System Test Lite to measure your storage:

• Set the File Size to 4 GB. (I did a quick test using 16 GB files and got essentially the same results to using 4 GB.)
• Set the Codec Type to 16bit RGB
• Run the test at least three times, because the first result tends to be the least accurate

These settings correlate test results more closely to the storage speeds necessary to stream multicam edits without dropping frames, which is how I determine whether test numbers approach reality.

Finally, some consistency is emerging. What we still don’t know, though, is whether these consistent results are accurate. That’s still to come.

EXTRA CREDIT

• Here’s a link to AJA System Test Lite, in the Mac App store. It’s free.
• I ran 64 speed tests for this report. Here are the details in a 3-page PDF (150 KB).

---

Bookmark the permalink.