cart
become a member

Significant Test Result Differences Between AJA System Test Lite and Blackmagic Disk Speed Test

Update – Nov. 28, 2022. I’ve heard back from OWC, Blackmagic Design and AJA. All three are looking into this problem.

Update – Dec. 5, 2022. Blackmagic PR wrote: “Thanks so much for sharing this information – I’ve passed this along to the Blackmagic Design team. As always, I’ll keep you posted as soon as I hear back from them.” Since then, I’ve heard nothing.

Update – Dec. 11, 2022>. Here’s an updated article looking at the differences between measurement tools. So far, no developer is able to explain these differences. Assume whatever test results you are looking at are wrong. (Errors also exist in Apple Activity Monitor.)

Update – Dec. 30, 2022. AJA is much more responsive. Today their tech support emailed me: “I did forward your test results to Product Development for review and they did acknowledge the Lite version has an issue. I don’t have any information on why there is a discrepancy between the full AJA system test and BMD, however I will let you know as soon as I have update from Product Development. By the way, thanks for all your work testing and documenting the test results, very much appreciated.”

This last week, I was testing the speed and capacity of the new OWC Thunderblade SSD RAID. (Read that review here.) This hardware speed demon is designed to maximize data transfer speeds using Thunderbolt 3 or 4. Except, I get significantly different results when testing storage speeds using AJA System Test Light versus Blackmagic Disk Speed Test.

Since we use both these tools daily to measure and verify the performance of our system, these differences trouble me — a lot.

EXECUTIVE SUMMARY

There is a 40% difference when measuring write speeds and a 22% difference when measuring read speeds between these two software tools.

Some variation during tests is expected, but this amount of difference is ridiculous. Clearly, one – or both- of these software tools is wildly wrong. But… which?

That, I have not yet determined.

MY COMPUTER

THE TESTS

I created a RAID 0 using an 8 TB OWC Thunderblade SSD, which uses four NVMe SSDs. This RAID format maximizes performance.

I ran four tests using each software to measure the read and write speeds of the Thunderblade.

NOTE: As part of my Thunderblade review, I ran twelve speed tests on each software and got very similar results. I used four tests here simply to save space.

(Click to see larger image.)

Same day, same computer, same storage, same data file size (4 GB), same connection (Thunderbolt 4)  — wildly different results.

THE DETAILS

NOTE: While the difference between read and write speeds is due to the Thunderblade, the differences in speed measurements are due to the software design.

Another troubling aspect of these results is that AJA System Test Lite reported a much wider fluctuation in speed measurements – especially in write speeds – than Blackmagic.

NOTE: The lower the standard deviation, which is measured in MB/second, the more consistent the speed. BMD reported far more consistent speeds than AJA. However, consistency is not the same as accuracy.

Here are the detailed results of my tests.

SUMMARY

Clearly, there is a significant difference between these two tools – and there shouldn’t be. Some variation is to be expected, but not this much. The problem is that we don’t know which one is the most accurate.

As I mentioned at the beginning,  I reached out to OWC, AJA and Blackmagic Design to get their interpretation of these results. All three said they would look into it and get back to me. For now, however, be careful relying on either of these software tools until we learn more.

EXTRA CREDIT

Apple’s Activity Monitor can be used to measure data transfer rates. However, unlike AJA or BMD’s tools, Activity Monitor can’t be isolated to report results on a single device. Activity Monitor shows aggregate disk or network traffic, which would almost always inflate the results.

Bookmark the permalink.

22 Responses to Significant Test Result Differences Between AJA System Test Lite and Blackmagic Disk Speed Test

Newer Comments →
  1. Tom Knoff says:
    November 28, 2022 at 4:56 am

    I noticed the same when performing tests on raid 0 ssd’s several weeks ago. My money is on AJA being wrong.

    Reply
  2. Christopher Campbell says:
    November 28, 2022 at 6:42 am

    I’ve been using an 8TB ThunderBlade for a year (primarily for a very large Lightroom catalog), and love the drive. However, I just replicated Larry’s results on an iMac Pro:

    AJA average Write: 1657 MB/s
    AJA average Read: 1773
    Blackmagic average Write: 2560
    Blackmagic average Read: 2437

    Speed Difference Write: -36.3%
    Speed Difference Read: -25.7&

    Reply
  3. DM says:
    November 28, 2022 at 10:28 am

    The difference is likely due to drive caching. I’d lean towards the AJA results as being more accurate…

    Reply
  4. Olaf says:
    November 28, 2022 at 3:07 pm

    Hello Larry,
    What data shows activity monitor at the same time, as this can be monitored in parallel with either of the two?
    I use Softraid‘s driver on my Blade and would like to know if you used Apple Raid 0?
    Olaf

    Reply
    • Larry says:
      November 28, 2022 at 3:46 pm

      Olaf:

      While is it easy – especially on a larger monitor – to have both Activity Monitor and one of the speed test applications open at the same time, the problem is that they don’t sample data rates the same way. BMD and AJA appear to sample in real-time. Activity Monitor samples every 6 – 10 seconds. I also don’t know if Activity Monitor is averaging data rates over that time or simply providing spot samples.

      I’ll see what I can find out from Apple.

      Larry

      Reply
      • olaf says:
        November 29, 2022 at 6:04 am

        Your are right Larry,
        Apple Monitor is so much off the TestApps.
        And interestingly my results show big differences to yours, too.
        I dropped you a mail with the graph.
        Olaf

        Reply
        • Larry says:
          November 29, 2022 at 6:11 am

          Olaf:

          Thanks for the chart. You showed speed differences, but not as large as mine. I should mention that I used OWC SoftRAID to create the SSD RAID, rather than Apple Disk Utility. The reason is that I also needed to test performance speeds of the Thunderblade using RAID 4 and RAID 10, which Disk Utility doesn’t support.

          Larry

          Reply
  5. Patrick DonEgan says:
    November 28, 2022 at 8:46 pm

    Have you attempted this same test on some other hard drives or RAID systems?

    Reply
  6. Jonah Lee Walker says:
    November 29, 2022 at 4:44 pm

    I ran it on my Thunderbay using Softraid 5, and it AJA showed 6.7% slower write, and 13.2 Slower on Read.

    Reply
    • Larry says:
      November 29, 2022 at 6:06 pm

      Jonah.

      Thanks for testing and reporting this.

      AJA is working with me to help figure out this problem. Due to my travel schedule this probably won’t get resolved until next week.

      I’ll keep everyone informed.

      Larry

      Reply
  7. Lee says:
    November 30, 2022 at 10:49 am

    Larry, take a look at AmorphousDiskMark. It is a close port of CrystalDiskMark.

    Reply
  8. Patrick DonEgan says:
    December 5, 2022 at 6:26 pm

    You are still on Catalina?
    Is that allowed?

    Reply
    • Larry says:
      December 5, 2022 at 7:15 pm

      Patrick:

      GREAT catch! Wrong screen shot. I did all these tests on macOS Monterey – 12.6.1.

      I replaced the screen shot with the correct one.

      My apologies for screwing this up.

      Larry

      Reply
  9. Larry Jordan has reported some huge test results differences between AJA Test Lite and Blackmagic Disk Speed Test – Jonah Lee Walker, Editor, Colorist and Motion Graphics says:
    December 16, 2022 at 9:00 am

    […] Larry is reporting huge speed measurement differences on a Blackmagic Thunderblade, with Aja showing…. […]

    Reply
  10. Eric says:
    January 25, 2023 at 8:01 am

    Hi Larry, any updates?

    Reply
    • Larry Jordan says:
      January 25, 2023 at 8:20 am

      Eric:

      Both BMD and AJA tech staff are looking at this. AJA has told me that AJA System Test Lite is incorrect.

      I’m waiting to hear more, but, so far, no other updates.

      Larry

      Reply
Newer Comments →

Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.

LarryJordan.com is owned by Axle.ai

2018-2024 © ALL Rights Reserved.