Why Is My Hard Disk So Slow?

Yesterday, I was under a deadline and needed to copy a 700 GB folder containing about 21,000 files from one hard drive to another. The first drive was a RAID, the second was a single drive.

The transfer started out fine, averaging about 100 MB/second, as measured by Apple’s Activity Monitor. The Finder reported the transfer would take “about 2 hours.”

However, an hour later, less than 200 GB of data had been transferred and Activity Monitor showed a transfer rate around 20 MB/second. Another hour later and the data rate had dropped below 10 MB/second.

What’s going on here?

FREE SPACE

First, when dealing with traditional hard disks, often called “spinning media,” the more free space it has, the faster it goes. Or, conversely, data transfer rates slow down as the disk fills up.

This is because, as a hard disk fills, the heads need to spend more time seeking out open places to write data. As the number of open places fills up, the heads spend more and more time trying to figure out where to store your stuff.

As a general rule, try to keep around 20% free space on each hard disk. To find out how much free space you have on your hard disk, select the drive in the Finder and type Cmd+I. Free space is listed near the top next to “Available.”

NOTE: When a hard disk is empty, it writes data to the outside edges first. These are both the fastest to access and hold the most data. As a disk fills, in addition to more seek time, data gets stored closer to the center, which takes longer for the heads to get there and holds less than the outer edges.

IT AIN’T JUST ONE THING

File transfer speed is actually dependent upon several factors:

• How big the files are (big files transfer faster than smaller files)
• How empty the hard drive is
• Which protocol is used to connect the drive (Thunderbolt, USB 3, FireWire, USB 2)
• The physical health of the drive
• Whether the storage uses spinning media or SSD
• Whether your computer is busy doing multiple things
• Whether you are using Finder or a separate utility to transfer files
• (And probably more that I’m forgetting at the moment)

IOPS MATTER

I sent a note to Larry O’Connor, CEO of OWC, the maker of the RAID, asking his advice. He pointed out that when we work with large media files the data transfer rate, called “bandwidth,” is most important. This is the transfer speed of moving a file from one drive to another and is generally measured in MB/s (MegaBytes per second).

However, smaller files require less bandwidth and more “IOPS” (Input/Output operations per second). Wikipedia states: “In isolation, IOPS can be considered analogous to “revolutions per minute” of an automobile engine i.e. an engine capable of spinning at 10,000 RPMs with it’s transmission in neutral…” However, to “meaningfully describe the performance characteristics of any storage device, it is necessary to specify a minimum of three metrics simultaneously, IOPS, response time, and (application) workload.”

As Mr. O’Connor writes: “Small files are what bog down transfers because they have the same kind of I/O load as a large contiguous file but only might be 4K or 120K in file size. Also, there is load between the OS and the two drives too – it’s read-then-write; it isn’t really a simultaneous process.”

“Small files can transfer VERY, VERY, VERY slowly – hard drives can do about 350-500 IOP/s vs an SSD that can do tens of thousands of IOP/s…. That’s why an SSD in real world can be up to 100X faster than an HD and also why peak SSD speed is pretty pointless,” because the increase in IOPs has a bigger impact on performance than pure data transfer speed.

Most of the time, when we transfer a small file, we are only transferring a few of them at a time and the slower performance isn’t really noticeable. But, when we are transferring thousands of smaller files, the drop in performance can be significant.

Mr. O’Connor adds: That data transfer speeds vary is “true for any media. An important aspect, also, is that there is correlation to the weakest link. Typically on consumer and prosumer systems you max out on operating system overhead way before you max the IOPS on today’s SSDs

“[Keep in mind that] peak performance is wonderful – but the majority of real time work is well under 500 mb/s and how well a drive sustains under real load is why a ‘slower’ peak drive (whether IOPS or max throughput) actually gets [the job] done better

“It’s kind of like racing – [a car can be] tuned to qualify and take the pole or tuned to come out tops in the actual real race. It’s rare that the pole setter is also the race winner.”

GET SMART

All drives today ship with diagnostic sensors on them that allows utility software – such as Disk Utility, SoftRAID, TechTool Pro and others to monitor the hardware status of your drive.

S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology) is a monitoring system built into both hard drives and SSD drives that detects and reports on various indicators of drive reliability, with the intent of anticipating hardware failures.

Wikipedia states: “Hard disk failures fall into one of two basic classes:

• Predictable failures, resulting from slow processes such as mechanical wear and gradual degradation of storage surfaces. Monitoring can determine when such failures are becoming more likely.
• Unpredictable failures, happening without warning and ranging from electronic components becoming defective to a sudden mechanical failure (which may be related to improper handling).”

You can check the hardware status of all your drives using Disk Utility. Simply open Disk Utility, select the drive you want to check on the left, and check the SMART status in the lower-center of the window.

NOTE: Mr. O’Connor states that SMART tests don’t always work reliably for USB 3 drives.

SUMMARY

Transferring big files is easy. Transferring a large number of files of different sizes to a hard drive that’s mostly full can take a lot longer than you expect.

SSDs can solve many of these problems, but they are still about ten times more expensive than spinning hard disks that hold the same amount of data. Hopefully, SSD prices will continue to come down over time.

In the meantime, keep an eye on your drives. Make sure they have adequate free space. Check their SMART status every month of two using Disk Utility. And, when you buy new storage, take a look at the IOPS specs as well as total storage.

The more you know, the better your storage will perform.

---

Bookmark the permalink.