[ Updated Oct. 12, 2020, with details on how to connect to a server using 10G, while still accessing the internet using two Ethernet ports. ]
In this article, I will:
I’ve wanted to have the gear necessary to write this article for more than a year. I’m looking forward to sharing what I’ve learned.
The absolute fastest storage you can have is that which is directly connected to your computer. No shared storage comes close. As well, the least expensive storage you can have, on a per user basis, is also that which is directly connected.
If high performance is your goal, the best option is to direct-connect a RAID to your computer. Even a 4-drive RAID filled with spinning hard disk drives easily exceeds the speed of a 10 Gb connection to a server when comparing both read and write speeds. Replace the HDDs with SSDs and direct-connect speeds blow any Ethernet connected device out of the water.
If sharing files is more important than performance, the most cost-effective option is to connect the server to the switch via 10 Gb Ethernet, then have individual editors connect to the switch via 1 Gb Ethernet. This configuration supports up to 10 editors getting data at close to the maximum 100 MB/second speed of 1 Gb Ethernet, which is enough for most HD productions.
The fewer users on the network, the faster data transfer speeds will be, up to the maximum speed of the communications protocol.
There are also benefits to connecting a few key users to shared storage via 10 Gb Ethernet to take advantage of the extra speed. To do so, however, requires servers, switches, cables and computer interfaces that support both 10 Gb Ethernet and jumbo frames.
For many of us, though, the ability to quickly share files within a workgroup is an equally high priority. In fact, we are willing to trade off raw speed for the flexibility of file sharing. Sharing requires a server connected to each computer via Ethernet.
Ethernet was first “created in 1973 by a team at Xerox Corporation’s Palo Alto Research Center (Xerox PARC) in California. The team, led by American electrical engineer Robert Metcalfe, sought to create a technology that could connect many computers over long distances. Metcalfe later forged an alliance between Xerox, Digital Equipment Corporation, and Intel Corporation, creating a 10-megabit-per-second (Mbps) standard, which was ratified by the Institute of Electrical and Electronics Engineers (IEEE).” (Britannica.com)
“Ethernet networks have grown larger, faster, and more diverse since the standard first came about. Ethernet now has four standard speeds: 10 Mbps (10 Base-T), 100 Mbps (Fast Ethernet), 1,000 Mbps (Gigabit Ethernet), and 10,000 Mbps (10-Gigabit Ethernet). Each new standard does not make the older ones obsolete, however. An Ethernet controller runs at the speed of the slowest connected device, which is helpful when mixing old and new technology on the same network.” (Britannica.com)
I especially want to point out this phrase: “An Ethernet controller runs at the speed of the slowest connected device.” This will become important in a bit.
In order to share files, we need to use a server. To connect multiple computers to the server we need to use a switch. Switches can handle multiple users connecting at different speeds. If we want a user to connect at the fastest Ethernet speed available today, they need to use 10 Gb Ethernet. This either requires a computer with a built-in 10 Gb Ethernet connection or getting an adapter that converts Thunderbolt to 10 Gb Ethernet. In this article, I’m using an adapter.
While Ethernet is normally used in a shared environment, with multiple users, this article looks at the maximum speeds we can achieve using 10 Gb Ethernet as measured by one user. This allows me to better explain the best use cases for 10 Gb Ethernet for one user or a group.
THE PIPE VS. THE DATA
Ethernet, like FireWire, SCSI, USB and Thunderbolt is a protocol, a pipe, that carries data from one place to another. Each of these pipes has a maximum speed that it can support. However, the actual speed of the data inside the pipe is dependent upon both the maximum size of the pipe and the data transfer speeds of the devices attached to it.
For example, a spinning hard disk transfers data around 150 MB/s. However, if you connect that hard disk to your computer using USB 2, you’ll only get about 25 MB/s, because that is the speed limit of USB 2. Connect the same hard drive via FireWire 800 and data transfer increases to around 80 MB/s, again due to the speed limit of FireWire. Gigabit Ethernet maxes out, as we will see shortly, around 110 MB/sec.
In all these cases, the limiting factor on performance is the size of the pipe that carries the data.
NOTE: The theoretical maximum speed for 1 Gb Ethernet is 128 MB/s. However, general system overhead reduces that to 100 – 110 MB/s.
With Thunderbolt and 10 Gbps Ethernet, the pipe is finally larger than any single device connected to it. So, measuring performance now requires an understanding of the speed of the storage device as well as the size of the pipe.
NOTE: In the case of servers, speed also depends upon the number of users who are accessing that server at the same time. The more users, the slower the data transfer rate for each user.
THE MYTH OF BIGGER PIPES
We just learned that how devices are connected, such as using Thunderbolt, is a gating factor in determining speed. Another gating factor is the speed of the device itself.
NOTE: Here’s an article that compares the speed of internal, external and shared storage.
A third gating factor – as you will discover there are many factors that determine network speed – is the speed of the connection between your storage and the switch, and, again, between your computer and the switch.
NOTE: If you want to connect multiple devices so that you can share files, you need to use a switch. Some wireless networks act as switches allowing different computers to send files to printers, for example, but, most of the time, switches are at the heart of any network.
The way ALL networks work is that an end-user connects to a switch which then distributes data requests to various servers, printers, the internet or any other connected device.
When you need to pull data from the server, the request goes from your computer to the switch to the server, then back. Additionally, because servers are shared between multiple users, the more users accessing the server at the same time, the slower the data transfer. This is true for all shared storage.
Here’s are some examples:
Example 1 illustrates one user connected to a 1 Gb switch via 1 Gb Ethernet. The server is also attached to the switch via 1 Gb Ethernet. When a single user requests data, the switch, server and computer all transfer data at the full 1 Gb speed.
Example 2 adds a second user, also making a request for data via 1 Gb Ethernet. Now, both users talk to the switch at 1 Gb speeds, the switch feeds both requests to the server, but the server can only transfer data at a maximum of 1 Gb, because that’s the speed of its connection. This means each user only gets their data a 1/2 Gb speed, or, roughly, 50 MB/s, because half the server output is going to user 1, while the other half is going to user 2. The speed of the server’s connection to the switch becomes a gating factor for speed.
NOTE: This example assumes each user is asking for a large file and both are accessing the server at the same time.
Example 3 increases this access to four users. Now, the connection between the server and the switch becomes a real bottleneck, because each user now only gets data at 1/4 Gb speed, or, roughly, 25 MB/sec. Obviously, adding more users to the same server makes any kind of media editing very difficult as the data transfer rate changes constantly.
NOTE: These are idealized examples, but you get the picture.
The easiest way to fix this speed bottleneck is to provide a faster connection between the server and the switch. This can be done with port aggregation (another long article) or, related to this article, connecting the server to the switch via 10 Gb Ethernet; provided the switch supports it. Some do, most don’t.
Examples 4 – 6 illustrate the speed differences to the individual user by having a faster connection between the server and the switch. In this example, even with four users, every user is getting its full bandwidth of media. Why? Because the server can feed data to the switch faster than the combined data needs from all four users (10 Gb vs. 4 Gb).
The point of these examples is to illustrate that there are many ways of leveraging the speed of 10 Gb Ethernet. At a minimum, you need a high-speed connection between your server and the switch.
In an ideal world, we would get the best performance by having all switches, servers and computers support 10 Gb Ethernet. To achieve this, would require:
However, a full 10 Gb system is very expensive. Example 7 illustrates an intermediate step: high-speed connections from the server to the switch, then ordinary 1 Gb Ethernet from the switch to individual computers.
But, let’s do something even more basic. Examples 8 illustrates bypassing the switch and connecting the computer directly to the server. This allows me to see what kind of maximum speeds we can get with 10 Gb Ethernet. While this direct connection would not work in a shared environment, it removes any potential data slowdowns caused by the switch.
NOTE: Cables also make a difference. The high-speed of 10 Gb Ethernet requires cables specially constructed to support it. Those are rated as Cat 6A or 7. Lower-rated cables can carry data, but not at the full 10 Gb speed. 10 Gb Ethernet also requires all cable runs to be less than 100 meters.
(Apple 2018 Mac mini)
For the purpose of this tutorial, I’m connecting a Mac Mini (2018) to a Synology DS 1517+ server. The Synology has five drives, configured as a RAID 5. If that RAID were directly attached, I would expect at least 600 MB/sec in read/write speeds. I’m using the internal 1 Gb Ethernet port on the Mac mini to measure baseline speeds.
Because the Mac mini does not have a built-in 10 Gb port, I need to add some extra gear to jump to 10 Gb speeds.
(Synology DS 1517+ 5-bay server)
NOTE: The Synology SNV3500-400G SSD is not needed for 10 Gb Ethernet, but, as I wrote in this article, the addition of the SSD card made a big difference in the responsiveness of the server; especially in displaying directories.
SIDEBAR: OWC THUNDERBOLT 3 10G ETHERNET ADAPTER
(OWC Thunderbolt 3 10G Ethernet Adapter)
Just a quick word on the OWC Thunderbolt 3 10G Internet Adapter. Using it could not be easier: connect it to a Thunderbolt 3 port on your computer, then plug the Ethernet cable into the other end.
The unit instantly self-configures for: 10 / 100 Mbps Ethernet, as well as 2.5 / 5 / and 10 Gb Ethernet. It could not be easier to use. About the size of two decks of cards stacked on top of each other, it weighs a pound (16 oz / 453 grams) and, when operating runs warm to hot.
It is dead quiet and just works. Plug it in, then worry about something else.
PREPPING THE HARDWARE
To get the hardware ready:
NOTE: The Synology adapter was easy to install and worked without any hardware hassles. The server software, though, needs to be configured to work with the adapter, which I cover below.
Ethernet sends data in packets, called a “frame.” A normal Ethernet frame holds 64 bytes of data. For 1 Gb Ethernet or slower, this is fine. But as speeds get faster, performance improves as the size of the frame increases. This new, bigger frame is called a “jumbo” frame and is 9,000 bytes in size.
So, to configure a Mac for this faster, 10 G connection, here’s what we are about to do:
ON THE MAC
NOTE: If you have a built-in 10 Gb Ethernet connection on your Mac, select it, then skip to the second bullet.
NOTE: The largest a jumbo frame can be is 9,000 bytes. You can make it smaller, but there’s no benefit to doing so in most cases.
UPDATE – HIGH-SPEED PLUS THE INTERNET!
Because we now have a direct connection to the server, we lost our connection to the Internet. To fix this, we need to enable two Ethernet ports, then change the service order so that the system uses the high-speed port to connect to the server and the slower speed port to connect to the web.
NOTE: Internet connections are far, far, FAR below 1 Gb Ethernet speeds so the slower port won’t make any difference in your Internet connection.
After configuring the 10 Gb connection, click the Gear icon and select Set Service Order. A Mac always attempts to connect to something via the highest listed service. If that fails, it jumps down to the second, and so on.
In the Service Order window, drag the name of the 10 Gb connection to the top, then make sure the built-in Ethernet port is listed second.
With this configuration, you have a high-speed connection to the server, with full access to all other connected devices via the switch.
CONFIGURE THE SYNOLOGY SERVER
NOTE: The specific menus and screens will vary depending upon the server you are using. In my example, I’m using the Synology operating system.
NOTE: Resetting the MTU value requires a restart of network services. Also, jumbo frames must be turned on for all devices that connect to this port.
(Browse for servers using the server name, not the IP address.)
We now have configured two different Ethernet connections on the same server: the new 10 Gb connection, via the adapter card, plus the older 1 Gb connection built into the unit. For this reason, when you Browse for the server, look for the Server Name, rather than the IP address. This may make finding the server easier.
What we’ve just done is create a direct connection between the computer and the server. Because we are bypassing the switch, we lose access to any other connected devices or the web. As well, the Mac mini can not be discovered using Remote Desktop or similar software.
So, this is an ideal setup for testing speeds, however, as discussed earlier in this article, unless the switch supports 10 Gb Ethernet in some form, all connected devices are limited to 1 Gb Ethernet speeds, regardless of the speed of the server connection.
This section is the reason for all our work: Just how fast is all this new gear? Well, to set some initial expectations, here’s what I expect will happen:
As you’ll discover, some of my expectations are correct, but others aren’t.
1 Gb ETHERNET BASELINE
(The baseline speed of 1 Gb Ethernet. Tests done using AJA System Test Lite.)
Here’s the baseline speed with the Mac mini connected using its native 1 Gb Ethernet port through the switch to the server via 1 Gb Ethernet. This is right in-line with the maximum speed 1 Gb Ethernet can provide.
(Test results using Blackmagic Disk Speed Test.)
As you can see, 1 Gb speed is fast enough for some media formats up to 1080p. But larger codecs, including HDR media, require more speed. This is also a good example of the efficiency of ProRes for video editing.
(10 Gb Ethernet speed using the default Mac settings; jumbo frames are turned off.)
Changing the connection to 10 Gb Ethernet but NOT changing the default jumbo frame settings roughly tripled the speed. In other words, if you just connect the OWC unit and keep all the defaults, this is the speed you will get.
(10 Gb Ethernet speed with jumbo frames enabled and set to 9000.)
But, enabling jumbo frame settings REALLY boosted read speeds, while write speeds were triple that of 1 Gb Ethernet. This is far faster than I was expecting the server to deliver data!
NOTE: For video editing, slower write speeds are OK, because rendering and exporting are based upon how fast the CPU can calculate each frame. In general, I’ve found write speeds during export to range between 50-80 MB/second. So slower write speeds are not a big concern in terms of overall editing performance.
When two computers access the server at the same time, 10 Gb speeds slow dramatically, though 1 GB speeds remain about as fast as ever. The top image is the 10 Gb connection, the bottom image is the 1 Gb connection from a second computer running at the same time.
NOTE: The second computer is connected through the switch, while the 10 Gb computer is directly connected to the server. I’m wondering if the speed drop is caused by combining jumbo and non-jumbo frames. I plan to spend more time optimizing the 10 Gb connection. This speed drop may be excessive. I need to do more homework.
The main reason for the slowdown, as we see here, is the wide variation in access speed caused by the drives inside the RAID hustling to read and write data for both systems accessing the server at the same time. In other words, the drives can’t devote themselves to fulfilling just one data request at a time.
NOTE: This would not be an issue using a server containing SSDs, as SSDs contain no moving parts and would respond more quickly to all requests. However, SSDs don’t hold nearly as much as HDDs.
One VERY interesting thing I learned was when I replaced the high-speed Cat6A cable with Cat5e cable. Theoretically, Cat5e should show more errors and a slower speed. What I discovered, though, was that it delivered the same throughput during the test as the Cat6A cable.
The lesson here is that, while it is always a good idea to use cable rated for the speed you need, in a pinch, you could use Cat5e cable without significant degradation; at least for short distances and the short term. I still plan to use Cat6A cable – just to be safe.
BIG NOTE: Both cable runs were 50 feet in length. 10 Gb speeds require cable runs shorter than 100 meters. 1 Gb supports cable runs up to 300 meters. Longer cable runs force speed to decrease.
I learned a lot doing this exercise. First, I’m going to keep all my 10 Gb gear and continue to look for ways to optimize speeds. When transferring hundreds of gigabytes, the faster speed will make a big difference.
However, I also need to learn how to fully integrate both 10 Gb with 1 Gb Ethernet across all my devices. There is always more to learn. Still, 10 Gb Ethernet is not a panacea. It is a tool, which has benefits, but you also need to understand the limitations.
Final Cut Pro X 10.4
Edit smarter with Larry’s brand-new webinars, all available in our store.