Testing 10 Gigabit Ethernet Over Copper on a Shoestring Budget

Testing 10 Gigabit Ethernet Over Copper on a Shoestring Budget

Written by Don Schultz, trueCABLE Senior Technical Advisor, Fluke Networks Copper/Fiber CCTT, BICSI INST1, INSTC, INSTF Certified

Are you ready to take the plunge into 10 Gigabit networking?

If yes, the next question is often “How do I test this stuff?” After all, you spent all of this money on new network gear and want to know you got your money’s worth. This is where everything gets quite hairy. Not just hairy, but off-the-rails, train-wreck-style hairy.

You see, while the rest of the technology world has been seeing price reductions there is one corner of it that will still cost dearly: test equipment. Secondarily, there is a distinct lack of information out there about how to test this technology to make sure it’s performing as it should. Up until now, this area has been the purview of professional network engineers and for the most part, they are not talking. At the very least, they have not documented their trials and tribulations for the rest of us! My goal is to help change that, so please keep reading.

The main focus of this article is how to test 10 Gigabit Ethernet without having to do something illegal, just to afford the test equipment. 

First, A Little Background

10 Gigabit Ethernet, or 10GBASE-T, is typically the fastest Ethernet link technology found outside of large data centers and still not common. This technology has been around since 2004 (ANSI/TIA ratified in 2008). Extremely high costs kept it out of reach for all but the wealthiest individuals and smaller businesses.

Until now, that is. 10 Gigabit over copper-twisted-pair Ethernet cable is now becoming less expensive to deploy. Companies like Aquantia, amongst others, are making this happen. Aquantia is one of the firms making microchips and Network Interface Cards (NICs) that enable this technology at a lower cost. These newer microchips add additional perks like support for 2.5GBASE-T and 5GBASE-T, which is another great boon to Local Area Networks. 

See more here about Just What is NBASE-T. The cabling costs have never been the primary roadblock to 10 Gigabit Ethernet. It is the cost of the associated network infrastructure like switches that impart “sticker shock.” Stringent installation requirements don’t help either, but that is for another article.

Cat6A Ethernet cable is readily available and less expensive 10GBASE-T compatible switches are making a debut. Reasonable access to a true 10 Gigabit compatible network is now becoming a reality. Even folks with existing Cat6 Ethernet cable can get into the game as long as they obey the distance limits as detailed in The Difference Between Cat6 vs Cat6A Ethernet Cable.

How can 10 Gigabit networking benefit the so-called “Average Joe”?

Good question. The answer is that it might have an immediate benefit in the Local Area Network (LAN) due to file sharing needs internally, like video editing

The article The Need for Speed dives deeply into where 10GBASE-T might come in handy. No, this technology won’t make your Internet connection any faster, but paying a large amount of money to your Internet Service Provider (ISP) every month might do the trick!

How to get from Point A to Point B: Testing

This white paper is going to deep dive into some key areas such as:

  • What Ethernet testers are commonly available.
  • What the Ethernet tester can really do, and what the costs are.
  • Certification versus verification.
  • Verification and validation of 10 Gigabit Ethernet on a budget.

What is all of this about test equipment?

Isn’t the typical “blinky” tester you have used all along good enough? Only as a bare basic starting point. The typical low-cost physical layer “blinky” tester is only going to confirm whether there are any shorts and miswires. In ANSI/TIA jargon, this is called a Wire Map. This is not enough to find out if the cable will actually perform to 10 Gigabit speed. In fact, this small basic tester is not enough to tell you if your cable can reach any speed. Now, the tester on the right is one where the price tag might induce immediate convulsive reactions!

blinky tester and high-end Ethernet Certifier

The typical “blinky” tester (left)...not even close to what is needed, but is a start point . The Fluke Certifier (right) costs $7,000 and up.

Let’s talk money.

Ethernet Tester Price ScalingTo the right is a chart I put together which details the current estimated market price point for each type of tester by capability (very generalized).

Consider each tester type on the bottom axis the mere starting point for each general type of tester. There are many variations and brands of testers, but the main takeaway here is the costs range from $10.00 to more than $8,000.00 depending upon additional accessories.

Given the price scaling chart above, one can quickly see that the actual Certification of 10 Gigabit Ethernet to the ANSI/TIA 568 specification will cost quite a bit of money. In fact, unless you specifically install and/or Certify Ethernet cable for a living, it makes little sense to buy one.

That all said, purchasing the Ethernet Certifier might make sense for some. Scenario: You are a contractor that regularly installs and tests Ethernet cables from 1 Gigabit speed to 10 Gigabit speed. Your clients, for various reasons, demand ANSI/TIA Certification results and a written warranty of the Ethernet link speed capability that you guarantee. In this scenario, you have no choice and the path is clear...get out the credit card and try not to wince.

For all other scenarios, there are a few options that don’t require a second mortgage:

  • Rent a high-end Ethernet Certifier instead of purchasing one. This is not a simple device; the user will require training. Ethernet installers who perform a 10 Gigabit installation from time to time are likely to use this option.
  • Borrow a high-end Ethernet Certifier from a friend, especially if this friend owes you a favor. You might get them to do the actual tests too, and you won’t need extensive training. If you have a friend that owns such a tool, consider yourself lucky!
  • Pay a cable contractor to Certify your Ethernet installation (your best option if you are not technically savvy).
  • Hybrid testing/verification. If you have two modern and powerful desktop PCs along with a less capable tester device you can confirm your speeds and spend a fraction of the money. This method is commonly used by folks who are not only proficient with PCs but desire a true DIY approach. I strongly recommend this approach for highly technical and motivated individuals.

You may have noticed that I keep using a capital “C” for Certify. Indeed, and here is why...

Ethernet cable Certification vs. verification.

The word “Certify” means something in the world of Ethernet cable installation and testing. It involves a test device that measures many aspects of your cable according to the ANSI/TIA 568 (or applicable) specification to achieve a specific Category rating. In order to be called a Certifier, this test device must measure required areas via field testing criteria, and use sine wave technology in the process. The ANSI/ TIA organization sets the standards for the industry and not only dictates how cables are to behave in order to achieve a Category rating such as Cat5e, Cat6, or Cat6A, but also physical compatibility and even best practices for how the cable is to be run and tested.

The word “Certify” means something in the world of Ethernet cable installation and testing.

 An Ethernet cable Certifier has an internal database of the required ANSI/TIA performance metrics that it uses to perform tests. The device must also be able to store test results and have updatable firmware for when the standards change. This drives up the complexity and price of the tester. Such a device will also export test results in a printable format for the required documentation.

Ethernet cable testers that call themselves testers or verifiers may not measure the entire required range of ANSI/TIA performance metrics in order to speed Certify an Ethernet cable. Some of the fancy ones will generate a quasi-Certification report, and actually test some of the field parameters required, but use digital signals to accomplish the task. This should NOT be confused with a true ANSI/TIA Certification result. Prices for these better verifiers range from $350.00 to $2,500.00. They are a great option for installers who regularly need to test 1 Gigabit installations, and would like to generate a professional test report. This only works for customers who are not demanding actual Certification.

What are some of these elusive items that are measured?

The metrics in the table below are for Category 6 field Certification. So now you know what you get in a tester at a given price. Take notice of how rapidly the price increases to add just one additional feature.

 r Category 6 field Certification chart

The lowest price of testers are under $100 (with or without the tone trace tool).

Next, you have testers/verifiers that cost less than $2,500. These devices fall into the quasi-Certifier classification, especially if they can store and print reports. These devices make use of digital testing, but really do check the cables and terminations well enough for installers and customers alike. Many cable installers use these devices to provide proof to their customers that the cable works as it should, but it won’t fly with a customer who demands actual pedigreed Certification. 

Lastly, we have items that cost over $2,500. These are true Certifiers that will pedigree each link. Names are taken. Dates are recorded. Records are stored. Sine waves are used. Warranties might get exercised.

 

 

Ok, So how much testing is enough?

Here you are, and you spent less than $1,500 on your cable, keystone jacks, RJ45 plugs, network equipment, and associated necessary installation tools. Now, you are being told in order to guarantee that you can reach 10 Gigabit speeds you might need to spend nearly 6X that just for the tester? That is a hard sell. The expensive test equipment is complex to operate. Costly specialized training classes are on offer too, where you can also pay to obtain Certification for the person using the Certifier. See where this is going? The fact is, most DIY individuals are not going to have the money to go the “full monte”, and don’t have a customer demanding that they have to.

 

My Solution: Hybridized Verification

 

Disclaimer: This solution was specifically developed by me to address the DIY crowd. After extensive research, testing, and basic trial and error this is the most effective way of getting from Point A to Point B, if you are a tech-savvy hands-on type of person. This method will not be accepted as a legitimately documented installation for a customer that demands actual speed Certification to ANSI/TIA 568 specifications for warranty and legal purposes.

Step 1

Purchase or rent a 1 Gigabit rated quasi-Certifier. If you wish to go the Fluke Networks (Fluke Networks CIQ-100 Copper Qualification Tester) route, their 1 Gigabit quasi-Certifiers will run about $1,250 if you wish to purchase it directly. A far less expensive route is the Triplett Byte Brothers Real World Certifier 2 (~$305).

Step 2

Perform the Ethernet cable verification test for each and every run. They should all rate at Category 6, and show bandwidth exceeding 1 Gigabit (1000Mb). If you don’t get past this point with a PASS result, then something is likely wrong with the run or terminations. For more information on troubleshooting, please see Basic Network Cable Troubleshooting: What Did I Do Wrong?

Alternative to Step 1 and Step 2

Use your existing “blinky” tester if that is all you have. You will have much less insight into cable length and less upfront warning about potential issues such as “on the edge” terminations that might cause a problem reaching 10 Gigabit. One item that your “blinky” tester really needs to have is the additional light for “G” or Ground, or “S” for Shield. They mean the same thing...a continuous path to ground is “good”.

That additional light is critical when testing out shielded cable. A functioning cable shield is critically important to avoid issues when installing shielded Ethernet cable. The Shielded vs Unshielded Cable blog I wrote will explain why.

Step 3

For this step, you will need two relatively modern desktop PCs with multi-core processors. I recommend a minimum of an Intel-based Core i5 desktop produced within the last three years. On the AMD side, I consider the minimum a first-generation Ryzen 5 series CPU, such as the Ryzen 5 1600. Both computers need to be equipped with a 10GBASE-T over copper Ethernet NIC card, installed into a PCI-Express 2.0 or higher X4 lane slots. Since you are already setting up for a 10 Gigabit network, then you likely have these network cards, and probably the PCs too. This could also be a great excuse to build a new PC! Take heed that the Intel CPU has a built-in GPU (graphics) so you can get away without a separate video card in that setup. The AMD machine will require a separate graphics card unless you purchase a “G” variation of the first generation or second-generation Zen+ integrated processor, called an APU (Accelerated Processing Unit) such as the AMD Ryzen 5 2400G.

Backup all data on any existing Microsoft Windows installation (just to be safe), because in the next step we won’t be using Windows at all. We will use the Ubuntu Linux Desktop x64 version. The latest supported version will work out of the box with Aquantia 10GBASE-T Ethernet NICs.

You may set up your computer to dual-boot Windows and the Ubuntu Linux installation right from the Ubuntu installer. This is easy to do, and easy to un-do if you need to down the road. Don’t stress about using Ubuntu Linux Desktop, it is actually quite easy to use and I will guide you through it.

For those who simply do not want to use Ubuntu Linux to perform this testing, and would like to try a “first stop” with Windows, then there is a software tool called Throughput Test that I utilize. It is free to download and use. It is aimed at measuring Wi-Fi throughput but will work for this type of testing. Both computers need to be loaded with this software and connected by Cat6A or higher cables to 10 Gigabit ports on your switch. I have tested this tool with the exact same hardware used for the testing performed under Linux. The Windows 10 results were as expected. Windows networking is simply not up to the task of 10 Gigabit speed testing with any degree of accuracy. If the test had been conducted with the “client” hitting multiple servers at once (more than a single TCP stream) than results might have been different. Here are those results, recorded from the “client” side PC. Both computers are fully documented in Addendum A.

Well, this is simply just not what we want to see. Both desktop PCs are indeed connected at 10 Gigabit to the switch, and this test is from RAM to RAM from the “client” to the “server” and back. All bottlenecks have been removed, especially considering how powerful the PCs are. As you can see, a steady-state download speed of 6.6 Gb/s is achievable, with rare spikes to 8.8 Gbp/s. Not good enough for me.

Good enough for you? I suspect you just said “NO”. Ok, back to the drawing board.

Now, for those more adventurous and OCD types. Grab a bootable USB copy of the latest version of Ubuntu Linux Desktop x64. You will need a tool to “burn” the bootable ISO copy onto a USB drive. Instructions for this are found in a well-written tutorial found here

Install this OS on both PCs, making sure you boot to your USB drive to get the installation underway. Many motherboards have an option button (F10, F11, etc.) on the screen at boot that allows you to select a boot device. If you are not presented with this option, or if you don’t see the option at boot you should consult your motherboard manual or ask for help. You may “nuke and pave” which is a tongue-in-cheek term used by technical people to describe the total removal of the previous OS (Windows, for example) and replacement with Linux. Secondarily, the Linux installer allows a dual boot configuration so you can choose between Windows or Linux upon the reboot of your computer. The final way is to run a “Live Preview” of Linux by installing it into your computer’s temporary memory. It is designed for those who wish to make no alterations to their PC and are “wiped” upon the reboot of your computer. I did not test this third method.

Step By Step How-to for iPerf3

Perform the following steps for both PCs.

Upon entering your brand new Linux desktop, the very first thing you need to do is update the OS as their will likely be quite a bit of updates. If you wait about 15 minutes Linux itself will present you with an update screen, or you can force the update via Terminal (and issue commands by command line).

If you wish to force the updates to happen faster, you can bring up the Terminal console by pressing CTRL+ALT+T. Another easy way is to simply use the mouse and right-click on the desktop, and select Open Terminal at the bottom. This will present you with a terminal screen, like below:

 

Now, issue the command (warning, Linux is case sensitive!): sudo apt update

The system will ask you for a password if you supplied one during installation. A lot of text will flash by, letting you know Linux is checking the update repositories. This does not perform the update yet.

When the update check is finished, it will return you to a command line like above. Now issue the command: sudo apt upgrade

The system will not ask you for a password this time. Your password is good until you close the Terminal. Again, a lot of command line stuff will fly by. In the end, it will ask you if you want to perform the upgrade, and the default is Y for yes, so simply press ENTER.

After all updates are complete you should reboot. Issue the command: sudo reboot

Upon reboot and login you need to find out what computer you want to use as the “server” and which will be the “client”. This is important because you need to find out the IP address of the “server”. The tool that is required, called iperf3, is likely already installed but can be installed manually by using Terminal to issue the following command: sudo apt install iperf3

If the program is not already installed, it will install it. Leave your Terminal open, as you will need it. It might ask for your password, so input it if necessary.

Now for decision time...which machine will serve as the “server”? Hint: it doesn’t really matter. Flip a coin. The important thing is to grab the IP address off of that “server” so you can use it when running iperf3 on the “client”.

Now, using the much prettier graphical user interface (GUI), find your IP address of the Aquantia Ethernet Adapter. This is not only to find out what your IP address is, but it is also the first part of our software test.

 


In the picture to the left, you navigate to settings (Gear Icon), select Network, and open the properties for the Wired connection that applies. In this screenshot, it shows a 1 Gigabit connection, which is NOT what you want. Find the wired connection that shows 10,000 Mb/s (10 Gigabit). Once you locate the correct wired connection, you can obtain the IPv4 Address. It will likely be 192.168.X.X (X changes from network to network). You now have your “server” IP address. Oh, it’s a good sign the NICs connect up at 10 Gigabit speeds.

 

 



Ready for the real 10 Gigabit test? Here you go:

On the “server”, where the Terminal screen is already open, issue the command: sudo iperf3 -s

The program will launch and go into “listen” mode. You are done here.

On the “client”, open a Terminal screen as you have been doing all along. Issue the command: sudo iperf3 -c 192.168.X.X

Once you press enter (remember, those X’s are specific to your setup), the test will run for 10 seconds and you will be presented with the results. You now know what your link is capable of! Any errors found will also show up. Here is a screenshot of how you WANT this to look:

Wait a minute. It is only showing 9.42 Gbits/sec. Shouldn’t we get 10? Sorry folks. Due to TCP/IP and Ethernet Protocol overhead, this is a normal result. Under “Retr” it says “0” which is very good--no errors. Reissue the command a few times to make sure you are satisfied.

There you have it! You sir or madam have now been verified.

If things didn’t go well, and you did not see the above results (or something better) then you have an issue somewhere. It could be the Ethernet terminations, or something else, like a PC. It could be just about anything, and you will need to peel back every possibility. That is the nature of the game, unfortunately. Remember, we are doing this on a “shoestring budget”.

There is never, ever, a free lunch. Not in this business. Happy networking!

 

 

 

 

Addendum A

Test PC Hardware Configuration:

 Test PC Configuration table

Test PC Software Configuration (both machines):

  • Windows 10 version 1903, latest updates and patches.
  • Aquantia adapters using the latest driver updates found on manufacturer website Motherboard BIOS versions are the latest as of this writing, AGESA ABB for the Gigabyte X570 board.
  • XMP Profile enabled on both machines for maximum XMP profile RAM performance.
  • For Linux testing, version 19.04 Ubuntu Desktop x64 was used (Disco Dingo). No additional drivers compiled. All latest updates from the Ubuntu Repositories applied.

Switch used for testing:

Netgear GS110MX unmanaged switch with 2 x 10 Gigabit ports and 8 x 1 Gigabit ports.

 

trueCABLE presents the information on our website, including the “Cable Academy” blog and live chat support, as a service to our customers and other visitors to our website subject to our website terms and conditions. While the information on this website is about data networking and electrical issues, it is not professional advice and any reliance on such material is at your own risk.

    1 out of ...