Posted on by Jean-Luc Aufranc (CNXSoft) - 10 Comments on Testing ntttcp as an iperf3 alternative in Windows 11 (and Linux)

Testing ntttcp as an iperf3 alternative in Windows 11 (and Linux)

ntttcp (Windows NT Test TCP) is a network benchmarking utility similar to iperf3 that works in both Windows and Linux written and recommended by Microsoft over iperf3, so we’ll test the alternative in this mini review.

iperf3 is a utility of choice for our reviews of single board computers and mini PCs running either Windows or Linux, but we’ve noticed that while Ethernet (up to 2.5GbE) usually performs just as well in Windows and Linux, WiFi is generally much faster in Ubuntu 22.04 than in Windows 11. So when XDA developers noticed a post by Microsoft saying iperf3 should not be used on Windows 11, it caught my attention.

Microsoft explains iperf3 should not be used in Windows for three main reasons:

  1. The maintainer of iperf – ESnet (Energy Sciences Network) – says “iperf3 is not officially supported on Windows, but iperf2 is. We recommend you use iperf2. Some people are using Cygwin to run iperf3 in Windows, but not all options will work”
  2. iPerf3 is Emulated on Windows – iPerf3 does not make Windows native API calls as it only knows how to make Linux/POSIX calls, and this may impact performance.
  3. I usually download iperf3 3.1.3 for Windows released in 2016, and Microsoft notes the one offered by ESnet (version 3.16)  is more recent but still 15 versions behind, so users are not running the latest version of the utilities.

So what’s the alternative for iperf3? Microsoft maintains two:

  • ntttcp (Windows NT Test TCP) open-source utility for Windows and Linux with a command line similar to iperf3 according to Microsoft in the sense it aims to isolate network stack throughput.
  • ctsTraffic for Windows-to-Windows testing only, also open-source and maintained on Github. ctsTraffic focuses on end-to-end goodput scenarios.

We can disqualify ctsTraffic immediately here at CNX Software since our tests typically involve a mix of Linux and Windows machines. Microsoft compared iperf3 to ntttcp utilities with high-speed network interfaces (10 GbE+), and the latter reports much higher performance. I only have hardware with 2.5GbE and WiFi 6, but I still wanted to test it, especially to check WiFi. So I decided to give ntttcp a try following the instructions on Microsoft Learn. It ended up being a challenge as those would not work on my system, and it took me a while to find the right parameters…

ntttcp vs iperf3 testbed

My testbed is comprised of four main components:

The Khadas Mind Premium was selected as it’s the only spare Windows system with 2.5GbE and WiFi 6 that I own and have already reviewed. You’ll notice the network tests in Windows 11 and Ubuntu 22.04 using iperf3 reveal much lower performance in the Microsoft OS as summarized in the table below.

WiFi 6 TxWiFi 6 Rx2.5GbE Tx2.5GbE Rx
WIndows 11 Home712 Mbps590 Mbps700 Mbps2.30 Gbps
Ubuntu 22.041.40 Gbps991 Mbps2.35 Gbps2.35 Gbps

The Khadas mini PC is an outlier when it comes to 2.5GbE upload performance, but WiFi is faster in Linux in all mini PC reviews we’ve done. Since the Mind Premium review was done a while ago (August 2023), I updated Windows to the latest version and drivers and tested networking performance again with iperf3 and ntttcp in Windows 11 Home using the same command line as in the Microsoft blog post.

The first step was to install ntttcp (Linux) on UP Xtreme i11 mini PC running Ubuntu 20.04:


We can run the receiver command as follows.


Only one core is used to emulate iperf3 and the V (verbose) options help a lot to troubleshoot issues. After downloading the ntttcp.exe binary to Windows we can run it immediately as a sender in a command prompt:


But as you can see from the log above, it did not quite work as expected. It turns out that for Windows to Linux tests, we need to use the “ns” (No Sync) parameter. It’s mentioned in the Microsoft’s blog post

There is a known interoperability limitation when testing between Windows and Linux. Details can be found in this ntttcp for Linux wiki article on GitHub.

It took me a couple of hours to find out, but once I did that, I could complete the WiFi 6 Tx (upload) test:


98.125 MB/s or about 785 Mbps, slightly better than with iperf3, but still far from the performance in Linux.

I then connected an Ethernet cable to test 2.5GbE upload:


88 MB/s or 704 Mbps, so it’s basically the same as with iperf3 even after having updated the drivers.

iperf3 has a reverse transfer option, but I could not see any such option on ntttcp. So I had to type the commands to run ntttcp.exe in receiver mode on Windows and ntttcp in sender mode on Linux. We’ll need to run CMD as administrator, open the firewall, and the networking benchmark tool with other parameters:


The Linux command for the Ubuntu sender is quite different than the same command for the Windows sender as parameters are different:


But I still managed it. Needless to say, I did have an overly positive view of ntttcp utility so far. The only benefit I see is that we have some extra data such as CPU usage during the transfer.  2.28 Gbps is about what we would expect for a 2.5GbE connection.

I usually disconnect the Ethernet cable to test WiFi 6 with iperf3 and run the same command. But here we also need to change the IP address on both the server and client side to test it again:


604 Mbps so there’s no improvement here.

WiFi 6 TxWiFi 6 Rx2.5GbE Tx2.5GbE Rx
iperf3551 Mbps608 Mbps736 Mbps2.30 Gbps
ntttcp785 Mbps604 Mbps704 Mbps2.28 Gbps

The table above summarizes the results after I ran iperf3 again. While ntttcp is faster for WiFi 6 upload, the results are not conclusive as the other results are more or less unchanged.  I suppose it only matters for high-speed networking with 10GbE or greater connections.

The tests above were done to compare ntttcp to iperf3 with similar parameters, but Microsoft says multithread and larger buffer sizes should be used to test bandwidth. Let’s try again with WiFi 6 download using 8 threads and 1024KB buffer size:


564 Mbps is slower than with only one thread and a 128KB buffer, although I reckon WiFi results can be fairly volatile.

Based on the tests done above there’s very little difference between iperf3 and ntttcp results, ntttcp Linux has not been updated for over three years, so I’m not convinced, and we’ll keep using iperf3 for the networking tests in reviews of Windows mini PCs…

Share this:
FacebookTwitterHacker NewsSlashdotRedditLinkedInPinterestFlipboardMeWeLineEmailShare

Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress

ROCK Pi 4C Plus

10 Replies to “Testing ntttcp as an iperf3 alternative in Windows 11 (and Linux)”

  1. What matters in any case is not to get *the* most accurate tool but a collection of different tools so that discrepancies can be spotted and investigated. Among the regular issues not tested by such tools by default is small packets (which happen with visioconfs for example), where it’s visible that all NICs are not equal at all.

    Reply

  2. Have you tried using iPerf with -w 4M or alternatively -P xx ?
    It allows you to change the window size or run parallel streams once you have more bandwidth available than the default settings are suitable for.

    Reply

    1. I haven’t tried, but considering there’s no issue with 2.5GbE (on most platforms), I’m not sure how increasing the buffer size may help with the slower WiFi on Windows vs Linux.

      Reply

      1. I find at home that my iperf3 test between windows clients gets ~ 2.3Gbps, I was surprised to see your results lower.

        However what I do see is that in general Windows does not raise it’s receive window high enough for latency even as low as 12ms. Linux will in that case and deliver the full 1Gpbs to one connection over the Internet.

        This is increasingly important as VPNs usually are single connections, and usually UDP. Which using a parallel tests can get windows to go the whole way. Too bad I cannot tell VPN clients to bond several connections.

        That brings up my next issue. I found this article as I’m searching for examples as the recent article stating that MS says don’t use iperf (I’m just going to shortly rant and say why don’t they fix it then? Apple seems to have their own version – iperf-darwin, iperf source is even hosted on github, they’ve got the talent and time, fork it) Getting back to the point, my testing so far with UDP is unreliable. It give an immediate result and the receiving side stays active.

        While this sub comment is getting off track, ntttcp is just a pale and inadequate replacement for iperf that is needlessly complicated. It’s good that there is a linux fork but iperf is everywhere. I can also use it on Android and IOS, I even asked a hardware vendor would they implement it.

        Reply

        1. I also get about 2.30 to 2.35 Gbps on most devices with iperf3 using 2.5GbE. It’s unclear why WiFi is slower in Windows compared to Linux, but as noted in the article, even using a parallel test with 8 threads does not improve the results

          Reply

          1. Forgive me, I should have been more specific with that comment. I mean the 2.5GbE TX numbers. Though my test devices are Server 2022 and Windows 10 pro, the numbers are usually very similar between the two systems. I do have a 2.5GbE USB adapter but I’m not sure if my NUC has USB 3 ports. If not, one day I’ll live boot my laptop and test.

            I’ve only seen parallel test help when the connections leave the local network and the problems are probably in the Internet connection’s router. My guess is that a lot of it has to do with the windows network congestion system, but it’s a bigger item than I can wrap my head around and I’m not a programmer.

            I’ve not tested via WiFi – with a full desktop, via mobile devices I do that all the time and if it’s 100Mbps, that’s good enough. I do have some .11AC/WiFi 6E USB adapters but for other reasons, testing is a bit difficult.

            I just wish I had the ear of MS or there was a VPN project that used multiple connections. Last night in testing with speedtest.net, a single connection got me around 150Mpbs upload, while multi got me 1072!

            Oh, and more on topic, I feel that I need to do more understanding ntttcp but my results between it and iperf 3.1.3 and even 3.16 were similar. I got my newer version from https://github.com/ar51an/iperf3-win-builds/releases I will try the build you linked in your other comment, 3.1.3 was from iperf.fr. Only one of my 2.5 targets are awake right now though and I’m remote to them.

          2. Results won’t differ with 2.5GbE between various tools.

            For the purpose of this test, we should ignore the low 2.5GbE Tx transfer rate on the Khadas Mind Premium because it’s a device-specific issue. I wanted to use another mini PC with fully working 2.5GbE in both directions, but the Mind Premium was the only device with Windows and WiFi 6 available to me at the time.

  4. After checking with Linixium, his own WiFi 6 tests show that ntttcp is faster than iperf 3.1.3 (downloaded from free.fr that typically shows up first in a web search), but iperf 3.16 (downloaded from https://files.budman.pw/) has the same performance as ntttcp.

    This could explain the difference in WiFi 6 throughput between Windows (iperf 3.1.3) and Linux (iperf 3.9 in Ubuntu 22.04).

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Khadas VIM4 SBC
Khadas VIM4 SBC