FreeBSD on the POWER8: it's alive!

A post to freebsd-ppc from a couple of months ago asked if we had support for POWER8 and offered to provide remote access to anyone interested in working on it. I was sufficiently intrigued that I approached the FreeBSD powerpc hackers to ask about it, and was informed that it'd be nice, but we didn't have hardware.

After a bit of wrangling of hardware logistics and with the FreeBSD Foundation purchasing a box, a Tyan POWER8 evaluation server appeared. Nathan Whitehorn started poking at it and managed to get a basic "hello world" going, but stalled on issues with the Linux KVM virtualisation environment.

Fast forward a few weeks - he's figured out the KVM issues, their lack of support for some mandated hypervisor APIs and other bugs - FreeBSD now boots inside of the hypervisor environment and seems stable enough to do development on.

He then found the existing powerpc pmap (physical memory management) code wasn't very SMP friendly - it works fine on one and two CPU powerpc machines, but this POWER8 evaluation board is a 4-core, 32-thread CPU. So a few days of development went by and he rewrote most of the pmap code to be much more fine grained locked and scale much, much better than the existing code. (He also found the PS3 hypervisor layer isn't thread-safe.)

What's been done thus far?

• FreeBSD boots inside the hypervisor environment;
• Virtualised console, networking and storage all work;
• (in progress) new, scalable pmap implementation;
• Initial support for the Vector-Scalar Extension (VSX) that's found on POWER7 and POWER8.

So, I'm impressed. Nathan's done a fantastic job bringing the whole thing up. There's some further work on the new powerpc technology that needs doing (things like the new vector processing units, performance counter support and such) and I'm sure Justin and Nathan will poke powerpc dtrace support into further good shape. I'm going to see if we can fix a chelsio 40G NIC into one of these and work with their developers to fix any endian/busdma issues that creep up, and then do some network stack scaling testing with it. There's also the missing hardware/hypervisor support to run FreeBSD on the bare metal, which would be a fantastic achievement.

Now I kind of want some larger POWER8 hardware.

TDMA (somewhat) working on AR9380 chips

(Wow, I have a lot of posts to write to catch up on things.)

I've just brought up FreeBSD's TDMA support on the AR9380 chipset. Specifically, the AR9331, since I have a Carambola 2 on me today.

It was pretty simple to bring up - I was missing the beacon configuration HAL call that the TDMA code expected. It's only used by the TDMA code - the STA and AP modes rely on the normal HAL beacon methods that date back to the Atheros HAL.

The only problem - it seems something is up with ANI (noise immunity) and sensitivity on at least the AR9331. It doesn't seem to behave well on slightly loaded channels and thus the beacons don't always go out when they're supposed to.

But, if you've been wanting to play with TDMA on the later Atheros chips, now you can!

On profiling HTTP, or "god damnit people, why are all the open source tools slow?"

Something that's been a challenge at work (and at other things in the past) has been "how do I generate enough traffic to test this thing?"

If you're running some public facing boxes then sure, you can do A/B testing. But what if you're not able to test it in the real world? What if you need to do testing before you ship, and the traffic levels have to be stupid high?

So, what do you do?

I've done this a few times. When doing squid and other reverse proxy development, I would run tools like apachebench, httperf, even web polygraph - but these things scaled poorly. They didn't handle tens of thousands of concurrent connections and scale to both slow and fast clients - their use of poll() and select() just wouldn't work out well.

Something I did at Netflix was to start building TCP testing tools that more than 65,000 concurrent sockets. My aim is much higher, but one has to start somewhere. I was testing out the network stack rather than specifically doing HTTP testing. Here at my current job, I'm much more interested in real HTTP and all processing.

I looked at what's out there, and it's not very pretty. I need to be able to do 10G of traffic, looking upwards towards 20G and 40G of HTTP in the future. After a little more digging into what was out there - and finding httperf actually reverted my changes to use libevent and went back to poll/select! - I decided it was about time I just started writing something minimal to stress test things and build upon it as the need arose. I want something that eventually ends up like web polygraph - multiple client/server sets with different URL choices from a pool, a variety of client IP addresses, and other things like how often to make the requests and other request pacing.

So, I grabbed libevent, libevhtp from Mark Ellzey and threw them together. It turned out okish - libevent/libevhtp still does a bunch of memcpy()'ing inside the buffer management routines that makes 40G on one box infeasible at the moment, but it's good enough to get a few gigabit of client traffic on one core. There were some hiccups which I'll cover below, but it's good enough to build upon.

What did I learn?

• Well, it turns out the client code in libevhtp was a bit immature. Mark and I talked a bit about it on IRC and then I found there was an outstanding pull request that found and fixed a bunch of these. So, my code has turned into another thing - a libevhtp client and server test suite.
• The libevhtp threading model is fine for a couple of CPUs, but it's the standardish *NIX model of "one thread does accept, farms work off to other threads." So it's not going to scale well at high request rates to multiple CPUs. That's cool; that's what the FreeBSD-HEAD RSS work is for.
•  There's memcpy()'ing in the libevhtp body handling code. It's not a big deal at 1G, but at 10G it's definitely noticeable. I've spoken to Mark about it.

But, it's a good starting point. Once the rest of the bugs get shaken out, it'll be a good high throughput HTTP traffic tester.

What would I do next, after the bugs?

• the server will eventually grow the ability to generate controllable sized responses. That way the client can control how big a response to send and thus can create a mix of requests/replies.
• .. and HTTP request body testing would be nice.
• The client side needs to grow the ability to create client pools, like web polygraph, where certain subsets of clients get certain behaviours (like a pool of IPs to use, separate pool of URLs to fetch from, the time between each HTTP request, etc.)

The other trick is how to simulate lots (and I do mean lots) of IP addresses. I don't want to create separate loopback connections for each - that would be crazy. Instead, it'd be good to use the transparent interception support in FreeBSD IPFW that allows both connections from and connections to arbitrary IP addresses. A little trickery with IP routing so we don't need more than 1 ARP entry for each server and voila!

Oh, and the code?

https://github.com/erikarn/libevhtp-http/

... mmm emulators.

I occasionally get asked to test out FreeBSD/MIPS patches for people, as they don't have physical hardware present. I can understand that - the hardware is cheap and plentiful, but not everyone wants to have a spare access point around just to test out MIPS changes on.

However QEMU does a pretty good job of emulating MIPS if you're just testing out non-hardware patches. There's even instructions on the FreeBSD wiki for how to do this! So I decided to teach my wifi build system about the various QEMU MIPS emulator targets so it can spit out a kernel and mfsroot to use for QEMU.

Thus:

https://github.com/freebsd/freebsd-wifi-build/wiki/MipsQemuEmulatorImages

It turns out that it wasn't all that hard. The main trick was to use qemu-devel, not qemu. There are bugs in the non-development QEMU branch that mean it works great for Linux but not FreeBSD.

The kernel configurations in FreeBSD had bitrotted a little bit (they were missing the random device, for example) but besides that the build, install and QEMU startup just worked. I now have FreeBSD/MIPS of each variety (32 bit, 64 bit, Little-Endian, Big-Endian) running under QEMU and building FreeBSD-HEAD as a basic test.

Next is figuring out how to build gdb to target each of the above and have it speak to the QEMU GDB stub. That should make it very easy to do MIPS platform debugging.

I also hear rumours about this stuff working somewhat for ARM and PPC, so I'll see how hard it is to run QEMU for those platforms and whether FreeBSD will just boot and run on each.

More RSS UDP tests - this time on a Dell R720

I've recently had the chance to run my RSS UDP test suite up on a pair of Dell R720s. They came with on-board 10G Intel NICs (ixgbe(4) in FreeBSD) so I figured I'd run my test suite up on it.

Thank you to the Enterprise Storage Division at Dell for providing hardware for me to develop on!

The config is like in the previous blog post, but now I have two 8-core Sandy Bridge Xeon CPUs to play with. To simply things (and to not have to try and solve NUMA related issues) I'm running this on the first socket. The Intel NIC is attached to the first CPU socket.

So:

• CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (2000.04-MHz K8-class CPU) x 2
• RAM: 64GiB
• HTT disabled

/boot/loader.conf:

# ... until ncpus is tunable, make it use 8 buckets.

net.inet.rss.bits=3

net.isr.maxthreads=8

net.isr.bindthreads=1

This time I want to test with 8 streams, so after some trial and error I found the right IPv4 addresses to use:

• Server: 10.11.2.1/24
• Client: 10.11.2.3/24, 10.11.2.2/24, 10.11.2.32/24, 10.11.2.33/24, 10.11.2.64/24, 10.11.2.65/24, 10.11.2.17/24, 10.11.2.18/24

The test was like before - the server ran one rss-udp-srv program that spawns one thread per RSS bucket. The client side runs rss-clt programs to generate traffic - but now there's eight of them instead of four.

The results are what I expected: the contention is in the same place (UDP receive) and it's per-core - it doesn't contend between CPU cores.

Each CPU is transmitting and receiving 215,000 510-byte UDP frames a second. It scales linearly - 1 CPU is 215,000 TX/RX frames a second. 8 CPUs is 215,000 TX/RX frames a second * 8. There's no degrading as the CPU core count increases.

That's 1.72 million packets per second. At 510 bytes frames it's about 7 gigabits/sec in and out.

The other 8 cores are idle. Ideally we'd be able to run an application in those cores - so hopefully I can get my network / rss library up and running enough to prototype an RSS-aware memcached and see if it'll handle this particular workload.

It's a far cry from what I think we can likely achieve - but please keep in mind that I know I could do more awesome looking results with netmap, PF_RING or Intel's DPDK software. What I'm trying to do is push the existing kernel networking subsystem to its limits so the issues can be exposed and fixed.

So, where's the CPU going?

In the UDP server program (pid 1620), it looks thus:

# pmcstat -P CPU_CLK_UNHALTED_CORE -T -w 1 -p 1620

PMC: [CPU_CLK_UNHALTED_CORE] Samples: 34298 (100.0%) , 155 unresolved

%SAMP IMAGE      FUNCTION             CALLERS

  8.0 kernel     fget_unlocked        kern_sendit:4.2 kern_recvit:3.9

  7.0 kernel     copyout              soreceive_dgram:5.6 amd64_syscall:0.9

  3.6 kernel     __mtx_unlock_flags   ixgbe_mq_start

  3.5 kernel     copyin               m_uiotombuf:1.8 amd64_syscall:1.2

  3.4 kernel     memcpy               ip_output:2.9 ether_output:0.6

  3.4 kernel     toeplitz_hash        rss_hash_ip4_2tuple

  3.3 kernel     bcopy                rss_hash_ip4_2tuple:1.4 rss_proto_software_hash_v4:0.9

  3.0 kernel     _mtx_lock_spin_cooki pmclog_reserve

  2.7 kernel     udp_send             sosend_dgram

  2.5 kernel     ip_output            udp_send

In the NIC receive / transmit thread(s) (pid 12), it looks thus:

# pmcstat -P CPU_CLK_UNHALTED_CORE -T -w 1 -p 12

PMC: [CPU_CLK_UNHALTED_CORE] Samples: 79319 (100.0%) , 0 unresolved

%SAMP IMAGE      FUNCTION             CALLERS

 10.3 kernel     ixgbe_rxeof          ixgbe_msix_que

  9.3 kernel     __mtx_unlock_flags   ixgbe_rxeof:4.8 netisr_dispatch_src:2.1 in_pcblookup_mbuf:1.3

  8.3 kernel     __mtx_lock_flags     ixgbe_rxeof:2.8 netisr_dispatch_src:2.4 udp_append:1.2 in_pcblookup_mbuf:1.1 knote:0.6

  3.8 kernel     bcmp                 netisr_dispatch_src

  3.6 kernel     uma_zalloc_arg       sbappendaddr_locked_internal:2.0 m_getjcl:1.6

  3.4 kernel     ip_input             netisr_dispatch_src

  3.4 kernel     lock_profile_release __mtx_unlock_flags

  3.4 kernel     in_pcblookup_mbuf    udp_input

  3.0 kernel     ether_nh_input       netisr_dispatch_src

  2.4 kernel     udp_input            ip_input

  2.4 kernel     mb_free_ext          m_freem

  2.2 kernel     lock_profile_obtain_ __mtx_lock_flags

  2.1 kernel     ixgbe_refresh_mbufs  ixgbe_rxeof

It looks like there's some obvious optimisations to poke at (what the heck is fget_unlocked() doing up there?) and yes, copyout/copyin are really terrible but currently unavoidable. The toeplitz hash and bcopy aren't very nice but they're occuring in the transmit path because at the moment there's no nice way to efficiently set both the outbound RSS hash and RSS bucket ID and send to a non-connected socket destination (ie, specify the destination IP:port as part of the send.) There's also some lock contention that needs to be addressed.

The output of the netisr queue statistics looks good:

root@abaddon:/home/adrian/git/github/erikarn/freebsd-rss # netstat -Q

Configuration:

Setting                        Current        Limit

Thread count                         8            8

Default queue limit                256        10240

Dispatch policy                 direct          n/a

Threads bound to CPUs          enabled          n/a

Protocols:

Name   Proto QLimit Policy Dispatch Flags

ip         1    256    cpu   hybrid   C--

igmp       2    256 source  default   ---

rtsock     3    256 source  default   ---

arp        4    256 source  default   ---

ether      5    256    cpu   direct   C--

ip6        6    256   flow  default   ---

ip_direct     9    256    cpu   hybrid   C--

Workstreams:

WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled

   0   0   ip         0    25        0 839349259        0       49 839349308

   0   0   igmp       0     0        0        0        0        0        0

   0   0   rtsock     0     2        0        0        0       92       92

   0   0   arp        0     0      118        0        0        0      118

   0   0   ether      0     0 839349600        0        0        0 839349600

   0   0   ip6        0     0        0        0        0        0        0

   0   0   ip_direct     0     0        0        0        0        0        0

   1   1   ip         0    20        0 829928186        0      286 829928472

   1   1   igmp       0     0        0        0        0        0        0

   1   1   rtsock     0     0        0        0        0        0        0

   1   1   arp        0     0        0        0        0        0        0

   1   1   ether      0     0 829928672        0        0        0 829928672

   1   1   ip6        0     0        0        0        0        0        0

   1   1   ip_direct     0     0        0        0        0        0        0

   2   2   ip         0     0        0 835558437        0        0 835558437

   2   2   igmp       0     0        0        0        0        0        0

   2   2   rtsock     0     0        0        0        0        0        0

   2   2   arp        0     0        0        0        0        0        0

   2   2   ether      0     0 835558610        0        0        0 835558610

   2   2   ip6        0     0        0        0        0        0        0

   2   2   ip_direct     0     0        0        0        0        0        0

   3   3   ip         0     1        0 850271162        0       23 850271185

   3   3   igmp       0     0        0        0        0        0        0

   3   3   rtsock     0     0        0        0        0        0        0

   3   3   arp        0     0        0        0        0        0        0

   3   3   ether      0     0 850271163        0        0        0 850271163

   3   3   ip6        0     0        0        0        0        0        0

   3   3   ip_direct     0     0        0        0        0        0        0

   4   4   ip         0    23        0 817439448        0      345 817439793

   4   4   igmp       0     0        0        0        0        0        0

   4   4   rtsock     0     0        0        0        0        0        0

   4   4   arp        0     0        0        0        0        0        0

   4   4   ether      0     0 817439625        0        0        0 817439625

   4   4   ip6        0     0        0        0        0        0        0

   4   4   ip_direct     0     0        0        0        0        0        0

   5   5   ip         0    19        0 817862508        0      332 817862840

   5   5   igmp       0     0        0        0        0        0        0

   5   5   rtsock     0     0        0        0        0        0        0

   5   5   arp        0     0        0        0        0        0        0

   5   5   ether      0     0 817862675        0        0        0 817862675

   5   5   ip6        0     0        0        0        0        0        0

   5   5   ip_direct     0     0        0        0        0        0        0

   6   6   ip         0    19        0 817281399        0      457 817281856

   6   6   igmp       0     0        0        0        0        0        0

   6   6   rtsock     0     0        0        0        0        0        0

   6   6   arp        0     0        0        0        0        0        0

   6   6   ether      0     0 817281665        0        0        0 817281665

   6   6   ip6        0     0        0        0        0        0        0

   6   6   ip_direct     0     0        0        0        0        0        0

   7   7   ip         0     0        0 813562616        0        0 813562616

   7   7   igmp       0     0        0        0        0        0        0

   7   7   rtsock     0     0        0        0        0        0        0

   7   7   arp        0     0        0        0        0        0        0

   7   7   ether      0     0 813562620        0        0        0 813562620

   7   7   ip6        0     0        0        0        0        0        0

   7   7   ip_direct     0     0        0        0        0        0        0

root@abaddon:/home/adrian/git/github/erikarn/freebsd-rss # 

It looks like everything is being dispatched correctly; nothing is being queued and/or dropped.

But yes, we're running out of socket buffers because each core is 100% pinned:

root@abaddon:/home/adrian/git/github/erikarn/freebsd-rss # netstat -sp udp

udp:

        6773040390 datagrams received

        0 with incomplete header

        0 with bad data length field

        0 with bad checksum

        0 with no checksum

        17450880 dropped due to no socket

        136 broadcast/multicast datagrams undelivered

        1634117674 dropped due to full socket buffers

        0 not for hashed pcb

        5121471700 delivered

        5121471044 datagrams output

        0 times multicast source filter matched

There's definitely room for improvement.

UDP RSS update: ixbge(4) turned out to have issues..

I started digging deeper into the RSS performance on my home test platform. Four cores and one (desktop) socket isn't all that much, but it's a good starting point for this.

It turns out that there was some lock contention inside netisr. Which made no sense, as RSS should be keeping all the flows local to each CPU.

After a bunch of digging, I discovered that the NIC was occasionally receiving packets into the wrong ring. Have a look at tihs:

Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100034:
m=0xfffff80047713d00; flowid=0x21f7db62; rxr->me=3
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100034:
m=0xfffff8004742e100; flowid=0x21f7db62; rxr->me=3
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100034:
m=0xfffff800474c2e00; flowid=0x21f7db62; rxr->me=3
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100034:
m=0xfffff800474c5000; flowid=0x21f7db62; rxr->me=3
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100034:
m=0xfffff8004742ec00; flowid=0x21f7db62; rxr->me=3
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100032:
m=0xfffff8004727a700; flowid=0x335a5c03; rxr->me=2
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100032:
m=0xfffff80006f11600; flowid=0x335a5c03; rxr->me=2
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100032:
m=0xfffff80047279b00; flowid=0x335a5c03; rxr->me=2
Sep 12 08:04:32 adrian-hackbox kernel: ix0: ixgbe_rxeof: 100032:
m=0xfffff80006f0b700; flowid=0x335a5c03; rxr->me=2


The RX flowid was correct - I hashed the packets in software too and verified the software hash equaled the hardware hash. But they were turning up on the wrong receive queue. "rxr->me" is the queue id; the hardware should be hashing on the last 7 bits. 0x3 -> ring 3, 0x2 -> ring 2.

It also only happened when I was sending traffic to more than one receive ring. Everything was okay if I just transmitted to a single receive ring.

Luckily for me, some developers from Verisign saw some odd behaviour in their TCP stress testing and had dug in a bit further. They were seeing corrupted frames on the receive side that looked a lot like internal NIC configuration state. They figured out that the ixgbe(4) driver wasn't initialising the flow director and receive units correctly - the FreeBSD driver was not correctly setting up the amount of memory each was allocated on the NIC and they were overlapping. They also found a handful of incorrectly handled errors and double-freed mbufs.

So, with that all fixed, their TCP problem went away and my UDP tests started properly behaving themselves. Now all the flows are ending up on the right CPUs.

The flow director code was also dynamically programming flows into the NIC to try and rebalance traffic. Trouble is, I think it's a bit buggy and it's likely not working well with generic receive offload (LRO).

What's it mean for normal people? Well, it's fixed in FreeBSD-HEAD now. I'm hoping I or someone else will backport it to FreeBSD-10 soon. It fixes my UDP tests - now I hit around 1.3 million packets per second transmit and receive on my test rig; the server now has around 10-15% CPU free. It also fixed issues that Verisign were seeing with their high transaction rate TCP tests. I'm hoping that it fixes the odd corner cases that people have seen with Intel 10 gigabit hardware on FreeBSD and makes LRO generally more useful and stable.

Next up - some code refactoring, then finishing off IPv6 RSS!

 
 

Receive side scaling: testing UDP throughput

I think it's about time I shared some more details about the RSS stuff going into FreeBSD and how I'm testing it.

For now I'm focusing on IPv4 + UDP on the Intel 10GE NICs. The TCP side of things is done (and the IPv6 side of things works too!) but enough of the performance walls show up in the IPv4 UDP case that it's worth sticking to it for now.

I'm testing on a pair of 4-core boxes at home. They're not special - and they're very specifically not trying to be server-class hardware. I'd like to see where these bottlenecks are even at low core count.

The test setup in question:

Testing software:

• http://github.com/erikarn/freebsd-rss
• It requires libevent2 - an updated copy; previous versions of libevent2 didn't handle FreeBSD specific errors gracefully and would early error out of the IO loop.

Server:

• CPU: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz (3292.59-MHz K8-class CPU)
• There's no SMT/HTT, but I've disabled it in the BIOS just to be sure
• 4GB RAM
• FreeBSD-HEAD, amd64
• NIC:  '82599EB 10-Gigabit SFI/SFP+ Network Connection
• ix0: 10.11.2.1/24

/etc/sysctl.conf:

# for now redirect processing just makes the lock overhead suck even more.
# disable it.
net.inet.ip.redirect=0
net.inet.icmp.drop_redirect=1
/boot/loader.conf:

hw.ix.num_queues=8

# experiment with deferred dispatch for RSS
net.isr.numthreads=4
net.isr.maxthreads=4
net.isr.bindthreads=1
 
kernel config:

include GENERIC
ident HACKBOX

device netmap
options RSS
options PCBGROUP

# in-system lock profiling
options LOCK_PROFILING

# Flowtable - the rtentry locking is a bit .. slow.
options   FLOWTABLE

# This debugging code has too much overhead to do accurate
# testing with.
nooptions         INVARIANTS
nooptions         INVARIANT_SUPPORT
nooptions         WITNESS
nooptions         WITNESS_SKIPSPIN

The server runs the "rss-udp-srv" process, which behaves like a multi-threaded UDP echo server on port 8080.

Client

The client box is slightly more powerful to compensate for (currently) not using completely affinity-aware RSS UDP transmit code.

• CPU: Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz (3192.68-MHz K8-class CPU)
• SMT/HTT: Disabled in BIOS
• 8GB RAM
• FreeBSD-HEAD amd64
• Same kernel config, loader and sysctl config as the server
• ix0: configured as 10.11.2.2/24, 10.11.2.3/32, 10.11.2.4/32, 10.11.2.32/32, 10.11.2.33/32

The client runs 'udp-clt' programs to source and sink traffic to the server.

Running things

The server-side simply runs the listen server, configured to respond to each frame:

$ rss-udp-srv 1 10.11.2.1

The client-side runs four couples of udp-clt, each from different IP addresses. These are run in parallel (i do it in different screens, so I can quickly see what's going on):

$ ./udp-clt -l 10.11.2.3 -r 10.11.2.1 -p 8080 -n 10000000000 -s 510
$ ./udp-clt -l 10.11.2.4 -r 10.11.2.1 -p 8080 -n 10000000000 -s 510
$ ./udp-clt -l 10.11.2.32 -r 10.11.2.1 -p 8080 -n 10000000000 -s 510
$ ./udp-clt -l 10.11.2.33 -r 10.11.2.1 -p 8080 -n 10000000000 -s 510

The IP addresses are chosen so that the 2-tuple topelitz hash using the default Microsoft key hash to different RSS buckets that live on individual CPUs.

Results: Round one

When the server is responding to each frame, the following occurs. The numbers are "number of frames generated by the client (netstat)", "number of frames received by the server (netstat)", "number of frames seen by udp-rss-srv", "number of responses transmitted from udp-rss-srv", "number of frames seen by the server (netstat)"

• 1 udp-clt process: 710,000; 710,000; 296,000; 283,000; 281,000
• 2 udp-clt processes: 1,300,000; 1,300,000; 592,000; 592,000; 575,000
• 3 udp-clt processes: 1,800,000; 1,800,000; 636,000; 636,000; 600,000
• 4 udp-clt processes: 2,100,000; 2,100,000; 255,000; 255,000; 255,000

So, it's not actually linear past two cores. The question here is: why?

There are a couple of parts to this.

Firstly - I had left turbo boost on. What this translated to:

• One core active: ~ 30% increase in clock speed
• Two cores active: ~ 30% increase in clock speed
• Three cores active: ~ 25% increase in clock speed
• Four cores active: ~ 15% increase in clock speed.

Secondly and more importantly - I had left flow control enabled. This made a world of difference.

The revised results are mostly linear - with more active RSS buckets (and thus CPUs) things seem to get slightly more efficient:

• 1 udp-clt process: 710,000; 710,000; 266,000; 266,000; 266,000
• 2 udp-clt processes: 1,300,000; 1,300,000; 512,000; 512,000; 512,000
• 3 udp-clt processes: 1,800,000; 1,800,000; 810,000; 810,000; 810,000
• 4 udp-clt processes: 2,100,000; 2,100,000; 1,120,000; 1,120,000; 1,120,000

Finally, let's repeat the process but only receiving instead also echoing back the packet to the client:

$ rss-udp-srv 0 10.11.2.1

• 1 udp-clt process: 710,000; 710,000; 204,000
• 2 udp-clt processes: 1,300,000; 1,300,000; 378,000
• 3 udp-clt processes: 1,800,000; 1,800,000; 645,000
• 4 udp-clt processes: 2,100,000; 2,100,000; 900,000

The receive-only workload is actually worse off versus the transmit + receive workload!

What's going on here?

Well, a little digging shows that in both instances - even with a single udp-clt thread running which means only one CPU on the server side is actually active! - there's active lock contention.

Here's an example dtrace output for measuring lock contention with only one active process, where one CPU is involved (and the other three are idle):

Receive only, 5 seconds:

root@adrian-hackbox:/home/adrian/git/github/erikarn/freebsd-rss # dtrace -n 'lockstat:::adaptive-block { @[stack()] = sum(arg1); }'

dtrace: description 'lockstat:::adaptive-block ' matched 1 probe

^C

              kernel`udp_append+0x11c

              kernel`udp_input+0x8cc

              kernel`ip_input+0x116

              kernel`netisr_dispatch_src+0x1cb

              kernel`ether_demux+0x123

              kernel`ether_nh_input+0x34d

              kernel`netisr_dispatch_src+0x61

              kernel`ether_input+0x26

              kernel`ixgbe_rxeof+0x2f7

              kernel`ixgbe_msix_que+0xb6

              kernel`intr_event_execute_handlers+0x83

              kernel`ithread_loop+0x96

              kernel`fork_exit+0x71

              kernel`0xffffffff80cd19de

         46729281

Transmit + receive, 5 seconds:

dtrace: description 'lockstat:::adaptive-block ' matched 1 probe

^C

              kernel`knote+0x7e

              kernel`sowakeup+0x65

              kernel`udp_append+0x14a

              kernel`udp_input+0x8cc

              kernel`ip_input+0x116

              kernel`netisr_dispatch_src+0x1cb

              kernel`ether_demux+0x123

              kernel`ether_nh_input+0x34d

              kernel`netisr_dispatch_src+0x61

              kernel`ether_input+0x26

              kernel`ixgbe_rxeof+0x2f7

              kernel`ixgbe_msix_que+0xb6

              kernel`intr_event_execute_handlers+0x83

              kernel`ithread_loop+0x96

              kernel`fork_exit+0x71

              kernel`0xffffffff80cd19de

             3793

              kernel`udp_append+0x11c

              kernel`udp_input+0x8cc

              kernel`ip_input+0x116

              kernel`netisr_dispatch_src+0x1cb

              kernel`ether_demux+0x123

              kernel`ether_nh_input+0x34d

              kernel`netisr_dispatch_src+0x61

              kernel`ether_input+0x26

              kernel`ixgbe_rxeof+0x2f7

              kernel`ixgbe_msix_que+0xb6

              kernel`intr_event_execute_handlers+0x83

              kernel`ithread_loop+0x96

              kernel`fork_exit+0x71

              kernel`0xffffffff80cd19de

          3823793

              kernel`ixgbe_msix_que+0xd3

              kernel`intr_event_execute_handlers+0x83

              kernel`ithread_loop+0x96

              kernel`fork_exit+0x71

              kernel`0xffffffff80cd19de

          9918140

Somehow it seems there's less lock contention / blocking going on when both transmit and receive is running!

So then I dug into it using the lock profiling suite. This is for 5 seconds with receive-only traffic on a single RSS bucket / CPU (all other CPUs are idle):

# sysctl debug.lock.prof.enable = 1; sleep 5 ; sysctl debug.lock.prof.enable=0

root@adrian-hackbox:/home/adrian/git/github/erikarn/freebsd-rss # sysctl debug.lock.prof.enable=1 ; sleep 5 ; sysctl debug.lock.prof.enable=0

debug.lock.prof.enable: 1 -> 1

debug.lock.prof.enable: 1 -> 0

root@adrian-hackbox:/home/adrian/git/github/erikarn/freebsd-rss # sysctl debug.lock.prof.stats | head -2 ; sysctl debug.lock.prof.stats | sort -nk4 | tail -10

debug.lock.prof.stats: 

     max  wait_max       total  wait_total       count    avg wait_avg cnt_hold cnt_lock name

    1496         0       10900           0          28    389      0  0      0 /usr/home/adrian/work/freebsd/head/src/sys/dev/usb/usb_device.c:2755 (sx:USB config SX lock)

debug.lock.prof.stats: 

       0         0          31           1          67      0      0  0      4 /usr/home/adrian/work/freebsd/head/src/sys/kern/sched_ule.c:888 (spin mutex:sched lock 2)

       0         0        2715           1       49740      0      0  0      7 /usr/home/adrian/work/freebsd/head/src/sys/dev/random/random_harvestq.c:294 (spin mutex:entropy harvest mutex)

       1         0          51           1         131      0      0  0      2 /usr/home/adrian/work/freebsd/head/src/sys/kern/sched_ule.c:1179 (spin mutex:sched lock 1)

       0         0          69           2         170      0      0  0      8 /usr/home/adrian/work/freebsd/head/src/sys/kern/sched_ule.c:886 (spin mutex:sched lock 2)

       0         0       40389           2      287649      0      0  0      8 /usr/home/adrian/work/freebsd/head/src/sys/kern/kern_intr.c:1359 (spin mutex:sched lock 2)

       0         2           2           4          12      0      0  0      2 /usr/home/adrian/work/freebsd/head/src/sys/dev/usb/usb_device.c:2762 (sleep mutex:Giant)

      15        20        6556         520        2254      2      0  0    105 /usr/home/adrian/work/freebsd/head/src/sys/dev/acpica/Osd/OsdSynch.c:535 (spin mutex:ACPI lock (0xfffff80002b10f00))

       4         5      195967       65888     3445501      0      0  0  28975 /usr/home/adrian/work/freebsd/head/src/sys/netinet/udp_usrreq.c:369 (sleep mutex:so_rcv)

Notice the lock contention for the so_rcv (socket receive buffer) handling? What's going on here is pretty amusing - it turns out that because there's so much receive traffic going on, the userland process receiving the data is being preempted by the NIC receive thread very often - and when this happens, there's a good chance it's going to be within the small window that the receive socket buffer lock is held. Once this happens, the NIC receive thread processes frames until it gets to one that requires it to grab the same sock buffer lock that is already held by userland - and it fails - so the NIC thread sleeps until the userland thread finishes consuming a packet. Then the CPU flips back to the NIC thread and continues processing a packet.

When the userland code is also transmitting frames it's increasing the amount of time in between socket receives and decreasing the probability of hitting the lock contention condition above.

Note there's no contention between CPUs here - this is entirely contention within a single CPU.

So for now I'm happy that the UDP IPv4 path is scaling well enough with RSS on a single core. The main performance problem here is the socket receive buffer locking (and, yes, copyin() / copyout().)

Next!