FreeBSD on a tiny system; what's missing

Now that I'm trying to use more userland services, there are some obvious shortcomings which need addressing.

The first is a lack of real service management. FreeBSD doesn't have a service management daemon - the framework assumes that daemons implement their own background and monitoring. It would be much nicer if init or something similar to init could manage services and start/restart them where appropriate. Yes, this is one of those arguments for systemd. Eg, maybe I want to only start telnetd or dropbear/sshd whenever a connection comes in. But I'd also like to be able to add services for monitoring, such as dnsmasq and hostapd.

The next is a lack of suitable syslog daemon. Yes, I'd like to be able to log some messages locally - even if it's only a couple hundred kilobytes of messages. I'd also like to be able to push messages to a remote service. Unfortunately the FreeBSD syslog daemon doesn't do log rotation or maximum log file sizes itself - it's done by "newsyslog" which runs out of cron. This isn't any good for real embedded systems with limited storage.

Then yes, there's a lack of a cron service. It'd be nice to have that integrated into the service management framework so things could be easily added/removed. I may just use cron, but that means cron is also always running which adds memory footprint (~1.3 megabytes) for something that is almost never actually active. When you have 32MB of RAM, that's quite a bit of wasted memory.

Finally, there's a lack of some message bus and notification mechanism for device changes. For example, openvpn-client creates a tunnel device - ok, so what should then check to see if a NAT configuration needs updating? Or updated firewall rules? It can be done with shell scripts (which I'll write tomorrow) but ideally there'd be something like dbus (a dirty word, I know) where these systems could push updates to and events could be triggered from them. I'd like to be able to run ntpdate whenever an interface comes up, because yes, there is no RTC on this hardware.

With all of the above in mind, I'll start working on some of it tomorrow. Hopefully I can automate the openvpn NAT configuration a little bit more so I can optionally have wifi NAT or openvpn NAT, depending upon the current requirement. Fixing ntpdate to run out of dhclient as part of the 'up' script may be helpful. I'll see what else I can do to tidy things up before I start the process of merging all of this back into freebsd-wifi-build.

At least this year I can now use the defcon wireless with all of my devices.

Musings on bringing up services on freebsd-wifi MIPS devices, or "why cross compiling is important"

FreeBSD has run on these MIPS routers for quite some time, but it was limited to what ships in base. There's not been any cross-built packages as part of the image building, which means we can't easily have third-party functionality.

Now, some of this third-party functionality is pretty important these days. Relying on telnet sucks; I'd like to have dropbear as an SSH server so we at least have SSH. Not having a DNS relay or DHCP server also sucks; dnsmasq would solve this problem. I'd also like some VPN services, so openvpn would be nice.

So, I eventually snapped a few months ago and started integrating some external toolchain compiler use with the freebsd-wifi-build scripts. bapt@freebsd did a whole lot of work to build ports of cross-compilers to be used by the FreeBSD ports and base system so I'm leveraging that for doing MIPS cross compiling. A bit of hacking later, and I'm cross compiling dnsmasq, dropbear, openvpn and lua.

Then I needed to integrate things. I wrote up a bunch of simple startup script wrappers to generate suitable config files for these services. Everything except the openvpn server/client configuration is in the rc.conf file, which will eventually make it much easier to turn into a configuration database.

OpenVPN was the most amusing. It cross compiled fine, save needing liblzo for compression (so that's disabled for now.) However, FreeBSD's openvpn package is version 2.3 but the easyrsa component is actually from 3.0 - which means all the documentation is out of date.

I used this for the OpenVPN config:

https://airvpn.org/topic/15096-verify-error-depth1-errorcertificate-is-not-yet-valid-using-router-with-tomato-shibby-firmware/

And this for easyvpn:

https://community.openvpn.net/openvpn/wiki/EasyRSA3-OpenVPN-Howto

And digitalocean have a writeup for how to convert the config file into a combined config file and certification bundle:

https://www.digitalocean.com/community/tutorials/how-to-configure-and-connect-to-a-private-openvpn-server-on-freebsd-10-1

A few things tripped me up:

• as mentioned before - the lack of freebsd openvpn documentation that works with easyvpn 3.0 made things frustrating;
• openvpn really wants valid system time, so I am going to have to run ntpdate when the WAN comes up;
• there's no RTC on many of these router boards, making time keeping very much reliant on NTP;
• kernel NAT works pretty well, but it needs interfaces to be up before you can add them. I'll have to add some scripts to openvpn-client to setup the NAT state once the link comes up so this stops being a problem;

I'll go into a little more detail about the details in a future post. But, hi from being behind an openvpn-client LAN gateway!

Fixing up the QCA9558 performance on FreeBSD, or "why attention to detail matters when you're a kernel developer."

When I started with this Atheros MIPS 11n stuff a few years ago, my first test board was a Routerstation Pro with a pair of AR9160 NICs. I could get ~ 150mbit/sec bridging performance out of it, and I thought I was doing pretty good.

Fast forward to now, and I've been bringing FreeBSD up on each of the subsequent boards. But the performance never improved. Now, I never bothered to look into it because I was always too busy with my day job, but finally someone trolled me correctly on the FreeBSD embedded IRC channel and I took a look.

It turns out that.. things could've benefitted from a lot of improvement.

First up - I'm glad George Neville-Neil brought up PMC (performance counters) on the MIPS24k platform. It made it easier for me to bring it up on the MIPS74k platform and it was absolutely instrumental in figuring out performance issues here. No, there's no real ability to get DTrace up on these boards - some have 32MB of RAM. Heck, the packet filter (bpf) consumes most of a megabyte of RAM when you first start it up.

My initial tests are on an AP135 reference design board from Qualcomm Atheros. It's a QCA9558 SoC with an AR8327 switch on board. Both on-chip ethernet ports (arge0, arge1) are available. I set it up as a straight bridge between arge0 and arge1 and then I used iperf between two laptops to measure performance.

The first test - 130mbit bridging performance. That's terrible for this platform.

So I fired up hwpmc, and I found the first problem - packets were being copied in the receive and transmit path. Since I'm more familiar with the transmit path, I decided to look into that.

The AR7161 MAC requires both transmit and receive buffers to be both DWORD (32 bit or 4 bytes) aligned. In addition, all transmit frames save the last frame are required to be a multiple of DWORD in length. Plenty of frames don't meet this requirement and end up being copied.

The AR7240 and later MAC relaxed this - transmit/receive buffers can now be byte-aligned. So that particular workaround can be removed. It still needs to do it for multi-descriptor transmits that aren't DWORD sized (eg if you just prepend a fresh ethernet header) but that doesn't happen in the bridging path in the normal case.

Fixing that got bridging performance from 130mbit to 180mbit. That's not a huge difference, but it's something.

Next up is the receive path.  This was more .. complicated. The receive code copies the whole buffer back two bytes in order to ensure that the IP payload presented to the FreeBSD network stack is aligned. This is a problem in FreeBSD's network stack - it assumes the hardware handles unaligned accesses fine. So if your receive engine is DWORD aligned, the 14 byte ethernet header will result in the start of the IP payload being non-DWORD aligned, and .. the stack blows up. Now, I have vague plans to start fixing that as a general rule, but I did the next worst hack - I grabbed a buffer, set its RX start point to two bytes in, so the ethernet header is unaligned but the IP header is. Now, the ethernet stack in FreeBSD handles unaligned stuff correctly, so that works.

Except it wasn't faster. It turns out that the MIPS busdma code was doing very inefficient things with mbuf handling if everything wasn't completely aligned. Ian Lepore (who does ARM work) recently fixed this for armv6, so he ported it to MIPS and I added it.

The result? bridging performance leaped from 180mbit to 420mbit. Quite nice, but not where Linux was.

I left it for a few days, and someone on the freebsd-mips mailing list pointed out big stability issues with his tests. I started looking at the Linux OpenWRT driver and the MIPS24K/MIPS74K memory coherency operations. I found a couple of interesting things:

• The busdma sync code never did a "SYNC" operation if things weren't being copied or invalidated; and
• I was using cache-writethrough instead of cache-writeback for the cached memory attribute for MIPS74K.

The former is a problem with driver memory / driver access sync - you need to ensure that the changes you've made are actually in memory before you tell the hardware to look at it. So I fixed that in the busdma routines.

The latter makes everything slow. It means each write is going through the cache and into memory - the cache hardware doesn't get to batch writes to memory. I changed that, and found more instability in some parts of the arge ethernet driver - the MDIO bus accesses started misbehaving. After looking at the Linux code and the sync operations, I reimplemented the MDIO code correctly and I added explicit read/write barriers as needed. The MDIO code does lots of same-register accesses in loops to look for things, and the hardware may subtly reorder things. I committed this, flipped on the correct cache attribute to support cache-writeback, and things got .. faster. Much faster in fact.

So, that worked - and I hit the hardware instability issue. But, I hit it at a higher traffic rate. The final thing was fixed by looking at the OpenWRT driver (ag71xx_main.c) and going "Aha!" - the transmit side was buggy.

Specifically - the transmit side is a linked list of descriptors, but it's formed into a ring. The TX DMA engine stops when it hits a descriptor that isn't marked "ready" (ie, has ARGE_DESC_EMPTY set.) Now, we didn't see this before when we were copying transmit buffers for a packet into a single correct transmit buffer, but now that I am doing multi-descriptor transmit more frequently, this bug was hit. The bug is that because the TX descriptors are in a big ring, it's possible the hardware will transmit everything and hit the end of the ring before we've completely setup the descriptors for the next packet. If this happens, and it hits ARGE_DESC_EMPTY, then it stops. But if we have say a 3 descriptor packet, and we set the descriptors up in order, the hardware may hit that first descriptor out of three before we've finished setting things up, and start transmitting. It hits a descriptor we've not setup yet, thinks we're done, and transmits what it's seen. Then when I finish the setup and hit "transmit" on the hardware, it stalls, and everything sticks.

The fix was to initialise the first descriptor as EMPTY, then when we're done setting them up, flip that first descriptor to non-empty.

And voila! The bug is fixed and things perform now at a much faster rate - 720mbit. Yup, it bridges at 720mbit and it routes at around 320mbit. I'd like to get routing up from 320mbit to somewhere near bridge performance, but that'll have to wait a while.

120mbit -> 720mbit. Yup, I'm happy with that.

... mmm emulators.

I occasionally get asked to test out FreeBSD/MIPS patches for people, as they don't have physical hardware present. I can understand that - the hardware is cheap and plentiful, but not everyone wants to have a spare access point around just to test out MIPS changes on.

However QEMU does a pretty good job of emulating MIPS if you're just testing out non-hardware patches. There's even instructions on the FreeBSD wiki for how to do this! So I decided to teach my wifi build system about the various QEMU MIPS emulator targets so it can spit out a kernel and mfsroot to use for QEMU.

Thus:

https://github.com/freebsd/freebsd-wifi-build/wiki/MipsQemuEmulatorImages

It turns out that it wasn't all that hard. The main trick was to use qemu-devel, not qemu. There are bugs in the non-development QEMU branch that mean it works great for Linux but not FreeBSD.

The kernel configurations in FreeBSD had bitrotted a little bit (they were missing the random device, for example) but besides that the build, install and QEMU startup just worked. I now have FreeBSD/MIPS of each variety (32 bit, 64 bit, Little-Endian, Big-Endian) running under QEMU and building FreeBSD-HEAD as a basic test.

Next is figuring out how to build gdb to target each of the above and have it speak to the QEMU GDB stub. That should make it very easy to do MIPS platform debugging.

I also hear rumours about this stuff working somewhat for ARM and PPC, so I'll see how hard it is to run QEMU for those platforms and whether FreeBSD will just boot and run on each.

The gymnastics required to just do a "HALT" for MIPS..

So, it turns out there's no nice, guaranteed way to implement a HALT style setup for MIPS in the idle thread in the kernel. I'll braindump what happened about two years ago to try and address this.

(And I do hope that the implementation actually works!)

When the kernel isn't running any work, it will schedule the "idle" thread to run. This has the simple(!) task of entering a sleep state, only to wake up when another interrupt fires. This saves power consumption and generates less heat.

However, it's a little trickier than just "enter idle state."

The only way to exit it is to receive an interrupt. Now, this can be device interrupts; this can be timer interrupts (and yes; the timer is a kind of device, tsk..) but without it, the CPU will stay halted. This isn't a big deal - the UNIX kernel tends to be one big event processing "thing" and these include both software and hardware events. If it's a software event that needs to happen "now", the kernel won't go into the idle loop - it'll just run the needed event. If it has to happen in the future, it'll schedule it to occur after some time has elapsed - and this will be driven by a timer interrupt. An interrupt would wake up the system from the HALT state. If the interrupt occured just before the HALT instruction was executed, there'd be a very small window where the interrupt would occur - the interrupt routine would be called, complete, and then the HALT instruction would next execute.

Now, imagine you're an ethernet driver on FreeBSD-10 where the interrupt handler just scheduled some ithread to run in the future. Here's one example of what may happen:

• The idle thread is executed;
• An interrupt occurs, say to signal completion of some ethernet frame transmission;
• The CPU kicks off the interrupt code;
• The interrupt code doesn't find a fast interrupt handler but finds an ithread; so it schedules the ithread to run;
• The scheduler goes to choose another thread to run and finds an ithread scheduled at a higher priority, so it schedules that;
• The ethernet driver code in the ithread runs;
• The scheduler then exits and re-enters the idle loop.

Everything is fine, right?

But, with the event timer changes that came in during the 9.x time frame, the halt code is now in a critical section.

So now, the following happens:

• The idle thread is executed;
• critical_enter() is called;
• An interrupt occurs, which calls the fast handler, which schedules the ithread to run;
• .. but critical_enter() has set a flag that says "no preemption", so the ithread doesn't get to run;
• Control is returned to the idle function;
• The idle thread gets to the point where it executes "HALT";
• The idle thread continues to run, executing HALT.

In this instance, the ithread is scheduled but can't run before HALT runs. The ithread only runs after the next interrupt occurs.

I noticed this was happening when doing traffic tests on FreeBSD-HEAD between 802.11 and ethernet interfaces. The atheros 802.11 hardware implemented interrupt moderation so there wouldn't be tens of thousands of interrupts a second being generated. But what I saw was occasionally the receive queue being filled by packets and not drained fast enough. When digging into it, I found that due to interrupt moderation, if the interrupt came in just before WAIT was executed, the 802.11 receive function was taking a long time (sometimes up to milliseconds) to run after the interrupt came in and the ithread was actually scheduled. If I had an interrupt for each received packet, the amount of time between interrupts would have been very small (20,000 packets a second, so around 1/20000 sec per interrupt) and this problem would've been masked. But with moderated interrupts, it would be 750 microseconds or so before the next receive interrupt was generated.

Now, this is messy. There's some hacks in the idle loop code to try and skip the halt bit if the scheduler detects there's something to run. But there's still a small race window there which needs to be closed.

How can this be solved?

Apparently - the two instructions STI;HLT on x86 are atomic. Ie, there's no race window between them. If an interrupt comes through, HLT doesn't run. This doesn't happen for MWAIT or the ACPI sleep states and I am concerned we're still possibly hitting this race window from time to time. The specific behaviour is that the STI causes interrupts to be enabled following the next instruction. So yes, there's no window.

All that's left now is to make sure that interrupts are disabled before you do the scheduler check so no new interrupt processing on that thread can be scheduled. Ie, when entering cpu_idle():

• Call critical_enter();
• Disable interrupts;
• See if the scheduler has anything to do - if so, enable interrupts and skip calling the idle loop
• If there's nothing to do - and since interrupts are disabled, nothing new will have happened (like an interrupt scheduling an ithread) then just call the idle function
• Which may or may not enable interrupts before entering the idle loop (you enter ACPI with interrupts disabled, but you enter HLT with interrupts enabled)
• .. and then call critical_exit(), which will let the kernel continue preempting.

For MIPS however, there's a clever hack. (No, not from me.)

Here's how it works:

• In the idle loop, it calls mips_wait()
• mips_wait() is a bit of assembly code that will:
• disable interrupts
• see if the scheduler sees anything running
• .. and if so, it doesn't bother running the WAIT instruction! Just enable interrupts and jump over the WAIT;
• .. but the bit of code that re-enables interrupts and calls WAIT is aligned to a 16 byte boundary and the address is a symbol (MipsWaitStart).
• Then in the exception handling code (MipsKernIntr), it sees if the instruction pointer where the exception occured is in the 16 bytes (4 instructions) at MipsWaitStart.
• .. if it is, it adjusts the return address from the interrupt to be after the WAIT instruction.

It's totally dirty and to be quite honest, I haven't at all tested it. Yes, it should be tested.

bsdbox - take #1

One of the nice things about building embedded Linux "stuff" is busybox. The other is "uClibc", which isn't really the focus of this post.

I've fleshed out a basic busybox style thing based on the crunched binary stuff which generates sysinstall and the rescue binaries. Thankfully, most of the hard work (read: the hacks needed) are done for you - both in the rescue Makefile and the general FreeBSD build framework.

So, I give you "bsdbox" - a not-so-small static binary busybox style solution to help build a standalone image.

# ls -l /sbin/bsdbox
-rwxr-xr-x 1 0 0 2958668 Sep 16 17:44 /sbin/bsdbox
# /sbin/bsdbox
usage: bsdbox ..., where is one of:
ls cat dd df cp hostname kill mkdir sleep sh -sh dmesg sysctl init reboot
mount mdmfs mount_mfs mdconfig newfs ifconfig route ping true false hexdump
tail netstat chown chgrp arp hostapd hostapd_cli bsdbox

And with that, I have this TP-Link device happily running FreeBSD as a WPA-enabled wireless access point.

Combined with the GEOM uzip module, the entire filesystem (which is the above binary and a few shell scripts) - is just slightly above 1 megabyte in size. (geom_lzma will make that even smaller.)

The bsdbox stuff is in my GIT repository in "bsdbox". It's built as part of the base system and installed in /bsdbox/. The binary can then be cherry picked and built into an image with whatever symlinks are needed. There's more to do - more binaries for a start; but also making it a bit easier to configure bits than a single Makefile - but it's useful right now.

FreeBSD/MIPS: AR9132 porting

I decided to avoid work a little by porting FreeBSD/MIPS to the AR9132 in the TP-LINK WN-1043nd 802.11n wireless AP I have here. There's existing OpenWRT code for it so I figured the port wouldn't be that difficult.

It turns out I was right. The AR91xx support took a couple of days to massage into a form which was good enough to commit to FreeBSD-HEAD. I also included some basic AR713x SoC support - not enough to be completely useful, but enough to get a head-start on porting.

I broke out various CPU specific operations into a set of function pointers which a CPU detection function sets up early during the MIPS init code. The main differences between the SoCs are:

* Different base frequencies for setting up RAM, peripheral bus (eg UART), etc
* Different register locations for a few things
* The AR713x has a PCIe bus; the AR71xx has a PCI bus; the AR91xx doesn't have a PCI bus
* The AR9132 has different PLL values for the gigabit ethernet MACs
* Each SoC has a different USB peripheral setup path
* Each SoC has different GPIO layouts
* There's slightly different on-board peripheral reset registers

I've tested the code on the AR71xx and AR9132 and things work fine. I now need to port the AR9100 wireless MAC support from Linux ath9k to complete things. The AR9100 looks a lot like the AR5416/AR9160 (802.11n, 3 radio chains) but it has a few annoying differences:

* It isn't on a PCI bus, so it has to be manually attached;
* A few registers differ in locations (one of which returns 0xdeadbeef if the AR5416 register location is read!)
* The EEPROM is not in the same location(s) as the AR5416 - it needs to be copied and then shoe-horned into the AR5416 eeprom access code.

Also, there's an RTL8133RB gigabit switch device on-board. There's also code in Linux/OpenWRT to support this. I'll look at porting this once the AR9100 support is done.

FreeBSD/MIPS on AR71xx / Routerstation Pro: NIC alignment/size bug fixed!

I found and squished a small bug in the gigabit NIC driver in FreeBSD/MIPS for the AR71xx chipset. It wasn't all that complicated - TX buffers weren't being thoroughly enough checked for alignment/size constraints before being handed to the DMA engine.

It did however fix a few niggling issues. My tunneling stuff was fixed. IPv6 frames are now correctly handled. And ng_nat doesn't cause a panic in the NIC driver. So there's at least three people who are happy. :)

FreeBSD-current on the RouterStation Pro

I've been working on shoehorning FreeBSD-current onto the RouterStation Pro. It's a rather cheap but nice Atheros MIPS board by Ubiquity. It was tricky, but doable. I now have a cut down kernel and memory root filesystem stored on the on-board flash - enough to run it as a basic access point.

FreeBSD-current has support for the chipset - it's the AR71XX kernel config file. I've added a few more devices (USB, disk, MSDOS, redboot) to the default kernel. The default cross-build system works just fine.

I've been using TFTP loaded kernel+mdroot images to test out standalone functionality, along with TFTP + NFS root images to run a mostly-full development environment.

The board uses RedBoot as a bootloader. There's a tool by Ubiquity which generates firmware images which are understood by the bootloader and will overwrite the non-system area of the on-board flash. This makes dumping an image onto the system highly trivial. The bootloader can also boot the (uncompresed) kernel+mdroot images via TFTP as well as having a compressed version written to flash - so I can easily test the standalone images without constant re-flashing.

RedBoot makes it easy to TFTP upload a replacement flash image written out by the Ubiquity tool. It takes care of erasing, repartitioning and copying into the onboard flash without overwriting or damaging the RedBoot code, flash partition and configuration areas.

There was a bug with the PCI probe code until a couple of days ago where PCI cards (ie, stuff like the mini-PCI radio cards!) wouldn't be enumerated unless bootverbose was specified. Gonzo traced it down to a missing DELAY() - Linux was waiting a lot longer between probes than FreeBSD was. PCI devices now probe and attach correctly.

The geom_redboot module doesn't attach to the on-board flash (device "spi") - it only probes "cfi" flash devices. A quick patch to src/sys/geom/geom_redboot.c to also probe flash/spi has fixed this. I've asked Gonzo/Warner to investigate this and possibly commit a fix. With this in place, FreeBSD can mount a compressed, read-only filesystem from the "rootfs" flash slice rather than needing to pack a kernel+mdroot into the kernel flash slice.

So far, so good. I've written out a basic image that can be coaxed to be an open (no authentication) access point.

Interesting links:

Instructions on re-flashing the unit (but it doesn't mention that the reset button must be held down during power on!): http://www.usualcoding.eu/post/2010/01/15/Flash-OpenWRT-on-the-Ubiquiti-RouterStation-Pro

My FreeBSD router-station pro wiki: http://wiki.freebsd.org/AdrianChadd/UbiquityRouterstationPro

My work-in-progress tarball of "stuff" for building firmware images for the RouterStation Pro: http://people.freebsd.org/~adrian/rspro/