Some more Lusca Profiling..

More Lusca profiling of a simple test workload:

samples % image name symbol name

234266 6.7052 libc-2.3.6.so _int_malloc

138031 3.9507 libc-2.3.6.so vfprintf

111831 3.2008 libc-2.3.6.so calloc

104393 2.9879 libc-2.3.6.so malloc_consolidate

98984 2.8331 libc-2.3.6.so memcpy

91783 2.6270 libc-2.3.6.so _int_free

72578 2.0773 libc-2.3.6.so memset

69068 1.9769 libc-2.3.6.so free

50473 1.4446 squid clientTryParseRequest

50064 1.4329 squid memPoolAlloc

45211 1.2940 libc-2.3.6.so re_search_internal

40469 1.1583 squid httpRequestFree

39227 1.1228 squid statHistBin

38916 1.1139 squid comm_select

36974 1.0583 libc-2.3.6.so _IO_default_xsputn

36260 1.0378 squid memPoolFree

.. CPU is still taken up in the standard areas: memory allocation, stdio, and memcpy.

memcpy() I can't tackle right now. malloc() and friends I can tackle, but it may cause strange bugs and I'm not willing to commit anything which upsets stability too much right now. But vfprintf() is fun.

51285 22.7881 libc-2.3.6.so vsprintf

168714 74.9667 libc-2.3.6.so vsnprintf

156434 100.000 libc-2.3.6.so vfprintf

.. so most of the CPU is vsnprintf().

2549 1.0775 squid httpHeaderPutStrf

78253 33.0790 squid memBufVPrintf

153322 64.8121 libc-2.3.6.so snprintf

13601 100.000 libc-2.3.6.so vsnprintf

.. memBufVPrintf():

33231 36.8652 httpHeaderPutStrf

55577 61.6549 packerPrintf

7225 100.000 memBufVPrintf

.. snprintf():

5021 2.8271 squid clientAccessCheckDone

31897 17.9597 squid clientSendHeaders

53339 30.0327 squid xitoa

85548 48.1681 squid urlCanonical

7870 100.000 libc-2.3.6.so snprintf

.. so it's likely that eliminating the random calls through the printf() code to do "stuff" like assembling the URL and request/reply strings will shave off about 4% of this workload. But the biggest issue right now is the stupidly large amount of CPU being used in the memory allocation routines.

But the annoying one is memcpy():

10557 7.3665 squid httpAccept

20779 14.4992 squid stringDup

37010 25.8250 squid httpHeaderEntryPackInto

47882 33.4113 squid connStateFree

143177 100.000 libc-2.3.6.so memcpy

.. there's no memcpy() call in connStateFree. Which means I need to go hunting to figure out what's going on. Grr.

Why are some Squid/Lusca ACL types slower than others? And which ones?

This post should likely be part of the documentation!

One thing which hasn't really been documented is the relative speed of each of the Squid/Lusca ACL types. This is important to know if you're administering a large Squid/Lusca install - it's entirely possible that the performance of your site will be massively impacted with the wrong ACL setup.

Firstly - the types themselves:

1. Splay trees are likely the fastest - src, dst, myip, dstdomain, srcdomain
2. The wordlist checks are linear but place hits back on the top of the wordlist to try and speed up the most looked up items - portname, method, snmp community, urlgroup, hiercode,
3. The regular expression checks are also linear and also reshuffle the list based on the most popular items - url regex, path regex, source/destination domain regex, request/reply mime type

Now the exceptions! Some require DNS lookups to match on the IP of the hostname being connected to - eg "dst", "srcdom_regex", "dstdom_regex".

A lot of places will simply use URL regular expression ACLs ("url_regex") to filter/forward requests. Unfortunately these scale poorly under high load and are almost always the reason a busy proxy server is pegging at full CPU.

I'll write up an article explaining how to work around these behaviours if enough people ask me nicely. :)

Open Source Economy?

I'm reposting this from a Buzz (eww) that I responded to; I'd appreciate feedback and comments.

The article:

Lobby Group Says Open Source Threatens Capitalism

My response:

That article is .. well, a silly summary. It's comparing apples to oranges in a way.

Open/Free Source Software creates a very low entry barrier to a variety of interesting possibilities. It means companies can leverage this to create their own solutions without necessarily having to spend a large amount of money on in-house development or expensively licenced solutions. In a way, Open/Free Source is forcing a large part of the commercial market to compete better.

But the question is how you make that sustainable. Do I think the current method that Open/Free Source is used in companies is sustainable? It can be. It can not be. I've worked on open source projects in both camps.

Everything looks fine right now with a large wad of Open/Free projects because the popular ones have a lot of inertia behind them. But I wonder if there's longer-term flow-on effects in the economy. Plenty of companies which use and abuse open/free software don't contribute very much back to the projects. The cost savings they're passing on sound great in theory, but in practice the "community" is mostly covering this without their investment.

So the interesting question is - at what point (if any) does open/free software use tip the software economy to the point where developing new solutions is just not cost effective enough to compete with the established open source base; and what does that mean for future software development.

Lusca: more reorganisation, new features

More Lusca updates!

I've further split up the client-side processing. Refresh, IMS handling, ETag handling, location rewriting and various range request support code is now in separate modules. The code isn't all that much easier to follow if you don't have a rough idea what is going on but it's getting there. One of the long-standing issues in the Squid/Lusca codebase is how much of the caching logic is done as part of the client-side handling; I'd like to continue slowly unwinding this so I can start sliding in processing hooks in useful places. I have eventual evil plans here; I'll talk more when they're coming closer to fruition.

I've also been whittling away a little at the lack of code documentation for the client-side request and reply framework. I'll commit some more comments to client_side.c and the associated routines as I explore things some more.

I've got a very overdue contract to sort out a TOS bit tagging feature. I'm putting in the basic framework to make this happen. The first step is a new ACL type - dstfwdip - which is the destination IP being handed the request. There's a bunch of other side-changes which need to occur before I can slot in the TOS marking map logic (and I still haven't figured out where -that- will be!) so stay tuned. The aim is to provide a simple way to tag client (and maybe also server) requests with TOS bits based on some property of the request - and this includes the destination IP/network the request is being forwarded to.

ibook hackery, or "why apple probably did the magnetic connector stuff."

I didn't have the foresight to take pictures of the incident beforehand so I'll have to make do with a verbose description of the problem.

Symptom: the iBook just doesn't seem to handle power very well. "Jiggling" the connector around helps. But if it doesn't get jiggled right - and further jiggling added to the problem - the laptop won't charge fully or .. well, stay on.

It also can't be good for the power electronics to be given bursts of a couple amps of power every few seconds or so if I'm not paying attention.

So, after much anger, and Yet Another CD Not Ejecting Properly From The DVD Drive, I decided to strip the iBook down (again!) to take a look.

It takes quite a bit of disassembly to be able to reach the CD drive to replace it. It doesn't take all that much to get to the power input daughterboard. Then, removal of the daughterboard showed a couple of interesting issues. Firstly, there's some cracked/dry joints rocking hard around the power connector. Secondly, those now dry joints are black with what I'm guessing is a whole lot of DC electrical arc'ing going on.

A little bit of hot soldering iron action later and the damage was reversed. The power feed seems much more stable now. Thank god! I can stave off replacing the G4 iBook for another day.

This leads me to the subject of this post. Yes, the magnetic DC power connector Apple introduced into their laptop range is very nifty but it does make sense. Even just the normal insert/remove cycle of these power connectors could cause joints to crack and this sort of damage to occur. But there's a flipside - I've seen at least one magnetic connector blackened with DC arcing after something unknown to me had occured. So there's still apparently a slight chance of DC arc'ing happening - but my pet cat won't be destroying anything by dashing across the room and taking the laptop with it.

Lusca: more reorganisation

I've been doing some further Lusca re-organisation in preparation for the next round of evil changes.

The client-side code has been split up a little bit. The breakup has almost exclusively been shifting functions into separate source files to better delineate function - the only refactoring was to extract some common client-side connection creation from the HTTP and HTTPS connection paths.

My next little bit of fudgery will be to extract out some more of the storage related code into a top-level library so some further unit testing and code reuse becomes easier.

There's still far, far too much code in the client-side request and reply path. I think I've mentioned it earlier - there's very little code involved in the data pump between the server->client reply handling code; the majority of the performance issues stem from the initial request and reply processing.

FreeBSD-current on the RouterStation Pro

I've been working on shoehorning FreeBSD-current onto the RouterStation Pro. It's a rather cheap but nice Atheros MIPS board by Ubiquity. It was tricky, but doable. I now have a cut down kernel and memory root filesystem stored on the on-board flash - enough to run it as a basic access point.

FreeBSD-current has support for the chipset - it's the AR71XX kernel config file. I've added a few more devices (USB, disk, MSDOS, redboot) to the default kernel. The default cross-build system works just fine.

I've been using TFTP loaded kernel+mdroot images to test out standalone functionality, along with TFTP + NFS root images to run a mostly-full development environment.

The board uses RedBoot as a bootloader. There's a tool by Ubiquity which generates firmware images which are understood by the bootloader and will overwrite the non-system area of the on-board flash. This makes dumping an image onto the system highly trivial. The bootloader can also boot the (uncompresed) kernel+mdroot images via TFTP as well as having a compressed version written to flash - so I can easily test the standalone images without constant re-flashing.

RedBoot makes it easy to TFTP upload a replacement flash image written out by the Ubiquity tool. It takes care of erasing, repartitioning and copying into the onboard flash without overwriting or damaging the RedBoot code, flash partition and configuration areas.

There was a bug with the PCI probe code until a couple of days ago where PCI cards (ie, stuff like the mini-PCI radio cards!) wouldn't be enumerated unless bootverbose was specified. Gonzo traced it down to a missing DELAY() - Linux was waiting a lot longer between probes than FreeBSD was. PCI devices now probe and attach correctly.

The geom_redboot module doesn't attach to the on-board flash (device "spi") - it only probes "cfi" flash devices. A quick patch to src/sys/geom/geom_redboot.c to also probe flash/spi has fixed this. I've asked Gonzo/Warner to investigate this and possibly commit a fix. With this in place, FreeBSD can mount a compressed, read-only filesystem from the "rootfs" flash slice rather than needing to pack a kernel+mdroot into the kernel flash slice.

So far, so good. I've written out a basic image that can be coaxed to be an open (no authentication) access point.

Interesting links:

Instructions on re-flashing the unit (but it doesn't mention that the reset button must be held down during power on!): http://www.usualcoding.eu/post/2010/01/15/Flash-OpenWRT-on-the-Ubiquiti-RouterStation-Pro

My FreeBSD router-station pro wiki: http://wiki.freebsd.org/AdrianChadd/UbiquityRouterstationPro

My work-in-progress tarball of "stuff" for building firmware images for the RouterStation Pro: http://people.freebsd.org/~adrian/rspro/