Unintentional side-effects - IPv4/IPv6 load balancing..

Here's a fun one:

1310397101.830 416 X TCP_MISS/200 700 GET http://www.freebsd.org/layout/images/front_get_back.png adrian DIRECT/[2001:4f8:fff6::22] image/png
1310397101.835 413 X TCP_MISS/200 851 GET http://www.freebsd.org/layout/images/front_get_tr.png adrian DIRECT/69.147.83.34 image/png

Since the ipcache code already balances outbound connections between multiple hosts, any IPv6 hosts get similarly load balanced.. between IPv4 hosts as well.

I may end up adding in logic to connect to "only v4" and "only v6" in forward.c...

Lusca update: IPv6 server work is now working

I'm now using the Lusca IPv6 branch for browsing both IPv4 and IPv6 sites.

There's a bunch of little things to update and test, including testing the neighbor selection, peer and ICP code. It's still not yet doing transparent interception or TPROXY - that'll be among the last thing to work on. Finally, there's a bunch of connection timeout handling code which currently is being ignored - there's no convenient place to attach a timeout handler now since the file descriptor isn't created up front; it's created only once the DNS resolution is complete.

My main focus at the moment is testing and verifying the peer, neighbor selection and ICP code out.

But yes, it is actually working!

Lusca development update - IPv6 is almost working

Now that I've (hopefully!) completely finished with university, I can get back into using my hard-earned money from Xenion to get more Lusca development done. (And yes, I'll also be doing wireless development too, fear not.)

The IPv6 branch is a bit messy at the moment, but it's almost able to handle IPv6 server requests.

The problem? The existing code which handles connecting to remote hosts (ie, src/comm.c) doesn't "know" about IPv6. It assumes all sockets are IPv4 and that all hostnames which are returned are also IPv4.

There's unfortunately a lot of dirty code in there - commReuseFD() is the main culprit and a good example of this. The 30 second version - since the commConnectStart() API assumes the socket is already created before the connect() occurs, any "connect retry" (for multiple hostnames and multiple attempts at the same end-host) requires the socket to be closed and recreated. But the FD has to stay the same. So commReuseFD() manually creates a new FD, makes it look like the old FD, then calls dup2() to get it into the same FD as the old one.

The "Correct" Fix is to modify the API to not take an FD, but to return an FD on successful connect() to the remote destination. There's some problems with this though, most notably in the request forwarding layer where the FD is created and comm close handlers are assigned before the connection is attempted. I need to make sure that there's no code which calls comm_close() on the active connection whilst connect() is going on - as said code expects the FD to be valid and assigned by this point.

The "Dirty" fix is to modify commConnectStart() and commReuseFD() to check the FD address family and destroy/create a "new" socket with the correct address family.

The "Problem" is that the code allows the outgoing address to be specified, both for transparent interception (source address spoofing) and to be set via an ACL match. Since IPv4 and IPv6 addresses are now possible, the API will have to be modified to handle this case.

What I'm likely going to do is something inspired by Squid-3. I'll teach the forwarding layer about "try v4 destinations" and "try v6 destinations". The administrator can then configure whether to try v4 or v6 destinations first. Only one outgoing address has to be provided - either "v4" or "v6"; and commConnectStart() will only try connecting to IP addresses that match the family of the outgoing address. That way if a host resolves to a mix of v4 and v6 addresses, they'll be tried in a "v4" group, then a "v6" group (or vice versa.) It's a bit dirty, but it's likely doable in the short-term.

In the long term, I'd like to fix the API up to be less messy and return an FD, rather than take an existing FD and abuse that. But that can come later.

LUSCA ipv6 - almost there (again!)

I've managed to now get the Lusca IPv6 code to perform IPv6 lookups, open a socket to the remote host and then try an IPv6 connection.

But this doesn't work - the legacy Squid-2 comm code does this:

• Create an AF_INET socket via comm_open();
• Pass that socket to commConnectStart() with the relevant hostname or IPv4 address to connect to.

If the host is an IPv6 host, the IPv4 socket won't connect to it.

So now I'll have to change the comm API slightly to make commConnectStart() return a FD on completion, rather than taking a socket to connect to. This will also fix a long-standing hatred of mine - where Squid/Lusca opens up a socket that just sits there until DNS lookups succeed. Ew.

Lusca and IPv6 - almost there

Today's task - tying together the bits of Lusca's IPv6 work to be able to do an IPv6 AAAA lookup, join it with an IPv4 A lookup, then try to connect to IPv6/IPv4 hosts.

The result:

IP Cache Contents:

Hostname Flg lstref TTL N
www.freebsd.org 4 21595 2( 1) [2001:4f8:fff6::22]-BAD 69.147.83.34-OK

So far, so good. I don't have it yet connecting to IPv6 destinations, but all the right bits are in play to be able to do so.

There's plenty of bits in the Squid-3 DNS code that work around potential bugs/gotchas in DNS vs EDNS handling when handling some of the larger IPv6 record replies seen in the real world. I'm going to commit what I have thus far and then look at leveraging what they've done.

LUSCA and IPv6

The IPv6 development branch (/playpen/LUSCA_HEAD_ipv6 in subversion) now handles IPv6 HTTP clients. It is still limited to IPv4 servers and other protocols (snmp, icmp, htcp, ftp) are all still IPv4.

But it's a (long overdue) start.

I'll look at next fleshing out IPv6-aware SNMP, ICP and HTCP as they're reasonably easy - I just need to change the socket handling to be IPv4/IPv6 agnostic.

I'll begin merging some of the simple framework changes back to mainline Lusca once I've committed that memory leak fix that I've been discussing.

More IPv6 hackery..

I've been spending a little time fleshing out some more of the IPv6 preparation work in Lusca. Since they're rather intrusive patches, I'm doing the work in a separate branch (/playpen/LUSCA_HEAD_ipv6) and will merge back bits and pieces as needed.

I've migrated the client db, external URL rewriters, access logging and the client-facing connection management code over to be IPv6 aware. I still have the request_t state, ACL lookups (which is luckily done - but sitting in a branch!), further DNS work and the protocol-facing stuff (HTTP, FTP.)

There isn't much more work involved in getting LUSCA_HEAD ready enough to serve IPv6-facing clients. That'll let me push out Cacheboy content over IPv6.

But for now, it's back to hacking on commercial, customer code.

Preparation for next release; IPv6 checklist

I've been slowly working on tidying up the codebase before the next snapshot release. I've been avoiding doing further large scale code reorganisation until I'm confident that this codebase is as stable and performs as well as it should.

I'll hopefully have the next stable snapshot online tonight. I'll then re-evaluate where things are at right now and come up with a short-list of things to do over the next couple of weeks. It'll almost certainly be the remainder of the IPv6 preparation work - I'd like to prepare the last few bits of infrastructure for IPv6 - and make certain that is all stable before I start converting the client-side and server-side code to actively using the IPv6 routines.

The current IPv6 shortlist, if I decide to do it:

1. client database code - convert to a radix tree instead of a hash on the IP address; make IPv4/IPv6 agnostic.
2. persistent connection code - up the pconn hash key length to fit the text version of the IPv6 address. I'll worry about migrating the pconn code to a tree later on.
3. Importing the last remaining bits of the IPv6 related code into the internal DNS code.
4. Make sure the internal and external DNS choices both function properly when handling IPv6 addresses for forward and reverse lookups.
5. Import the IP protocol ACL type and IPv6 address ACL types - src6 and dst6.
6. Modify the ACL framework to use the IPv6 datatype instead of "sockaddr_in" and "inaddr" structs; then enable src6/dst6.
7. Make certain the source and destination hostname ACLs function correctly for both IPv4 and IPv6.
8. Test, test, test!

The last time I did a "hack" conversion to support IPv6 client side code I found a number of places which expected a newly-allocated struct to be zero'ed, and thus the "in_addr" embedded inside it to be INADDR_ANY. This caused some crashes to occur in production testing. I'm thus going to hold off on pushing through the IPv6 client side changes (which are actually surprisingly simple once the above is done!) until I've enumerated and fixed all of those particular nightmares.

The IPv6 server-side stuff is a whole different barrel of fun. I'm going to ignore a lot of that for now until I've made certain the client-side code is stable and performing as well as the current IPv4-only code.

I don't even want to think about the FTP related changes that need to occur. I may leave the FTP support IPv4 only until someone asks (nicely) about it. The FTP code is rife with C string pointer manipulations which need to be rewritten to use the provided string primitives. I'd really like to do -that- before I consider upgrading it to handle IPv6.

Anyway. Lots to do, not enough spare time to do it all in.

Serving IPv6 from Cacheboy-1.6.PRE2

I've done the very minimum amount of work required to get Cacheboy-1.6.PRE2 to the point where it'll handle IPv6 client requests. I've put it in front of http://www.cacheboy.net/ which now has v4 and v6 records.

There's still plenty of work to do to bring it up to par with the Squid-3 IPv6 support but that will have to wait a while. Specifically, (if anyone feels up to handling it), the dns, ipcache and fqdncache code all needs to be massaged to support IPv4 and IPv6 handling. It shouldn't be that much work.

Cacheboy-1.6 is definitely now in the "freeze and fix bugs as they creep up" stage. I'll continue the memory allocator and HTTP parser code reimplementation in their respective branches and get them ready for merge once I'm happy 1.6 is stable. The rest of the IPv6 support will also have to wait.

Cacheboy-1.5: IPv6 DNS servers

I'm just debugging the last couple of issues with the IPv6-aware UDP/TCP DNS code. The Internal DNS resolver still only understands IPv4 code (and, more importantly, the ipcache/fqdncache layer too!) but the code itself will communicate with IPv4/IPv6 DNS servers.

I think I'll stop the development here and concentrate on getting the Cacheboy-1.5 release out the door. I'll then work on IPv6 record resolution in a seperate branch in preparation for Cacheboy-1.6. I may even break out the ipcache/fqdncache code into external libraries so I can reuse/debug/test that code during development.