Standalone HTTP header parser!

I've finally broken out enough of the HTTP header parsing code from src/ into libhttp/ to run the http header parser standalone.

This allows me to write some test cases to make sure I don't break things whilst changing how the HTTP header parser and HTTP header entry code uses (ie, abuses!) the memory allocator. It's also one step closer to being able to reuse bits of the Squid internals in a "simpler" HTTP proxy core.

I'll commit this code reorganisation to Cacheboy trunk after I've released and tested a few developer previews.

So, without further delay:

test1b: test parsing sample headers
| init-ing hdr: 0x7fffffffe6f0 owner: 2
| parsing hdr: (0x7fffffffe6f0)
Host: www.creative.net.au
Content-type: text/html
Foo: bar


| creating entry 0x60ed40: near 'Host: www.creative.net.au'
| created entry 0x60ed40: 'Host: www.creative.net.au'
| 0x7fffffffe6f0 adding entry: 27 at 0
| creating entry 0x60eda0: near 'Content-type: text/html'
| created entry 0x60eda0: 'Content-Type: text/html'
| 0x7fffffffe6f0 adding entry: 18 at 1
| creating entry 0x60ee00: near 'Foo: bar'
| created entry 0x60ee00: 'Foo: bar'
| 0x7fffffffe6f0 adding entry: 68 at 2
  retval from parse: 1
  Parsed Header: Host: www.creative.net.au
  Parsed Header: Content-Type: text/html
  Parsed Header: Foo: bar
| cleaning hdr: 0x7fffffffe6f0 owner: 2
| destroying entry 0x60ed40: 'Host: www.creative.net.au'
| destroying entry 0x60eda0: 'Content-Type: text/html'
| destroying entry 0x60ee00: 'Foo: bar'

IPv6 core merged into cacheboy trunk

I've just completed merging the IPv6 core into the cacheboy trunk. This doesn't mean it handles IPv6 client/server requests yet - there's a lot more to do before that can happen!

I'll next merge in the IPv6 DNS changes from husni's Squid-2.6 IPv6 patch and do up a basic test suite for all of that. Once done, I'll roll the first Cacheboy-1.5 pre-release.

Merging sockaddr_rework into Cacheboy trunk

I'm slowly cherrypicking bits and pieces of the Cacheboy sockaddr_rework into trunk. I've merged in the no_addr/any_addr tidyup which makes those comparisons and sets much clearer. I'll next bring over the sqinet_ routines as just files, ignoring the change history. I'll then bring over the changesets implementing the sqinet_ changes to the comm code and main codebase, retaining the basic change history.

I now need some live testing under decent amounts of real traffic so I can make sure I haven't missed some silly corner condition in the base comm code.

All of this work exposed some of the ugliness that happens in the IPC code with filedescriptor creation that bypasses the comm layer. Basically, the IPC helper code creates file descriptors itself and uses fd_open() to tell Squid about them but then unconditionally uses comm_close() to close them. This is .. stupid.

I may drop in some debugging code to ensure that only sockets created by the comm layer are closed by comm_close(). I wonder how many bad uses of file descriptors will be caught out by that..

Cacheboy IPv6 (phase 1): More Updates!

The IPv6 code hackery is going along well. I'm just sorting out making a few loose ends .. well, slightly less loose.

I'll run polygraph PolyMix-4 over this codebase in the next few days to make sure I haven't busted anything and then I'll start preparing to merge it back into CACHEBOY_PRE.

I'm not quite sure how to conditional-compile IPv6; I'm not bothering to do it at the moment (ie, its always included.) Thats a later problem.

The IPv6 TCP proxy is still happily chugging along. FreeBSD's IPv6 stack seems still partially Giant locked but I'm still pushing ~ 350mbit through this Core 2 Duo test server.

This has been too easy. What the hell have I missed!??

IPv6 tcp proxy success

I feel like an undergraduate computer science student after all of this. I've managed to coax the cacheboy core to support v4/v6 and am using it in the tcpproxy test application.

I've got a modified apachebench speaking IPv6 to tcpproxy, listening on a :8080 IPv6 socket. It then forwards all requests to a thttpd instance running on IPv4.

Tomorrow's job - making sure the squid proxy codebase is still happy with these latest changes, and then preparing for some further testing and the implementation of some unit tests for the comm and inet libraries. Then it back to commercial projects for a few weeks.

Server Software:        thttpd/2.25b                                       
Server Hostname:        [2a01:348:XXX:3207]
Server Port:            8080

Document Path:          /test8k
Document Length:        8192 bytes

Concurrency Level:      1000
Time taken for tests:   21.690 seconds
Complete requests:      100000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      844171764 bytes
HTML transferred:       819841632 bytes
Requests per second:    4610.42 [#/sec] (mean)
Time per request:       216.90 [ms] (mean)
Time per request:       0.22 [ms] (mean, across all concurrent requests)
Transfer rate:          38919.86 [Kbytes/sec] received

Cacheboy IPv6 (phase 1): Updates!

I've been working on the IPv6 core support in a cacheboy branch (Changes: http://code.google.com/p/cacheboy/source/list?path=playpen/sockaddr_change) and it seems to be coming along swimmingly.

The current goal is just to get basic IPv6 support into the base libraries and keep the rest of the codebase IPv4-only.

I've converted commBind(), comm_connect_addr() and comm_accept() to my new IPv4/IPv6 address type and nothing seems amiss at the present time. comm_open() and comm_openex() will take a little more time as there are plenty of places which create a new outgoing socket.

My next move is to modify my tcp proxy to listen on both IPv4 and IPv6 incoming ports and proxy to an IPv4 destination. I can then fire off some HTTP clients at it and see what happens.

(I may have to modify apachebench-adrian to support IPv6 though; I'm not sure what other stupidly-high-traffic open source http benchmarking clients exist at the present time.)

I hope to get all of this sorted out in the next week or so and head over to the Sydney Squid developers meet with my "alternate" IPv6 core for Squid-2 and better understand the IPv4/IPv6 requirements before discussing them with Amos.