More IPv6 hackery..

I've been spending a little time fleshing out some more of the IPv6 preparation work in Lusca. Since they're rather intrusive patches, I'm doing the work in a separate branch (/playpen/LUSCA_HEAD_ipv6) and will merge back bits and pieces as needed.

I've migrated the client db, external URL rewriters, access logging and the client-facing connection management code over to be IPv6 aware. I still have the request_t state, ACL lookups (which is luckily done - but sitting in a branch!), further DNS work and the protocol-facing stuff (HTTP, FTP.)

There isn't much more work involved in getting LUSCA_HEAD ready enough to serve IPv6-facing clients. That'll let me push out Cacheboy content over IPv6.

But for now, it's back to hacking on commercial, customer code.

Cacheboy: Firefox Release 3.6.4

Cacheboy is currently pushing a good 800-1200mbit of firefox 3.6.4 updates. It has about 6% of the total mozilla mirror weight so I predict that there's currently around 16 gigabits of total mozilla updates going out.

Lusca is holding up fine - a single process happily pushed ~ 400mbit on some hosts during the initial peak. I'd obviously like to be able to push a lot more than that but I'm still doing baby steps in the Lusca performance department.

More issues with Lighttpd

So occasionally Lighttpd on FreeBSD-7.x+ZFS gets all upset. I -think- there's something weird going on where I hit mbuf exhaustion somehow when ZFS starts taking a long time to complete IO requests; then all socket IO fails in Lighttpd until it is restarted.

More investigation is required. Well, more statistics are needed so I can make better judgements. Well, actually, more functional backends are needed so I can take one out of production when something like this occurs, properly debug what is going on and try to fix it.

Cacheboy Update / October/November 2009

Howdy,

Just a few updates this time around!

• Cacheboy was pushing around 800-1200mbit during the Firefox 3.5.4 release cycle. I started to hit issues with the backend server not keeping up with revalidating requests and so I'll have to improve the edge caching logic a little more.
• Lusca seems quite happy serving up 300-400mbit from a single node though; which is a big plus.
  
• I've found some quite horrible memory leaks in Quagga on only one of the edge nodes. I'll have to find some time to login and debug this a little more.
  
• The second backend server is now offically toast. I need to acquire another 1ru server with 2 SATA slots to magically appear in downtown Manhattan, NY.
  

Cacheboy downtime - hardware failures

Howdy,

I've had both backend servers fail today. One is throwing undervolt errors on one PSU line and is having disk issues (most likely related to an undervoltage); the other is just crashed.

I'm waiting for remote hands to prod the other box into life.

This is why I'd like some more donated equipment and hosting - I can make things much more fault tolerant. Hint hint.

My current wishlist

I'm going to put this on the website at some point, but I'm currently chasing a few things for Cacheboy:

• More US nodes. I'll take anything from 50mbit to 5gbit at this point. I need more US nodes to be able to handle enough aggregate traffic to make optimising the CDN content selection methods worthwhile.
• Some donations to cover my upcoming APNIC membership for ASN and IPv4/IPv6 space. This will run to about AUD $3500 this year and then around AUD $2500 a year after that.
• Some 1ru/2ru server hardware in the San Francisco area
• Another site or two willing to run a relatively low bandwidth "master" mirror site. I have one site in New York but I'd prefer to run a couple of others spread around Europe and the United States.

I'm sure more will come to mind as I build things out a little more.

New project - sugar labs!

I've just put the finishing touches on the basic sugar labs software repository. I'll hopefully be serving part or all of their software downloads shortly.

Sugar is the software behind the OLPC environment. It works on normal intel based PCs as far as I can tell. More information can be found at http://www.sugarlabs.org/

Cacheboy presentation at AUSNOG

I've just presented on Cacheboy at AUSNOG in Sydney. The feedback so far has been reasonably positive.

There's more information available at http://www.creative.net.au/talks/.

Cacheboy status update

So by and large, the pushing of bits is working quite well. I have a bunch of things to tidy up and a DNS backend to rewrite in C or C++ but that won't stop the bits from being pushed.

Unfortunately what I'm now lacking is US hosts to send traffic from. I still have more Europe and Asian connectivity than North American - and North America is absolutely where I need connectivity the most. Right now I'm only able to push 350-450 megabits of content from North America - and this puts a big, big limit on how much content I can serve overall.

Please contact me as soon as possible if you're interested in hosting a node in North America. I ideally need enough nodes to push between a gigabit and ten gigabits of traffic.

I will be able to start pushing noticable amounts of content out of regional areas once I've sorted out North America. This includes places like Australia, Africa, South America and Eastern Europe. I'd love to be pushing more open source bits out of those locations to keep the transit use low but I just can't do so at the moment.

Canada node online and pushing bits!

The Canada/TORIX node is online thanks to John Nistor at prioritycolo in Toronto, Canada.

Thanks John!

Cacheboy is on WAIX!

Yesterday's traffic from mirror1.au into WAIX:

ASN	MBytes	Requests	% of overall
AS7545	17946.77	7437	29.85	TPG-INTERNET-AP TPG Internet Pty Ltd
AS4802	12973.47	4476	21.58	ASN-IINET iiNet Limited
AS4739	8497.92	2947	14.13	CIX-ADELAIDE-AS Internode Systems Pty Ltd
AS9543	2524.57	1241	4.20	WESTNET-AS-AP Westnet Internet Services
AS4854	2097.32	941	3.49	NETSPACE-AS-AP Netspace Online Systems
AS17746	1881.17	1050	3.13	ORCONINTERNET-NZ-AP Orcon Internet
AS9822	1425.44	456	2.37	AMNET-AU-AP Amnet IT Services Pty Ltd
AS17435	1161.01	411	1.93	WXC-AS-NZ WorldxChange Communications LTD
AS9443	1140.62	701	1.90	INTERNETPRIMUS-AS-AP Primus Telecommunications
AS7657	891.93	1187	1.48	VODAFONE-NZ-NGN-AS Vodafone NZ Ltd.
AS7718	740.74	272	1.23	TRANSACT-SDN-AS TransACT IP Service Provider
AS7543	732.11	423	1.22	PI-AU Pacific Internet (Australia) Pty Ltd
AS24313	527.38	252	0.88	NSW-DET-AS NSW Department of Education and Training
AS9790	436.80	389	0.73	CALLPLUS-NZ-AP CallPlus Services Limited
AS17412	365.13	228	0.61	WOOSHWIRELESSNZ Woosh Wireless
AS17486	349.27	116	0.58	SWIFTEL1-AP People Telecom Pty. Ltd.
AS17808	311.65	248	0.52	VODAFONE-NZ-AP AS number for Vodafone NZ IP Networks
AS24093	303.40	114	0.50	BIGAIR-AP BIGAIR. Multihoming ASN
AS9889	288.85	197	0.48	MAXNET-NZ-AP Auckland
AS17705	282.49	84	0.47	INSPIRENET-AS-AP InSPire Net Ltd

Query content served: 54878.07 mbytes; 23170 requests.
Total content served: 60123.25 mbytes; 28037 requests.

BGP aware DNS

I've just written up the first "test" hack of BGP aware DNS.

The basic logic is simple but evil. I'm simply mapping BGP next-hop to a set of weighted servers. A server is then randomly chosen from this pool.

I'm not doing this for -all- prefixes and POPs - it is only being used for two specific POPs where there is a lot of peering and almost no transit. There are a few issues regarding split horizon BGP/DNS and request routing which I'd like to fully sort out before I enable it for everything. I don't want a quirk to temporarily redirect -all- requests to -one- server cluster!

In any case, the test is working well. I'm serving ~10mbit to WAIX (Western Australia) and ~ 30mbit to TORIX (Toronto, Canada.)

All of the DNS based redirection caveats apply - most certainly that not all client requests to the caches will also be over peering. I'll have to craft some method(s) of tracking this.

Updates - or why I've not been doing very much

G'day! Cacheboy has been running on autopilot for the last couple of months whilst I've been focusing on paid work and growing my little company. So far (mostly) so good there.

The main issue scaling traffic has been the range request handling in Squid/Lusca, so I've been working on fixing things up "just enough" to make it work in the firefox update environment. I think I've finally figured it out - and figured out the bugs in the range request handling in Squid too! - so I'll push out some updates to the network next week and throw it some more traffic.

I really am hoping to ramp traffic up past the gigabit mark once this is done. We'll just have to see!

VLC 1.0 released

VLC-1.0 has been released. The CDN is pushing out between 550 and 700mbit of VLC downloads. I'm sure it can do more but as I'm busy working elsewhere, I'm going to be overly conservative and leave the mirror weighting where it is.

Graphs to follow!

Current Downtime/issues

There's a current issue with content not being served correctly. It stemmed from a ZFS related panic on one of the backend servers (note to self - update to the very latest FreeBSD-7-stable code; these are all fixed!) which then came up with lighttpd but no ZFS mounts. Lighttpd then started returning 404's.

I'm now watching the backend(s) throw random connection failures and the Lusca caches then cache an error rather than the object.

I've fixed the backend giving trouble so it won't start up in that failed mode again and I've set the negative caching in the Lusca cache nodes to 30 seconds instead of the default 5 minutes. Hopefully the traffic levels now pick up to where its supposed to be.

EDIT: The problem is again related to the Firefox range requests and Squid/Lusca's inability to cache range request fragments.

The backend failure(s) removed the objects from the cache. The problem now is that the objects aren't re-entering the cache because they are all range requests.

I'm going to wind down the Firefox content serving for now until I get some time to hack up Lusca "enough" to cache the range request objects. I may just do something dodgy with the URL rewriter to force a full object request to occur in the background. Hm, actually..

New mirror node - italy

I've just turned on a new mirror node in Italy thanks to New Media Labs. They've provided some transit services and (I believe) 100mbit access to the local internet exchange.

Thanks guys!

And the GeoIP summary..

And the geoip summary:

From Sun Jun 7 00:00:00 2009 to Sun Jun 14 00:00:00 2009

Server	Country	MBytes	Requests
	us	5163783.09	6533162
	de	1514664.22	2307222
	ca	1152095.00	917777
	fr	948433.27	1451105
	uk	945640.71	1136455
	it	818161.03	770164
	br	542497.79	1426306
	se	482932.15	229559
	es	445444.34	647321
	pl	397755.30	1021083
	nl	373185.13	306023
	ru	368124.64	749924
	tr	293627.27	484965
	mx	276775.12	463252
	be	249088.62	213460
	ch	201782.33	209530
	ro	190059.45	274216
	fi	172399.75	204630
	ar	170421.77	374071
	no	169351.46	155258

A quick snapshot of Cacheboy destinations..

The following is a snapshot of the per destination AS traffic information I'm keeping.

If you're peering with any of these ASes and are willing to sponsor a cacheboy node or two then please let me know. How well I can scale things at this point is rapidly becoming limited to where I can push traffic from, rather than anything intrinsic to the software.

From Sun Jun 7 00:00:00 2009 to Sun Jun 14 00:00:00 2009

Time	Site	ASN	MBytes	Requests	% of overall
		AS3320	602465.01	1021975	3.26	DTAG Deutsche Telekom AG
		AS7132	583164.05	778259	3.16	SBIS-AS - AT&T Internet Services
		AS19262	459322.30	603127	2.49	VZGNI-TRANSIT - Verizon Internet Services Inc.
		AS3215	330962.95	553299	1.79	AS3215 France Telecom - Orange
		AS3269	317534.06	333114	1.72	ASN-IBSNAZ TELECOM ITALIA
		AS9121	259768.32	434932	1.41	TTNET TTnet Autonomous System
		AS22773	244573.65	283427	1.32	ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc.
		AS12322	224708.25	343686	1.22	PROXAD AS for Proxad/Free ISP
		AS3352	206093.84	305183	1.12	TELEFONICADATA-ESPANA Internet Access Network of TDE
		AS812	204120.74	166633	1.10	ROGERS-CABLE - Rogers Cable Communications Inc.
		AS8151	198918.22	328632	1.08	Uninet S.A. de C.V.
		AS6327	197906.53	152861	1.07	SHAW - Shaw Communications Inc.
		AS3209	191429.18	303787	1.04	ARCOR-AS Arcor IP-Network
		AS20115	182407.09	225151	0.99	CHARTER-NET-HKY-NC - Charter Communications
		AS2119	181719.20	117656	0.98	TELENOR-NEXTEL T.net
		AS577	181167.02	152383	0.98	BACOM - Bell Canada
		AS12874	172973.42	108429	0.94	FASTWEB Fastweb Autonomous System
		AS6389	165445.73	236133	0.90	BELLSOUTH-NET-BLK - BellSouth.net Inc.
		AS6128	165183.07	210300	0.89	CABLE-NET-1 - Cablevision Systems Corp.
		AS2856	164332.96	219267	0.89	BT-UK-AS BTnet UK Regional network

Query content served: 5234195.61 mbytes; 6878234 requests (ie, what was displayed in the table.)

Total content served: 18473721.25 mbytes; 26272660 requests (ie, the total amount of content served over the time period.)

Seeking a few more US / Canada hosts

G'day everyone!

I'm now actively looking for some more Cacheboy CDN nodes in the United States and Canada. I've got around 3gbit of available bandwidth in Europe, 1gbit of available bandwidth in Japan but only 300mbit of available bandwidth in North America.

I'd really, really appreciate a couple of well-connected North American nodes so I can properly test the platform and software that I'm building. The majority of traffic is still North American in destination; I'm having to serve a fraction of it from Sweden and the United Kingdom at the moment. Erk.

Please drop me a line if you're interested. The node requirements are at http://www.cacheboy.net/node_requirements.html . Thankyou!

Another day, another firefox release done..

The June Firefox 3.0.11 release rush is all but over and Cacheboy worked without much of a problem.

The changes I've made to the Lusca load shedding code (ie, being able to disable it :) works well for this workload. Migrating the backend to lighttpd (and fixing up the ETag generation to be properly consistent between 32 bit and 64 bit platforms) fixed the initial issues I was seeing.

The network pushed out around 850mbit at peak. Not a lot (heck, I can do that on one CPU of a mid-range server without a problem!) but it was a good enough test to show that things are working.

I need to teach Lusca a couple of new tricks, namely:

• It needs to be taught to download at the fastest client speed, not the slowest; and
• 
  
• Some better range request caching needs to be added.

The former isn't too difficult - that is a weekend 5 line patch. The latter is more difficult. I don't really want to shoehorn in range request caching into the current storage layer. It would look a lot like how Vary and Etag is currently handled (ie, with "magical" store entries acting as indexes to the real backend objects.) I'd rather put in a dirtier hack that is easy to undo now and use the opportunity to tidy up the whole storage layer a whole lot. But the "tidying up" rant is not for this blog entry, its for the Lusca development blog.

The hack will most likely be a little logic to start downloading full objects that aren't in the cache when their first range request comes in - so subsequent range requests for those objects will be "glued" to the current request. It means that subsequent requests will "stall" until enough of the object is transferred to start satisfying their range request. The alternative is to pass through each range request to a backend until the full object is transferred and this would improve initial performance but there's a point where the backend could be overloaded with too many range requests for highly popular objects and that starts affecting how fast full objects are transferred.

As a side note, I should probably do up some math on a whiteboard here and see if I can model some of the potential behaviour(s). It would certainly be a good excuse to brush up on higher math clue. Hm..!