I've made some good progress with the AR9160 802.11n hostap mode in FreeBSD.
Firstly, the A-MPDU density setting needed tuning. "0" or "N/A" isn't appropriate for the AR9160 (and likely not appropriate for anything earlier than the AR9300/AR9400 series.) Once set to 8 microseconds, the number of A-MPDU retransmits dropped dramatically.
I found that there was some missing code to do with disabling RIFS (reduced inter-frame spacing) RX on the AR9160 which I committed to FreeBSD-HEAD. This has eliminated all of the baseband lockups I've been seeing, rare as they were now.
Then I found there were packets being dropped by if_arge on TX. It turns out the default TX/RX ring buffer size for if_arge was 128 packets - definitely not enough given the uncoupled interrupt/process driven forwarding engine in UNIX these days.
The background: since NIC drivers aren't doing their work in an interrupt or shared process context, they only do their work when their TX/RX tasks get scheduled. Since they don't have any way to signal each other to "back off" via flow control when the queues are getting filled, it's quite possible to have a very busy device (in this instance wlan0 RX'ing anything more than around 100mbit) queue enough packets to the devices' TX queue faster than the TX task can be scheduled to process packets.
Bumping if_arge's default TX/RX ring buffer size from 128 to 1024 did the trick - no more packet drops on TX.
Now I'm getting ~ 105mbit TCP RX in HT/40 5GHz hostap mode (since A-MPDU RX is implemented and working) and ~ 210mbit UDP RX before I begin to drop packets. Anything more than 210mbit starts resulting in the infamous "stuck beacon", but that could be for a variety of reasons. I don't think I'm saturating the PCI bus (read: the CPU scheduler does show about 30% idle clock cycles during a UDP RX test) so there's something else to be blamed.
Next is figuring out whether there's some scheduling issues with the device TX/RX tasks.
I find it quite creepy that I can get ~ 100mbit of 802.11n throughput to my macbook pro whilst lying in bed. But that's what 802.11n is for, right? :)