Category Archives: atheros

Don’t let anyone tell you that FreeBSD doesn’t "do" 802.11n:

This is from my FreeBSD-HEAD 802.11n access point, currently doing ~ 130MBit/s TCP:



# athstats -i ath0
41838297 data frames received
31028383 data frames transmit
78260 short on-chip tx retries
3672 long on-chip tx retries
197 tx failed 'cuz too many retries
MCS13 current transmit rate
8834 tx failed 'cuz destination filtered
477 tx frames with no ack marked
239517 rx failed 'cuz of bad CRC
10 rx failed 'cuz of PHY err
10 OFDM restart
42043 beacons transmitted
143 periodic calibrations
-0/+0 TDMA slot adjust (usecs, smoothed)
45 rssi of last ack
51 avg recv rssi
-96 rx noise floor
812 tx frames through raw api
41664029 A-MPDU sub-frames received
42075948 Half-GI frames received
42075981 40MHz frames received
13191 CRC errors for non-last A-MPDU subframes
129 CRC errors for last subframe in an A-MPDU
2645042 Frames transmitted with HT Protection
351457 Number of frames retransmitted in software
23299 Number of frames exceeding software retry
30674735 A-MPDU sub-frame TX attempt success
374408 A-MPDU sub-frame TX attempt failures
8676 A-MPDU TX frame failures

443 listen time
6435 cumulative OFDM phy error count
161 ANI forced listen time to zero
3672 missing ACK's
78260 RTS without CTS
1469003 successful RTS
239605 bad FCS
2 average rssi (beacons only)
Antenna profile:
[0] tx 1466665 rx 1
[1] tx 0 rx 41838296

A tale of two sequence numbers, or "when QoS seqno and CCMP PN don’t match up"..

Many moons ago (say, 3 or 4 weeks - so hm, most-of-a-moon-ago actually) I found a rather curious failure condition in the ath(4) TX aggregation path. The colourful history is documented in FreeBSD kern/166190. In short - there are situations where sequence numbers were allocated in a different order to how frames were being added to the block-ack window tracking, and if you got unlucky, you'd cause the stack to think a frame was (far) outside the BAW.

The 30 second explanation:

Imagine you allocated four frames - sequence numbers 1, 2, 3 and 4. They have to be added to the block-ack window in precisely that order. Ie:

  1. Starting condition: Window is at 0:63 (64 frame window, starting at 0, so ending at 63)
  2. Add 1: Window is now at 0:63, starting at 1
  3. Add 2: Window is now at 0:63, starting at 2
  4. Add 3: Window is now at 0:63, starting at 3.
The reason the window pointer isn't moving along is because although you've sent the frames (or you're about to), you can't advance it until the other end has ACKed it (via a block-ack or a normal ACK.) For more information, google how 802.11n aggregation works.

The important bit here is that the window is still 0:63 and the starting point is now '3'. This continues all the way to trying to queue frame 64, where it will be outside of the current BAW and not be allowed to be transmitted. It'll sit in the software queue and wait until frame '0' has been ACKed and the BAW has been advanced to be 1:64 - at which point 63 will fall inside the window and will be transmitted.

So yes, the sender is tracking two things - the BAW and what the starting point is that they've added to the BAW.

Now, imagine instead of (1, 2, 3, 4) on the software queue, I somehow get preempted (or race between two sending threads, when using SMP) between 'allocated seqno' and 'queue to software queue'. In the existing code, a lock was held when:
  • Allocating a sequence number, then it was dropped; then
  • Adding it to the software queue.
Now because there was a period where no lock was being held, it's quite possible that what ends up on the software queue is (2, 1, 3, 4.) So:
  1. Starting condition: Window is at 0:63
  2. Add 2: Window is now 0:63, starting at 2.
  3. Add 1: Window is 0:63, starting at 2; 1 is outside of the BAW (it's treated as a 'wraparound', so imagine it's 4095 seqno's away) so TX stalls.
This was the cause of the TX stalls that I was seeing originally in kern/166190. I "fixed" it by only allocating sequence numbers when the frame was about to be transmitted for the first time, and then adding it to the BAW right there. Since both sequence number allocation and adding to the BAW happened inside the same lock, everything was sweet.

Except, I totally forgot about CCMP PN. So under high enough UDP TX loads (say, > 200MBit), I'd hit the same race, but between 802.11 sequence numbers and CCMP PN sequence numbers.

CCMP PN is assigned during 802.11 encapsulation time, in the driver. In the ath(4) case, it's done during transmit and before being queued to the software queue. And it was being done outside of any locking.  So it's very possible that frames would end up on the software queue with 802.11 and CCMP PN sequence numbers out of lock-step.

What would happen?

Simply - after the 802.11n reordering occured on the receive side, the CCMP PN replay detector would notice sequence numbers out of order, and start tossing said out of order frames. Lots of packet loss ensued.

So, I sat down and started trying to address it. The simplest thing - wrap the whole encapsulation path between ieee80211_crypto_encap(), 802.11 sequence number assignment and software/hardware queueing behind the TID (well, hardware TXQ) lock. It took some time; I had to revert two earlier commits which introduced the delayed sequence number allocation.

This didn't fix things. So I was back to square one.

I started looking at all the places where the frames were being queued to the software queue and .. well, let's just say I spent Sunday swearing _at myself_ for all the weird and wonderful stupid mistakes I had made when writing/porting this code over.

The short version follows (the long version is "read the sys/dev/ath/ commit logs and the PR history"):
  1. When I was queueing frames to the software queue, I'd check how deep the hardware queue was. If the hardware queue was shallow/empty, I'd direct dispatch up to two frames to the hardware to get things 'busy'. That will (hopefully) let further frames come along in the meantime and be aggregated. However, I was queueing the new frame to the hardware rather than queueing the new frame to the tail of the queue, and queueing the head frame of the queue to the hardware. That led to some out of order behaviour.
  2. ath_tx_xmit_aggr() would check if the sequence number was within the block-ack window and if it wasn't, it'd queue the frame to the tail of the queue. This meant that any new frames that came along would be queued to the end of the queue, even if they had been dequeued from the head of the queue. This lead to frames on the software queue being out of order.
    1. Frames on the software queue don't have to be in-sequence (as retries are prepended to the beginning of the list, and new frames are appended to the end) however they have to be in-order. If they end up being out of order, the BAW logic fails.
So, now that I allocate sequence numbers at packet queue time, I have to be triply sure that what ends up on the software queue is correctly in order, or the BAW logic will cause traffic stalls and potentially duplicate sequence number issues. Yes, this means that the old behaviour, whilst it now works right with all the right locking, requires me to correctly handle putting frames on the software queue. (Or, as I like to say, "keeping the bastard (me) honest.")

TL;DR - 802.11n aggregation works again. Now, to fix those pesky "queue full and I want to send a BAR frame so I can unblock the full queue and transmit" problems. At least that one is more tractable and easier to solve. Or is it.


Fixing BAR handling and handle corner cases of things..

Part of the general 802.11n requirements is correctly handling TX aggregation and failures - where we need to pause the TX queue, send a BAR to forcibly move the remote end block-ack window to be in sync with the transmitters idea, and then continue sending frames.

This exposed a very annoying problem - what if the driver runs out of ath_buf entries to schedule TX frames? Or, what if the network stack runs out of mbufs? If we need to allocate an ath_buf/mbuf to send a BAR frame, but they're all allocated and unavailable, the driver/wireless stack will come to a grinding halt. Typically these allocated ath_buf's are allocated in the software queue, waiting for the BAR TX (or power-save wakeup) to send a frame.

So, I haven't fixed this. It's on my (very) short term to-do list. But it did expose some issues in how the net80211 BAR send code (ieee80211_send_bar()) works. In short - it didn't handle resource allocation failures at all. It worked fine if the driver send method (ic->ic_raw_xmit()) succeeded and just failed to TX the frame. But if it couldn't allocate an mbuf, or if the driver send method failed.. things just stopped. And when the BAR TX just stopped, the ath(4) software TX queue would just keep buffering frames, right until all the TX ath_buf entries were consumed.

This is obviously .. sub-optimal.

But this raises an interesting point - how much of your kernel and/or userland application handle resource shortages correctly? I've seen plenty of userland software just not check whether malloc() returned NULL and I've seen some that specifically terminate (non-gracefully) if malloc()/calloc() fails - Squid does this. But what about your network stack? How's it handle mbuf shortages? What about the driver stack? What about net80211 (ew) ? What if the kernel malloc() API has to sleep because there's no free memory available?

I don't (currently) have an answer - it's a difficult, cross-discipline problem. What I -can- do though (at least in my corner of the FreeBSD world - net80211 and ath(4)) is to start testing some of these corner cases, where I  force some resource shortages and ensure that the wireless stack and driver(s) recover somewhat gracefully. 802.11n is very unforgiving if you start dropping frames involved in an active aggregation session. So it's best I try and address these sooner rather than later.

And the winner of the most committing committer to src/sys over the last 12 months is ..

.. well, it was me. Up until last week. marius@ snuck in to take my place. I guess all of those commits I did about this time last year to start bringing FreeBSD's ath(4) support up to scratch are finally expiring out of the 12 month window.

(Source: http://people.freebsd.org/~peter/commits.html)

.. but I wouldn't call myself the most important committer. Or the most active. What I'd call myself is the "most active fixing a sorely needed corner of the codebase."

What I _could_ have done is simply do all my work in a branch and then merge it back into -HEAD when I was done. And, for about 6 months, this is what I did. The "if_ath_tx" branch is where I did most of the initial TX aggregation work.

But as time goes on, your branch diverges more and more from the master branch (-HEAD in FreeBSD) and you are faced with some uncomfortable decisions.

If you stay on the same branch point and never merge in anything from your master branch, you _may_ have a stable snapshot of code, but who knows how stable (or relevant) your work will be when you merge it back into master.

You have no idea if your work will break anything in master and you have no idea if changes in master have broken your work.

As time goes on, the delta between your branch point and the master branch increases, making it even more difficult to do that final merge back. It also has the side effect of making it increasingly likely that problems will occur with the merge (your code breaking master, master breaking your code, etc.)

So as uncomfortable as it was - and as much as I wanted things to stay stable - I did press through with relatively frequent merging. This means:
  • I would pick specific development targets to work towards, at which point I'd stop developing and go into a code review/tidyup/testing phase;
  • I'd do frequent merges from master back into my branch during active development - I wouldn't leave this until I was ready to merge my work back into master;
  • Once I reached my development target and had done sufficient testing - including integrating changes from master back into my branch - I'd kick off a semi-formal review (read: email freebsd-wireless@) and call for testers/review;
  • Only _then_ would I merge what was suitable back into master.

I wouldn't merge everything from my branch into master. In my instance, there were some debugging extensions that were easy to maintain (read: lots of device_printf() calls) but weren't suitable for FreeBSD-HEAD. But I merged the majority of my work each time.

But that doesn't always work. I managed to merge a bunch of ath(4), ath_hal(4) and net80211 fixes back into -HEAD as appropriate. But the TX aggregation code was .. well, rather large. So I attempted to break up my commit into as many small, self-contained functional changes as possible. Yes, there was a big "here's software TX queue and aggregation" as a big commit at the end but I managed to peel off more than 30% of that in the lead-up commits.

Why bother doing that?

Two words - version bisection. Once I started having users report issues, they would report something like "FreeBSD-HEAD revision X worked, revision Y didn't." (If I were lucky, of course.) Or, they'd note that a certain snapshot from a certain day worked, but the next day had a regression. If I had committed everything as one enormous commit after having spent 6 + months on the branch, I'd be in for a whole lot of annoying line-by-line debugging of issues. Instead, I was able to narrow down most of the regressions by trying all the different commits.

Now that 802.11n ath(4) TX aggregation and general 802.11n support is in the tree, I only use branches for larger scale changes that take a couple of weeks. For example, when fixing up the reset path to not drop any TX/RX frames. I do most of the bugfixing in FreeBSD-HEAD. I could do it in a branch and then do monthly merges, but I then have the same problems I've listed above.

In summary: don't underestimate how helpful it is to break down your commits into little, piecemeal, self-contained functional changes. It has the side effect of making you look really good in the committer statistics.

The initial introduction into "it’s the NIC, stupid!"

I've finally hit that dreaded condition which hardware guys hate - where a NIC is behaving badly.

In my case, an IBM/Lenovo Thinkpad T60 has been modified (not by me) to take an Atheros AR9280 NIC. Unfortunately, the NIC was proving to be very unstable when doing 802.11n throughput. The investigations did show I was doing something slightly incorrect with TX descriptors (and I've since fixed that) but the stability issues remained.

The Atheros NICs can expose some host interface error conditions via the AR_INTR_SYNC_CAUSE register. These include PCI(e) transaction timeouts, illegal chip access (eg whilst the MAC is asleep), parity errors, and other rather nice things. FreeBSD's HAL and Linux ath9k does have the register definition for what the bits do - but unfortunately we don't keep statistics.

In my particular case:
  • I'd see AR_INTR_SYNC_LOCAL_TIMEOUT occur. This is because a PCI(e) transaction didn't complete in time. I can tune these timeouts via a local register but that's not the point - I was seeing these errors when receiving only beacons from the access point. That's a bit silly.
  • I'd also see AR_INTR_SYNC_RADM_CPL_DLLP_ABORT, which is an indication that the PCIe layer isn't behaving well.

I swapped it out with another AR9280 based NIC and suddenly all the instabilities have gone away. No TX hangs, no missed TX interrupts. Everything looks great.

So as an open source developer, I want to try and put some tools into the hands of the community to be able to debug what's going on - or, if that's not possible, at least get some indication that things are going wrong. Right now the only thing people see is "I see TX timeouts, it must be the driver/chip fault." There's too much going on to be able to conclude that.

My game plan is this:

  • Implement statistics keeping for each of the SYNC interrupts and expose those via a diagnostic interface. Ben Grear has done something similar for Linux ath9k after a private email discussion. He's also seeing MAC sleep accesses, so it's quite likely we'll start finding/squishing these.
  • Take the offending laptop/NIC to the office and attach it to a very expensive and fancy looking PCIe analyser. I'm hoping we'll find something really silly occuring - like lots of sleep state transitions, or a high number of parity errors.
  • Try documenting this a lot better so users are able to understand what's going on when their NIC is misbehaving.

Concurrency in the TX path and when it all falls down..

I'm still (yes, still) hacking on the FreeBSD 802.11n ath(4) TX path. I'm trying to find and fix issues that creep up before I can flip on the 11n code by default.

I've become increasingly aware of the lack of locking in net80211, including some of the A-MPDU TX session management code. I know that I'll eventually have to plan out and implement some locking, but for now I'm just trying to squish whatever issues are showstoppers.

A user approached the freebsd-wireless list about two weeks ago and noticed that his 802.11n session was hanging after a bit of use. If he tried 802.11g, things worked fine. He tried SMP - and things worked fine. The symptoms? A number of frames seemed to be "stuck" and sitting in a software TX queue, not being transmitted. But other frames would be TXed fine, so it wasn't as simple as a totally confused TX BAW (Block-ack Window.)

After I managed to reproduce the issue locally, I discovered what was going on.

It was concurrency.

Specifically, that there's multiple places where ath_start() is being called, thus multiple concurrent TX is occuring.

Now, in the non-aggregate method, net80211 is doing all the sequence number assignment. I'm not so sure that in the normal case, the sequence numbers are being allocated in a consistent, sequential way - if the net80211 TX code is able to be called concurrently from multiple threads, sequence numbers can and will be occasionally "raced" and allocated in the reverse order that they're submitted to the driver. But I'm not here to fix that (however I'll eventually have to.)

In the aggregate method, the driver is doing the aggregation and sequence number assignment. For now, the driver is also doing the TX BAW tracking and frame queuing. So imagine this sequence of events:

* a frame is submitted via ath_start();
* since aggregation is enabled, it's allocated a sequence number;
* it's then thrown into the software queue;
* then at some later stage, the software queue is checked, the frame is popped out of the list and if it's inside the BAW, it's added to the BAW and TX;ed
* adding the frame to the TX BAW slides the left hand edge of the BAW to be at that sequence number. Any frames TXed with a seqno _less_ than this will be treated as outside of the BAW and put back onto the software queue for now.

Now, the locking only occurs at:

* the time the frame has the sequence number allocated;
* then when the queue is checked and the frame is popped off.

If two or more threads are allocating sequence numbers and doing work, it's quite possible that thread A will allocate (say) seqno 5, thread B will allocate seqno 6 and then add it to the BAW before thread A can. Then when thread A tries to do some work, it finds the queue has a frame with seqno 5 in it - and since it's before the left hand edge of the BAW (which is at 6, as it was successfully pushed to the hardware), it won't be transmitted and will stay in the software queue until the BAW sequence numbers wrap around to 0 and catch up.

Now, linux ath9k/mac80211 doesn't have this problem. The TX pathway is totally serialised, which means that even if multiple threads are trying to TX, only one thread will be able to enter the TX code at any time. The other threads get blocked.

So how can I solve this? The easy solution would be to serialise FreeBSD's net80211 and ath driver TX code. That way all of this nonsense will go away. For the net80211 side of things this may work - the legacy TX path, where sequence numbers are allocated by the stack, could benefit from serialisation. The throughput isn't ever going to be that great, so we wouldn't really hurt from it. But the trouble is making absolutely sure that the driver also does the same - even if I push the frames into the queue in order and ensure that they have sequential sequence numbers, there's no guarantee that a driver with concurrent entry paths into XXX_start() will de-queue the frames and push them into the hardware in the same order.

So what I've chosen to do instead is to ignore the legacy part for now, and not serialise anything. Instead, I'm doing the sequence number allocation (for aggregation, remember) -at the time I'm about to add the frame to the BAW and TX it to the hardware-. Ie, until the frame actually is able to be added to the BAW, it won't _have_ a sequence number. Since this action is done behind a lock, it's guaranteed to be sequential. The trick here is to only allocate the sequence numbers once it's known for certain that the frame _will_ be going out to the hardware.

For the legacy path, it's also likely worthwhile delaying the sequence number allocation until it's just about to go to the ifnet queue. That way the frames on the ifnet queue have sequence numbers that are in order. Then I need to fix each driver (ugh) to make sure they're dequeued fine.

I've written up the aggregation change and this so far works quite well. I don't want to tackle the legacy path yet or fix other drivers, not until I've verified this works. What I also should do is write some test cases to verify that indeed sequence numbers are being presented to the driver in order, so I can identify this happening in the wild.

FreeBSD on the TP-Link TL-WR1043nd!

I've now included (almost) all of the support needed to run FreeBSD natively on the TP-Link TL-WR1043nd. It's a 3-antenna, 2x2 stream 2.4ghz 802.11n AP which you can get for under AUD $100.

It supports hostap mode (which is what I bet most of you want to use) and I'm currently using it at home alongside my Ubiquiti Routerstation Pro based hostap (which is what I use to test out all the other pre-11n and 11n NICs that I currently own.)

I currently get around 50mbit TCP throughput - but I leave full FreeBSD-HEAD debugging on. I'm sure I can push the unit closer to 100mbit. (Compare to the Routerstation Pro + AR9160 hostap - where I routinely get 160mbit of TCP throughput.)

What works (read: what I've tested):
  • Ethernet (at least the WAN port);
  • Wireless - 802.11bgn - 20/40mhz operation as well as legacy operation (and both, if that's what you need);
  • Serial console - if you've soldered in one.

The firmware image stores the configuration in a 64k flash partition which is read upon boot. You can modify files in /etc and then save these to flash via "cfg_save".

What isn't supported:
  • The onboard switch - so I believe the only port available at the present moment is the WLAN port;
  • The GPIO lines aren't being configured, so the WLAN, status and USB/QSS buttons don't function.
I haven't tested out Multi-SSID mode yet. The earlier AR9130 revisions have some issues with multi-SSID mode and handling block-ack tracking, so I _think_ I'll need to somehow disable aggregation on the second and later VAP interfaces. Just keep that in mind if you're tinkering.

Further details about the hardware and how to build the software for yourself can be found here in my FreeBSD wifi development project wiki.

No, I won't (yet) be putting up firmware images for people to test. Things are changing quite rapidly and there's no easy way to reflash a unit once you've placed FreeBSD on it - you'll need to have added a serial console to the device.

FreeBSD 802.11n update: 27 November 2011.

I've merged in most of the reset related fixes from my git tree into FreeBSD-HEAD. This means that normal resets (eg stuck beacon, calibration resets, etc) shouldn't drop frames any longer.

Frames are still dropped during things like channel/operation mode changes and channel scanning (which does do a channel change.) I'll have to look into that at a later stage. If you're using this in station mode you will likely need to disable background scanning or your aggregation sessions may occasionally drop. You'll have random messages logged when frames are dropping during a flush or reset, so just check your system dmesg log for anything from the ath driver.

I'll be next working on correctly handling failed/filtered frames and then adding some transition stuff to net80211 so the TIM/ATIM bitmaps can be kept correctly up to date. This should fix some of the power saving issues that I'm sure exist.

Unfortunately transmitting BAR frames is still quite a bit off. There's a lot more tidying up that I'd like to do before I start down the path of handling BAR TX, including trying to figure out how to better handle packet transmission and reception when the NIC is off-channel (eg when doing a background channel scan.)

I also have a long list of things I'd like to do to the rate control code and all the surrounding code which sets up rates and creates aggregates. The code I ported/wrote is a little too verbose and duplicate-y for me. That likely will occur after the christmas break.

Enjoy!

FreeBSD is now doing (even more) 802.11n..

I held off merging in my 802.11n work as much as possible but I decided that I'd like to get it done before the end of the year. Even though 9.0-RELEASE is still around the corner, I decided that it would be better to merge in what I have into -HEAD and then tidy that up then wait for what could be a few more months.

So, it's in there, bugs and all, supporting both station and hostap mode. No, wds, adhoc, mesh and TDMA aren't currently supported (I have enough bugs to worry about for the time being, without trying to debug the other operating modes. But I'd like to.)

What works:
  • TX and RX aggregation!
  • The rest of the 802.11n negotiation stuff, mostly thanks to Bernhard Schmidt who fixed up a lot of the net80211 quirks.
  • Lots of ANI changes which hopefully make noisy environments more stable.

What doesn't yet work:

  • Interface resets cause frames to be dropped from the RX and TX queues. This messes up aggregation and causes sessions to hang. I'm fixing that up in a git branch at the moment.
  • BAR TX - I'll implement BAR TX soon - it's just tricky to get right.
  • Filtered frames - ie, TX failed frames from the hardware. Instead of the current method of "always try", the hardware supports failing the current and subsequent frames in a set. That way a hostap seeing a station going into power saving mode can quickly abort all TX frames to said station and then only retransmit them when the station indicates it's again awake. If I don't do this then the hardware will constantly fail a lot of frames, causing BAR frames to be TXed when they likely shouldn't be.

But it's enough to try. So if you have an AR5416, AR9160, AR9220, AR9280, AR9285, AR9227 or AR9287, give it a whirl. If you have a pre-11n NIC then please, give it a go too. I'd like to ensure that the hardware support for earlier chipsets hasn't broken.

If you'd like to use this in production on a hostap then please keep in mind that power saving support isn't entirely functional and featured, so stations which go into frequent power saving mode may have some performance issues. I'll tinker with this some more soon.

Finally, thank you very much to Hobnob, Inc. for sponsoring this work and Qualcomm Atheros for providing me source code, documentation and assistance in understanding how all of this works.

FreeBSD vendor summit – November 2011; Sunnyvale, California

Since this doesn't seem to have gotten much notice - there's a FreeBSD vendor summit being held in Sunnyvale, California on the 3rd and 4th of November 2011. This is an opportunity for developers and vendors to share project direction and goals, collaborate on various projects, and likely hit each other with a big "why can't you do X" stick :-)

NetApp is sponsoring the event.

I'll be there with someone from Qualcomm Atheros, who will be presenting on Qualcomm/Atheros' open source direction and strategy. There'll likely be some (informal) discussion about how developers and vendors can sign up to their open source developer programme and obtain documentation/reference code for the Atheros wireless and embedded chipsets.

(This is what I've signed up to and it's been a huge help in getting Atheros 802.11n support up and going in FreeBSD.)

More details can be found here:

http://wiki.freebsd.org/201111VendorSummit

If you're a vendor using FreeBSD, or you're a vendor thinking about using FreeBSD in a project (wireless or otherwise) this mini-conference is just for you. Personally I'm interested in talking with the nvidia representative about trying to get some CUDA stuff going on FreeBSD.

Wireless update – 802.11n development

A few of us have been working towards 802.11n support in FreeBSD. Bernhard Schmidt and I were hoping to get 802.11n support up to scratch before 9.0-RELEASE but a combination of timing constraints and work constraints has hampered things somewhat.

But that said, we the net80211 support for 802.11n is in a lot better shape than it was in previous releases. Bernhard has been working out the kinks in the intel driver (if_iwn) and has 802.11n support mostly stable for those NICs. Someone else has been working on if_ral 802.11n support and has had quite a bit of success there. And I've been working on if_ath/ath_hal support.

(As a side-note, I've ordered some Marvell if_mwl compatible NICs to do some testing with.)

The 802.11n support in the FreeBSD atheros driver is already mostly usable for testing - 802.11n TX and RX works, but 802.11n TX aggregation doesn't work. So if you'd like to test it out in 9.0-RELEASE, you can do this:

* build a kernel with "options ATH_ENABLE_11N";
* ifconfig wlan0 -ampdutx
* associate as per normal to an 802.11n network.

Your download speeds will be good (as RX aggregation works) but as TX aggregation doesn't, you won't be getting full speed.

Now, as for the current work I'm doing. I've been porting over (and reimplementing in places) the 802.11n TX aggregation support, based on a combination of the atheros reference/carrier codebases and what's filtered through to the Linux ath9k driver.

In short - so far, so good. There's a lot to do, but basic TX aggregation is working in both station and hostap mode. My current test setup is:
  • A routerstation pro with an AR9160 NIC, running locally built firmware (FreeBSD) in hostap mode;
  • An EEEPC 701 retrofitted with an AR9280 NIC, running FreeBSD, as a station;
  • My macbook pro (broadcom 802.11n) as a station.
So far it's performing well - 200mbit TX/RX UDP in 5ghz HT/40 2x2 stream mode; and 130-140mbit TX/RX TCP.

I'm currently doing the work in a FreeBSD user svn branch. The details are in the Wiki - http://wiki.freebsd.org/AdrianChadd/AtherosTxAgg . I currently have exactly one tester (AR5416 STA, AR9280 hostap) who is reporting excellent success. I'd obviously like a few more. :)

There's still a lot of work to be done before this can be merged into HEAD. I'm sorry to say it won't happen before 9.0-RELEASE as there are some upcoming net80211 changes which will be rather intrusive and a bit risky to throw into the release at this late stage. It may also be difficult to backport as some changes will break kernel ABI.

But don't let anyone tell you FreeBSD doesn't support 802.11n. 9.0-RELEASE may not have much support, but all the key pieces are there. Once the release has been cut, I'll do some chasing up to get the Intel, Marvell and Ralink 802.11n support updated. (If someone would like to take charge of the broadcom NIC drivers, please drop freebsd-wireless@ a line.)

The next 6 months will be very interesting in the FreeBSD 802.11 world. :)

Finally, none of this current work would've been possible without the support of the sponsor paying me to get FreeBSD's net80211 and ath support updated - Hobnob, Inc. They contacted me a few months ago to help work out kinks in the Atheros NIC they're using in FreeBSD and seem impressed enough by my work to sponsor general net80211/ath improvements which they'll likely roll into future products. So when you're using Atheros 802.11n hardware in FreeBSD and getting that nice 140-150mbit TCP throughput, please give them thanks. :)

And finally finally, Qualcomm Atheros and the Linux ath9k developers have been instrumental in this work. The documentation, reference code and general interactive discussions have allowed me to get all of this work done in such a short period of time. (Yes, I did say "Atheros", "Documentation" and "Source Code" there, in the context of open source. Honest.)

Wireless update – DFS wrapup

Now that the DFS work has been completed, tested and paid for, I guess I can now write about it.

The DFS work was sponsored by KBC Networks, a wireless product company based in the USA. The eventual aim is to integrate this DFS work into their mesh product. I can't talk any further about what's going on there, except to say that they very graciously allowed the FreeBSD work to be released back to the community.

So FreeBSD now has:

  • Updated DFS master (hostap) and DFS slave (station) support in net80211;
  • Updated regulatory domain entries for the USS ("FCC3");
  • Changes to the ath(4) driver to fix corner cases in DFS master and DFS slave modes.
All of this work is now in FreeBSD-HEAD and will be available in FreeBSD-9.0.

Future work will hopefully include DFS IBSS and DFS mesh functionality. This work didn't include 802.11n DFS - but that's not too difficult to do (hint: if someone would like the mini-project, please contact me!)

Along with that was a port of the Atheros radar detection code (from Fusion, their earlier carrier codebase) which is currently proprietary. The Linux wireless developers have been working towards DFS compliance for a while, but haven't yet integrated working radar detection into any drivers. So this was a good exercise in uncharted territory to see how difficult it would be.

It turns out that it wasn't terribly difficult at all. After spending a bit of time chasing down missing bits in the FreeBSD atheros HAL driver, the third party doing said radar code porting managed to get FreeBSD doing radar detection at the same level as the commercial firmware.

During this particular exercise we discovered a few things about the software radar detection, which I think the community at large should know about.

The process of getting a device FCC/ETSI certified is just that - the whole device. This includes the NIC, the internal cabling, the enclosure, the antenna(s) and any external cabling. Maybe even whether it's indoors or outdoors, I wasn't involved in that. This likely has repercussions for any open source DFS implementation - although the DFS machinery in a specific FreeBSD release could be certified, the software based radar matching code may need tuning for the specific hardware used. So it's likely that Linux/FreeBSD can't just publish a single "ath_dfs" source module that will work for all Atheros NICs - or even a single NIC. It's likely that things will need tuning based on how a given NIC is used, the environment it's intending to be used in, and a variety of other things I haven't even thought about yet.

So if/when this does occur, the open source wireless community (FreeBSD, Linux, NetBSD, OpenBSD, DragonflyBSD, whatever..) will have to be responsible and not just simply run an "ath_dfs" radar module, assuming that it works. For complete, correct, certified behaviour, it's going to likely need the community to work with vendors and users in order to create "working, certified" combinations of hardware and software. So, if a vendor (call them U) wishes to release a new "SR-M" NIC, they could work with the open source wireless community to create a certified NIC+board+enclosure+antenna combination and publish a set of radar patterns and settings. They could leverage the proposed existing open source radar pattern matching code and simply add whatever configuration is needed. Users should then be responsible - they should only use the given radar dfs code with the correct setup. If they wish to run a modified setup, they should work with the vendor and upstream community to get things re-certified.

This way the DFS and radar pattern matching code is open source - what then changes is the radar patterns and configuration, which depends upon the NIC, cabling, antenna and environment.

For home users/developers, having a set of certified hardware combinations will make things easier for them.

For commercial vendors, if they adopt open-source FreeBSD or Linux on their platforms then (in an ideal world) they could get their equipment certified once (which as I note, they have to do anyway) and then publish the relevant radar patterns and configuration. The rest of the driver (including the radar pattern matching source and the general DFS support) would already be open source. Users can then use this when running their own kernel/distribution/OS on that platform.

I know this is all very fluffy and assumes that vendors/users act in a responsible, open manner. But I think this is what needs to occur if the open source community wishes to see stable, supported DFS regulatory compliance in their open source project.

To wrap this up, if you're a company wishing to leverage open source on their (Atheros) wireless platform and are interested in regulatory compliance, please drop me a line. I'll put you in contact with the relevant people inside Qualcomm Atheros to discuss how to best achieve this.

FreeBSD: DFS update

I've spent the last few weeks sorting out the DFS support in FreeBSD. I've made some decent progress with it however there's still a long way to go. This has been paid work and I'm glad to see increasing interest in 802.11 support under FreeBSD.

I've debugged DFS channel switching behaviour in station and access point mode. This now functions correctly in both instances.

The ath driver now behaves "better" when doing a DFS channel change. The driver would change channels and wait for the first beacon to program the beacon timers - which are then used to signal when there's a lack of beacons (and thus net80211 marks the state as "SCAN" and begins hunting for another AP.) If a channel change occured to a channel that required a CAC (clear access check) and a radar event occured, a second channel switch would occur - and there'd never be any beacons on that channel. The NIC would sit there forever waiting to hear a first beacon that never came.

I've updated the FCC3 regulatory domain to include frequency bands that require DFS.

I've also introduced some radar device setup and event parsing code into the ath HAL layer which would allow an interested party to tinker with radar detection/classification code on the AR5212 series NICs. I don't have permission to commit the radar frame decoding code for the 11N NICs - I'm working on this.

Finally, I've introduced a radar detection layer - called "ath_dfs" - which currently does nothing (hence the only module is "dfs_null".) This provides all the hooks needed to do radar detection, classification and signaling.

Now, what I haven't done:
  • The project also includes working porting the radar classification code and getting it to work. This unfortunately won't be open sourced. I'm going to try and get as much of the radar hardware setup and radar event frame parsing code committed to FreeBSD so an interested party can begin tinkering with this.
  • I haven't yet ported a lot of the "channel interference" support. It's there, but I haven't really fleshed it out to be useful and then used it.
  • There's no DFS support for 11N yet - in particular, 11N HT/40 channels aren't correctly marked as "interference" or "radar"; 11N channels also aren't scanned for during DFS channel selection. That's a later project.
  • I need to extend things a bit to handle the weather radar stuff when doing DFS - the ETSI requirements include different channel quiet/scan times.
  • There's no support in net80211 (yet) for the DFS, TPC and Quiet elements. This is going to be required for a fully compliant DFS implementation.
  • I've not investigated implementing/fixing up the DFS support for IBSS and mesh modes.
However, even though there are shortcomings and some missing support in the DFS support, FreeBSD-9.0 will support DFS STA mode well enough to correctly obey instructions from a DFS-compliant access point.

Implementing 11n TX aggregation, why it’s not so easy..

The only real missing bit from the initial 11n support for ath(4) in FreeBSD is now TX aggregation. This is also quite a tricky thing to do right, because (like everything related to these Atheros NICs) it's all done in software.

I'm about half way through the first part - implementing per-node, per-TID software queues. Since A-MPDU sessions are implemented using TIDs (although in net80211 they're linked to WME access classes (AC), hm!) traffic for each TID has to be individually managed.

The basic premise is quite simple. Frames are queued based on the destination node and TID. Non-QoS frames to a unicast destination (ie, a known station) go to the "non-QoS TID" (TID 16 in FreeBSD's net80211 stack.) Multicast frames go to the software-driven multicast/content-after-beacon (cabq) queue. Then a separate task handles de-queuing packets form these queues and queuing them to the hardware.

The reality however is a little more difficult than that.

Firstly, frames are queued from multiple contexts. The most obvious is the per-interface queue (ie wlanX). These may run in parallel and target the same hardware, so now I need to add locking to the per-node structure. Previously this wasn't (much) of a problem since the TX code simply directly dispatched to the hardware, and these queues are correctly locked. However now there's software-driven queues (which need new locks) along with some more per-node state. That all needs locking.

Next, the rate control code wasn't at all locked. I've begun wrapping them in per-node locks. Making sure I get this correct isn't easy.

Then there's the question of marking nodes and TIDs that have traffic to send. Right now that's done by adding each node to a linked list of "ready nodes". I then check all the TIDs for that node and dequeue what packets I have to. It's ugly code, but it works well enough for testing. I'll finish off implementing what ath9k and the Atheros reference code does (ie, maintaining both a "node" and "TID" linked list, so I can directly walk which TIDs need walking), but there's implications here for getting "QoS" right. (Ie, which order do you dequeue packets? How many do you dequeue from which queue in order for things to be "fair" ?) I'm going to leave all of these questions unanswered for now (as they're not handled in the current code anyway) and hope someone else steps up to it!

At this point I now have a mostly-clean (for values of "clean") implementation of per-node, per-TID software queues. I'm still in the process of tidying all of this up in preparation to commit to -HEAD before I begin the aggregation code. I'll cover that in a moment.

Then there's handling the aggregation session stuff.

Firstly, there's handling the ADDBA exchange (ie, establishing an A-MPDU session.) When this is done, any unicast traffic pending to that node/TID needs to be paused and kept around until the ADDBA request/response occurs. This is known as "pausing" the queue. I've a hack in place to do this and will implement a correct "pause" API soon to clean it up.

Next there's handling packets already queued to the hardware. This crept up on me without warning.

Firstly - a little background. The ADDBA exchange establishes the initial sequence number and window size to use as the "window" for subsequent data exchanges. After that, any packets being queued to the destination must fall within this window (typically say 64 sequence numbers long.) The window stays a fixed size, but the window itself slides along when TX packets are successfully completed. Since packets may be lost at any time (whether aggregate or not), the window doesn't move until the sequence numbers at the beginning of the window are successfully received.

So for example, say my window begins at "0", my window size is "4" and the station sends 4 packets with sequence numbers "0", "1", "2", "3". If I receive all four packets correctly and thus send back a block-ACK indicating this, the new window position will begin at "4". I can then send packets "4", "5", "6" and "7". I can't send "8", even if I have it in my queue.

But say the other end only received "0", "1" and "3". "2" wasn't received, so I have to retransmit it. I can only advance the window to "2". I can't advance it to "3", since I didn't receive "2". I could advance it to "1", but that'd be pointless as I've already received it. So in this instance, both the station and access point would advance the window position to "2" and the station would then retransmit "2". If it's coded correctly, it could transmit "2", "4", "5". It can't transmit "6", as that's outside the window size of 4 packets. It could just retransmit "2" if it wanted to and not the others. In fact, it could transmit "4", "5" and "2" - the receiver would then be responsible for re-ordering the packets and only passing them up to the network layer once all the packets in the correct order are available.

Finally, if I don't receive "2" (say I just can't seem to transmit it), the STA need to do a few things. It first has to pause the queue so no further packets are queued to the hardware. At this point, packets may be queued to the hardware, but they'll be for sequence numbers greater than "2" but still within the block-ack window. Once the hardware has finished completing packets (successfully or not), it needs to send a "BAR" frame to the access point to establish a new starting point for the sequence number window. In this instance it could set it to "3", but what if "3" was already in the hardware queue and was successfully transmitted? That would confuse the hell out of the other end. So it looks at the last successfully transmitted sequence number was, then sends the "BAR" with a starting sequence number 1 after that.

So, if "2" came back as having failed too many times, but "3" and "4" were in the hardware TX queue, the driver will pause the queue, wait for pending frames to complete, then look at what happened. If "3" and "4" were successfully transmitted, it would send a BAR with the sequence beginning at "5". If "3" was successfully transmitted but not "4", it would send a BAR with the sequence beginning at "4". But if "3" failed, and "4" didn't - it would set the BAR at "3" and then try retransmitting it. Finally (and this is another strange corner case I have to handle!) if "3" came back as having failed - and it failed too many times! - but "4" TX'ed ok, then I have to set the initial sequence number at "5" and send the BAR.

The other end (in this instance, the hostap) would then receive the BAR, set the new expected sequence number to be whatever was sent (say "5" in this instance), flush all pending packets in the reorder queue - even if there are missing packets (in the above example, "3" may be missing but "4" was received ok, so the receive stack would've been waiting for "3" to be received before sending it all up to the network layer!) and send back a block-ACK to the STA confirming the new window position.

Now (phew!) given all of that, you can see what kind of complicated stuff is needed to properly handle all corner cases when doing software (re)transmitting of packets. I haven't even begun to talk about how to handle sending and re-sending aggregate frames! All of the above needs to be implemented before I can do that.

So! The above is what I'm now working on. Once that's done, I'll work on handling what's known as "filtered frames" (and that'll be brain-dumped in a future blog post, but it has to do with handling power save stations correctly, or A-MPDU sessions will be torn down prematurely!) and then when THAT is all done and stable, I'll look at handling aggregates.

And when I handle aggregates correctly, we'll finally have fast 11n TX in FreeBSD. Then I can enable "ATH_ENABLE_11N" by default. :-)

AR9287 update

I finally gave in and whacked a FreeBSD-HEAD snapshot on my EEEPC so I can test the ath 802.11n code out with the various mini-PCIe NICs I have.ath0: mem 0xfbef0000-0xfbefffff irq 18 at device 0.0 on pci1ath0: [HT] enabling HT modesath0: [HT] 2 RX streams; 2 TX streamsath0: AR9287 mac 384.2 RF5133 phy 15.15.. andwlan0: flags=8843 metric 0 mtu 1500 ether b4:82:fe:33:95:53 inet 10.61.8.27 netmask 0xffffff00 broadcast 10.61.8.255 media: IEEE 802.11 Wireless Ethernet MCS mode 11ng status: associated ssid CACHEBOY_RS channel 1 (2412 MHz 11g ht/40+) bssid 00:1b:b1:58:f6:f0 regdomain ROW country AU indoor ecm authmode WPA2/802.11i privacy ON deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 30 bmiss 7 scanvalid 450 bgscan bgscanintvl 300 bgscanidle 250 roam:rssi 7 roam:rate 64 protmode CTS -ampdu ampdulimit 32k ampdudensity 8 -amsdu shortgi wme burst roaming MANUAL(Yes, there's no TX aggregation support for now so I've disabled ampdu.)And:ADDR AID CHAN RATE RSSI IDLE TXSEQ RXSEQ CAPS FLAG00:1b:b1:58:f6:f0 1 1 216M 24.5 0 30271 30064 EPS AQEHTRS RSN HTCAP WPA WMEIt's happily sitting at MCS12 from my bedroom (which is what I expect given the noise on the 2.4ghz band where I live.)

AR9287 update

I finally gave in and whacked a FreeBSD-HEAD snapshot on my EEEPC so I can test the ath 802.11n code out with the various mini-PCIe NICs I have.

ath0: mem 0xfbef0000-0xfbefffff irq 18 at device 0.0 on pci1
ath0: [HT] enabling HT modes
ath0: [HT] 2 RX streams; 2 TX streams
ath0: AR9287 mac 384.2 RF5133 phy 15.15

.. and

wlan0: flags=8843 metric 0 mtu 1500
ether b4:82:fe:33:95:53
inet 10.61.8.27 netmask 0xffffff00 broadcast 10.61.8.255
media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
status: associated
ssid CACHEBOY_RS channel 1 (2412 MHz 11g ht/40+) bssid 00:1b:b1:58:f6:f0
regdomain ROW country AU indoor ecm authmode WPA2/802.11i privacy ON
deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 30 bmiss 7
scanvalid 450 bgscan bgscanintvl 300 bgscanidle 250 roam:rssi 7
roam:rate 64 protmode CTS -ampdu ampdulimit 32k ampdudensity 8 -amsdu
shortgi wme burst roaming MANUAL

(Yes, there's no TX aggregation support for now so I've disabled ampdu.)

And:

ADDR AID CHAN RATE RSSI IDLE TXSEQ RXSEQ CAPS FLAG
00:1b:b1:58:f6:f0 1 1 216M 24.5 0 30271 30064 EPS AQEHTRS RSN HTCAP WPA WME

It's happily sitting at MCS12 from my bedroom (which is what I expect given the noise on the 2.4ghz band where I live.)

AR9160 11n performance – finally!

I've made some good progress with the AR9160 802.11n hostap mode in FreeBSD.

Firstly, the A-MPDU density setting needed tuning. "0" or "N/A" isn't appropriate for the AR9160 (and likely not appropriate for anything earlier than the AR9300/AR9400 series.) Once set to 8 microseconds, the number of A-MPDU retransmits dropped dramatically.

I found that there was some missing code to do with disabling RIFS (reduced inter-frame spacing) RX on the AR9160 which I committed to FreeBSD-HEAD. This has eliminated all of the baseband lockups I've been seeing, rare as they were now.

Then I found there were packets being dropped by if_arge on TX. It turns out the default TX/RX ring buffer size for if_arge was 128 packets - definitely not enough given the uncoupled interrupt/process driven forwarding engine in UNIX these days.

The background: since NIC drivers aren't doing their work in an interrupt or shared process context, they only do their work when their TX/RX tasks get scheduled. Since they don't have any way to signal each other to "back off" via flow control when the queues are getting filled, it's quite possible to have a very busy device (in this instance wlan0 RX'ing anything more than around 100mbit) queue enough packets to the devices' TX queue faster than the TX task can be scheduled to process packets.

Bumping if_arge's default TX/RX ring buffer size from 128 to 1024 did the trick - no more packet drops on TX.

Now I'm getting ~ 105mbit TCP RX in HT/40 5GHz hostap mode (since A-MPDU RX is implemented and working) and ~ 210mbit UDP RX before I begin to drop packets. Anything more than 210mbit starts resulting in the infamous "stuck beacon", but that could be for a variety of reasons. I don't think I'm saturating the PCI bus (read: the CPU scheduler does show about 30% idle clock cycles during a UDP RX test) so there's something else to be blamed.

Next is figuring out whether there's some scheduling issues with the device TX/RX tasks.

I find it quite creepy that I can get ~ 100mbit of 802.11n throughput to my macbook pro whilst lying in bed. But that's what 802.11n is for, right? :)

AR9160 11n performance – finally!

I've made some good progress with the AR9160 802.11n hostap mode in FreeBSD.Firstly, the A-MPDU density setting needed tuning. "0" or "N/A" isn't appropriate for the AR9160 (and likely not appropriate for anything earlier than the AR9300/AR9400 series.) Once set to 8 microseconds, the number of A-MPDU retransmits dropped dramatically.I found that there was some missing code to do with disabling RIFS (reduced inter-frame spacing) RX on the AR9160 which I committed to FreeBSD-HEAD. This has eliminated all of the baseband lockups I've been seeing, rare as they were now.Then I found there were packets being dropped by if_arge on TX. It turns out the default TX/RX ring buffer size for if_arge was 128 packets - definitely not enough given the uncoupled interrupt/process driven forwarding engine in UNIX these days.The background: since NIC drivers aren't doing their work in an interrupt or shared process context, they only do their work when their TX/RX tasks get scheduled. Since they don't have any way to signal each other to "back off" via flow control when the queues are getting filled, it's quite possible to have a very busy device (in this instance wlan0 RX'ing anything more than around 100mbit) queue enough packets to the devices' TX queue faster than the TX task can be scheduled to process packets.Bumping if_arge's default TX/RX ring buffer size from 128 to 1024 did the trick - no more packet drops on TX.Now I'm getting ~ 105mbit TCP RX in HT/40 5GHz hostap mode (since A-MPDU RX is implemented and working) and ~ 210mbit UDP RX before I begin to drop packets. Anything more than 210mbit starts resulting in the infamous "stuck beacon", but that could be for a variety of reasons. I don't think I'm saturating the PCI bus (read: the CPU scheduler does show about 30% idle clock cycles during a UDP RX test) so there's something else to be blamed.Next is figuring out whether there's some scheduling issues with the device TX/RX tasks.I find it quite creepy that I can get ~ 100mbit of 802.11n throughput to my macbook pro whilst lying in bed. But that's what 802.11n is for, right? :)