Author Archives: kientzle

Watchdogs

I’ve been having a lot of fun over the last few months with FreeBSD on BeagleBone.  Most recently, that’s involved working on the CPSW ethernet driver.

One nasty bug has been eluding me for quite some time: The controller just stops sending packets after about 20 minutes.

Eventually, I will track down this problem. However, I’ve managed to make the driver quite usable even with such an unpleasant bug: I can leave SSH sessions open for days, download port tarballs, use NFS mounts, and generally do the things you expect a network to do.

The key was to find a really good watchdog strategy. Even with the controller locking up regularly, a good watchdog notices this and resets the driver within a few seconds, fast enough that network protocols simply retry and keep going after the reset. A less effective watchdog can leave the controller non-functioning for a minute or more, resulting in failed transfers and dropped connections.

Network Driver Basics

Three functions that appear in every network driver play a part in the watchdog process:

  • The “start send” routine hands packets to the controller.
  • The “transmit completion interrupt” is invoked by the controller when a packet has finished being sent; this routine recycles the memory and other resources for subsequent packets.
  • The “watchdog ticker” wakes up once a second and decides whether or not to reset the controller.

I’ll refer to these three functions as the “start”, “interrupt”, and “watchdog” functions to match how they are named in the source code of most drivers.

The Standard Watchdog

The standard logic that appears in many FreeBSD network drivers uses a single “timer” variable that is updated in each of the above functions:

  • The start routine sets it to 5 whenever a new packet is added to the controller.
  • The interrupt routine sets it to zero whenever it reclaims the last outstanding packet.
  • The watchdog subtracts one and resets the controller when the counter changes from one to zero. (If the counter is already zero, it’s left alone.)

Remember the goal here is to detect when the network is no longer running.  That is, we want to know when the interrupt has stopped getting called.  In fact, because the interrupt isn’t getting invoked, we can entirely ignore that function for now.

To understand the standard logic, consider three different scenarios:

Scenario One:  An almost idle network, with more than 5 seconds between packets. In this case, the start routine queues some packet and sets the timer to 5. The watchdog function counts this down every second until it hits zero, then resets the controller.

Scenario Two:  A very busy network. It can take only a few milliseconds for a busy machine to completely fill the transmit queue. In this environment, each new packet will cause the timer to get reset to 5.  The watchdog may tick and reduce the timer to 4, but a new packet will immediately reset it to 5. Once the transmit queue is full, however, new packets stop getting added and the start function stops resetting the timer. Again, the watchdog function counts down and resets the controller fairly promptly.

Scenario Three:  A lightly used network. Suppose “ping” is running and the transmitter stops. In this case, one packet is getting sent every second. If the transmit queue holds 100 packets, it will take 100 seconds before the transmit queue fills up.  During that time, the start routine and the watchdog routine alternately set the timer count to 5 and 4.

This last scenario is troublesome. In the first two scenarios, the watchdog function detects and resets the failed transmitter in about 5 seconds.  But in the third scenario, the standard logic can leave the network broken for more than a minute, long enough for TCP sessions to time out and reset.

After a few days of not finding the cause for the transmitter stalls, I decided to spend a little time trying to improve the watchdog itself. I tried a variety of different approaches:  only a few handled all of these scenarios well.

A Better Watchdog

I spent almost a week experimenting with different watchdog logic before I formulated two key questions:

  • Is there something to be done?  If there’s no work to be done, then the watchdog should sit quietly.  For a network driver, this just requires checking whether there are packets in the queue waiting to be sent.
  • Has progress been made?  For network drivers, we have “progress” when any packet completes sending.

Translating these questions into code gives a watchdog function that looks something like this:

cpsw_tx_watchdog(struct cpsw_softc *sc)
{
  if (sc->tx_in_queue == 0) {
    sc->tx_wd_timer = 0; /* Nothing to do. */
  } else if (sc->tx_completed > sc->tx_completed_at_last_tick) {
    sc->tx_wd_timer = 0;  /* More stuff got done. */
  } else {
    /* Something should have been done! */
    ++sc->tx_wd_timer;
    if (sc->tx_wd_timer > 3) {
      ... reset controller ...
    }
  }
  sc->tx_completed_at_last_tick = sc->tx_completed;
}

Notice that the watchdog timer is no longer touched in either the start or interrupt routines.  The start and interrupt routines only need to maintain two statistics:  a count of how many packets have been taken off the queue by the interrupt routine (tx_completed), and a count of how many packets are still on the controller’s queue (tx_in_queue).

There’s another interesting feature of this new logic. The timer variable now has a comparatively simple interpretation: It is the number of seconds that have elapsed since we noticed work being missed. (As an exercise, try formulating a single sentence that accurately describes the timer variable in the standard logic.)

Most importantly, of course, this logic works well in all of the scenarios described above, including the lightly-used network scenario that causes problems for the standard logic.

Of course, all of the above will be considerably less interesting once I figure out how to keep the controller from stalling every 20 minutes…