Category Archives: arm

VCHIQ drivers work again

I synced both vchiq-freebsd and userland to latest and greatest.

As I mentioned earlier – OS compatibility shim was removed from upstream sources so I had to create Linux KPI implementation layer which turned out not that awful task because I managed to reuse a lot of code from Max Khon’s DAHDI port. I had to implement (in somewhat hackish fashion) kthread API, re-implement semaphores support using condvar and mutex in order to get _interruptible part of API working properly and create dumb implementation of rather small subset of Linux list.h API.

With latest code I got pretty much all demos in hello_pi working except hello_jpeg(crashes system) and hello_encode(didn’t test). The most exciting bit for me was watching H.264 video playing on Raspberry Pi in hello_video demo. Network throughput still sucks so I had to copy file to tmpfs partition in order to get smooth playback though.

If you want to test VCHIQ – in addition to sources you’ll need latest firmware files. For demos you’ll also have to install freetype2 and manually hack Makefile.include in hello_pi. I’m planning to create ports/packages for both drivers and userland some time next week.

On the related note: Aleksandr Rybalko got XOrg working on Efika MX Smartbook so FreeBSD/Pi will get graphic interface soon :)

Watchdogs

I’ve been having a lot of fun over the last few months with FreeBSD on BeagleBone.  Most recently, that’s involved working on the CPSW ethernet driver.

One nasty bug has been eluding me for quite some time: The controller just stops sending packets after about 20 minutes.

Eventually, I will track down this problem. However, I’ve managed to make the driver quite usable even with such an unpleasant bug: I can leave SSH sessions open for days, download port tarballs, use NFS mounts, and generally do the things you expect a network to do.

The key was to find a really good watchdog strategy. Even with the controller locking up regularly, a good watchdog notices this and resets the driver within a few seconds, fast enough that network protocols simply retry and keep going after the reset. A less effective watchdog can leave the controller non-functioning for a minute or more, resulting in failed transfers and dropped connections.

Network Driver Basics

Three functions that appear in every network driver play a part in the watchdog process:

  • The “start send” routine hands packets to the controller.
  • The “transmit completion interrupt” is invoked by the controller when a packet has finished being sent; this routine recycles the memory and other resources for subsequent packets.
  • The “watchdog ticker” wakes up once a second and decides whether or not to reset the controller.

I’ll refer to these three functions as the “start”, “interrupt”, and “watchdog” functions to match how they are named in the source code of most drivers.

The Standard Watchdog

The standard logic that appears in many FreeBSD network drivers uses a single “timer” variable that is updated in each of the above functions:

  • The start routine sets it to 5 whenever a new packet is added to the controller.
  • The interrupt routine sets it to zero whenever it reclaims the last outstanding packet.
  • The watchdog subtracts one and resets the controller when the counter changes from one to zero. (If the counter is already zero, it’s left alone.)

Remember the goal here is to detect when the network is no longer running.  That is, we want to know when the interrupt has stopped getting called.  In fact, because the interrupt isn’t getting invoked, we can entirely ignore that function for now.

To understand the standard logic, consider three different scenarios:

Scenario One:  An almost idle network, with more than 5 seconds between packets. In this case, the start routine queues some packet and sets the timer to 5. The watchdog function counts this down every second until it hits zero, then resets the controller.

Scenario Two:  A very busy network. It can take only a few milliseconds for a busy machine to completely fill the transmit queue. In this environment, each new packet will cause the timer to get reset to 5.  The watchdog may tick and reduce the timer to 4, but a new packet will immediately reset it to 5. Once the transmit queue is full, however, new packets stop getting added and the start function stops resetting the timer. Again, the watchdog function counts down and resets the controller fairly promptly.

Scenario Three:  A lightly used network. Suppose “ping” is running and the transmitter stops. In this case, one packet is getting sent every second. If the transmit queue holds 100 packets, it will take 100 seconds before the transmit queue fills up.  During that time, the start routine and the watchdog routine alternately set the timer count to 5 and 4.

This last scenario is troublesome. In the first two scenarios, the watchdog function detects and resets the failed transmitter in about 5 seconds.  But in the third scenario, the standard logic can leave the network broken for more than a minute, long enough for TCP sessions to time out and reset.

After a few days of not finding the cause for the transmitter stalls, I decided to spend a little time trying to improve the watchdog itself. I tried a variety of different approaches:  only a few handled all of these scenarios well.

A Better Watchdog

I spent almost a week experimenting with different watchdog logic before I formulated two key questions:

  • Is there something to be done?  If there’s no work to be done, then the watchdog should sit quietly.  For a network driver, this just requires checking whether there are packets in the queue waiting to be sent.
  • Has progress been made?  For network drivers, we have “progress” when any packet completes sending.

Translating these questions into code gives a watchdog function that looks something like this:

cpsw_tx_watchdog(struct cpsw_softc *sc)
{
  if (sc->tx_in_queue == 0) {
    sc->tx_wd_timer = 0; /* Nothing to do. */
  } else if (sc->tx_completed > sc->tx_completed_at_last_tick) {
    sc->tx_wd_timer = 0;  /* More stuff got done. */
  } else {
    /* Something should have been done! */
    ++sc->tx_wd_timer;
    if (sc->tx_wd_timer > 3) {
      ... reset controller ...
    }
  }
  sc->tx_completed_at_last_tick = sc->tx_completed;
}

Notice that the watchdog timer is no longer touched in either the start or interrupt routines.  The start and interrupt routines only need to maintain two statistics:  a count of how many packets have been taken off the queue by the interrupt routine (tx_completed), and a count of how many packets are still on the controller’s queue (tx_in_queue).

There’s another interesting feature of this new logic. The timer variable now has a comparatively simple interpretation: It is the number of seconds that have elapsed since we noticed work being missed. (As an exercise, try formulating a single sentence that accurately describes the timer variable in the standard logic.)

Most importantly, of course, this logic works well in all of the scenarios described above, including the lightly-used network scenario that causes problems for the standard logic.

Of course, all of the above will be considerably less interesting once I figure out how to keep the controller from stalling every 20 minutes…

FreeBSD/armv6: what’s new and exciting?

It’s been a while since last update on the project status so it might seem as there was no progress in this area. The reality is: there is a bunch of activities happening with various levels of success. So I decided to give kind of end-of-the-year round-up of ongoing projects, plans and obstacles ARM hackers face.

First of all we tried switching default cache type from write-through to write-back type. It should have increased performance but instead opened a can of worms. Memory corruption debugging led to L2 cache driver on Pandaboard, EHCI driver code and subsequently to busdma code. Whole process took quite a few days full of hair-pulling and nagging various people and ended up in committing USB fixes and Ian Lepore’s busdma patches. PL310 (L2 cache controller) driver is being tested at this very moment. Original issue (WB caches) still stands and postponed till next year.

Then there are two projects by Andrew Turner aimed at modernizing FreeBSD/armv6 subsystem: switching to EABI and clang support for ARM. Daisuke Aoyama took both of them and produced working image for Raspberry Pi. He also fixed two issues with event timers on Raspberry Pi so now the platform is much more stable. I ran buildkernel in a loop overnight and by the morning Pi had survived 7 cycles and still was alive and kicking. I also managed to get python built and working on it. Didn’t have 100% success with perl 5.14/5.16, ports were built but failed at install stage segfaulting in do_clean_objs function.

My Pandaboard survived overnight buildkernel loop with L2 cache disabled, but acting up if I enable it. Investigating.

Then there are also several platform bring-ups in progress. Alexander Rybalko works on getting FreeBSD running on Efika MX Smartbook. Ganbold Tsagaankhuu hacks on Allwinner 10. Alexander Dutkowski’s hardware of choice is BeagleBoard-xM.
Ruslan Bukin experiments with Exynos4412 and Thomas Skibo reported about FreeBSD running on Zedboard (Xilinx Zynq-7000).

But what about devices/platform we have in tree? I have limited knowledge about some platforms so here is summary of the ones I’m aware of. If you have more information on any of these targets (or any other ARM-related projects) – let me know, I’ll update post.

  • BCM2835 Raspberry Pi the most accessible and therefore the one that gets the most exposure and testing. Pretty stable, considering. Supported devices: USB, network, MMC, GPIO, framebuffer, GPU. The rest is on ToDo list. VCHIQ driver is BSD-licensed now and I’m planning on getting it to sys/contrib. Userland bits of OpenGL ES should be added as a port though.
  • (update) LPC32x0 No first hand experience but judging by the code it supports MMC, FB, GPIO and USB
  • Marvel Armada XP I don’t have information about this one, sorry
  • Nvidia Tegra2 Just barebone boot stuff.
  • TI AM335x Examples: BeagleBone, TI Sitara EVM. Network was reported working but unstable on BeagleBone. USB is not supported. Haven’t tested GPIO yet.
  • TI OMAP3 Example: BeagleBoard-xM. See Alexander Dutkowski’s project
  • TI OMAP4 The hw I have – Pandaboard ES. Supported devices: USB, network, MMC, GPIO. Some issues with L2 cache
  • Versatile Platform Board Exists only as emulation target for QEMU. Supported hardware: PCI, network, framebuffer. Seems to be fairly stable, no extensive testing performed.

BeagleBone, PandaBoard ad Raspberry Pi images can be built using Tim Kientzle’s scripts.

Not really stellar list of supported peripherals I’d say. I tend to blame several things.

First – experimental and unstable state of FreeBSD/armv6 in general. It’s no fun adding new hardware support when you’re not confident in underlying subsystems stability. “I flush cache for this TX descriptor but is it really gets flushed?”. Been there, no fun at all. That’s why I believe task #1 for nearest future is maximum performance and rock-solid stability of what we have.

Then there is the case of syscons. It’s old, it’s inflexible and it’s mostly i386-centric. Just until recently most of our so-called embedded targets were headless so there were no pressure from this side to reorganize things. My experience with coding two framebuffer drivers or trying to add PS/2 keyboard support on non-i386 platform was not very pleasant. It’s messy and there is a lot of code duplication. newsyscons project may be the way to go, I haven’t looked at it yet. We just need someone(tm) to finish it and get into the tree.

Fix these two issues should make bring-up process easier. It leaves us with question of GPU support. But it’s different story for different post…

Happy New Year, everybody!

FreeBSD/armv6 in QEMU

First take at getting FreeBSD/armv6 running in simulators. Simulators are great for tracking down nasty bugs and building packages.

So here is support for Versatile Platform Board machine supported by QEMU. Most likely this code will not run on real VersatilePB because I do not have this hardware and timing code (or lack of it) on CLCD driver and Keyboard/Mouse interface (PL050) is pure guesswork.

Back to gory details:

Build

You’ll need this patch and this script. Apply patch, use script to get freebsd-versatilepb.flash.

As for userland – it’s fully compatible with Raspberry Pi’s userland, or Pandaboard’s one. So you can use latest RPi SD card image. As for now it’s freebsd-pi-r243778.img.gz (124Mb)

Run QEMU

I believe that at least QEMU 1.2.0 is required. It’s still 1.1.1 in ports due to some blockers that prevent upgrade to 1.3.0. This patch updates port to 1.3.0 and it worked for me. Also I tested images with QEMU on windows and OS X – works fine.

qemu-system-arm -M versatilepb -m 128M -kernel freebsd-versatilepb.flash  -cpu arm1176 -hda freebsd-pi-r243778.img

Caveats

  • Serial console is off by default, use graphics console. If you need headless mode, rebuild image with “device sc” and related options disabled or use prebuilt flash image for headless mode
  • root device name is hardcoded so if you’re using some other image or building your own – be sure that’s ROOTDEV actually match real root
  • Memory size is hardcoded – 128M. For getting this information run-time we’ll need uboot and ubldr added to boot chain

Prebuilt kernels

freebsd-versatilepb.flash (4Mb)
freebsd-versatilepb-headless.flash (4Mb)
MD5 (freebsd-versatilepb-headless.flash) = 24a41807bf94c5fec0565adcfef48678
MD5 (freebsd-versatilepb.flash) = 085dedae67895ac1d1a7c04c7cda8468

FreeBSD on Pi: more stuff

Long overdue update on how the things are going with FreeBSD on Raspberry Pi. We've made some good progress so far:

  • Hans Petter Selasky fixed low-speed interrupt endpoints problem which means we have working USB keyboard now
  • GPIO driver by Luiz Otavio O Souza. So now you can blink OK LED (gpioctl -f /dev/gpioc0 -t 16). Not the most productive activity though.
  • Kernel now obtains information about display resolution, memory layout, MAC address from firmware
  • Framebuffer/syscons support added
  • Some stability fixes for SDHCI/li>
  • Initial port of VCHIQ interface (vchiq-freebsd)
  • Port of userland libraries (userland)

Overall stability and performance is still a problem, but it's what we're going to work on next.

And if you missed previous post: freebsd-pi is no more, use HEAD from FreeBSD subversion repository.

Boot process has been changed and now it looks like: firmware → uboot → ubldr → kernel. So old script for building image is no longer relevant. Here is new one. Tim Kientzle's scripts collection for building images for BeagleBone, Pandaboard and RPi uses more systematic approach but RPi part hasn't caught up to latest boot chain changes yet. Once it is up to date I suggest using Tim's scripts.

Building FreeBSD does not require any additional tools but if you want VideoCore bits you'll need following packages installed:

  • devel/cmake
  • devel/git
  • devel/gmake

If you don't need VideoCore binaries, just comment build_videocore and install_videocore calls. This script will also install OpenGL ES hello_triangle demo to /root folder. To run it run perform following steps:

# cd /root
# kldload vchiq
# ./hello_triangle.bin

I tried to build Qt5 with OpenGL ES support, but build choked on compile-time assert triggered by FreeBSD using OABI. Good news though: EABI work is almost done, so there is a fat chance we'll see Qt5 with eglfs backend running on FreeBSD in near future.

You can try pre-built image (124Mb, MD5 sums). Login is "root", no password. Use dd to write it to SD card. U-Boot seems to be somewhat finicky about SD cards, so if you get "** Unrecognized filesystem type **" message try another card. First boot might take some time because sshd will generate keys. U-Boot output goes to serial port and monitor, FreeBSD console messages go only to monitor, but by the end of boot sequence you should get login prompt on serial.

This image is a snapshot of work in progress and by no means a production system.

UPDATE

** Unrecognized filesystem type ** U-Boot issue seems to be more widespread then I thought. I'm working on it.

Rearranging the deck chairs in the ARM port

Recently, I've been trying to catch up with some of the technical debt that the FreeBSD/arm port has accumulated.  Some of that technical debt was my fault, of course, but some of it wasn't.  I don't really much care whose fault things were,  but I would like to get things cleaned up.  Some of these are paths not taken.  Some are porting shims that never got properly connected.  There's also some features that were poo-poo-ed by some (including me) years ago that seem to make more sense now.

None of this stuff is terribly sexy.  Some of it will be addressed by the summer of code project that's working its way through some of the maze of twisty passages in initarm() that are subtly different between the different boards for no good reason.

We have no common place for boot loader code.  Some arm ports do use a common routine to fake up just enough metadata that debugging and other symbols work.  Some arm ports assume a full /boot/loader is present.  However, there's no common facility for getting information from via the Linux boot protocol, nor is there any provisions for having multiple boards supported by a single kernel.  There's no uniform interface to get this information that might be passed in by other means.  There's no real way for a board to get control early enough to get at meta-data a custom boot loader might pass in.

The first big thing that I'd like to comment on is a small change to the interface between locore.S' _start routine and the first port-specific code in initarm.  I've preserved the first 4 registers that were passed in on boot to _start().  Before, there'd be no way to access the Linux ATAGs, for example, by a port since these are passed in as arg 3.  To allow for future expansion, I'm just passing in one structure now.

In the coming weeks, I'll be implementing routines to parse these arguments based on standard boot protocols, while allowing customization for special needs.

In the coming months, I plan to expand our support for having multiple boards based on a single Atmel SoC.  Once I have the one at a time SoC code working, I hope to get multiple SoCs working in one kernel.  While the extra 'bloat' isn't optimal for many users, the convenience of being able to boot one kernel for installation, or trying out the system, is a feature that's been much requested.  I've already eliminated the need to have a compiled-in master clock frequency, with improved main clock detection with fallback for people using unconventional frequencies.

I'm also interested in sweeping into the tree many of the Atmel arm changes that have been accumulating in the PR database.  Some of these will be easy to integrate, while others may be difficult due to drift from the original submission.

Once we get a good baseline, I hope to merge these changes into 9.x and 8.x.  To date, the interfaces that have change have been purely internal ones we do not support.  External users may experience some bumpy waves if they haven't been working to get their changes merged upstream.  The more that you've submitted, the more I'm likely to go the extra mile to help smooth any transitions.  With luck, 8.3 and 9.2 will both benefit from these efforts.

FreeBSD/arm -

I've just found out that FreeBSD/armv6 now runs on this:

http://www.embeddedartists.com/products/kits/lpc3250_kit.php


This is excellent news. There's also been some recent work to improve pandaboard support (TI-OMAP) focusing on pmap and SMP fixes.

I'm so very glad that the FreeBSD community is pushing this ARM project along. Yes, the armv6 branch is not very well named (it supports more recent ARM platforms, not just armv6 improvements) and I'm hoping this will all make it into FreeBSD-HEAD soon.

What is all the TBEMD stuff

I've been getting questions about what TBEMD means in my commits to FreeBSD head.First, I'll tell you the name. TBE stands for TARGET_BIG_ENDIAN. For the MIPS and ARM ports, it is an environment variable that you have to set to build these targets for big endian (otherwise they default to little endian). MD is Must Die. A few years ago, there was an action movie whose title was more memorable than the file "Romeo Must Die." So I just stole the name for this branch.So why must TARGET_BIG_ENDIAN die? It breaks things. For all the other platforms that FreeBSD supports, you just set TARGET and/or TARGET_ARCH to do cross builds. You know everything you need to know from the MACHINE and MACHINE_ARCH in the resulting image. In addition, the build system segregates things so you can build all the targets in one tree..But for mips and arm, you don't know what endian a binary is, the obj tree will collide if you try to build both little endian and big endian binaries at the same time. There's also other, more subtle issues. For example, TARGET_BIG_ENDIAN isn't set on big endian mips, so a native biuldworld tries to build little endian binaries (oops).So, mips will be moving to mipsel (for little endian) and mipseb (for big endian). Arm will move to arm (for little endian) and armeb (for big endian). You could fill an entire email archive with all the possible other names and why they are better or worse than these names. History, however, trumps all those arguments: these are the names used elsewhere and we're just following convention.Once tbemd is completely merged, you'll be able to build both endians of MIPS in the same object tree, for example, and all the other issues I've discovered. It also helps with new architectures as we move into powerpc64 (already merged) and mips64 (to be done after tbemd is collapsed). It will also mean we can have different packages for the different endians now.

What is all the TBEMD stuff

I've been getting questions about what TBEMD means in my commits to FreeBSD head.

First, I'll tell you the name. TBE stands for TARGET_BIG_ENDIAN. For the MIPS and ARM ports, it is an environment variable that you have to set to build these targets for big endian (otherwise they default to little endian). MD is Must Die. A few years ago, there was an action movie whose title was more memorable than the file "Romeo Must Die." So I just stole the name for this branch.

So why must TARGET_BIG_ENDIAN die? It breaks things. For all the other platforms that FreeBSD supports, you just set TARGET and/or TARGET_ARCH to do cross builds. You know everything you need to know from the MACHINE and MACHINE_ARCH in the resulting image. In addition, the build system segregates things so you can build all the targets in one tree..But for mips and arm, you don't know what endian a binary is, the obj tree will collide if you try to build both little endian and big endian binaries at the same time. There's also other, more subtle issues. For example, TARGET_BIG_ENDIAN isn't set on big endian mips, so a native biuldworld tries to build little endian binaries (oops).

So, mips will be moving to mipsel (for little endian) and mipseb (for big endian). Arm will move to arm (for little endian) and armeb (for big endian). You could fill an entire email archive with all the possible other names and why they are better or worse than these names. History, however, trumps all those arguments: these are the names used elsewhere and we're just following convention.

Once tbemd is completely merged, you'll be able to build both endians of MIPS in the same object tree, for example, and all the other issues I've discovered. It also helps with new architectures as we move into powerpc64 (already merged) and mips64 (to be done after tbemd is collapsed). It will also mean we can have different packages for the different endians now.

Moving towards board files

I've spent a little bit of time implementing the start of board files for the arm port. The initial push has been for the at91 subport only, and many improvements could be made to this. I've written up my initial thoughts on this on the FreeBSD wiki FreeBSD Arm Boards. It could use much improvement, I'm sure.One idea that hasn't been reflected there yet, was shown to me by Sam Laffler who suggested using linker sets to allow boards to 'probe', 'init' and other standardized functions. This is an interesting idea and I plan on working on adding it to the above links when Sam has results to share.I'd also like to expand the above wiki page to be a 'best practices' guide for all architectures where there's great diversity of boards/cpus/etc (eg, not the homogeneous env that x86 provides).I'm also soliciting comments on the above boards in addition to the above. Send them to me, or post them here.

Moving towards board files

I've spent a little bit of time implementing the start of board files for the arm port. The initial push has been for the at91 subport only, and many improvements could be made to this. I've written up my initial thoughts on this on the FreeBSD wiki FreeBSD Arm Boards. It could use much improvement, I'm sure.

One idea that hasn't been reflected there yet, was shown to me by Sam Laffler who suggested using linker sets to allow boards to 'probe', 'init' and other standardized functions. This is an interesting idea and I plan on working on adding it to the above links when Sam has results to share.

I'd also like to expand the above wiki page to be a 'best practices' guide for all architectures where there's great diversity of boards/cpus/etc (eg, not the homogeneous env that x86 provides).

I'm also soliciting comments on the above boards in addition to the above. Send them to me, or post them here.

Marvell ARM support in svn

Rafal Jaworowski just committed support for a number of Marvell Processors. There are a number of commits going into the tree. He's just completed the first of these:Introduce low-level support for new Marvell core CPUs: 88FR131, 88FR571.From an earlier email message, we know that Rafal is working on support for Marvell 88F5182, 88F5281, 88F6281, and MV78100 SoCs.The following peripherals will likely be supported:
  • EHCI USB 2.0
  • Ethernet
  • GPIO
  • Interrupt controller
  • L1, L2 cache
  • Timers, watchdog, RTC
  • TWSI (I2C)
  • UART
This is a fairly mature port that is going into the tree. It works with NFS or USB mounted root filesystems. The port is self hosted. Kernel and world builds succeed on the box. The box also boots to multiuser mode. Rafal has created a web page with all this information on it on the FreeBSD wiki at http://www.wiki.freebsd.org/FreeBSDMarvellThis is way cool news! Good Job Rafal.Note: Much of this information was taken from a posting Rafal made to [email protected]

Marvell ARM support in svn

Rafal Jaworowski just committed support for a number of Marvell Processors. There are a number of commits going into the tree. He's just completed the first of these:
Introduce low-level support for new Marvell core CPUs: 88FR131, 88FR571.

From an earlier email message, we know that Rafal is working on support for Marvell 88F5182, 88F5281, 88F6281, and MV78100 SoCs.

The following peripherals will likely be supported:
  • EHCI USB 2.0
  • Ethernet
  • GPIO
  • Interrupt controller
  • L1, L2 cache
  • Timers, watchdog, RTC
  • TWSI (I2C)
  • UART

This is a fairly mature port that is going into the tree. It works with NFS or USB mounted root filesystems. The port is self hosted. Kernel and world builds succeed on the box. The box also boots to multiuser mode. Rafal has created a web page with all this information on it on the FreeBSD wiki at http://www.wiki.freebsd.org/FreeBSDMarvell

This is way cool news! Good Job Rafal.

Note: Much of this information was taken from a posting Rafal made to [email protected]

FreeBSD embedded (mips, powerpc, arm) update

The last week or so has show a spike up in FreeBSD embedded platform work. Fixes for mips and powerpc have gone into the tree. Support for a new ARM platform is getting ready to be committed. FreeBSD/mips is almost self-hosting in the svn tree (it has been in the p4 tree for some time). FreeBSD/powerpc and FreeBSD/arm continue to mature.Thanks to a flurry of activity over the weekend by David O'Brien, the number of patches needed for FreeBSD/mips has been reduced to a set for libpam and for binutils. David cleaned up the diffs for gcc and pushed them into the tree in a manner that allows for easier upstream committing.In addition, David has cleaned up a number of dangling nits in the build. This is the result of a slight mismatch between p4 and the code in FreeBSD's svn tree. Oleksandr Tymoshenko has looped back most of these changes into p4, and is working on resolving the conflicts that were generated. Once this is sorted out, Oleksandr or myself will generate a new set of diffs and go for the final push into the tree.I'm looking forward to having a good base for FreeBSD/mips in the tree. I've been busy with a number of other projects and haven't given it the time that I really want to give.On the PowerPC side of the ledger, Marcell Moolenaar and Rafal Jaworowski have been polishing uboot boot loader support in the FreeBSD/powerpc subversion tree. In addition, some bug fixes to and code polishing of the low level Book E PowerPC support have been committed. Many of these improvements were obtained from Semihalf consulting.On deck to be committed soon is support for Marvell's Orion NAS chips. This code has been in perforce for a while now and is nearly ready for the FreeBSD tree. I'll keep everybody posted when that happens. I hope to have my DLINK DIR-615 Rev A1 working by then with this code, but time has been tight and there's already too many different projects afoot in my life.

FreeBSD embedded (mips, powerpc, arm) update

The last week or so has show a spike up in FreeBSD embedded platform work. Fixes for mips and powerpc have gone into the tree. Support for a new ARM platform is getting ready to be committed. FreeBSD/mips is almost self-hosting in the svn tree (it has been in the p4 tree for some time). FreeBSD/powerpc and FreeBSD/arm continue to mature.

Thanks to a flurry of activity over the weekend by David O'Brien, the number of patches needed for FreeBSD/mips has been reduced to a set for libpam and for binutils. David cleaned up the diffs for gcc and pushed them into the tree in a manner that allows for easier upstream committing.

In addition, David has cleaned up a number of dangling nits in the build. This is the result of a slight mismatch between p4 and the code in FreeBSD's svn tree. Oleksandr Tymoshenko has looped back most of these changes into p4, and is working on resolving the conflicts that were generated. Once this is sorted out, Oleksandr or myself will generate a new set of diffs and go for the final push into the tree.

I'm looking forward to having a good base for FreeBSD/mips in the tree. I've been busy with a number of other projects and haven't given it the time that I really want to give.

On the PowerPC side of the ledger, Marcell Moolenaar and Rafal Jaworowski have been polishing uboot boot loader support in the FreeBSD/powerpc subversion tree. In addition, some bug fixes to and code polishing of the low level Book E PowerPC support have been committed. Many of these improvements were obtained from Semihalf consulting.

On deck to be committed soon is support for Marvell's Orion NAS chips. This code has been in perforce for a while now and is nearly ready for the FreeBSD tree. I'll keep everybody posted when that happens. I hope to have my DLINK DIR-615 Rev A1 working by then with this code, but time has been tight and there's already too many different projects afoot in my life.

Slimming down the NSLU kernel

A few days ago, I wrote about the NSLU kernel config file being committed to FreeBSD's svn tree. The default compressed kernel is about 1.6MB, but the size of the partition of the NSLU2's flash is 1.25MB, a gap of just under 300kB. With some hacking, I've been able to reduce the size of the kernel to fit.

The default kernel's text+data size is 3,425,844 bytes. Compressed, with the appropriate headers, this drops to 1,591,109 bytes. However, the goal is 1,310,720 bytes. We're short by at least 294,553 bytes of fitting into the kernel portion of the flash on the NSLU.

My first thought was to start cutting things out. I removed IPv6 (options INET6) since I didn't need it for my setup. This saved 210,797 bytes in the uncompressed kernel and 74,115 bytes in the compressed kernel. This one option shaved 5% (compressed) of the kernel size. A move in the right direction, but more was needed.

Next, I recalled from previous attempts that there's many inline functions that don't need to be inlined. There's a small performance gain from inlining these functions, but it can add a lot to the kernel. So I added three options to the kernel config:
options MUTEX_NOINLINE
options RWLOCK_NOINLINE
options SX_NOINLINE
This resulted in a size reduction of 432,984 bytes (uncompressed) and 87,289 bytes (compressed). This is also a significant savings, and meant that we'd saved 161,404 bytes in the compressed kernel so far. We were still ~133,000 bytes shy of the mark.

At this point I noticed that there were a lot of things in the kernel that weren't required for boot, and that had shown up as being larger. When doing the size reduction, often times I use the 'size' command to get the top 20 .o files to see if there's junk that can just be omitted from the kernel. In this case, I saw that usb, scsi and releated drivers consumed a lot of space. This could be loaded from a number of modules after boot if I was booting from a RAM disk. Since the plan was to put an initrd-like ram disk in the other partition of the flash, I went ahead and removed these pieces:
#device usb
#options USB_DEBUG
#device ohci
#device ehci
#device ugen
#device umass
#device scbus # SCSI bus (required for SCSI)
#device da # Direct Access (disks)
This trimming saved 278,837 bytes (uncompressed) and 133,520 bytes (compressed). This was huge, and brought the total trimmed up to 294,924 bytes. This is just barely over the line, so I thought I'd go for a little more.

Another big area of the kernel is the code brought in with 'options miibus' configuration line. This brings in all the phy drivers on the off chance you'll need it. In many embedded systems, you know which phy driver you have, and can wire in only the one or ones you need. Since I wasn't planning on using this box as a general purpose router, that needed to work with any Ethernet dongle that was plugged into the USB port, I went ahead and transitioned to this configuration. The NSLU has a RealTek Phy that needs the rlphy driver. This means that I changed the config file like so:
#device miibus # NB: required by npe
device mii
device rlphy
This option shaved a few more bytes off the total. 80,012 bytes (uncompressed) and 25,156 bytes (compressed).

At the end of all this, the compressed kernel had been reduced from 1,625,125 bytes down to 1,286,193 bytes, for a savings of 320,080 bytes. This was enough margin for me to feel comfortable trying to burn it into my NSLU. Unfortunately, I ran out of time tonight to try it, so the story of that experiment will wait for another post. Also, I don't know how to make blogspot do tables, or I'd have made some nice tables with all this data for this log.

There's likely more options that can be explored to squeeze even more bytes out of the kernel. bzip2 might be a good place to get maybe 10% more space back (at these sizes, an extra 100kB compressed, or 250kB uncompressed). The exploder for bzip2 is bigger and uses more memory.

Looking at the .o files that are laying around that pci.o is kinda big, suggesting that we need to break it up some more. In previous experiments, separating out the MSI and PCIe stuff helped quite a bit. I'm unsure what the best newbusly way to extract this functionality might be, but it could save us 20kB uncompressed (maybe 8k compressed, iirc).

There's a number of other places code is agresively inlined, these could be investigated. there's
a few things in the tree tagged as standard that likely could be made to be options. There's much room here to explore a number of different options to reduce the kernel size even further if the need presents itself.

The nfs server option is about 45kB. Soft updates run about the same. nfsv4 also eats a lot of memory (looks to be 55kB).

However, for all these options, it would be nice is the NAS could have RAID, FAT, ntfs, smbfs and ext2fs support as well. Continuing to trim would allow this more easily. Or at least eliminate the need for loadable modules.

This is to say, there's a lot of areas for further research and documentation. I didn't even look at using a /boot/loader either. Maybe that can be helpful... Maybe I'll turn that into a research paper for one of the conferences... Either the continued size reduction thing, or using /boot/loader.

Finally, I've uploaded the changed NSLU config file in case my verbal descriptions here were insufficient.

NSLU2 support committed to FreeBSD/arm

I just committed the basic Linksys NSLU2 support to the subversion tree. It boots and runs on an NFS root. We're currently lacking support for the leds, the buttons and the on-board flash. Here's some basic instructions if you want to try it on your NSLU2. They are adapted from Sam Leffler's README file for the AVILA, which is adapted from other sources, including Olivier Houchard's and my instructions that have appeared.

  1. Build world
  2. Setup NFS root
  3. Configure diskless system
  4. Build and Install Kernel
  5. Put Kernel in tftp area
  6. Boot from redboot


Build world

The FreeBSD handbook can help you find the sources, etc Once you have them, and they are from -CURRENT after 8am on 2008-08-03, you can proceed to build the world. Each section is cumulative from the previous section for simplicity sake.


setenv TARGET arm
setenv MAKEOBJDIRPREFIX /tmp/$USER/obj
setenv TARGET_BIG_ENDIAN t
setenv TARGET_CPU_TYPE xscale
make buildworld


Setup NFS root

Find a server that has some disk space. You'll need about 250MB of space for a full FreeBSD image.

setenv ROOT /data/freebsd/roots/slug
make installworld DESTDIR=$ROOT
mergemaster -m $SRC/etc -D $ROOT -i -A arm


Setup diskless system
diskless(8) gives a good overview in how to do this. It is left as an exercise for the read to come up with all the settings. Usually /etc/rc.conf and /etc/fstab must be configured. /etc/ttys is another one that usually needs some tlc. If there's enough demand, I'll write about this in the future.

Build and install the kernel

make KERNCONF=NSLU buildkernel
make KERNCONF=NSLU DESTDIR=$ROOT installkernel


Note: the NSLU kernel is configured with the root filesystem mounted via NFS over the npe0 wired interface.

You should now have an NFS-mountable root filesystem with a kernel. The final step is to setup network diskless booting from the board. It is assumed you have a DHCP server operating on your network and the server is configured to supply the necessary information in the DHCP lease. If you run the ISC DHCP server the following configuration information is an example of how to do this:

option root-opts code 130 = string; # NFS / mount options

host slug {
hardware ethernet 00:d0:12:02:47:68;
fixed-address 10.0.0.221;
next-server 10.0.0.251;
filename "kernel-slug.nfs";
option root-path "10.0.0.251:/data/freebsd/roots/slug";
option root-opts "nolockd";
}

Note: the root-opts item specifies the root filesystem should be mounted with the nolocked option; this just short-circuits file locking requests so you don't get complaints from programs that use the pidfile(3) routine (e.g. devd)

Install kernel in tftp area

Place the kernel in the TFTP area for booting from the prom monitor. If your TFTP server returns file from /tftpboot (default) then do something like:
cp $ROOT/boot/kernel/kernel /tftpboot/kernel-slug.nfs


Boot using Redboot

+(hit control C here)
RedBoot> ip -h 10.0.0.251 -l 10.0.0.1
IP: 10.0.0.1/255.255.255.0 Gateway: 0.0.0.0
Default server: 10.0.0.251
RedBoot> load -b 0x200000 kernel-avila.nfs
Using default protocol (TFTP)
Address offset = 0x40000000
Entry point: 0x00200100, address range: 0x00200000-0x004db2d4
RedBoot> go


dmesg
Here's a complete transcript of my doing this. The .gz file listed below really isn't the simple gz file, but rather kernel.gz.tramp from the kernel build. Also, I've hacked my NSLU to have a serial port, which make development like this easier. See the web for instructions on how to do a no-mod breakin.

+Ethernet eth0: MAC address 00:14:bf:68:47:ba
IP: 192.168.0.1/255.255.255.0, Gateway: 192.168.0.1
Default server: 0.0.0.0, DNS server IP: 0.0.0.0

RedBoot(tm) bootstrap and debug environment [ROMRAM]
Red Hat certified release, version 1.92 - built 15:16:07, Feb 3 2004

Platform: IXDP425 Development Platform (XScale)
Copyright (C) 2000, 2001, 2002, Red Hat, Inc.

RAM: 0x00000000-0x02000000, 0x000723a0-0x01ff3000 available
FLASH: 0x50000000 - 0x50800000, 64 blocks of 0x00020000 bytes each.
== Executing boot script in 2.000 seconds - enter ^C to abort
^C
RedBoot> ip -l 10.0.0.12 -h 10.0.0.8
IP: 10.0.0.12/255.255.255.0, Gateway: 192.168.0.1
Default server: 10.0.0.8, DNS server IP: 0.0.0.0
RedBoot> load -b 0x200000 kernel.nslu.gz
Using default protocol (TFTP)
Address offset = 0xf0000000
Entry point: 0x00200074, address range: 0x00200000-0x003830fc
RedBoot> go
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #12: Sun Aug 3 00:09:07 MDT 2008
imp@lighthouse:/tmp/imp/obj/arm/cache/svn/head/sys/NSLU
CPU: IXP425 266MHz rev 1 (ARMv5TE) (XScale core)
DC enabled IC enabled WB enabled LABT branch prediction enabled
32KB/32B 32-way Instruction cache
32KB/32B 32-way write-back-locking Data cache
real memory = 33554432 (32 MB)
avail memory = 25894912 (24 MB)
ixp0: on motherboard
pcib0: on ixp0
pci0: on pcib0
ohci0: irq 28 at device 1.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0
usb0: on ohci0
usb0: USB revision 1.0
usbd_get_string: getting lang failed, using 0
uhub0: on usb0
uhub0: 3 ports with 3 removable, self powered
ohci1: irq 27 at device 1.1 on pci0
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb1: OHCI version 1.0
usb1: on ohci1
usb1: USB revision 1.0
usbd_get_string: getting lang failed, using 0
uhub1: on usb1
uhub1: 2 ports with 2 removable, self powered
ehci0: irq 26 at device 1.2 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb2: EHCI version 1.0
usb2: companion controllers, 3 ports each: usb0 usb1
usb2: on ehci0
usb2: USB revision 2.0
uhub2: on usb2
uhub2: 5 ports with 5 removable, self powered
ixpclk0: on ixp0
ixpiic0: on ixp0
iicbb0: on ixpiic0
iicbus0: on iicbb0 master-only
iicbus0: at addr 0
iic0: on iicbus0
ixpwdog0: on ixp0
uart0: on ixp0
uart0: [FILTER]
uart0: console (115200,n,8,1)
uart1: on ixp0
uart1: [FILTER]
ixpqmgr0: on ixp0
ixpqmgr0: [ITHREAD]
npe0: on ixp0
npe0: [ITHREAD]
npe0: remember to fix rx q setup
miibus0: on npe0
rlphy0: PHY 1 on miibus0
rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
npe0: Ethernet address: 00:14:bf:68:47:ba
npe1: on ixp0
npe1: [ITHREAD]
npe1: remember to fix rx q setup
npe1: Cannot find my PHY.
device_attach: npe1 attach returned 6
ixpclk0: [FILTER]
Timecounter "IXP425 Timer" frequency 66666600 Hz quality 1000
Timecounters tick every 10.000 msec
bootpc_init: wired to interface 'npe0'
Sending DHCP Discover packet from interface npe0 (00:14:bf:68:47:ba)
npe0: link state changed to UP
Received DHCP Offer packet on npe0 from 0.0.0.0 (accepted) (no root path)
Sending DHCP Request packet from interface npe0 (00:14:bf:68:47:ba)
Received DHCP Ack packet on npe0 from 0.0.0.0 (accepted) (got root path)
npe0 at 10.0.0.12 server 0.0.0.0 boot file kernel.slug
subnet mask 255.255.255.0 router 10.0.0.1 rootfs 10.0.0.8:/pe/slug rootopts nolockd
Adjusted interface npe0
Trying to mount root from nfs:
NFS ROOT: 10.0.0.8:/pe/slug
warning: no time-of-day clock registered, system time will not be set accurately
warning: no time-of-day clock registered, system time will not be set accurately
Interface npe0 IP-Address 10.0.0.12 Broadcast 10.0.0.255
Loading configuration files.
No suitable dump device was found.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
mount_nfs: can't update /var/db/mounttab for 10.0.0.8:/pe/slug
Setting hostuuid: 1c728616-1dd2-11b2-9fa6-0014bf6847ba.
Setting hostid: 0xb1c1273a.
Mounting local file systems:.
Setting hostname: slug.
net.inet6.ip6.auto_linklocal: 1 -> 0
DHCPREQUEST on npe0 to 255.255.255.255 port 67
ip length 328 disagrees with bytes received 332.
accepting packet with data after udp payload.
DHCPACK from 10.0.0.1
bound to 10.0.0.12 -- renewal in 900 seconds.
lo0: flags=8049 metric 0 mtu 16384
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
npe0: flags=8843 metric 0 mtu 1500
ether 00:14:bf:68:47:ba
inet6 fe80::214:bfff:fe68:47ba%npe0 prefixlen 64 scopeid 0x1
inet 10.0.0.12 netmask 0xffffff00 broadcast 10.0.0.255
media: Ethernet autoselect (100baseTX )
status: active
Additional routing options:.
Starting devd.
Additional IP options:.
Mounting NFS file systems:.
Creating and/or trimming log files:.
Starting syslogd.
/etc/rc: WARNING: Dump device does not exist. Savecore not run.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Initial arm initialization:.
Additional ABI support:.
Clearing /tmp (X related).
Starting local daemons:.
Updating motd.
Mounting late file systems:.
Starting cron.
Local package initialization:.
Starting inetd.
Starting background file system checks in 60 seconds.

Thu Jan 1 00:00:39 UTC 1970

FreeBSD/arm (slug) (ttyu0)

login: root
Jan 1 00:03:24 slug login: ROOT LOGIN (root) ON ttyu0
Last login: Thu Jan 1 00:42:28 on ttyu0
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 8.0-CURRENT (NSLU) #12: Sun Aug 3 00:09:07 MDT 2008

Welcome to FreeBSD!

Before seeking technical support, please use the following resources:

o Security advisories and updated errata information for all releases are
at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
for your release first as it's updated frequently.

o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
along with the mailing lists, can be searched by going to
http://www.FreeBSD.org/search/. If the doc distribution has
been installed, they're also available formatted in /usr/share/doc.

If you still have a question or problem, please take the output of
`uname -a', along with any relevant error messages, and email it
as a question to the [email protected] mailing list. If you are
unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
manual page. If you are not familiar with manual pages, type `man man'.

You may also use sysinstall(8) to re-enter the installation and
configuration utility. Edit /etc/motd to change this login announcement.

nslug# ntpdate harmony
3 Aug 07:04:18 ntpdate[582]: step time server 10.0.0.6 offset 1217746845.716155 sec
slug# date
Sun Aug 3 07:04:19 UTC 2008
slug# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
10.0.0.8:/pe/slug 55871592 49741636 1660230 97% /
devfs 1 1 0 100% /dev
10.0.0.8:/pe 55871592 49741636 1660230 97% /pe

Minor mips reorg and plans for the future

In trying to bring the MIPS tree more into line with idealized FreeBSD practices, I've moved the SoC directories up one level in the hierarchy. Future changes will make it easier to define different boards, as well as stylizing these sorts of things across different CPU architectures. I need this for the Plat'Home OpenMicroServer work I've started on, so I'll use that to drive these changes.

For the curious, I've moved mips/mips32/adm5120 -> mips/adm5120, likewise with the idt, malta, and sentry5 directories. While all of these were, strictly speaking, mips32 CPUs, having them in their own subdirectory doesn't make a lot of sense. There's never going to be so many of them that having an extra layer of indirection is going to help. Also, mips32 is ambiguous. Does it mean the MIPS ISA known as MIPS32 and MIPS32r2? Does it mean that the ports are running in 32-bit mode? It wasn't clear where a 32-bit kernel for a mips64 processor would live. Also, the idt parts aren't completely mips32 compliant in a couple of minor details. Is that non-compliance enough to move them elsewhere?

The next set of changes will try to tease apart the boot loader environment from the SoC that it is running on. While many SoCs have their own bootloader, often times one may also run uboot on the same hardware. In addition, things like uboot are used in many different platforms, so duplicating that code will grow tiresome.

Once the boot loader stuff is better abstracted, the next set of problems will be how to create a system that can run on multiple different boards of the same SoC. At first, this will be statically compiled, but eventually it will be dynamic.

Another key component of this work, whose overall goal is to make it easier to add board support to FreeBSD embedded platforms, will be an overhaul of the hints system. While it can be made to work, there are some practical problems it solves poorly that are desirable to have solved better. For example, it has no good way to tie attributes or parameters to hardware. Today, one can only tie them to a driver instance, which itself can be tied to hardware.

Finally, the last planned component in this work is a simplification of creation of busses with attributes and fixed or semi-fixed children. You can't have variants of hints loaded today that say "If I'm running on a AT91RM9200, use this set, but if I'm runnning on an AT91SAM9620 use this other set of hints" to enumerate the busses. This makes it hard to achieve the dynamic selection of boot code to run needed to allow kernels to boot on a variety of boards.

Most of this is low-level code janitor stuff. However, it is code janitor stuff that's needed and will be important to FreeBSD's success in the future.

Adding a Serial Port to a D-LINK DIR-615

A while ago, I added a serial port to the D-LINK DIR-615 (HW Rev A1). Today, I'll document how I did it.

First, you'll need a level converter circuit. There are many on the network. You can find a good write up on them and links from adding a serial port to a NSLU2. I always use the Parallax USB2SER that I bought years ago. It is very reliable, and I've had several provide years of good service to me. They convert 3.3V directly into a USB serial port, which vastly simplifies power and cabling. Since that's what I used for this project, all the instructions are geared towards it.

First, you'll need to disassemble the D-LINK DIR-615. There's only two screws so this is easy. Place the D-Link with the label with the serial number face up. There are four rubber feet on the bottom of this unit. Carefully pry up the two nearest to the Ethernet ports. Remove the screws found under them. The screws are clearly visible in this photo.



Carefully remove the lid. It should pry off just above the plastic bar that runs over the ethernet ports. You'll see a green PC board. On the right hand side of the unit, you'll find a header with 4 pins labeled CON5. They are labeled VCC, TX, RX, GND. This is the serial port. In my unit, it was hidden under the internal antenna, so the photos show it moved over a bit. Note also in figure two the board says "DIR635B1" on it...



I connected these pins to a ribbon cable and ran them out to a header that my USB2SER could fit on. I connected them so that they would match up to my USB2SER which had pins in the order GND, RX, TX, RES. The last one is unconnected. TX should go to RX, RX to TX and GND to GND. I didn't connect VCC, since it wasn't needed. USB2SER gets its power from the USB port. I reconnected everything and plugged the USB2SER into my computer. I threaded the ribbon cable out through the holes in the plastic, which is what you see in the first photo. Here's everything hooked up. I didn't include a photo of the USB2SER plugged into the serial port of my laptop for obvious reasons...



I see that it has a uboot bootloader:

% tip ucom0
U-Boot 1.1.1 (Jan 19 2007 - 11:08:07)
CAMEO uBoot Linux Loader version: 1.3.0.0

DRAM CS[0] base 0x00000000   size  32MB
DRAM Total size  32MB
before entry mvFlashInit
Flash: flashStructGet manu 0x89 id 0x17
INTEL 28F640J3A (64 Mbit)
Size:  8 MB,Bus Width: 1, device Width: 1.
Flash base: 0xff800000,Number of Sectors: 64 Type: REGULAR.
[8192kB@ff800000] Flash:  8 MB
Addresses 20M - 0M are saved for the U-Boot usage.
Mem malloc Initialization (20M - 16M): Done
*** Warning - bad CRC, using default environment


Soc: 88F5181 B1
CPU: ARM926 (Rev 0) running @ 500Mhz
SysClock = 166Mhz , TClock = 166Mhz


USB 0: host mode
PCI 0: PCI Express Root Complex Interface
PCI 1: Conventional PCI, speed = 33000000
Net:   egiga0 [PRIME]
Hit any key to stop autoboot:  0
Marvell>>

So that's how you add a serial port.