freebsd-wifi-build, or "wait, you can run freebsd on atheros MIPS access points? where do I get that?"

I've been running FreeBSD at home as my primary internet/wifi access for a few years now. It's cheap, it's easy to do, and I've tried very hard to wrap up the whole process into a mostly-simple build system that spits out a useful image to use.

It's pretty simple in concept - I take FreeBSD-HEAD, build it with some cut-down options, create a custom filesystem image with some custom boot scripts and a custom configuration file, and provide an image that you can TFTP (using a serial console and ethernet cable) or upload directly to the AP if it supports it.

The supported hardware list is here:

https://github.com/freebsd/freebsd-wifi-build/wiki/Supported-Boards

Now, it's not a huge list like OpenWRT, but that's mostly because I don't have an infinite supply of Atheros MIPS based routers. I think I'll get some of the TP-Link Archer series stuff next.

Building it is pretty simple:

https://github.com/freebsd/freebsd-wifi-build/wiki

You checkout the build repo, check out FreeBSD-HEAD, install a couple of packages, and run the build for your board. Once it's done, the images for your board appear in ../tftpboot/. There's a wiki page for each of the supported boards with a walkthrough with how to get FreeBSD going on it.

It comes up on 192.168.1.20/24 with 'user' and 'root' users, with no password. So, the first thing you should do after installation is telnet in, configure /etc/cfg/rc.conf with your actual LAN IPs, set the user/root passwords, and then 'cfg_save' to save things. Then, reboot and voila!

The configuration file format looks like FreeBSD but it isn't. I'm keeping it somewhat hierarchical-looking in naming but flat in implementation so I can migrate it to something like a sqlite or luci backend in the future.

https://github.com/freebsd/freebsd-wifi-build/wiki/Config-Overview

It's good enough for me to be able to set up an AP to be a bridge with a management IP address and configure the ethernet switch. Others have added ipfw support to do NAT and firewalling - I'm going to add configuration rules for NAT, IPFW and routing soon so it's all integrated.

It's FreeBSD, all the way through:


$ uname -a
FreeBSD tl-wdr3600 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r282406M: Wed May 6 22:27:16 PDT 2015 adrian@lucy-11i386:/usr/home/adrian/work/freebsd/head-embedded/obj/mips/mips.mips/usr/home/adrian/work/freebsd/head-embedded/src/sys/TL-WDR4300 mips
$ ifconfig wlan0 list sta
ADDR AID CHAN RATE RSSI IDLE TXSEQ RXSEQ CAPS FLAG
18:ee:69:15:f4:12 2 1 26M 37.0 45 2703 51888 EPS AQEHTRM RSN HTCAP WME
04:e5:36:0d:1b:0d 1 1 19M 23.0 15 1524 47072 EPS AQEPHTR RSN HTCAP WME
cc:3a:61:0e:33:a0 3 1 19M 32.0 30 2585 43072 EPS AQEPHTR RSN HTCAP WME
40:0e:85:1a:f1:69 4 1 19M 25.0 30 1138 54800 EPS AQEPHTR RSN HTCAP WME
00:0f:13:97:14:54 5 1 54M 30.0 45 1808 57424 EPS AE RSN
00:22:fa:c2:d1:20 6 1 26M 24.5 0 574 57776 EPS AQEHTRS RSN HTCAP WME

So if you'd like a FreeBSD based device to act as your home gateway, this is where you can start. It's not pfsense, but it's designed to run on things much smaller than pfsense supports and it's a good introduction into the world of FreeBSD embedded.

bsdtalk253 – George Neville-Neil

An interview with George Neville-Neil about the recently published 2nd edition of The Design and Implementation of the FreeBSD Operating System.

File Info: 30Min, 15MB

Ogg Link: https://archive.org/download/bsdtalk253/bsdtalk253.ogg

PC-BSD 11.0-CURRENTMAY2015 images now available

The PC-BSD project is pleased to announce the availability of our 11.0-CURRENTMAY2015 images.

WARNING: These images are considered “bleeding-edge” and should be treated as such.

The DVD/USB ISO files can now be downloaded from this URL.

We hope to continue rolling these –CURRENT images as a way for testers and developers to tryout both FreeBSD and PC-BSD bleeding edge features, often months before a planned release. These images include a full PKG repository compiled for that months image. Users of this system will also be able to “upgrade” when the next monthly image is published.

This is a great way to test features and report bugs well before the release cycle begins for the next major .0 release.

For bugs in PC-BSD, please report to https://bugs.pcbsd.org.

For FreeBSD / Port / Kernel / World bugs, please report to https://bugs.freebsd.org/bugzilla/enter_bug.cgi.

Updating from 10.1-RELEASE

There was a bug we found in freebsd-update when upgrading from 10.1-RELEASE to 11.0.

This has been fixed in 10.1-RELEASE-p21 (Check the output of freebsd-version), please update to it before attempting to change branch to 11.0.

Once you have the –p21 fix, run the following to “upgrade” your system to 11.0-CURRENTMAY2015svn282515

# pc-updatemanager chbranch 11.0-CURRENTMAY2015svn282515

This process will take a while, downloading new packages / world / kernel for the system. When done you can reboot, and the updater will finish up the update process.

PC-BSD 10.1.2-RC1 Now Available

The PC-BSD team is pleased to announce the availability of RC1 images for the upcoming quarterly 10.1.2 release.

Please test these images out and report any issues found on our bug tracker.

PC-BSD 10.1.2 Notable Changes

  • New PersonaCrypt Utility allows moving all of users $HOME directory to an encrypted USB Drive. This drive can be connected at login, and used across different systems
  • Stealth Mode allows login to a blank $HOME directory, which is encrypted with a one-time GELI key. This $HOME directory is then discarded at logout, or rendered unreadable after a reboot
  • Tor mode switches the firewall to running transparent proxy, blocking all traffic except what is routed through Tor
  • Migrated to IPFW firewall for enabling VIMAGE in 10.2
  • Added sound configuration via the first boot utility
  • Support for encrypted iSCSI backups via Life-Preserver, including support for bare-metal restores via installer media
  • New HTML handbook, updated via normal package updates
  • Media Center support allowing direct login to Kodi and PlexHomeTheater for the 10ft user experience
  • Switch to new AppCafe interface, with remote support via web-browser
  • Improvements to Online Updater, along with GRUB nested menus for Boot-Environments
  • Migrate all ports to using LibreSSL instead of OpenSSL
  • Switch from NTP to OpenNTPD
  • Lumina desktop 0.8.4
  • Chromium 42.0.2311.90
  • Firefox 37.0.2
  • NVIDIA Driver 346.47
  • Pkg 1.5.1

Updating

Users currently running the EDGE package repo can now update their packages via the updater GUI or “pc-updatemanager” utility to be brought up to date with RC1.

Updates for users on the 10.1.1 / PRODUCTION repo will be available once 10.1.2-RELEASE is announced.

Getting media

10.1.2-RC1 DVD/USB media can be downloaded from here via HTTP or Torrent.

Reporting Bugs
———————————————
Found a bug in 10.1.2? Please report it (with as much detail as possible) to our bugs database.

PC-BSD 10.1.2-RC1, Lumina Desktop 0.8.4 Released!

The PC-BSD team is pleased to announce the availability of RC1 images for the upcoming quarterly 10.1.2 release. Please test these images out and report any issues found on our bug tracker.

What else is new in PC-BSD 10.1.2? How about a new version of the Lumina Desktop Environment! PC-BSD users who stick to the “Production” branch of packages will find that the Lumina desktop has evolved/improved an incredible amount since the last quarterly update for PC-BSD (10.1.1), so I highly recommend that you try it out! The release notes for this new version are also listed at the bottom of this announcement for those of you who have been tracking along with its development, so please try it out and let us know what you think!

 

PC-BSD 10.1.2 notable Changes

  • New PersonaCrypt Utility
    • Allows moving all of users $HOME directory to an encrypted USB Drive. This drive can be connected at login, and used across different systems
    • Stealth Mode — Allows login to a blank $HOME directory, which is encrypted with a one-time GELI key. This $HOME directory is then discarded at logout, or rendered unreadable after a reboot
  • Tor mode — Switch firewall to running transparent proxy, blocking all traffic except what is routed through Tor.
  • Migrated to IPFW firewall for enabling VIMAGE in 10.2
  • Added sound configuration via the first boot utility
  • Support for encrypted iSCSI backups via Life-Preserver, including support for bare-metal restores via installer media
  • New HTML handbook, updated via normal package updates
  • Media Center support allowing direct login to Kodi and PlexHomeTheater for the 10ft user experience
  • Switch to new AppCafe interface, with remote support via web-browser
  • Improvements to Online Updater, along with GRUB nested menus for Boot-Environments
  • Migrate all ports to using LibreSSL instead of OpenSSL
  • Switch from NTP to OpenNTPD
  • Lumina desktop 0.8.4
  • Chromium 42.0.2311.90
  • Firefox 37.0.2
  • NVIDIA Driver 346.47
  • Pkg 1.5.1 Updating

Package Availability:

  • Users currently running the EDGE package repo can now update their packages via the updater GUI or “pc-updatemanager” utility to be brought up to date with RC1. Updates for users on the 10.1.1 / PRODUCTION repo will be available once 10.1.2-RELEASE is announced.

Getting media

Reporting Bugs

  • Found a bug in PC-BSD 10.1.2 or Lumina 0.8.4? Please report it (with as much detail as possible) to our bugs database. https://bugs.pcbsd.org

 

Lumina Desktop changes since version 0.8.3:

Panel Improvements:

  • Add mouse tracking support
  • Add support for variable-length panels (a percentage of the screen edge length).
  • Add support for pinning the panel to a particular location on the screen edge (either corner, or centered)
  • Automatically re-scale the panel size if the monitor used in the previous session was a different screen resolution.
  • For hidden panels, have 1% of the panel size be visible on the screen while it is “hidden” (rather than a hard-coded pixel size). This is better for high-resolution screens.
  • Remove the restriction that panels be on opposite screen edges.

New options/usage for lumina-search:

  • Easily change file/dir search preferences on a temporary basis
  • New command-line flags for starting searches instantly
  • Search functionality integrated into the Insight file manager. The Ctrl-F keyboard shortcut or the “Search” menu option will start a search for a file/directory with the current directory as the starting point.
  • A “Search” button has been added to the  home directory browser in the user menu. This will allow the user to easily start searching for a file/dir within the selected directory.

New “Favorites” system backend:

  • This new backend is much faster and more reliable than the old system of sym-links.
  • Your favorites should be automatically converted to the new format when you log into the new version of Lumina.

New Utility: lumina-fileinfo

  • This utility allows the user to view basic file information, such as timestamps, owner/group info, file size, and read/write permissions.
  • If the file is a XDG desktop shortcut (that the user has permission to modify), this utility also provides the ability to make changes to that shortcut.
  • This can easily be used by right-clicking on files in the desktop view plugin or within the Insight file manager and selecting the “Properties” option.
  • A big thank you to contributor William (william-os4y on GitHub) for writing this utility!!

Other Random Improvements:

  • Better application recommendations for files/URL’s (especially for web browsers or email clients).
  • Major cleanup of XCB library usage.
  • Hardware-brightness controls now used for PC-BSD by default (if supported by the system hardware).
  • Putting the system into the suspend state is now supported for PC-BSD/Debian.
  • New clock display formats.
  • A large number of session cleanup improvements
  • A large number of session initialization improvements (including resetting the user’s previous screen brightness and audio volume settings).
  • New default keyboard shortcuts for tiling the open windows on the screen (new user configurations only)
  • Better support for the URL input format when required by an application.
  • Make the user’s “log out” window appear much faster when activated.

Errata:

  • There is a known bug in Lumina 0.8.4 regarding “unlocked” desktop plugins. The close/maximize buttons for the plugin are unresponsive when using Qt 5.4.1, preventing the user from easily removing/maximizing a desktop plugin. We are still looking into this, but at the moment it appears to be a bug in Qt itself. As a temporary workaround, you can simply right-click on the titlebar for the unlocked plugin and select close/maximize from the menu instead.

From the Trenches, Tips & Tricks Edition: Hacking "/ on ZFS" and GELI Encrypted Drives, the Old-School Way

Glen Barber is back to kick off our latest From The Trenches series: The Tips and Tricks Edition. 

All my personal machines run FreeBSD.

In fact, all my personal machines run FreeBSD-CURRENT. I do this primarily to keep track of changes that get committed to the head branch, so I can personally test changes (for the things I use, at least) before they get merged to the stable branches.

As one of the Release Engineers, I find it essential that, whenever possible, I find issues so they can be corrected before they are part of a release.

My primary work machine is a laptop, currently a Lenovo Thinkpad T540p. I picked this laptop, and all the other laptops before it, because it met my minimum requirements for a primary workstation: it is capable of supporting a large amount of RAM (16GB for my Thinkpad, 8GB for all previous laptops), an Intel Core i7 CPU, and I could replace the DVD drive with a second hard drive.

In addition to these hardware requirements, I also have a few personal requirements of any workstation - the drives must be encrypted, and the underlying filesystem must be ZFS.

For me, it is not so much about the data I have *on* the laptop that I need to protect, but the kinds of things within the FreeBSD Project I am permitted access. Without encrypted drives, a lost or stolen laptop would absolutely be my worst possible nightmare, because I only have my login passphrase protecting my data (GPG key, SSH keys, and so on).

Recent FreeBSD releases allow "/ on ZFS" installation with the option to enable GELI-based encryption. This predates my original installation, however, since each laptop I have purchased for the past several years used the hard drives from the previous laptop. According to zpool history, the installation was at least two and a half years ago, but I know it is much longer than that, because of zfs recv being one of the first things zpool history reports.

So, I needed to do things the old-fashioned way, and manually create the GELI-backed providers and perform the "/ on ZFS" installation myself.

While bsdinstall(8) may now cover the majority of use cases for such installations, there may be cases where someone specifically needs to do something a certain way that the installer does not provide.

Because I only had one hard drive in the system when the system was initially installed (a long time ago), I will only refer to one hard drive when describing the steps I used to perform the installation, for now.

I installed the system using the 9.0-RELEASE or 9.1-RELEASE memory stick installer (memstick.img), I cannot remember which, but that detail is not as important, since I did not use the installer anyway.

When I booted from the memory stick, the two drives recognized on the system were the internal hard drive, /dev/ada0, and the external USB flash drive for the installation, /dev/da0. The first menu screen has three options available: "Install", "Shell", "Live CD".

I selected "Live CD", and logged in as root (no password is necessary for the "Live CD" functionality). The hard drive did not have an operating system. Because I purchased the hard drive, in addition to the laptop, with the intention of replacing the laptop's drive, I did not need to remove any partitions from an existing installation. If I did need to remove partitions, I would have done so with:
# gpart destroy -F ada0
Here is where some technical details become important:
  • While you can install "/ on ZFS" on a drive partitioned with MBR (Master Boot Record), using GPT is far easier. In fact, I have forgotten much about how MBR partitioning is actually done.
  • When doing full disk encryption, you must keep /boot contents separate, otherwise loader(8) and the kernel will not be available when the BIOS hands over control to the operating system. As such, /boot should be given its own partition on the disk left unencrypted, and the rest of the system on its own encrypted partition.
I created four partitions on the drive. The first partition is for the boot blocks (not to be confused with the /boot contents), the second partition is for /boot, the third is for the encrypted system, and the fourth is for swap.
# gpart create -s gpt ada0
# gpart add -t freebsd-boot -s 512k -i 1 -l gptboot ada0
# gpart add -t freebsd-zfs -s 10G -i 2 -l bootfs ada0
# gpart add -t freebsd-swap -s 10G -i 3 -l swapfs ada0
# gpart add -t freebsd-zfs -s 180G -i 4 -l rootfs ada0
I decided to put the swap partition between the /boot partition and the rest of the system, in case I needed to increase or decrease the size of the /boot partition, it would be far easier (and safer) to do.

Then, I loaded the necessary kernel modules for ZFS and GELI:
# kldload /boot/kernel/opensolaris.ko
# kldload /boot/kernel/zfs.ko
# kldload /boot/kernel/geom_eli.ko
Now that GELI functionality is available, I created the backend provider for the ZFS dataset:
# geli init -b -a HMAC/SHA256 -e AES-CBC -l 256 
-s 4096 /dev/ada0p4
Then I attached the GELI provider, and wrote data from /dev/randomto the new device /dev/ada0p4.eli:
# geli attach ada0p4
# dd if=/dev/random of=/dev/ada0p4.eli bs=4096

This took a while on the system this hard drive was originally installed, so I probably got coffee at this point. :-)

When the dd(1) command finished, I continued the installation.

I created temporary directories to use to import the pools after they were created:
# mkdir /tmp/zroot
# mkdir /tmp/zboot
Keep in mind, I am installing from a memory stick image, which by default, is read-only. The /tmp directory is writable, however, because it is a md(4)-backed memory disk filesystem.
# zpool create -O checksum=fletcher4 -O atime=off 
-m /tmp/zboot zboot /dev/ada0p2
# zpool create -O checksum=fletcher4 -O atime=off
-m /tmp/zroot zroot /dev/ada0p4.eli
Then I made a few ZFS datasets for various paths:
# for i in var var/log var/tmp var/db usr usr/home 
usr/compat usr/ports
usr/local tmp; do
zfs create zroot/${i}
done
I also made a separate ZFS dataset for the "bootfs" contents, and set the mountpoint to the /boot directory in the temporary working directory:
# zfs create zboot/boot
# zfs set mountpoint=/tmp/zroot/boot zboot/boot
On the memory stick installation media, the distribution sets are located in /usr/freebsd-dist. I extracted their contents into the newly-created filesystem:
# cd /tmp/zroot
# for i in base kernel lib32; do
tar -xf /usr/freebsd-dist/${i}.txz -C .
done
Then I wrote the bootcode to the first partition of the drive:
# gpart bootcode -b /tmp/zroot/boot/pmbr 
-p /tmp/zroot/boot/gptzfsboot -i 1 ada0
Because the "bootfs" (/boot) and "rootfs" (everything else) are both ZFS, I needed to use the gptzfsboot bootcode for the "freebsd-boot" partition.

Now the system is installed, but I needed to make a few modifications before I was ready to reboot. In particular, set a root password, edit /etc/fstab to enable swap, edit /etc/rc.conf to enable the zfs rc(8) startup script, and edit /boot/loader.conf to load the geom_eli.ko, opensolaris.ko, and zfs.ko kernel modules at boot.
# chroot /tmp/zroot
# passwd root
[enter password]
# echo '/dev/gpt/swapfs none swap sw 0 0'
>> /etc/fstab
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'geom_eli_load="YES"' >> /boot/loader.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# exit
Before rebooting, I needed to make a few adjustments to where /boot from the zboot/boot dataset would be mounted at boot.
# zfs umount zboot/boot
# zfs set mountpoint=/realboot zboot/boot
This now makes the /boot directory mount as /realboot, so I then needed to point /boot in the zroot dataset to the correct place. This was easily solved with a symbolic link:
# cd /tmp/zroot
# ln -s boot /realboot
Now when the system boots, the filesystem will look something like this:
/bin
/sbin
/boot -> /realboot
/realboot
[...]
Finally, I needed to unmount the zroot dataset, and fix its mountpoints. I only needed to change the zroot mountpoint itself, since all children datasets adjusted their paths automatically.
# zfs umount -a
# zfs set mountpoint=/ zroot
At this point, the installation was complete. I rebooted the laptop, entered the GELI passphrase for /dev/ada0p4.eli when prompted, and was greeted by the "login: " prompt we have all grown to love.

PC-BSD and 4K — Oh my!

I had recently seen some chatter on IRC about 4K monitor support. One of the people discussing stated that PC-BSD didn’t support 4K monitors, which was curious to me, since I hadn’t tried it yet. This week I did something about that, and ended up getting two 4K monitors and a new video card (With triple DisplayPort) to do some testing.

The Hardware

2x — ASUS PB287Q 28-Inch Screen LED-Lit 4K Monitor (Amazon)

Zotac GeForce GTX 960 2GB GDDR5 PCI Express 3.0 HDMI DVI DisplayPort (Amazon)

The Setup

The setup process was thus:

1. Remove the old monitors and video card

2. Connect the new monitors and video card

3. Go into monitor menus — Enable DisplayPort 1.2 mode

4. Boot up, selecting the “Run Display Wizard” option at the boot-loader.

5. Click “Accept”

That’s it! The display wizard properly detected both 4k monitors, and set them up in a proper dual-head configuration. Logging into KDE resulted in it detecting the new monitors, and automatically re-configuring the display to take advantage of the new resolution. At this point the system was fully usable, font-scaling even worked properly for Firefox and such.

Having 4K setup I did find a few minor issues, which are already being fixed in time for PC-BSD 10.1.2 next week, such as changing the mouse cursor to something which supports larger sizes /  resolutions better. The only real hiccup was figuring out that the monitor ships with DisplayPort set to 1.1 mode, only giving 30hz mode out of box. Changing that in the monitors menus to DisplayPort 1.2 mode, enabled 60hz, giving a silky-smooth desktop experience at 4K resolutions. Not sure if that is something typical of all 4K monitors, or just this model in particular, but if you can’t go beyond 30hz, you may want to check that in your monitor documentation.

Anyway, I hope this information is of use to anybody else who is thinking of taking the 4K plunge. Of course I would love additional feedback from other users who are running 4K, to see if there are other tweaks we can make for optimal support out of box. Let us know on the bug tracker, mailing lists or new forums!

Intel DDIO, LLC cache, buffer alignment, prefetching, shared locks and packet rates.

I've been digging into the low level behaviour of high throughput packet classification and pushing for my job. The initial suggestions from everyone was "use netmap!" Which was cool, but it only seems to to fast packet work if you're only ever really flipping packets between receive and transmit rings. Once you start actually looking into the payload, you start having to take memory misses and things can slow down quite a bit. An L3 miss (ie, RAM access) on Sandybridge is ~50ns. (There's also costs involved in walking the TLB, but I won't cover that here.)

For background: http://7-cpu.com/cpu/SandyBridge.html .

But! Intel has this magical thing called DDIO. In theory (and there's a lot of theory here), DMA is done via a small (~10%) fraction of LLC (L3) cache, which is shared between all cores. If the data is already in cache when the CPU accesses it, it will be quick. Also, if you then wish to DMA out data from something in cache, it doesn't have to get flushed to memory first - it's just DMAed straight out of cache.

However! When I was doing packet bridge testing (using netmap + bridge, 64 byte payloads), I noticed that I was doing a significant amount of memory bandwidth. It wasn't quite at the rate of 10G worth of bridged data, but DDIO should be doing almost all of that work for me at 64 byte payloads.

So, to reproduce: run netmap bridge (eg 'bridge -i netmap:ix0 -i netmap:ix1') and run pkt-gen between two nodes.

This is the output of 'pcm-memory.x 1' from the intel-pcm toolkit (which is available as a binary package on FreeBSD.)

---------------------------------------||---------------------------------------
--                   System Read Throughput(MB/s):    300.68                  --
--                  System Write Throughput(MB/s):    970.81                  --
--                 System Memory Throughput(MB/s):   1271.48                  --
---------------------------------------||---------------------------------------

The first theory - the bridging isn't occuring fast enough to service what's in LLC before it gets flushed out by other packets. So, assume:

  1. It's 1/10th of the LLC - which is 1/10th of an 8 core * 2.5MB per core setup, is ~ 2MB.
  2. 64 byte payloads are being cached.
  3. Perfect (!) LLC use.
That's 32,768 packets at a time. Now, netmap is doing ~ 1000 packets a batch and it's keeping up line rate bridging on one core (~14 million packets per second), so it's not likely that.

Ok, so what if it's not perfect LLC usage?

Then I thought back to cache line aliasing and other issues that I've previously written about. What if the buffers are perfectly aligned (say, 2048 byte aligned) - the cache line aliasing effects should also manifest themselves as low LLC utilisation.

Luckily netmap has a twiddle - 'dev.netmap.buf_size' / 'dev.netmap.priv_buf_size'. They're both .. 2048. So yes, the default buffer sizes are aligned, and there's likely some very poor LLC utilisation going on.

So, I tried 1920 - that's 2048 - (2 * 64) - ie, two cache lines less than 2048.


---------------------------------------||---------------------------------------
--                   System Read Throughput(MB/s):    104.92                  --
--                  System Write Throughput(MB/s):    382.32                  --
--                 System Memory Throughput(MB/s):    487.24                  --
---------------------------------------||---------------------------------------

It's now using significantly less memory bandwidth to do the same thing. I'm guessing this is because I'm now using the LLC much more efficiently.

Ok, so that's nice - but what about when it comes time to actually look at the packet contents to make decisions?

I've modified a copy of bridge to do a few things, mostly inspired by netmap-ipfw:
  • It does batch receive from netmap;
  • but it then looks at the ethernet header do decap that;
  • then it gets the IPv4 src/dst addresses;
  • .. and looks them up in a (very large) traditional hash table.
I also have a modified copy of pkt-gen that will use completely random source and destination IPv4 addresses and ports, so as to elicit some very terrible behaviour.

With an empty hash set, but still dereferencing the ethernet header and IPv4 source/destination, handling a packet at a time, no batching, no prefetching and only using one core/thread to run:

buf_size=2048:
  • Bridges about 6.5 million pps;
  • .. maxes out the CPU core;
  • Memory access: 1000MB/sec read; 423MB/sec write (~1400MB/sec in total).
buf_size=1920:
  • Bridges around 10 million pps;
  • 98% of a CPU core;
  • Memory access: 125MB/sec read, 32MB/sec write, ~ 153MB/sec in total.
So, it's a significant drop in memory throughput and a massive increase in pps for a single core.

Ok, so most of the CPU time is now spent looking at the ethernet header in the demux routine and in the hash table lookup. It's a blank hash table, so it's just the memory access needed to see if the bucket has anything in it. I'm guessing it's because the CPU is loading in the ethernet and IP header into a cache line, so it's not already there from DDIO.

I next added in prefetching the ethernet header. I don't have the code to do that, so I can't report numbers at the moment. But what I did there was I looped over everything in the netmap RX ring, dereferenced the ethernet header, and then did per-packet processing. This was interesting, but I wanted to try batching out next. So, after some significant refactoring, I arranged the code to look like this:
  1. Pull in up to 1024 entries from the netmap receive ring;
  2. Loop through, up to 16 at a time, and place them in a batch
  3. For each packet in a batch do:
    1. For each packet in the batch: optional prefetch on the ethernet header
    2. For each packet in the batch: decapsulate ethernet/IP header;
    3. For each packet in the batch: optional prefetch on the hash table bucket head;
    4. For each packet in the batch: do hash table lookup, decide whether to forward/block
    5. For each packet in the batch: forward (ie, ignore the forward/block for now.)
I had things be optional so I could turn on/off prefetching and control the batch size.

So, with an empty hash table, no prefetching and only changing the batch size, at buf_size=1920:
  • Batch size of 1: 10 million pps;
  • Batch size of 2: 11.1 million pps;
  • Batch size of 4: 11.7 million pps.
Hm, that's cute. What about with prefetching of ethernet header? At buf_size=1920:
  • Batch size of 1: 10 million pps;
  • Batch size of 2: 10.8 million pps;
  • Batch size of 4: 11.5 million pps.
Ok, so that's not that useful. Prefetching on the bucket header here isn't worthwhile, because the buckets are all empty (and thus NULL pointers.)

But, I want to also be doing hash table lookups. I loaded in a reasonably large hash table set (~ 6 million entries), and I absolutely accept that a traditional hash table is not exactly memory or cache footprint happy. I was specifically after what the performance was like for a traditional hash table. Said hash table has 524,288 buckets, and each points to an array of IPv4 addresses to search. So yes, not very optimal by any measure, but it's the kind of thing you'd expect to find in an existing project.

With no prefetching, and a 6 million entry hash table:

At 2048 byte buffers:
  • Batch size of 1: 3.7 million pps;
  • Batch size of 2: 4.5 million pps;
  • Batch size of 4: 4.8 million pps.
At 1920 byte buffers:
  • Batch size of 1: 5 million pps;
  • Batch size of 2: 5.6 million pps;
  • Batch size of 4: 5.6 million pps.
That's a very inefficient hash table - each bucket is going to have around 11 IPv4 entries in it, and that's checking almost a cache line worth of IPv4 addresses in it. Not very nice. But, it's within a cache line worth of data, so in theory it's not too terrible.

What about with prefetching? All at 1920 byte buffers:
  • Batch size of 4, ethernet prefetching: 5.5 million pps
  • Batch size of 4, hash bucket prefetching: 7.7 million pps
  • Batch size of 4, ethernet + hash bucket prefetching: 7.5 million pps
So in this instance, there's no real benefit from doing prefetching on both.

For one last test, let's bump the bucket count from 524,288 to 2,097,152. These again are all at buf_size=1920:
  • Batch size of 1, no prefetching: 6.1 million pps;
  • Batch size of 2, no prefetching: 7.1 million pps;
  • Batch size of 4, no prefetching: 7.1 million pps;
  • Batch size of 4, hash bucket prefetching: 8.9 million pps.
Now, I didn't quite predict this. I figured that since I was reading in the full cache line anyway, having up to 11 entries in it to linearly check would be cheap. It turns out that no, that's not exactly true.

The difference between the naive way (no prefetching, no batching) to 4-packet batching, hash bucket prefetching is not trivial - it's ~ 50% faster. Going all the way to a larger hash bucket was ~75% faster. Now, this hash implementation is not exactly cache footprint friendly - it's bigger than the LLC, so with random flows and thus no real useful cache behaviour it's going to degrade to quite a few memory accesses.

This has been quite a fun trip down the optimisation peephole. I'm going to spend a bunch of time writing down the hardware performance counters involved in analysing this stuff and I'll look to write a follow-up post with details about that.

One final things: threads and locking. I wanted to clearly demonstrate the cost of shared read locks on a setup like this. There's been lots of discussions about the right kind of locking and concurrency strategies, so I figured I'd just do a simple test in this setup and explain how terrible it can get.

So, no read-locks between threads on the hash table, batch size of 4, hash bucket prefetching, buf_size=1920:
  • 1 thread: 8.9 million pps;
  • 4 threads: 12 million pps.
But with a read lock on the hash table lookups:
  • 1 thread: 7 million pps;
  • 4 threads: 4.7 million pps.
I'm guessing that as I add more threads, the performance will drop.

Even taking a rwlock as a reader lock in pthreads is expensive - it's purely just an atomic increment/decrement in FreeBSD, but it's still not free. I'm getting the lock once for two hash table lookups - ie, the source and destination IP hash table lookups are done under one lock. I'm sure if I took the lock for the whole batch hash table lookup it'd work out a little better on a small number of CPU cores, but I think this demonstrates my point - read locks aren't going to cut it when you have a frequently accessed thing to protect.

The best bit about this post? The prefetching, terrible (large) hash table performance and general cache abuse is not new. Doing batching on superscalar Intel CPUs is not new. Documenting DDIO effectiveness using non-power-of-two-aligned buffer sizes is new, but it's just a rehash of the existing cache aliasing effect. But, I now have a little test bed to experiment with these things without having to try and involve the rest of a kernel.

Yes, I'll publish code soon.

ICC profiles and “startx” on FreeBSD

Beyond photography, I find that a calibrated screen is more comfortable to use, especially if you have two or more monitors with uneven panels and colors rendering.

You can buy a professional colorimeter for a not-so-small amount of money, but fortunately, there’s an open-source alternative. Richard Hughes developed ColorHug, a free open-source colorimeter that you can build yourself or order from him. I bought one myself shortly after it was released. It gives very good results, comparable to professional devices.

To calibrate the screen, FreeBSD lacks the appropriate applications in the Ports tree. The ColorHug is provided with a modified Fedora live CD with all the required programs installed and configured. Just boot your computer using this CD, plug the device and the wizard starts automatically, offerring you to calibrate all connected monitors. The process is quite straightforward and takes from a few minutes to 20-25 minutes, depending on the accuracy you want. When it’s finished, you have your shiny new profile(s) in ~/.local/share/icc. Profiles are named after monitor’s name and the date of the calibration. You need to copy those files to the same location on your FreeBSD desktop.

Now, you need to configure your computer to use the generated profile(s). GNOME, KDE and any other modern desktop environment come with display configuration tools which are easy to use. For those preferring a more lightweight environment and startx, things are a bit more complicated. Here is what I had to do.

  1. ICC profiles are managed by colord (graphics/colord). This daemon is started by dbus automatically. However, it seems that, after a fresh install, the service fails to start as reported in bug 197946 by several users of darktable. Running the service manually once “fixed” the problem for me:

    /usr/local/libexec/colord

  2. colord finds ICC profiles located in ~/.local/share/icc without further configuration. However, monitors are probed and managed by the X.Org server so another tool is required to declare all monitors in colord during session startup. I used xiccd, compiled from source (I need to make a port for it).

  3. Now, you need to start xiccd but PolKit and ConsoleKit get in the way as soon as xiccd tries to communicate with colord. Debugging this is where I spent most of the time… So here is the solution:

    • ConsoleKit is started by a login manager such as XDM, GDM, KDM and so on. If you don’t use one, you need to enable ConsoleKit in /etc/pam.d/login:

      # ConsoleKit.
      session  optional  /usr/local/lib/pam_ck_connector.so  nox11

    • To test if ConsoleKit works, log out and log in again on the console and run ck-list-sessions. You should get something like:

      Session1:
      	unix-user = '...'
      	realname = '...'
      	seat = 'Seat1'
      	session-type = ''
      	active = TRUE
      	x11-display = ''
      	x11-display-device = ''
      	display-device = '/dev/ttyv0'
      	remote-host-name = ''
      	is-local = TRUE
      	on-since = '2015-04-06T08:12:57.167178Z'
      	login-session-id = ''
      	idle-since-hint = '2015-04-06T08:13:27.293605Z'

      The “active = TRUE” line is the important one.

    • You need to start your X session with ConsoleKit. Modify your ~/.xinitrc to start your session with ck-launch-session:

      exec ck-launch-session dbus-launch --exit-with-session ~/.xstart

      dbus-launch starts dbus under the ConsoleKit session and ~/.xstart (of course, choose the name you want) is a script running all applications requiring a working dbus and ConsoleKit environment.

  4. My .xstart launches xiccd, loads ICC profiles and starts a window manager:

    #!/bin/sh 
    
    set -e
    
    if [ -x $HOME/Projects/graphics/xiccd/install/bin/xiccd ]; then
            # xiccd, compiled from source, is present, let's start it.
            $HOME/Projects/graphics/xiccd/install/bin/xiccd &
    
            # It takes some time to declare all screens in colord. Wait here
            # until colord knows about the laptop's monitor.
            #
            # To know the name of the monitors, use "colormgr get-devices".
            while ! colormgr get-devices | grep -q xrandr_LVDS_(...); do 
                    sleep 1 
            done 
    
            # Load the ICC profile for the laptop's monitor.
            #
            # To list the ICC profile IDs, use "colormgr get-profiles".
            colormgr device-add-profile 
                /org/freedesktop/ColorManager/devices/xrandr_LVDS_(...) 
                /org/freedesktop/ColorManager/profiles/icc_(...)
    
            # Do this again for each possible external screens.
            if colormgr get-devices | grep -q xrandr_EA244UHD_(...); then 
                    # NEC EA244UHD. 
                    xrandr --output LVDS --off 
                    xrandr --output HDMI-0 --mode 1920x1080 --rate 60 
    
                    colormgr device-add-profile 
                        /org/freedesktop/ColorManager/devices/xrandr_EA244UHD_(...) 
                        /org/freedesktop/ColorManager/profiles/icc_(...)
            fi 
    
            if colormgr get-devices | grep -q xrandr_SONY_TV_(...); then 
                    # Sony Bravia KDL-40WE5. 
                    xrandr --output LVDS --off
    
                    colormgr device-add-profile 
                        /org/freedesktop/ColorManager/devices/xrandr_SONY_TV_(...) 
                        /org/freedesktop/ColorManager/profiles/icc_(...)
            fi
    fi
    
    # Start window manager.
    exec awesome

Now, after startx(1), all monitors display the same colors!

With a single monitor to manage, oyranos (graphics/oyranos) was working for me. There is no dependency to colord, ConsoleKit or PolKit and is way simpler to use. However, it didn’t want to manage my external monitor (or two monitors at a time, I don’t know). This made an opportunity to play with colord. I just didn’t expect ConsoleKit or PolKit to join the party :)

Huge Announcement for PC-BSD

Hey PC-BSDers!

Some of you may have heard the speculation surrounding the last couple weeks of development here at the PC-BSD project, and I’d like to go ahead and address those rumors in today’s post.  We know there has been a large amount of misinformation circulating so we recommend that all of our users please read the following explanation so the information is clear.

For the last couple of months we have been beating our heads against the wall trying to find a way to put us on par with the best Linux distros out there such as Ubuntu.  During a phone meeting on Monday between Kris, Dru, Ken and myself (Josh), we began tossing around ideas on how we can make that happen.  One of the ideas that was presented was making Lumina DE the only supported desktop environment in PC-BSD and by doing so focus all of our development time exclusively on it.  Ken Moore argued that if we could create a desktop environment that everyone loved, and “Unify” the user experience, that no one would have any reason ever to use anything else because it was the best.  Although we did toss the idea around for a while we thought that might be a little bit of an issue with a few of our users that like to use other crappy desktop environments.

After an hour of spirited back and forth debate, many good ideas were presented.  As we all took a moment during our video call to think about the ideas that were just discussed, I interrupted the awkward silence and told the rest of the group about an idea I had been thinking about for a few days.  “What if we could do something even better than Unify the user experience…What if we could make the system boot extremely fast”. I explained that if we could hard code a “boot shim” so to speak into the kernel, that we could be the fastest booting unix-like distribution out there.  We all looked at each other and started to realize this was the “holy grail” so to speak that we had been looking for.  After perusing the internet for about 3 and a half minutes we found out there was already a Linux solution available.

We hammered out the details and now want to present you with our new “Everything Manager” the new SystemBS-D.  Most of you are familiar with how we can port different types of Linux software and run them through our emulation layer.  Using the same emulation magic we have taken what one kid that lives on my street called “the best piece of software ever. The end.”, and ported it over to PC-BSD.  SystemBS-D not only makes your system boot faster, but it can basically manage everything on your system as well.  Sure it still crashes a lot and it has trouble displaying log files, but we’re working on that.  In a discussion with Kris Moore he stated “SystemBS-D may be unstable, but it DID make the system boot super fast.  I feel like that’s a pretty good trade.”

Taking cues from other popular software companies we have also thought about integrating a really cool storefront into PC-BSD and making people look at it before they can go to their desktop.  We could also consider locking down the user’s system with “grub-lock” so no other operating systems could be installed on it… oops I meant “secure” the user’s system.

We look forward to hearing your feedback on these new developments.

P.S. if you thought any of this was real look at the date on your computer.  Happy 1st!

Best Regards,

Josh

 

Lumina Desktop 0.8.3 Released!

The next version of the Lumina Desktop Environment has just been released!

This is mainly a bugfix release to correct an urgent issue with the system tray on FreeBSD 11, but there are a number of other slight improvements/updates included as well. The full list of changes is included at the bottom of this announcement, but the notable changes are as follows:

  • New Panel Plugin: “Application Launcher“
    • This allows the user to pin the shortcut for an application directly to a panel.
  • New Utility: “lumina-xconfig“
    • This utility allows the user to easily enable/disable additional monitors/screens within the desktop session.
  • Fix the issue with transparent system tray icons on FreeBSD 11
  • Add support for the XDG autostart specifications.

 

The FreeBSD port has already been updated and this version will be included in the next set of PC-BSD package updates (“Edge” packages being created now) as well as included in the next PC-BSD 11 image for April (coming soon).

 

Reminder: The Lumina desktop environment is still considered to be “beta-quality”, so if you find things that either don’t work or don’t work well, please report them on the PC-BSD bug tracker so that they can get fixed as soon as possible. Feel free to also post tickets for any feature requests or improvements that you think might be useful!

 

Lumina-0.8.3

 

Changes Since 0.8.2

FreeDesktop Standards Compliance:

  •  A number of bugs related to detecting/using XDG mimetypes were fixed.
  • Support for the XDG autostart specifications was added (usage only  — more work is still necessary to convert the current Lumina autostart spec over)
  •  Add some additional fallback routines to account for possible errors in *.desktop files.

 

New Utility: “lumina-xconfig”

  •  This utility is a graphical front-end to xrandr, and allows the user to easily add/remove screens from the current X session.
  • Shortcuts to this utility are available in the user button plugin, and the settings menu plugin

 

Insight File Manager:

  •  Add support for creating new (empty) files.
  •  Add an option for enabling/disabling the use of image thumbnails (useful if you have massive directories of pictures — just be sure you disable thumbnails *before* loading the directory).
  •  Add initial drag-and-drop support for moving files/dirs within a directory.
  •  Load the specific icon for any application shortcuts.
  •  Add the ability to view checksums of files.
  •  Add some additional checks/excludes for copy/move operations in the background to prevent the user from performing illegal operations (such as moving a directory into itself).
  •  Add support for listing statistics about the current directory on the window (number of files, total size of files, percent of the filesystem which is used).
  •  Streamline the frequency of the background directory checker — now it runs much less often.

 

Desktop Changes:

  •  Disable the shutdown/restart options on PC-BSD if the system is in the middle of performing updates. The system may still be shutdown/restart from within PCDM — this just adds an extra layer of safety for users.
  •  Have the shutdown/restart options use the “-o” option on FreeBSD/PC-BSD so that the system performs the action much faster.
  •  Add support for thumbnails, increasing/decreasing icon sizes, removing files, and  cut/copy files to the “desktopview” desktop plugin (the plugin which provides traditional desktop icons).
  •  Add support for increasing/decreasing the icon size for the application launcher desktop plugin.
  •  Update the icon used for the “favorites” system in the user button and the file manager.
  •  Add the ability to display alternate timezones in the system clock. This does *not* change the system time at the moment, it is just a setting for the visual clocks/plugins.
  •  Add a new panel plugin for pinning application shortcuts directly to the panel. (just like the “applauncher” desktop plugin, but on the panel).
  •  Perform the initial search for applications on the system within the session initialization. This ensure that buttons/plugins are responsive as soon as the desktop becomes visible.
  •  Fix an issue with transparent system tray icons on FreeBSD 11, and convert the system tray embed/unembed routines to use the XCB library instead of XLib.