Sometimes you have to sit down and write #FreeBSD documentation

When working on new projects or hacks, sometimes you just have to stop, think and start writing down your processes and discoveries. While working on bootstrapping the DLink DIR-825C1, I realized that I had accumulated a lot of new (to me) knowledge from the FreeBSD Community (namely, Adrian Chadd and Warner Losh).

There is a less than clear way of constructing images for these embedded devices that has an analogue in the Linux community under the OpenWRT project. Many of the processes are the same, but enough are different that I thought it wise to write down some of the processes into the beginning of a hacker’s guide to doing stuff and/or things in this space.

The first document I came up with was based on the idea that we can netboot these little devices without touching the on-board flash device. This is what you should use to get the machine bootstrapped and figure out where all the calibration data for the wireless adapters exist. This is crucial to not destroying your device. The wireless calibration data (ART) is unique to each device, destroying it will mean you have to RMA this device.

The second document I’ve created is a description of how to construct the flash device hints entries in the kernel hints file for FreeBSD. I found the kernel hints file to be cumbersome in comparison to the linux kernel way of using device specific C files for unique characteristics.

Its interesting stuff if you have the hankering to dig a bit deeper into systems that aren’t PC class machines.

Meraki Sparky boards, and constant resetting

There's a Mesh internet project at Sudo Room and they've been doing some great work getting a platform up and running. However, like a lot of volunteer projects, they're working with whatever time and equipment they've been donated.

A few months ago they were donated a few hundred Meraki Sparky boards. They're an Atheros AR2317 SoC based device with an integrated 2GHz 802.11bg radio, 10/100 ethernet and.. well, a hardware watchdog that resets the board after five minutes.

Now, annoyingly, this reset occurs inside of Redboot too - which precludes them from being (fully) flashed before the unit reboots. Once the unit was flashed with OpenWRT, the unit still reboots every five minutes.

So, I started down the path of trying to debug this.

What did I know?

Firstly, the AR2317 watchdog doesn't have a way of resetting things itself - instead, all it can do is post an interrupt. The AR7161 and later SoCs do indeed have a way to do a full hardware reset if the watchdog is tickled.

Secondly, redboot has a few tricksy ways to manipulate the hardware:

  • 'x' can examine registers. Since we need them in KSEG1 (unmapped, uncached) then the reset registers (0x11000xxx becomes 0xb1000xxx.) Since its hardware access, we should do them as DWORDS and not bytes.
  • 'mfill' can be used to write to registers.
Thirdly, there's an Atheros specific command - bdshow - which is surprisingly informative:

RedBoot> bdshow
name:     Meraki Outdoor 1.0
magic:    35333131
cksum:    2a1b
rev:      10
major:    1
minor:    0
pciid:    0013
wlan0:    yes 00:18:0a:50:7b:ae
wlan1:    no  00:00:00:00:00:00
enet0:    yes 00:18:0a:50:7b:ae
enet1:    no  00:00:00:00:00:00
uart0:    yes
sysled:   no, gpio 0
factory:  no, gpio 0
serclk:   internal
cpufreq:  calculated 184000000 Hz
sysfreq:  calculated 92000000 Hz
memcap:   disabled
watchdg:  disabled (WARNING: for debugging only!)

serialNo: Q2AJYS5XMYZ8
Watchdog Gpio pin: 6
secret number: e2f019a200ee517e30ded15cdbd27ba72f9e30c8


.. hm. Watchdog GPIO pin 6? What's that?

Next, I tried manually manipulating the watchdog registers but nothing actually happened.

Then I wondered - what about manipulating the GPIO registers? Maybe there's a hardware reset circuit hooked up to GPIO 6 that needs to be toggled to keep the board from resetting.

Board: ap61
RAM: 0x80000000-0x82000000, [0x8003ddd0-0x80fe1000] available
FLASH: 0xa8000000 - 0xa87e0000, 128 blocks of 0x00010000 bytes each.
== Executing boot script in 2.000 seconds - enter ^C to abort
^C
RedBoot> # set direction of gpio6 to out
RedBoot> mfill -b 0xb1000098 -l 4 -p 0x00000043
RedBoot> x -b 0xb1000098
B1000098: 00 00 00 43 00 00 00 00  00 00 00 00 00 00 00 03  |...C............|
B10000A8: FF EF F7 B9 7D DF 5F FF  00 00 00 00 00 00 00 00  |....}._.........|

RedBoot> # pat gpio6 - set it high, then low.
RedBoot> mfill -b 0xb1000090 -l 4 -p 0x00000042
RedBoot> mfill -b 0xb1000090 -l 4 -p 0x00000002

.. then I manually did this every minute or so.

RedBoot>
RedBoot> mfill -b 0xb1000090 -l 4 -p 0x00000042
RedBoot> mfill -b 0xb1000090 -l 4 -p 0x00000002
RedBoot> mfill -b 0xb1000090 -l 4 -p 0x00000042
RedBoot> mfill -b 0xb1000090 -l 4 -p 0x00000002

.. so, the solution here seems to be to "set gpio6 to be output", then "pat it every 60 seconds."

I hope this helps people bring OpenWRT up on this board finally. There seems to be a few of them out there!

The Short List #6: Make the CD drive do something useful on #FreeBSD

Noted today that while grip could access the CD drive on my machine, clemetine-player and xfburn could not.

Figure out which device node your CD drive is with camcontrol:

camcontrol devlist | grep cd
at scbus4 target 0 lun 0 (cd0,pass2)

Simply add the following to /etc/devfs.conf and restart devfs to get access to the CD device:

perm /dev/cd0 0666
perm /dev/xpt0 0666
perm /dev/pass2 0666

Now bear in mind, that this means any user of your machine has access to the device now. Hopefully, on a desktop computer, you know all the users of your machine.

Using Jenkins libvirt-slave-plugin with bhyve

I've played with libvirt-slave-plugin today to make it work with libvirt/bhyve and decided to document my steps in case it would be useful for somebody.

libvirt-slave-plugin

Assuming that you already have Jenkins up and running, installation of libvirt-slave-plugin is as follows. As we need a slightly modified version, we need to build it ourselves. I've made a fork which contains a required modification which could be cloned like that:

git clone -b bhyve [email protected]:jenkinsci/libvirt-slave-plugin.git

The only change I made is adding a single line with 'BHYVE' hypervisor type, you could find the pull request here. When that would be merged, this step will be not required.

So, getting back to the build. You'll need maven that could be installed from ports:

cd /usr/ports/devel/maven2 && make install clean

When it's installed, go back to the plugin we cloned and do:

mvn package -DskipTests=true

When done, login to the Jenkins web interface, go to Manage Jenkins -> Manage Plugins -> Advanced -> Upload Pluging. It'll ask to provide a path to the plugin. It would be target/libvirt-slave.hpi in our plugin directory.

After plugin is installed, please follow to Manage Jenkins -> Configure System -> Add new cloud. Then you'll need to specify hypervisor type BHYVE and configure credentials so Jenkins could reach your libvirtd using SSH. There's a handy 'Test Connection' you could use your configuration.

Once done with that, we can go to Manage Jenkins -> Manage Nodes -> New Node and choose 'libvirt' node type. Then you'll need to choose a libvirt domain to use for the node. From now on, node configuration is pretty straightforward, expect, probably an IP address of the slave. To find out an IP address, you'd need to find out its MAC address (just run virsh dumpxml and you'll find it there) and then find the corresponding file in dnsmasq/default.leases file.

Guest Preparation

The only thing guest OS needs is to have jdk installed. I preferred to download a package with java/openjdk7, but I had to configure network first. My VMs use bridged networking on virbr0, so NAT config looks like that in /etc/pf.conf:


ext_if="re0"
int_if="virbr0"

virt_net="192.168.122.0/24"

scrub all

nat on $ext_if from $virt_net to any -> ($ext_if)

Now openjdk could be installed from the guest using:

pkg install java/openjdk7

Finally, find the nodes in node management menu and press 'Launch slave agent' button. It should be ready for the builds now.

PS It might be useful to sync clock on both guest and host systems using ntpdate.

PPS libvirt version should be at least 1.2.2.


EuroBSDCon 2014

EuroBSDCon 2014 (http://2014.eurobsdcon.org/), InterExpo Congress Center, Sofia, Bulgaria 25 - 28 September, 2014. EuroBSDcon is the premier European conference on the open source BSD operating systems attracting about 250 highly skilled engineering professionals, software developers, computer science students and professors, and users from all over Europe and other parts of the world. The goal of EuroBSDcon is to exchange knowledge about the BSD operating systems, facilitate coordination and cooperation among users and developers. The dates for EuroBSDCon 2014 in Sofia have been set to September 25-26th for tutorials and September 27-28th for the main conference.

PC-BSD Weekly Feature Digest 23

Hey PC-BSDers! This week we’re coming at you with some pretty sweet updates to PC-BSD. The mount tray has seen some significant improvement and is now able to mount most audio / dvd formats without a problem. Also windows partition types are now showing up correctly on my test system after building the new mount tray from source. The mount tray will also prompt you to open your disc with a program and will offer you correct suggestions based on the proper package / PBI. Ultimately the mount tray will most likely replace the built in mounting systems in the desktop environments. This is still a little ways off in the future, but the direction we are heading in.

We heard that there were some users that were experiencing problems upgrading and believe we have found the guilty party. I was able to duplicate the same package upgrade problem that was causing updates to 10.0.1 to fail, and asked Allan over at Scale Engine to give us a hand. Allan was able to track down the issue to a faulty distribution server that was interrupting connections and preventing the upgrades randomly. This server has been removed from service at this time and further work is going into preventing this from happening again in the future.

Work has begun to localize PC-BSD into the Hindi language. We’d like to give a shout out to the newest member of our translation team Simran. Thanks for your help and we are excited at the prospect for even more people to be able to use PC-BSD. Our estimated date of completion is 3 weeks from now. If you have an interest in this language please help us spread the word!

Other News / Projects for this week:

* Merged latest ports and gnome3 patches into ‘master’
* Merged in latest VirtualBox versions
* Wrote a userland replacement for the FUSE module to execute PBIs in a faster and less unstable manner (about 90% complete)
* Kicked off new –STABLE builds
* Update 9.x PBI’s
* Add new XDG-compatibility classes in libpcbsd (scanning/listing/filtering system applications)
* New Utility: pc-systemflag (shell) — pc-systemflag is used to set a flag/message on the system for cross-application communication
* Rewrite the pc-systemupdatertray utility to use the new SystemFlagWatcher. Is much simpler and more streamlined now.
* Add system flag usage to pc-softwaremanager for PBI update availability
* Add system flag usage to the pbi-manager (“pbi_update –check-all” usage only)
* Add system flag usage to pc-updatemanager (for all package and system updates/checks)

bsdtalk239 – PkgNG with Baptiste Daroussin at vBSDCon

A recording of Baptiste Daroussin speaking at vBSDCon in October 2013.  He is a FreeBSD source committer and project developer for PkgNG.  PkgNG is a package management tool for FreeBSD. It is the replacement for the current pkg_info/pkg_create/pkg_add tools.

File Info: 55Min, 26MB.

Ogg Link: http://cis01.uma.edu/~wbackman/bsdtalk/bsdtalk239.ogg

Getting to know your portmgr-lurker — Frederic Culot

Name

Frederic Culot

Committer name

culot

Inspiration for your IRC nick

lack of inspiration actually…

TLD of origin

.fr

Current TLD (if different from above)

.lu

Occupation

IT consultant in the banking sector in Luxembourg, but I don’t always do IT.
I am also interested in business and management and my wife and I are working
on starting our own business.

When did you join portmgr@

Joined FreeBSD as a committer in October 2010 and the portmgr-lurkers program in
March 2014, but never been part of portmgr@.

Blog

http://people.freebsd.org/~culot is the closest thing I have to a blog

Inspiration for using FreeBSD

I was a longtime OpenBSD user until I worked in the same company as clement@
(former portmgr) who successfully managed to convert me to FreeBSD. I did not
feel the need to look into another system since then.

Who was your first contact in FreeBSD

clement@. But when I really started to get involved in FreeBSD it was jadawin@
who first contacted me. He is one of the kindest person I ever worked with and
while we’ve known each others for about 4 years now I’ve never been able to
meet him in person. But that’s the way it is with projects such as FreeBSD:
teams are virtual and gathering together might be difficult unfortunately.

Who was your mentor(s)

My mentors were sahil@ and wen@. Thanks to them I believe my mentorship at
FreeBSD was the best induction program I ever experienced. I was also amused to
realize that whereas companies spend huge amounts to design reward systems, it
is sometimes when nothing is to be expected in return that people are the most
caring and helpful.

What was your most embarrassing moment in FreeBSD

My first pointy hat: a bit after my first 700 commits when I started to feel
confident I finally managed to break INDEX :’(

Boxers / Briefs / other

Any 15-year old single material does it.

What is your role in your circle of friends

uncork the bottles usually…

vi(m) / emacs / other

vim

What keeps you motivated in FreeBSD

The people behind it. There are lots of great guys behind this project, and a
day when I could not meet with other developers on irc is a sad day for me :’(

But FreeBSD is also one of my sources of inspiration when it comes to how
organizations behave and innovate (which is a topic of interest I got into
during my MBA studies) and I find it very interesting to compare FreeBSD with
the for-profit companies I work for. I even wrote an article for BSDmag in case
some would also be interested in those aspects:

http://people.freebsd.org/~culot/BSDmag.pdf

> Favourite musician/band

I don’t listen to much music. The cause might be that I work in a very noisy
environment (large open-space), so I more and more enjoy silence and calm when
I’m back home. But recently when I listen to music I enjoy Moby’s “wait for me”
album (ambient edition), Erik Mongrain, or a bit of merengue to remind me of my
holidays.

What book do you have on your bedside table

Nietzsche’s Thus spoke Zarathoustra.

I even extracted my favorite quotes and created the
french/fortune-mod-zarathoustra port.

coffee / tea / other

Both, depends on the time of day

Do you have a guilty pleasure

To enjoy a 7-course meal with my wife at a 3-star michelin restaurant and
finish relaxing in a club chair in front of the fireplace with a 40 years old
armagnac.

How would you describe yourself

Sober, clever, and motivated in the morning.
Drunk, stupid, and depressed in the evening.
Or is it the opposite?

sendmail / postfix / other

sendmail as it’s in base, but not for long apparently so I could have to make a
more reasoned choice soon

Do you have a hobby outside of FreeBSD

Sports (I go to the gym almost everyday day, did quite some scuba diving and
snowboarding when I was younger), but I enjoy good food and wine when I’m done
training. I also enjoy traveling. My last trips were to India, Dominican
Republic, and Lapland: so many nice places to visit!

What is your favourite TV show

My favorites to day are twin peaks, the prisoner, and battlestar galactica

Claim to Fame

I spent one night at the pub with bapt@, and survived.

What did you have for breakfast today

oat flakes with water

What sports team do you support

If you want to torture me, just fasten me in front of the TV with a soccer game
on. I could even confess I enjoy tabthorpe’s jokes just to shorten the ordeal.

<Editors note: I have know idea what he means by this :>

What else do you do in the world of FreeBSD

Apart from my work on ports I also did some French translations (translated
the contributing-ports, linux-users, and building-products articles). I also
try to participate in IT exhibits and promote FreeBSD by managing booths such
as at Solutions Linux Paris for which I designed a poster to attract visitors:

http://people.freebsd.org/~culot/SolutionsLinux.png

But most importantly I offer beers and whisky to other FreeBSD developers when
I meet them :)

What can you tell us about yourself that most people don’t know

I actually enjoy tabthorpe’s jokes. Sometimes.

<Editor’s note: again, no clue what he is talking about :>

Any parting words you want to share

I repeat it but my main motivation to work on this project is to get in touch
with other FreeBSD enthusiasts, so do not hesitate to ping me on irc if you
feel like sharing some of your thoughts with me. I would be most pleased.

What is your .sig at the moment

Regards,
Frederic Culot

PC-BSD Weekly Feature Digest 22

The week is finally almost over and we’re back for another update on PC-BSD! The majority was spent squashing bugs and performing minor updates to PC-BSD utilities (as well as recovering from the Jet lag from AsiaBSDcon for Kris and Dru)! To check out pictures from the big event have a look at IXsystem’s facebook page here. For a list of some of the changes and updates this week have a look below.

Bug Fixes
* Fixed missing RDP support for krdc
* Fixed issue installing src / ports for server installs
* Enabled “lz4” compression on root FS by default
* Disabled some FUSE file-cache functionality in PBIFS
* Investigated issues with calls to “vflush” causing fuse to never finish unmounting
* Imported latest stable/10 and started builds
* Imported latest gnome3 / cinnamon changes
* Finished building next Edge package set
* Finished GUI updates and changes to bring them up to our new / current standards
* Added accessibility / shortcut keys for PC-BSD utilities

Adding chipset powersave support to FreeBSD’s Atheros driver

I've started adding some basic powersave support to the FreeBSD Atheros ath(4) driver. The NICs support putting parts of the device to sleep to conserve power but.. well, it's tricky.

In order to make things consistent, I either need to not do things when the NIC is asleep (for example, doing calibration when the NIC isn't running), but I also need to ensure that I force the NIC awake when the NIC may be asleep. During normal running, the NIC may have put itself into temporary sleep whilst waiting for some packets from the AP to signal that it needs to wake up. So I will also need to force the NIC awake before programming it.

So, before I start down the path of handling the whole dynamic power management stuff, I figured I'd tackle the initial bits - handling powering on the NIC at startup and powering it off when it's not in use. This includes powering it down during device detach and suspend, as well as when all of the VAPs are down.

This is turning out to be slightly more complicated than I'd like it to be.

The first really stupid thing I found was that during the interface down process, the VAP state change from RUN -> INIT would reset the BSS, which included re-programming the slot time. So, I have to wake up the hardware when programming that. It can then go back to sleep when I'm done with it.

Now there's some issues in the suspend path with the NIC being marked as asleep when it is being reset, which is confusing - the NIC should be woken up when ath_reset() is called. So, I'll have to debug these.

The really annoying bit is that if I read a register whilst the silicon is asleep, the reads return 0xDEADBEEF. So if I am storing the register contents anywhere, I'll end up storing and programming a potentially totally invalid value.

There's also some real problems with race conditions. I can put the power state changes behind a lock, but imagine something like this:

* ATH_LOCK; force awake; do something; ATH_UNLOCK .. ATH LOCK; do some more; put back to sleep; ATH_UNLOCK

Now, if a second thread puts the NIC back to sleep in between those two lock sections, the second "do some more" work may occur once the NIC was put to sleep by said second thread. So I have to correctly track if the NIC is being forced awake by refcounting how many times its being forced awake, then when the refcount hits zero and we can put it to sleep, put it back to sleep.

Once this is all done, I can start down the path of supporting proper network sleep - where the NIC stays asleep and wakes up to listen for beacons and received frames from the AP. I then choose to force the NIC awake and do more work. I have to make absolute sure that I don't queue things like transmitted frames or add more frames to the receive queue if it may fall asleep. There's also some mechanisms to have a transmit frame put the NIC to sleep - there's a bit that says "when this frame is transmitted, transition the NIC back to sleep." I have to go and figure out how that works and implement that.

But for now, let's keep it simple and debug just putting the NIC to sleep when it's not in use.

Upgrading Graphite

Recently swills@ upgraded Graphite and reconfigured how it works to fit more in to the FreeBSD file system layout.

So if you are upgrading from a graphite installation older than 0.9.12_1, you will need to follow the following instructions:

  1. Stop carbon
  2. Copy the old data from /usr/local/storage/whisper/* to /var/db/carbon/whisper/
  3. Copy the /usr/local/etc/carbon/carbon.conf.example over to carbon.conf
  4. Set the SECRET_KEY to something random in /usr/local/etc/graphite/local_settings.py
  5. Then follow the instructions after the install, including updating the httpd.conf per the message after the install
  6. Restart Carbon and Apache

Be careful that you do not miss any of the steps and you should have a working Graphite install.

Bhyve in libvirt

I continue my activities on improving libvirt FreeBSD support and I have some good news. Recent libvirt release, 1.2.2, is the first version to include the bhyve support!

Currently it's in its early stage and doesn't support some of the features and doesn't provide good flexibility, it's just a basic stuff at this point. I'll not provide a detailed description and instead will point you to the document: Libvirt: Bhyve driver. You'll find a sample domain XML which covers all the features currently supported by the driver.

TODO list

While there are lots and lots of things to be done, there are some specific ones I'm focusing on:

  • Console support through nmdm(4). This is very important feature for debugging and checking what's going on in the guest.
  • Domains autostart support. There's a patch already kindly provided by David Shane Holden that just needs review and testing.
  • A little more flexible slot ids allocation / device configuration.

Qemu/FreeBSD status

As a side note, I'll give an update what's changed since my previous blog post about qemu libvirt driver on FreeBSD. So, here's what's new:

  • Proper TAP interfaces cleanup
  • CPU affinity configuration support, check http://libvirt.org/formatdomain.html#elementsCPUAllocation for details
  • virsh console should now work if you run it from freebsd host and connect to libvirtd on Linux
  • Node status support (such as virsh nodecpustats, virsh nodememstats)

Some of these are available in already released versions, some are only in git version.


PC-BSD Weekly Feature Digest 21 — PCBSD 10.0.1 Released!

PC-BSD 10.0.1 Has been released! Check out the release notes from Kris below. Kris is currently out in the field attending AsiaBSDCon so make sure to stop by the FreeBSD booth if you’re in the area and show your support! Work has continued this week on the development of the new PC-BSD mixer although our primary goal this week was to get a bunch of trac tickets fixed, closed, or assigned to someone to take care of them. Thanks as always and enjoy the new updates!

- Josh

———————————————

The first PC-BSD 10.0 quarterly update is upon us, and 10.0.1 is now
available.

This update includes a number of important bugfixes, as well as newer
packages and desktops, such as KDE 4.12.2, Cinnamon 2.0 and more. For
more details and updating instructions, refer to the notes below.

Changes
———————————————

* KDE 4.12.2
* Cinnamon 2.0
* Samba 4.1.4
* Stability improvements to PBI subsystems
* Updated GRUB loader, fixing issues related to slow / hanging startup
* Updated AppCafe UI
* Updates to Life-Preserver, including “Classic” backup mode and
automatic snapshots
* Updated control panel with desktop settings buttons
* PulseAudio 5.0 integration
* Improved Video display auto-detection
* Bugfixes to mouse auto-detection
* Improved LDAP / AD support for login manager
* Misc other bugfixes

Updating
———————————————

Desktop users already running 10.0 can update via Control Panel ->
Package Manager -> Updates.
Server users can update via the “pc-updatemanager” utility.

If package updating fails due to conflict errors, please be sure to
apply all system updates first before trying again.

Installing
———————————————

10.0.1 DVD/USB media can be downloaded from the following URL:

http://www.pcbsd.org/en/download.html

Reporting Bugs
———————————————

Found a bug in 10.0.1? Please report it (in as much detail as possible)
to our Trac Database.

https://trac.pcbsd.org

On standards (and testing)

RFC 4648 defines the Base16, Base32 and Base64 encodings. Base16 (aka hex) and Base64 are widely known and used, but Base32 is an odd duck. It is rarely used, and there are several incompatible variants, of which the RFC acknowledges two: [A-Z2-7] and [0-9A-V].

One of the uses of Base32, and the reason for my interest in it, is in Google’s otpauth URI scheme for exchanging HOTP and TOTP keys. I needed a Base32 codec for my OATH library, so when a cursory search for a lightweight permissive-licensed implementation failed to turn up anything, I wrote my own.

My OATH implementation is currently deployed in an environment in which OTP keys for new users (or new OTP keys for existing users) are generated by the primary provisioning system, which passes them on to a smaller provisioning system in charge of firewalls and authentication (codenamed Nexus), which passes them on to a RADIUS server, which uses my code to validate user responses. When we transitioned from generating OTP keys manually to having the provisioning system generate them for us, we ran into trouble: some keys worked, others didn’t. It turned out to be a combination of factors:

  • The keys generated by the provisioning system were syntactically correct but out of spec. Most importantly, their length was not always a multiple of 40 bits, so their Base32 representation included padding.
  • Nexus performed only cursory validation of the keys it received from the provisioning system, so it accepted the out-of-spec keys.
  • The Google Authenticator app (at least the Android version, but possibly the iOS version as well) does not handle padded keys well. If I recall correctly, the original Android app rejected them outright; the current version simply rounds them down. (Why don’t the Android system libraries provide Base32 encoding and decoding?)
  • My Base32 decoder didn’t handle padding correctly either… and of course, I only had tests for the encoder, because I was in a rush when I wrote it and I didn’t need decoding until later. Yes, this is stupid. Yes, I fixed it and now have 100% condition/decision coverage (thanks to BullseyeCoverage, with a caveat: 100% C/D coverage of table-driven code does not guarantee correctness, because it only checks the code, not the table).

Having fixed both the provisioning system and the OATH verification tool, I decided to add stronger input validation to Nexus. The easiest way to validate a Base32-encoded key, I figured, is to decode it. And wouldn’t you know, there are not one but two Perl implementations of Base32!

Unfortunately, they’re both broken, and have been for years.

  • MIME::Base32 (the latest release is dated 2010-08-25, but the code hasn’t changed since the original release on 2003-12-10) does not generate padding, and decodes it into garbage. In addition, it does not accept lower-case code.
  • Convert::Base32 (the latest release is dated 2012-04-22, but the code hasn’t changed since the original release on 2001-07-17) does not generate padding, and dies when it encounters what it calls “non-base32 characters”. In addition, while it accepts lower-case code (which is commendable, even though the RFC specifies an upper-case alphabet), it also generates lower-case code, which is wrong.

Both packages ship with tests. MIME::Base32’s tests simply encodes a string, decodes the result, and checks that it got the original string back.

Convert::Base32’s tests are more complex and include length and padding tests, but it defines padding as the lower, unused bits of the last non-padding character in the output.

MIME::Base32 references RFC 3548 (the predecessor to RFC 4648) but does not come close to implementing it correctly. Convert::Base32 predates the RFC and conforms to the old RACE Internet draft, which is small consolation since RACE was never standardized and was eventually replaced by Punycode.

I wrote a script which runs the RFC 4648 test vectors through either or both MIME::Base32 and Convert::Base32, depending on what’s available. The first two columns are the input and output to and from the encoder, and the last two are the input and output to and from the decoder. Note that the script adds the correct amount of padding before feeding the encoded string back to the decoder.

MIME::Base32
 1 f            |  2 MY               |  8 MY======         |  7 fOOOOO     
 2 fo           |  4 MZXQ             |  8 MZXQ====         |  6 fo����
 3 foo          |  5 MZXW6            |  8 MZXW6===         |  6 foo���
 4 foob         |  7 MZXW6YQ          |  8 MZXW6YQ=         |  5 foob       
 5 fooba        |  8 MZXW6YTB         |  8 MZXW6YTB         |  5 fooba       
 6 foobar       | 10 MZXW6YTBOI       | 16 MZXW6YTBOI====== | 12 foobarOOOOO
Convert::Base32
Data contains non-base32 characters at base32-test.pl line 16
 1 f            |  2 my               |  8 my======         | %

(the final % is my shell indicating that the output did not end with a line feed).

The same test, with forced conversion to upper-case before decoding:

MIME::Base32
 1 f            |  2 MY               |  8 MY======         |  7 fOOOOO     
 2 fo           |  4 MZXQ             |  8 MZXQ====         |  6 fo����
 3 foo          |  5 MZXW6            |  8 MZXW6===         |  6 foo���
 4 foob         |  7 MZXW6YQ          |  8 MZXW6YQ=         |  5 foob       
 5 fooba        |  8 MZXW6YTB         |  8 MZXW6YTB         |  5 fooba       
 6 foobar       | 10 MZXW6YTBOI       | 16 MZXW6YTBOI====== | 12 foobarOOOOO
Convert::Base32
Data contains non-base32 characters at base32-test.pl line 17
 1 f            |  2 my               |  8 MY======         | %

Once again, with forced conversion to lower-case:

MIME::Base32
 1 f            |  2 MY               |  8 my======         |  8 my======    
 2 fo           |  4 MZXQ             |  8 mzxq====         |  7 mz{����
 3 foo          |  5 MZXW6            |  8 mzxw6===         |  7 mz{�O
 4 foob         |  7 MZXW6YQ          |  8 mzxw6yq=         |  6 mz{��^
 5 fooba        |  8 MZXW6YTB         |  8 mzxw6ytb         |  6 mz{��]
 6 foobar       | 10 MZXW6YTBOI       | 16 mzxw6ytboi====== | 14 mz{��]���zzzzz
Convert::Base32
Data contains non-base32 characters at base32-test.pl line 17
 1 f            |  2 my               |  8 my======         | %

sigh

FreeBSD Foundation Accepting Travel Grant Applications for BSDCan 2014

Calling all FreeBSD developers needing assistance with travel expenses to BSDCan 2014.

The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please fill out and submit  the Travel Grant Request Application by April 7th, 2014 to apply for this grant.

This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc).  In some  cases we are also able to fund non-developers, such as active community members and FreeBSD advocates. More details are available in the announcement.

Getting to know your portmgr-lurker — Alexy Dokuchaev

In this latest edition of Getting to know, we interview senior ports committer Alexy Dokuchaev as one of our newest portmgr-lurkers.

Name

Alexey Dokuchaev

Committer name

danfe@

TLD of origin

.ru (technically should be .su, but it’s now defunct)

Occupation

Software engineer and contractor

Inspiration for using FreeBSD

Wanted a Unix system that I could understand and that would not get bloated
as time goes by. In 1998 Linux was popular mount local folks, but I just
could not get it (trying to switch from DOS). Someone mentioned FreeBSD;
and it all started to make sense pretty much immediately. And even after
some 15 years, FreeBSD still feels like back in those good days.

Who was your first contact in FreeBSD

Max Khon (fjoe@), I guess…

Who was your mentor(s)

fjoe@ and krion@

vi(m) / emacs / other

nvi(1)

What keeps you motivated in FreeBSD

That in 2014 I can have a modern Unix system which I still can work with
like it’s 1999 again. I can still have text console. And start X11 with
startx(1). Configure things by editing /etc/rc.conf, but have support for
the latest hardware. Play sound via OSS, yet enjoy low-latency in-kernel
mixer. We’re so lucky to not have Lennart Poettering with his PulseAudio
and systemd crap…

coffee / tea / other

Both (but tea preferred) + beer

What is your favourite TV show

The X Files, I guess…

What is your .sig at the moment

./danfe (making it both answer and a sig this time)

FreeBSD Project to participate in Google Summer of Code 2014

The FreeBSD Project is pleased to announce its participation in Google's 2014 Summer of Code program, which funds summer students to participate in open source projects. This will be the FreeBSD Project's tenth year in the program, having mentored over 160 successful students through summer-long coding projects between 2005 and 2013.

Porting over the AR8327 support

It's been a while since I posted. I'll post about why that is at some point but for now I figure it's time I wrote up the latest little side project - the Atheros AR8327 switch support.

The AR8327 switch is like the previous generation Atheros switches except for a couple of very specific and annoying differences - the register layouts and locations have changed. So it's not just a case of pretending it's an AR8316 except for the hardware setup - there's some significant surgery to do. And no, I did try just ignoring all of that - the switch doesn't come up and pass packets.

So, the first thing was to survey the damage.

The Linux driver (ar8216.c) has a bunch of abstractions that the FreeBSD driver doesn't have, so that's a good starting point. The VLAN operations and VLAN port configuration stuff is all methods in the Linux driver, so that was a good starting point. I stubbed most of the VLAN stuff out (because I really didn't want it to get in the way) - this turned out to be more annoying than I wanted.

Next was the hardware setup path. There's more configurable stuff with the AR8327 - there's two physical ports that I can configure the PHY/MAC parameters on for either external or internal connectivity. I just took the code from Linux (which yes, I have permission to relicence under BSD, thanks to the driver authors!) and I made it use the defaults from OpenWRT for the DB120. The ports didn't properly come up.

I then realised that I was reading total garbage from the PHY register space, so I went looking at the datasheet and ar8216 driver for some inspiration. Sure enough, the AR8327 has the PHY MDIO bus registers in different locations. So after patching the arswitch PHY routines with this knowledge, the PHYs were probed and attached fine. Great. But it still didn't detect port status changes.

So, back to the ar8216 driver. It turns out that there were a few things that weren't methodized - and these were the bits that read the PHY status from the switch. Both drivers didn't just poll the PHYs directly - they read the switch registers which had a summary of the port status. So, I taught the driver about this and voila! Port status changes worked.

But, no traffic.

Well, there's a few reasons for this. It's a switch, so I don't have to setup anything terribly difficult. The trick here is to enable port learning and make sure they're all in the same VLAN group. Now, here's where I screwed up and I found a bug that needed working around.

The port setup code did enable learning and put things into a vlan group.

Firstly, I found this odd behaviour that I got traffic only when I switched the ethernet cable to another port. Then learning worked fine. I then found that the ar8216 driver actually triggers a forwarding table flush upon port status change, so I added that. This fixed that behaviour.

But then it was flooding traffic to all ports. This is kinda stupid. What did I screw up? I put each port in a separate vlangroup, rather than put them in the same vlangroup. Then, I programmed the "which ports can you see?" to include all the other ports. What this meant was:
  • The forwarding table (ie, what addresses were learnt) were linked to the vlangroup the port is in;
  • .. and when the switch did a lookup for a given MAC on another port, it wouldn't find it, as the address in the forwarding table showed it was for another vlangroup;
  • .. so it would do what switches do when faced with not knowing about the MAC (well, and how I had configured it) - it flooded traffic.
The solution was thankfully easy - I just had to change the vlangroup (well, "port vlan" here) to be '1', instead of the port id. Once this was done, all the ports came up perfectly and things worked great.

So, this now works great on the Atheros DB120 reference board. It's not working on other boards - there's likely some timing issues that need to be resolved. But we're making progress!

Finally, I spent a bunch of time porting over the port configuration and LED configuration stuff from OpenWRT so I didn't have the driver just hard-coded to the DB120 board. I'll update the configuration and code when I get my hands on other boards that use the AR8327 but for now this is all I have.

Enjoy!

bhyvecon 2014

bhyvecon 2014 (http://bhyvecon.org/), SAKURA Internet Research Center, Tokyo, Japan 12 March, 2014. See the bhyve hypervisor in action and ask a core bhyve developer your technical questions.

PC-BSD Weekly Feature Digest 20

New Sound Management

Work has began to fully port pulse audio into PC-BSD for 10.1, and we are quite pleased so far with the results.   Kris has been making headway this week getting pulse audio and it’s related utilities working.  In the meantime Ken has been working on an all new utility pc-mixer.  pc-mixer is a complete front-end to the FreeBSD “mixer” utility that will allow users a simple to use GUI and volume control for every day tasks.  There will also be an advanced tab allowing for more specific audio setups and control.

Other News

*New PBIs for 9.x versions and 10.x versions were released this week, so be sure to check out the AppCafe and see what’s new.

*Gnome 3 and Cinnamon 2.0 desktops have received updates this week.  These desktops are not 100% fully supported yet and as    such we can not make any guarantee on functionality.

*Grub 2.02 has been fully ported over and updated to GRUB 2.02-prerelease.

*Lastly The PC-BSD ports tree has been frozen in preparation for our quarterly package update.

Improvements for Life-Preserver
* Add new “Classic” backup dialog for custom exclusions and status updates
* Fix bug with restoring a file/dir into a missing directory on the main system.
* Clean up the restore tab

Bug Fixes
* Bugfixes to the FUSE “pbifs” file-system
* Fix bug showing HPLIP drivers in the main CUPS Manager.
* Fix seg-fault crash in EasyPBI when removing a non-selected item.