Category Archives: FreeBSD

Upgrading Graphite

Recently swills@ upgraded Graphite and reconfigured how it works to fit more in to the FreeBSD file system layout.

So if you are upgrading from a graphite installation older than 0.9.12_1, you will need to follow the following instructions:

  1. Stop carbon
  2. Copy the old data from /usr/local/storage/whisper/* to /var/db/carbon/whisper/
  3. Copy the /usr/local/etc/carbon/carbon.conf.example over to carbon.conf
  4. Set the SECRET_KEY to something random in /usr/local/etc/graphite/
  5. Then follow the instructions after the install, including updating the httpd.conf per the message after the install
  6. Restart Carbon and Apache

Be careful that you do not miss any of the steps and you should have a working Graphite install.

Bhyve in libvirt

I continue my activities on improving libvirt FreeBSD support and I have some good news. Recent libvirt release, 1.2.2, is the first version to include the bhyve support!

Currently it's in its early stage and doesn't support some of the features and doesn't provide good flexibility, it's just a basic stuff at this point. I'll not provide a detailed description and instead will point you to the document: Libvirt: Bhyve driver. You'll find a sample domain XML which covers all the features currently supported by the driver.

TODO list

While there are lots and lots of things to be done, there are some specific ones I'm focusing on:

  • Console support through nmdm(4). This is very important feature for debugging and checking what's going on in the guest.
  • Domains autostart support. There's a patch already kindly provided by David Shane Holden that just needs review and testing.
  • A little more flexible slot ids allocation / device configuration.

Qemu/FreeBSD status

As a side note, I'll give an update what's changed since my previous blog post about qemu libvirt driver on FreeBSD. So, here's what's new:

  • Proper TAP interfaces cleanup
  • CPU affinity configuration support, check for details
  • virsh console should now work if you run it from freebsd host and connect to libvirtd on Linux
  • Node status support (such as virsh nodecpustats, virsh nodememstats)

Some of these are available in already released versions, some are only in git version.

Getting to know your portmgr-lurker — Alexy Dokuchaev

In this latest edition of Getting to know, we interview senior ports committer Alexy Dokuchaev as one of our newest portmgr-lurkers.


Alexey Dokuchaev

Committer name


TLD of origin

.ru (technically should be .su, but it’s now defunct)


Software engineer and contractor

Inspiration for using FreeBSD

Wanted a Unix system that I could understand and that would not get bloated
as time goes by. In 1998 Linux was popular mount local folks, but I just
could not get it (trying to switch from DOS). Someone mentioned FreeBSD;
and it all started to make sense pretty much immediately. And even after
some 15 years, FreeBSD still feels like back in those good days.

Who was your first contact in FreeBSD

Max Khon (fjoe@), I guess…

Who was your mentor(s)

fjoe@ and krion@

vi(m) / emacs / other


What keeps you motivated in FreeBSD

That in 2014 I can have a modern Unix system which I still can work with
like it’s 1999 again. I can still have text console. And start X11 with
startx(1). Configure things by editing /etc/rc.conf, but have support for
the latest hardware. Play sound via OSS, yet enjoy low-latency in-kernel
mixer. We’re so lucky to not have Lennart Poettering with his PulseAudio
and systemd crap…

coffee / tea / other

Both (but tea preferred) + beer

What is your favourite TV show

The X Files, I guess…

What is your .sig at the moment

./danfe (making it both answer and a sig this time)

Porting over the AR8327 support

It's been a while since I posted. I'll post about why that is at some point but for now I figure it's time I wrote up the latest little side project - the Atheros AR8327 switch support.

The AR8327 switch is like the previous generation Atheros switches except for a couple of very specific and annoying differences - the register layouts and locations have changed. So it's not just a case of pretending it's an AR8316 except for the hardware setup - there's some significant surgery to do. And no, I did try just ignoring all of that - the switch doesn't come up and pass packets.

So, the first thing was to survey the damage.

The Linux driver (ar8216.c) has a bunch of abstractions that the FreeBSD driver doesn't have, so that's a good starting point. The VLAN operations and VLAN port configuration stuff is all methods in the Linux driver, so that was a good starting point. I stubbed most of the VLAN stuff out (because I really didn't want it to get in the way) - this turned out to be more annoying than I wanted.

Next was the hardware setup path. There's more configurable stuff with the AR8327 - there's two physical ports that I can configure the PHY/MAC parameters on for either external or internal connectivity. I just took the code from Linux (which yes, I have permission to relicence under BSD, thanks to the driver authors!) and I made it use the defaults from OpenWRT for the DB120. The ports didn't properly come up.

I then realised that I was reading total garbage from the PHY register space, so I went looking at the datasheet and ar8216 driver for some inspiration. Sure enough, the AR8327 has the PHY MDIO bus registers in different locations. So after patching the arswitch PHY routines with this knowledge, the PHYs were probed and attached fine. Great. But it still didn't detect port status changes.

So, back to the ar8216 driver. It turns out that there were a few things that weren't methodized - and these were the bits that read the PHY status from the switch. Both drivers didn't just poll the PHYs directly - they read the switch registers which had a summary of the port status. So, I taught the driver about this and voila! Port status changes worked.

But, no traffic.

Well, there's a few reasons for this. It's a switch, so I don't have to setup anything terribly difficult. The trick here is to enable port learning and make sure they're all in the same VLAN group. Now, here's where I screwed up and I found a bug that needed working around.

The port setup code did enable learning and put things into a vlan group.

Firstly, I found this odd behaviour that I got traffic only when I switched the ethernet cable to another port. Then learning worked fine. I then found that the ar8216 driver actually triggers a forwarding table flush upon port status change, so I added that. This fixed that behaviour.

But then it was flooding traffic to all ports. This is kinda stupid. What did I screw up? I put each port in a separate vlangroup, rather than put them in the same vlangroup. Then, I programmed the "which ports can you see?" to include all the other ports. What this meant was:
  • The forwarding table (ie, what addresses were learnt) were linked to the vlangroup the port is in;
  • .. and when the switch did a lookup for a given MAC on another port, it wouldn't find it, as the address in the forwarding table showed it was for another vlangroup;
  • .. so it would do what switches do when faced with not knowing about the MAC (well, and how I had configured it) - it flooded traffic.
The solution was thankfully easy - I just had to change the vlangroup (well, "port vlan" here) to be '1', instead of the port id. Once this was done, all the ports came up perfectly and things worked great.

So, this now works great on the Atheros DB120 reference board. It's not working on other boards - there's likely some timing issues that need to be resolved. But we're making progress!

Finally, I spent a bunch of time porting over the port configuration and LED configuration stuff from OpenWRT so I didn't have the driver just hard-coded to the DB120 board. I'll update the configuration and code when I get my hands on other boards that use the AR8327 but for now this is all I have.


portmgr-lurkers@ March 1 edition

The first intake of portmgr-lurkers@ is complete, and it is now time to start with the second round of our -lurkers.  Please join us in welcoming Alexey (danfe@) Dokuchaev and Frédéric (culot@) Culot to our ranks.

During this -lurker round, culot@ will be the shadow portmgr-secretary@, learning the finer points of the roles and responsibilities of the job.

on behalf of portmgr@

We can patch it for you wholesale

…but remembering costs extra.

Every once in a while, I come across a patch someone sent me, or which I developed in response to a bug report I received, but it’s been weeks or months and I can’t for the life of me remember where it came from, or what it’s for.

Case in point—I’m typing this on a laptop I haven’t used in over two months, and one of the first things I found when I powered it on and opened Chrome was a tab with the following patch:

diff --git a/lib/libpam/modules/pam_login_access/pam_login_access.c b/lib/libpam/modules/pam_login_access/pam_login_access.c
index 945d5eb..b365aee 100644
--- a/lib/libpam/modules/pam_login_access/pam_login_access.c
+++ b/lib/libpam/modules/pam_login_access/pam_login_access.c
@@ -79,20 +79,23 @@ pam_sm_acct_mgmt(pam_handle_t *pamh, int flags __unused,

        gethostname(hostname, sizeof hostname);

-       if (rhost == NULL || *(const char *)rhost == '') {
+       if (tty != NULL && *(const char *)tty != '') {
                PAM_LOG("Checking login.access for user %s on tty %s",
                    (const char *)user, (const char *)tty);
                if (login_access(user, tty) != 0)
                        return (PAM_SUCCESS);
                PAM_VERBOSE_ERROR("%s is not allowed to log in on %s",
                    user, tty);
-       } else {
+       } else if (rhost != NULL && *(const char *)rhost != '') {
                PAM_LOG("Checking login.access for user %s from host %s",
                    (const char *)user, (const char *)rhost);
                if (login_access(user, rhost) != 0)
                        return (PAM_SUCCESS);
                PAM_VERBOSE_ERROR("%s is not allowed to log in from %s",
                    user, rhost);
+       } else {
+               PAM_VERBOSE_ERROR("neither host nor tty is set");
+               return (PAM_SUCCESS);

        return (PAM_AUTH_ERR);

The patch fixes a long-standing bug in pam_login_access(8) (the code assumes that either PAM_TTY or PAM_RHOST is defined, and crashes if they are both NULL), but I only have the vaguest recollection of the conversation that led up to it. If you’re the author, please contact me so I can give proper credit when I commit it.

Burning all the bridges. Cleaning up jails with ezjail-admin on #FreeBSD

I noted that my updates on my jail host didn’t actually do a delete-old/delete-old-libs during the basejail process:

ezjail-admin update -i

I tend to update my jails with my base host svn updates to -current, so there’s a bit of churn and burn with regards to old files and such. This came to a head today as my src.conf on the base host declares WITHOUT_NIS to conserve my limited space.

The python port checks for the existence of the yp binaries to determine whether or not to build NIS support. So, if the old binaries are lying around and support for NIS is removed from your system, python’s build will abort with something like the following:

Install them as needed.
====> Compressing man pages (compress-man)
===> Installing for python27-2.7.6_2
===> Checking if lang/python27 already installed
===> Registering installation for python27-2.7.6_2 as automatic
pkg-static: lstat(/var/ports/basejail/usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/ No such file or directory
*** Error code 74

I realized that even though my host system was fairly clean (I do port rebuilds after each upgrade and delete-old delete-old-libs following that), the basejail was still filled with obsoleted files.

A super dangerous and super effective way to clean that up is the following:
yes | make delete-old DESTDIR=/usr/jails/basejail
yes | make delete-old-libs DESTDIR=/usr/jails/basejail

Dangerous, because you have to realize that your deleting binaries and libraries that might still be in use if you haven’t recompiled your ports packages. Effective, because it will cleanup and purge a lot of things if you haven’t done it in a while.

This also led me to understand that the /etc/src.conf tuneables WITHOUT_* don’t *stop* the buildsystem from creating the binaries and libraries. It doesn’t seem to shorten your build time. It *will* allow you to purge them from your system at install time with the delete-old make targets.

On testing, part III

I just got word of an embarrassing bug in OpenPAM Nummularia. The is_upper() macro, which is supposed to evaluate to true if its argument is an upper-case letter in the ASCII character set, only evaluates to true for the letter A:

#define is_upper(ch)                            \
        (ch >= 'A' && ch <= 'A')

This macro is never used directly, but it is referenced by is_letter(), which is referenced by is_pfcs(), which is used to validate paths and path-like strings, i.e. service names and module names or paths. As a consequence, OpenPAM does not support services or modules which contain an upper-case letter other than A. I never noticed because a) none of the services or modules in use on the systems I use to develop and test OpenPAM have upper-case letters in their names and b) there are no unit or regression tests for the character classification macros, nor for any code path that uses them (except openpam_readword(), which uses is_lws() and is_ws()).

The obvious course of action is to add unit tests for the character classification macros (r760) and then fix the bug (r761). In this case, complete coverage is easy to achieve since there are only 256 possible inputs for each predicate.

I have merged the fix to FreeBSD head (r262529 and r262530). Impatient users can fix their system by running the following commands:

% cd /usr/src/contrib/openpam
% svn diff -r758:762 svn:// | patch
% cd /usr/src/lib/libpam/libpam
% make && make install

Unsurprisingly, writing more unit tests for OpenPAM is moving up on my TODO list. Please contact me if you have the time and inclination to help out.

httperf tuning for #FreeBSD testing

Was playing around with httperf to excercise Apache / stunnel SSl benchmarks on FreeBSD this week and ran into the code that nerfs simultaneous connections down from the environment ulimit of maxfiles to the limit FD_SETSIZE as defined in <select.h>.

One can override this at compile time and push the system harder by passing in some ./configure foo:

env CC=”cc -DFD_SETSIZE=4096″

However, you will then be able to max out the number of ports in use very quickly if you try to use stunnel and apache in this configuration.  I noted that on our systems we raise the low port number and reduce the high port number for connections:



I set first down to 2000 and last up to 65534 for my testing.  This gives me quite a bit more ports to use in testing.  At this point I can run stunnel on 443 forwarding to apache on localhost:80 and get more than 8k simultaneous connections when using SSL accelerators on FreeBSD 10


Time to bid farewell to the old pkg_ tools

There comes a time in the life cycle of just about every software package that it has bee re-evaluated, refreshed, deprecated or just retired.

It is time that we bid farewell to the old pkg_* software that has been part of FreeBSD since the beginning, and has served us well. After years of development, testing, and playing, pkg(8) has become a suitable replacement.

Pkg is the Next Generation package management tool for FreeBSD. It is the replacement for the current pkg_info/pkg_create/pkg_add tools that ports use to register local packages and which provide remote packages. Its main goals are to facilitate remote binary package upgrades. It also works with ports without remote binary packages.

Pkg, combined with the quarterly release package sets, enables easy installation and safe upgrades for binary packages. Signed, binary packages are available for all supported FreeBSD releases on the i386 and  amd64 platforms from Additionally, for those compiling ports from source, pkg’s new database format gives more fine-grained querying and management of installed software.

New features on the drawing board, like automatic pkg-plist generation, sub-packages, creating multiple packages containing different parts of a port from one build process, and flavours, being able to ask for e.g. a webserver, without directly specifying a specific one, cannot be implemented in the old pkg_* tools and those plans are currently on hold.

You are not obligated to switch to binary packages, if you still prefer to compile your own ports, it is a simple matter of installing ports-mgmt/pkg, run pkg2ng, add WITH_PKGNG=yes to your make.conf and use pkg <action> instead of pkg_<action>.

You can read more about pkgng on the FreeBSD wiki,

The decision has been made to allow the old pkg_* software to be EoL’d 6 months from now, at September 1, 2014 in all active FreeBSD branches.

Please start testing pkg(8) in your test environments before taking it live, you will find the benefits of full binary updates for your ports to be beneficial in a very short amount of time. Even if you prefer to compile from source, you will still reap the benefits of the modern packaging system.

The short list #5: coredumping with sudo on #FreeBSD

Things I learned from a misbehaving pam module managing our sudo context at work.  sudo, for security, will not dump core files if it hits a segfault.  You need to tell the kernel to allow set uid root binaries to core dump *and* you have to let sudo know that its ok via a sudo.conf entry.


kern.sugid_coredump: 1

/etc/sudo.conf –> Set disable_coredump true

ref –>


Ion-Mihai Tetcu steps down from duties on portmgr@

Ion-Mihai Tetcu, aka itetcu@, aka ionut, has stepped down from his duties on the FreeBSD Ports Management Team.

Ion-Mihai was the one who initially gave us Ports QAT, which gave almost immediate response to a problematic commit, then he enhanced with with other tests, such as installation of all ports to a non-standard DESTDIR. These combined efforts paved the way for cleaner commits to the ports tree.

On behalf of the Ports Management team, we would like to thank Ion-Mihai for his many years of service and dedication.
on behalf of portmgr@

portmgr-lurkers@ promoted to voting members of portmgr@

The FreeBSD Ports Management team is pleased to announce that Mathieu Arnold (mat@) and Antoine Brodin (antoine@) have been promoted to full voting members of the team after a successful launch of the portmgr-lurkers pilot project.

Each of them brings new skills and vast experience to the team. Please join me in welcoming them aboard.
on behalf of portmgr@

Joe Marcus Clarke steps down from duties on portmgr@

Joe Marcus Clarke, aka marcus@, has stepped down from his duties on the FreeBSD Ports Management Team.

Joe was our longest serving member of the team. Among his many accomplishments was being the repocopy source of authority, instrumental in championing tinderbox development and maintaining portlint.

On behalf of the Ports Management team, we would like to thank Joe for his many years of service and dedication.
on behalf of portmgr@

Hacking on Mindwave for fun and .. fun

Allison (and others, like a game developer named Lat) showed interest in these Neurosky Mindwave headsets. They're little wireless (bluetooth, almost!) headsets that ship with a cheap USB dongle and expose their data via a binary protocol.

The protocol is not consistently and well documented. It's out there, if you can craft the right search queries. For the USB widget, you need to implement the basic handshake commands to attempt to connect to a given (or any) headset. Then you also need to implement the data decoding for the raw and processed data.

Now, I don't want to go into the details - you can read the documentation and my very bad, hacked up code.

The USB dongle didn't work with FreeBSD-9.x. It's a cheap chipset (CH341) and it just wouldn't transmit. It works fine on FreeBSD-HEAD though.

So, to explore it, I wrote a simple, hackish library to encapsulate pairing, parsing, data gathering. It needs a lot of improvement but it's there. Then, I (re-)learnt enough SDL and OpenGL to plot some data points. Finally, I grabbed a FFT library to poke at the returned data to see if it makes sense.

A few points thus far.

I still haven't found any correlation with the attention / meditation parameters the firmware returns. For the most part, you just have to stop any kind of muscular movements.

The raw values clip very easily with any kind of muscular movement. I can see how to decode say, "blink" as a muscular action though.

I've only started looking at the raw FFT results. Hopefully with a bit of filtering I'll see things that actually look like basic EEG results, or I'll concede these things are expensive muscular reaction devices.

The code:

And the obligatory screenshot:

Blanket approval to modernize the Ports Tree

In years gone by, and I am thinking of FreeBSD 7.0 specifically, portmgr@
gave some latitude to *ALL* committers to “just fix things” to get a port
into shape. In the case of 7.0, it was making ports build for gcc4.

What we have laying ahead of us is a ports tree in various states of modern
preparedness (new style USES=, stagefication, etc) and the old way of doing
ports (boo!).

We would like committers, and contributors to generate a PR and/or “just
fix” the old ports to update them to the new way of doing things regardless
of maintainership. We are looking for fixes in the following areas

- Convert to LIB_DEPENDS
- stagify ports
- convert things like USE_GMAKE -> USES=gmake USE_DOS2unix -> USES=dos2unix

This can be done with implicit portmgr@ blanket approval, and without
maintainer approval.

Please, however, respect some boundaries, do not change ports belonging to
kde@, gnome@ or x11@. These teams work in private repos that may have
changes pending.

Also, cross reference GNATS, to see if a port has an open PR that you can
factor into the fix. It is important to stress here that we *DO NOT* want
to invalidate existing patches that a maintainer has offered up or already

If the change is very trivial AND has been tested, “just fix it”. One of
the strengths of the Ports Collection is it’s volunteer maintainers, if you
make a change, regardless of how trivial, just send a courtesy email to the

Getting to know your portmgr@ — Bryan Drewery

In this interview, we talk to the newest member of the team, Bryan Drewery.  Bryan first came to the attention of many by adopting portupgrade and friends, and then jumping into pkgng and poudriere.


Bryan Drewery

Committer name


TLD of origin


<Editor’s note: while Bryan claims to be a netizen, it is believed he is originally from .us :)>


Software Engineer

When did you join portmgr@

March 2013. Joined FreeBSD as a committer in August 2012. As a contributor
in March 2012.

Blog is my blog. I have not made much effort on it yet but
have a lot of ideas and content to add eventually.

Inspiration for using FreeBSD

It took a long time for me to discover FreeBSD. I wish I had 10 years
sooner. My first experience with a computer was with MS-DOS 5. Then I worked
up through Windows 3.1, 95, 98, XP. Somewhere around here I discovered Redhat
5 and shortly after Debian and Gentoo. In high school I took an AP CS class
that used FreeBSD 4, which was my first introduction to it. My first
introduction to ports was in the same class where not having root and wanting
to install an application I went into /usr/ports/irc/BitchX and tried to ‘make
install’ and failed of course. I still wish this worked. Shortly after that I
started doing work for a Shell Hosting company that used FreeBSD 4.10. At this
time I was still much more fond of Linux though. When I met my Wife, she was
also doing Shell Hosting with FreeBSD. That’s when I started doing actual
development on it and customizing the system. I found that with FreeBSD I
could customize the system far more than I could with any Linux distribution.
This is what sold me the most and led me away from Linux. Though I do still
use Linux for Xen dom0 and some development.

Who was your first contact in FreeBSD

Probably garga@ qmail patches I sent in years ago. Once I discovered pkgng
though it was bapt@. That’s what led me to becoming a committer. I actually
knew zi@ outside of FreeBSD too from when I was an EFNet oper.

Who was your mentor(s)

Baptiste Daroussin (bapt@) and Eitan Adler (eitan@)

What was your most embarrassing moment in FreeBSD

Wiping systems of course. One of which was someone else’s system who ran my
bad code!

vi(m) / emacs / other

I was a longtime (6 years) pico/nano poweruser (haha) until I discovered vim
in my first real job.

What keeps you motivated in FreeBSD

I just like to write code. It’s hard for me to let go of things I put a lot
of effort into! I got into this with the intent to help get packaging working
for my own servers and to takeover ports that were abandoned that I felt were
critical for my servers.

Favorite musician/band

There’s so many. Lately I’ve been listening to metalcore and post-hardcore
bands. has all of the music I listen to.

What book do you have on your bedside table

Design and Implementation of FreeBSD, Kindle, C++ Standard Template Library

coffee / tea / other

I used to drink insane amounts of diet Mt. Dew. Not anymore though, no
caffeine for me.

Do you have a guilty pleasure

I get into games every now and then. Xbox 360, PS3, PC, Wii. I love Windows
7. I’ve been a Mac user for a few months now as well.

sendmail / postfix / other

qmail. My mail server is something that I setup years ago, hacked at quite a
bit with custom patches and never want to redo again.

What is your favorite TV show

Breaking Bad, Lost, The Wire, Dr. Who, Sons of Anarchy, Dexter.

What sports team do you support

I’m not really into rooting on sports teams. I enjoy playing much more than

What else do you do in the world of FreeBSD

I maintain upstream for portupgrade and poudriere, help with pkgng, qmail,
openssh-portable. I also am starting to work more in the src world. I maintain
portmaster but have not put much real effort into it. For portmgr I help test
Mk/ patches, I do exp-runs, help manage the package building systems, and fix
Mk/ bugs as I run across them.

What can you tell us about yourself that most people don’t know

I was involved with the eggdrop IRC bot project and have been maintaining a
pretty popular fork of my own for the past 10 years.

Any parting words you want to share

Getting involved with Open Source is really easy. We’re all volunteers like
you. Just start helping. Send patches, bug reports, code, documentation,
translations, typo fixes. Everything helps.

What is your .sig at the moment

Bryan Drewery

You need to construct additional pylons. Building wine for #FreeBSD

I’ve been playing Blizzards’s Starcraft 2 on Linux via wine emulation lately and thought I’d see if I can get the same thing working on FreeBSD via the emulators/i386-wine-devel port.  After talking with the fine folks in #bsdports on EFNet, I finally found a recipe that is poudriere friendly and seems to spit out something that sort of works.

David Naylor ([email protected]) has a working method for constructing wine on FreeBSD and this should work in most cases for using current.  The method is really designed for building a binary package for releases, most folks wouldn’t want to go down this route.

In order to begin, get poudriere configured and ready to go.  You’ll need to construct an i386 jail for the first part of this process.  Something like I show in my poudiere blog post

poudriere jail -c -j 11i386 -v head -a i386 -m svn

This will give you a build environment to get the 32bit binaries for wine built and packaged up for step 2.

poudriere builk -j 11i386 emulators/i386-wine-devel

If all goes well, you now have an i386 package of wine that will be consumed as a distfile for the amd64 package build.  I redefine PORTSDIR=/usr/local/poudriere/ports/default in /etc/make.conf.

If you are like me and use poudriere for everything, copy it to /usr/local/poudriere/ports/defaults/distfiles/freebsd:11:x86:64/

Now you’ll need to edit the emulators/i386-wine-devel distfile with the appropriate information generated from a sha256 and ls -l of your packagefile in your local i386 repo:

sha256 i386-wine-devel-1.7.7,1.txz

SHA256 (i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify this with the following to make it compatible with your system:

SHA256 (freebsd:11:x86:64/i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify the to exclude checks for the OS version:

—    (revision 335346)
+++    (working copy)
@@ -41,10 +41,10 @@

.include <>

-.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
-IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only
+#.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
+#IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only

RUN_DEPENDS+=   ${DATADIR}/gecko/wine_gecko-2.24-x86.msi:${PORTSDIR}/emulators/wine-gecko-devel

And now, you can try building the package in your *AMD64* poudriere build with:

poudriere bulk -j 11amd64 emulators/i386-wine-devel

If my instructions have succeeded, you now have a package suitable for installation on your amd64 machine that will now let you do wine things.

Now, I need to figure out what the Blizzard Network Installer is trying to do as it runs, self-updates and hangs.

Experimenting with zero-copy network IO in FreeBSD-HEAD

Back when I started all of this networking hacking, the "big thing" was the overhead of doing poll() and select(). Various operating systems came up with ways of eliminating these - FreeBSD grew the kqueue infrastructure; linux received epoll, Solaris received an epoll-like device and then ended up with some form of kqueue-like event mechanism. Windows has completion ports/overlapped IO which combined the event mechanism with a zero-copy way of doing network IO.

So the Free/Open operating systems have scalable event notification mechanisms for handling large numbers of concurrent sockets but they don't all have some nice, efficient way of doing zero-copy network IO.

Linux has splice()/tee()/vmsplice(). So yes, it effectively does have a way of doing zero-copy socket reading and writing.

OpenBSD does have a splice style syscall to copy data from a source to a destination TCP socket.

FreeBSD, however, has mostly focused on the "disk to network" path for content serving and thus has a lot of time invested in their sendfile() implementation. This is great if you're doing a lot of file to network sending (which Netflix does), but it has some serious shortcomings. The main one I'll address here is the lack of being able to do general zero-copy socket writes from userland. So it can only send data from disk files to the network. You can't implement a zero-copy intermediary proxy server, nor a memory cache that keeps things in pre-allocated memory regions. You have to use disk files (whether that be a real filesystem on disks, or a memory filesystem) and leverage VM hints to control caching.

Recently there was some new sendfile() work to allow sending from POSIX shared memory segments. This intrigued me - it's not the most effective way of doing zero-copy network IO from userland but it's a start. So I set off to write an updated version of my network library from yesteryear to implement some massively parallel network applications with.

The idea is simple - you allocate a POSIX shared memory segment. You then mmap() that region into memory and treat it as a place to allocate write-side network buffers from. Then you use the shared memory filedescriptor and offset to schedule a sendfile() from the shared memory segment to the destination network socket. It's not as elegant as having a write path that wires the memory down and just populates mbufs from that, but that'll come later.

Here's what I found.

Firstly, there's no asynchronous "I'm done!" notification for the sendfile path. So you have no explicit notification that the underlying memory has been freed so you can reuse it. sendfile() has the SF_SYNC flag which causes it to sleep until the transaction is done - primarily so users can be sure they can change the underlying file contents after the syscall completes. This is used by caches such as Varnish that leverage on-disk files as their cache filesystem space.

So I've been adding that. I have a working prototype that is scaling quite well under load and I'll look to commit it to FreeBSD-HEAD soon. It posts a knote to a kqueue filedescrpitor once a transaction has completed.

Once that was done, I started benchmarking the performance of this setup.

The first real roadblock I hit was massive VM contention on the shared memory segment. It turns out that a single POSIX shared memory segment is represented as a single vm_object and this is protected by a single lock. So when 8 threads are actively doing IO from the same shared memory segment it hits massive lock contention. I fixed this in my test suite by allocating one shared memory segment per thread. It's not elegant but it works well enough for benchmarking.

I next hit issues with contention on the VM page lists. Besides the per-object list, there's also a global per-type list (active, inactive, etc.) There's one lock protecting each of these lists. What I found was the VM was shuffling pages between active/inactive and at the traffic rates I was doing (20+gbit/sec) it was a few hundred thousand pages a second being shuffled around. The solution? mlock() the whole region into memory. This prevented the VM from having the pages change state so often and eliminated that overhead.

The code for doing this sendfile() work with posix shared memory is in my libiapp code - . It's terrible and hacky - I'm just experimenting with things for now. But with some tuning, I can get a good 35Gbit/sec out of 70,000 active TCP sockets. There's still a long way to go - I shouldn't be saturating an 8-core CPU with this traffic level when I'm doing no socket data copies. I'll write another update or two about that soon.

Now, what would I like to see? I did some experiments with physical disk IO using the FreeBSD AIO paths doing the same kinds of IO patterns as I am doing with network socket IO (4KiB to 64KiB random disk reads.) It turns out if you do everything correctly, the FreeBSD AIO code will turn physical disk IO into asynchronous disk buffer transactions by wiring the userland buffer into memory and then using that as the backing buffer memory. The overhead of doing the pmap work for this was not too high. So, I wonder if it's worth writing a new transmit path that uses the pmap code (and not the VM!) to wire in a region of memory and then use that for transmit buffers. Combined with an iovec style array of buffers and the above kqueue notification of the network IO completion, I think we can end up with a much more flexible method of doing network IO from userland without the shortcomings by using POSIX shared memory with sendfile().