There’s a NFS file server in my office, from which I mount stuff into my home directory on my workstation. There’s a small FreeBSD server in my living room which, among other things, serves Samba to my Windows desktop machine. Using sshfs on the home server, I mounted my home directory from my workplace as a subdirectory of my local user, and I’m accessing it from my Windows desktop over Samba.

Before the bytes hit the drives on the server, here’s the path they must take:

[Home desktop, Windows] -- Samba -- [Home server, FreeBSD] -- sshfs -- [Work desktop, Linux] -- NFS -- [Work server, FreeBSD]


And it works. Really. I’m editing OpenOffice files on my Work server right now.

Of course it should work – all of these individual components in the chain are tested and known to work so there’s practically no real concern, but seeing all this in operation made me think how many standards, interoperability specs and engineering went into making this possible, especially since the actual connections between the components are very varying: ADSL, Ethernet of various speeds and I’m sure there’s still ATM somewhere in the telco’s infrastructure. The number of different operating systems the bytes go through (if “embedded” ones on routers and similar equipment is also counted) is probably huge.

We live in great times.

(Of course, I won’t try anything that depends on file locking )

The only problem is that sshfs basically hangs the system when the IP changes on the ADSL side (file system lookups hang).

So I created a daemon that uses kqueue(2) to monitor which files changed and feed only those files to rsync (it’s not exactly a new idea, I’m sure somebody has also mentioned it somewhere on the FreeBSD lists). This is in many ways a suboptimal solution since it needs to keep an open file descriptor for all the monitored files (which ties in kernel resources and memory) so it won’t scale for really large directories, which could actually benefit the most from this approach. It will work reasonably well for a small number of files (up to several tens of thousands), with modifying kern.maxfiles and kern.maxfilesperproc sysctls and login session limits (if applicable).

Anyone who’s interested can download the adfs daemon and try it. This was hacked together over the weekend so it probably has some problems. I’ll fix those problems that prevent me from using it, but I’ll update the online archive only if there are interested users.

In this (again, emphasis on “small”) project, there were a couple opportunities where I could make use of a fast data-access structure like a hash table (since I need to store and retrieve a lot of data entries) or dynamic memory allocation (since I don’t want to artificially limit the number of these entries) but I just didn’t feel like writing all that code to implement a hash table in C (or use a heavy external library) and deal with memory reallocation and track all those pointers. Yes, I’m lazy. In a more abstract language I could just instantiate a dictionary and say d[i] = something and this would actually be very efficient and take care of memory allocation automagically. Since I limited myself to basic C, I chose simpler algorithms like linked lists and evil static arrays on stack. Ironically, these structures would be comparatively significantly more inefficient in Python.

Of course, at its roots this can be stripped down to be simply a choice between using pre-packaged routines instead of writing your own (aka the NIH problem), but in this case it would actually make my simple program faster and more efficient – despite the overhead of an interpreted, dynamic language.

There are many more similar cases – programmers write bubblesorts in C because they are easy to implement, while going to a higher level of abstraction they could just write mylist.sort() and would get QuickSort or some other efficient algorithm for free, etc. etc.

Does anyone know of a library / collection of algorithms for C similar to glib only BSD-licensed? (Yes, I know about C++ algorithms, I don’t want to use C++).

FreeBSD’s source CVS is one of the oldest and biggest in existence; it’s approximately 12 years old and has apparently had something like 180,000 commits over the years, or on average slightly more than 41 commits daily. A checkout of RELENG_7 branch holds more than 42,000 files (in 482 MB as du sees it).

This move was discussed extensively during the DevSummit at BSDCan 2008; there have been many issues with CVS over the years, most of which are minor enough to be overlooked, but some of which are just nasty (the inability of CVS to move/rename files, bad handling of branching in the event of constant new development and additions to the directory tree, non-atomic commits) and have frequently required manual interventions in the CVS repository (“repocopy” is one of the relatively frequently requested operations to the CVS admins).

Old infrastructure, of which cvsup / csup is probably the most important part, will continue to work as code will continually be mirrored from SVN to CVS, until suitable replacements or upgrades to the above tools are created. This is also the reason the name “CVS” will be present for some time in the system infrastructure, until all of it is updated. Ports will continue to use CVS for the foreseeable future.

To make a source base this large work efficiently on SVN, version 1.5 had to be used, since it creates its database files in a hash-tree of directories on the file system instead of one huge directory with all the files in it.

Also see the official announcement of Subversion and Peter Wemm’s notes about Subversion (very useful to developers).

In accordance with the specification expressed here:

... I’ve created a device driver that implements the functionality in kernel. In honour of the operating system by which this work was inspired, I name the driver random.debian. The kernel module creates a device entry (/dev/random.debian) which is a infinite source of random data with entropy compatible with the above specification. The source code tarball for the Debian-like random data source is of course available under the BSD license. This will work on any recent version of FreeBSD.

As much as I would like to have though of this first, the idea was actually put out by PHK or Robert Watson while we were waiting for dinner, so that part of the credit goes to them.

This DevSummit+BSDCan was very fun and educational and will definitely try to be here in the next years also.

All together, it was a very nice spent day with many opportunities to learn new things and speak to like-minded people working on interesting things.

This enables headless installs of FreeBSD in the following fashion:

1. Insert the CD with the finstall in the server (obviously, the server needs to have a CD/DVD reader; USB ones are mostly fine). The server can be headless, i.e. without a monitor, a keyboard or a mouse.


2. Connect the server to a LAN. No remote-network hops (routing) are allowed since UDP broadcasts are used to locate the headless nodes. Boot off the finstall CD.


3. On another machine (one that has monitor+keyboard+mouse or equivalent X11 devices) start the installer front-end.


4. In the front-end, opt to connect to a remote finstall node, choose the one you want. At this point you can see boot-time dmesg data from the nodes so you can locate the right one in case there are many of them.


5. Proceed to use the front-end GUI just like it was a local install.


6. Reboot, configure, use the server, etc.

As described, the primary usage for this is to setup headless servers.

PXE is supported in theory, but not tried. The idea is that, since the whole finstall setup is actually a live FreeBSD system, PXE can be configured manually once the CD is booted somewhere (possibly on a virtual machine), and remote systems can be booted from this CD-based file system, then installed as if they are booted locally. This is experimental and untried.

This mode of installation has many side-effect uses, such as scripting the remote install, etc.

More about this and other features of finstall will be presented on BSDCan 2008.

Update: The original ISO image posted above had a trivial but unfortunate bug. Download the new ISO image with MD5 fingerprint a9eebbdc546565a9eb9c6622bb948d75.

Marko Zec helped me with that, temporarily providing me a machine which dual boots 7.x and 4.x with his VIMAGE patches, as well as without the patches. Originally I just used the 7.0 system, and achieved something like 62,000 trans/s, which is too low for me. On his insistence I booted 4.11 and ported the client-side benchmark on it. Without any significant modifications except those needed for the difference between gcc 2.9x and gcc 4, the same client code rocketed to 81,000 trans/s! This is using libc_r, meaning the whole 40-threaded thing is visible to the kernel as one process (4.x doesn’t have kernel support for multithreading)! This number is still too low and I’ll probably need to find several machines that could work at the same time to overtax the server (which will be very hard) but just the raw difference between 4.x and 7.x is staggering. Network card is bge, gigabit, directly connected to the server via crossover cable.

On the bright side, VIMAGE patches don’t influence the performance noticeably.