The new blog system is very new and will probably take some time before all the kinks are fixed but it will do for now. I don’t intend to write here anymore, but I also have no plans of moving old posts to the new system, since blog posts are ephemeral anyway. So update your bookmarks, feed readers, etc. to the new address.

UnixBench results of running FreeBSD 7.0-RELEASE in VirtualBox (i386) on Windows XP host are:


INDEX VALUES
TEST BASELINE RESULT INDEX

Dhrystone 2 using register variables 116700.0 6484818.8 555.7
Double-Precision Whetstone 55.0 1644.1 298.9
Execl Throughput 43.0 271.3 63.1
File Copy 1024 bufsize 2000 maxblocks 3960.0 101938.0 257.4
File Copy 256 bufsize 500 maxblocks 1655.0 48434.0 292.7
File Copy 4096 bufsize 8000 maxblocks 5800.0 150300.0 259.1
Pipe Throughput 12440.0 908613.9 730.4
Pipe-based Context Switching 4000.0 15755.6 39.4
Process Creation 126.0 670.0 53.2
Shell Scripts (8 concurrent) 6.0 79.5 132.5
System Call Overhead 15000.0 714570.9 476.4
=========
FINAL SCORE 199.3


This is too good to be true, compared to what I got on the same hardware in VMWare previously (in short: a score of 70). The results above (and all other unixbench results) should really be analyzed line by line instead of just looking at the final score. For example, notice that the “shell scripts” score is 133 on VirtualBox and 1.7 in VMWare. Context switching has always been VMWare’s weak point and it looks like it doesn’t really have to be that way.

Now I only have to figure out if it can be used “headless” (without the GUI console). VMWare’s network support is much better because it can to “bridged” networking, putting the VM directly on the network, but setting up routing or bridging for VirtualBox is simple enough (like qemu).

Update: buildkernel finishes without problems but it takes 40 minutes – I think this is way too slow but will investigate tomorrow.

I tried VMWare Server 2.0 RC1 again. The first time was sometime early in beta so I just shrugged at the problems and returned to 1.0 to wait until they are solved. This time, it’s a release candidate. And it just sucks. It’s full of WTF moments:

* The install package downloaded from their site is > 500 MB. Really. WTF? A virtualization program? VMWare Server 1.0 had 150 MB and this was also completely over the top considering that the actual VM software (vmware-vmx.exe and companion libraries) is less than 7 MB uncompressed. When installed, VMWare Server 2.0 wants over 1.5 GB of disk space. Drives are cheap but this is ridiculous.
* Completely inappropriate use of technology 1: The VMWare Console (which was one of the best parts in 1.0 – small and reliable) is now replaced with a web interface, written in Java. Now we know how they filled 1.5 GB. The web interface is written as a servlet running in Tomcat 6. This means that, without doing anything, the host machine has a > 95 MB tomcat process in memory, waiting for you to connect. Again, ridiculous. Much more appropriate usage could be gained with practically any other technology ever. Java is particularly bad because of its sandbox – can’t do any system calls so a separate native layer must exist between it and the VM software proper. They could have written it in anything: Perl, Python, PHP, and got the same portability with a small fraction (2-3 MB) of used resources.
* Completely inappropriate use of technology 2: The web interface is AJAX-y. When you try to view the information about a VM you created, you get a mostly-blank form with cute little animated GIF images next to the word “Refreshing” in place of the actual data. At this time JavaScript code is asking the server (judging from the network traffic, once for each form field!) the data to fill in the fields. WTF? I can’t believe that this couldn’t have been done by parsing the .vmx file (typically consisting of 30 key-value lines!) on page load, instead of dynamically after it’s loaded. That’s completely what AJAX isn’t for and stinks like a design by committee.
* The web interface is on HTTPS, which is good, but the SSL server asks the browser for client certificates. As I actually have client certificates in my browser, created for some other purpose, and have set the browser to ask me each time someone asks it for the certificate, this means I get constantly nagged by popup boxes. There is NO WAY to generate a client certificate that would be applicable to the VMWare Server 2.0 from the web interface or the tools installed in the Start menu.

And then there are the bugs (you can skip this section, maybe they will be fixed by the time you read it):

* The web inteface has somehow picked up some of my old “registered” virtual machines (actually it picked the machines at random, whether they are registered or not), but made a complete mess with their names – I had 10 machines, with only 4 unique names. And, I couldn’t de-register all of them. The AJAX stuff broke every time I tried to de-register the last of the duplicates.
* The Console (implemented, naturally, as a native executable so all that effort in cross-platformness of the UI goes out the window; actually, the console looks somewhat like VMWare Player so again running it on FreeBSD will be a problem) works in Firefox 3.0 but not in 3.0.1
* The Console cannot use the hosts file (/etc/hosts and its equivalent in Windows) to resolve hosts. So no matter that i have accessed the AJAX-y web interface as http://myhost/, the console complains myhost can’t be resolved. Nice, really nice. Fortunately, we can pretend we’re in the early 80ies and use my IP address to connect both with the web interface and the console.
* And at last, when I got the new Console to work through the IP address (as suggested on VMWare forums), it couldn’t find the VM it was supposed to monitor – a message like “there’s no MyVMName.vmx running on the server” pops up.

The end effect is that, after two days of trying to get it to work, I couldn’t even boot a VM and attach the console to it. And I really tried. So far, it’s an awful product (actually the VM product itself could be really nice, but I didn’t even get the chance to try it because of the horrible user interface) and I reverted to using 1.0.

There’s a NFS file server in my office, from which I mount stuff into my home directory on my workstation. There’s a small FreeBSD server in my living room which, among other things, serves Samba to my Windows desktop machine. Using sshfs on the home server, I mounted my home directory from my workplace as a subdirectory of my local user, and I’m accessing it from my Windows desktop over Samba.

Before the bytes hit the drives on the server, here’s the path they must take:

[Home desktop, Windows] -- Samba -- [Home server, FreeBSD] -- sshfs -- [Work desktop, Linux] -- NFS -- [Work server, FreeBSD]


And it works. Really. I’m editing OpenOffice files on my Work server right now.

Of course it should work – all of these individual components in the chain are tested and known to work so there’s practically no real concern, but seeing all this in operation made me think how many standards, interoperability specs and engineering went into making this possible, especially since the actual connections between the components are very varying: ADSL, Ethernet of various speeds and I’m sure there’s still ATM somewhere in the telco’s infrastructure. The number of different operating systems the bytes go through (if “embedded” ones on routers and similar equipment is also counted) is probably huge.

We live in great times.

(Of course, I won’t try anything that depends on file locking )

The only problem is that sshfs basically hangs the system when the IP changes on the ADSL side (file system lookups hang).

So I created a daemon that uses kqueue(2) to monitor which files changed and feed only those files to rsync (it’s not exactly a new idea, I’m sure somebody has also mentioned it somewhere on the FreeBSD lists). This is in many ways a suboptimal solution since it needs to keep an open file descriptor for all the monitored files (which ties in kernel resources and memory) so it won’t scale for really large directories, which could actually benefit the most from this approach. It will work reasonably well for a small number of files (up to several tens of thousands), with modifying kern.maxfiles and kern.maxfilesperproc sysctls and login session limits (if applicable).

Anyone who’s interested can download the adfs daemon and try it. This was hacked together over the weekend so it probably has some problems. I’ll fix those problems that prevent me from using it, but I’ll update the online archive only if there are interested users.

In this (again, emphasis on “small”) project, there were a couple opportunities where I could make use of a fast data-access structure like a hash table (since I need to store and retrieve a lot of data entries) or dynamic memory allocation (since I don’t want to artificially limit the number of these entries) but I just didn’t feel like writing all that code to implement a hash table in C (or use a heavy external library) and deal with memory reallocation and track all those pointers. Yes, I’m lazy. In a more abstract language I could just instantiate a dictionary and say d[i] = something and this would actually be very efficient and take care of memory allocation automagically. Since I limited myself to basic C, I chose simpler algorithms like linked lists and evil static arrays on stack. Ironically, these structures would be comparatively significantly more inefficient in Python.

Of course, at its roots this can be stripped down to be simply a choice between using pre-packaged routines instead of writing your own (aka the NIH problem), but in this case it would actually make my simple program faster and more efficient – despite the overhead of an interpreted, dynamic language.

There are many more similar cases – programmers write bubblesorts in C because they are easy to implement, while going to a higher level of abstraction they could just write mylist.sort() and would get QuickSort or some other efficient algorithm for free, etc. etc.

Does anyone know of a library / collection of algorithms for C similar to glib only BSD-licensed? (Yes, I know about C++ algorithms, I don’t want to use C++).

FreeBSD’s source CVS is one of the oldest and biggest in existence; it’s approximately 12 years old and has apparently had something like 180,000 commits over the years, or on average slightly more than 41 commits daily. A checkout of RELENG_7 branch holds more than 42,000 files (in 482 MB as du sees it).

This move was discussed extensively during the DevSummit at BSDCan 2008; there have been many issues with CVS over the years, most of which are minor enough to be overlooked, but some of which are just nasty (the inability of CVS to move/rename files, bad handling of branching in the event of constant new development and additions to the directory tree, non-atomic commits) and have frequently required manual interventions in the CVS repository (“repocopy” is one of the relatively frequently requested operations to the CVS admins).

Old infrastructure, of which cvsup / csup is probably the most important part, will continue to work as code will continually be mirrored from SVN to CVS, until suitable replacements or upgrades to the above tools are created. This is also the reason the name “CVS” will be present for some time in the system infrastructure, until all of it is updated. Ports will continue to use CVS for the foreseeable future.

To make a source base this large work efficiently on SVN, version 1.5 had to be used, since it creates its database files in a hash-tree of directories on the file system instead of one huge directory with all the files in it.

Also see the official announcement of Subversion and Peter Wemm’s notes about Subversion (very useful to developers).

In accordance with the specification expressed here:

… I’ve created a device driver that implements the functionality in kernel. In honour of the operating system by which this work was inspired, I name the driver random.debian. The kernel module creates a device entry (/dev/random.debian) which is a infinite source of random data with entropy compatible with the above specification. The source code tarball for the Debian-like random data source is of course available under the BSD license. This will work on any recent version of FreeBSD.

As much as I would like to have though of this first, the idea was actually put out by PHK or Robert Watson while we were waiting for dinner, so that part of the credit goes to them.

This DevSummit+BSDCan was very fun and educational and will definitely try to be here in the next years also.

All together, it was a very nice spent day with many opportunities to learn new things and speak to like-minded people working on interesting things.