Author Archives: Dag-Erling Smørgrav

On testing, part II

About a year ago, I blogged about writing unit tests for OpenPAM to weed out bugs in the configuration file parser, and mentioned a specific bug which my unit tests caught, and which I wouldn’t have noticed without them.

Yesterday, this came back to bite me.

The patch in question fixed the following edge case:

hello "" world

This is supposed to be parsed as three words (“hello

Not too proud to beg

I’ve added two items to the FreeBSD donations wantlist.

The first, and most important, is hosting for a new source tinderbox  / continuous integration cluster. The source tinderbox currently run on three servers graciously donated, maintained and hosted by Sentex which can handle approximately two full builds per day. The University of Oslo recently replaced its old HPC cluster, and I’ve secured 28 1U nodes (including rack-mount kits but no power cords) from the old one which I hope to be able to use for a new source tinderbox cluster with a completely new, more efficient implementation, but I have nowhere to host them. As noted in the wantlist entry, once installed, the cluster will not require large amounts of external bandwidth—just enough to keep an svn mirror regularly updated and to serve the web frontend—but it will need a dedicated gigabit switch or partition for internal communication. The hosting center needs to be within driving distance of Oslo, Norway.

The second item is a request for parts to upgrade my main FreeBSD development box: an Intel H77-based motherboard, an Intel Core i7-3770S CPU and 32 GB of PC3-12800 RAM. I already have a case, a sufficiently powerful PSU and plenty of disks. I’d also like to replace my old Chieftec hot-swap frames with a pair of trayless four-disk frames such as this one or preferably this one which has one large fan instead of two small ones (and is therefore presumably quieter) but is not available in Norway.

You can also use the button in the sidebar to make a donation which will go towards hardware, software and hosting costs related to my FreeBSD and Open Source work.

Transition

If you can see this, I have successfully moved my blog from Blogger to a self-hosted WordPress instance.

This was not a trivial endeavor. I have had this blog since 2006, and there are links to it all over the net. I did not want to break those links, nor did I want them to point to an abandoned Blogger instance. In particular, I wanted readers to still be able to comment on old posts.

I had already set up Blogger to redirect maycontaintracesofbolts.blogspot.com to blog.des.no, so if I changed blog.des.no to point to my WordPress instance and made sure the slugs matched, most of the links should still work.

The first step was to configure WordPress to use the same style of permalinks as Blogger, i.e. year/month/slug.html. There was a catch, though: Blogger views posts as individual pages, whereas WordPress views them as directories, and is more comfortable with year/month/slug/ (note the final slash). Therefore, instead of trying to get WordPress to mimic Blogger more closely, I used the following mod_rewrite hack to convert Blogger-style URLs in incoming requests to WordPress-style URLs:

RewriteEngine On
RewriteRule ^/([0-9]{4}/[0-9]{2}/.*)\.html$ /$1/ [R,L]

Next, I used the Blogger Importer plugin to import posts and comments from Blogger. Several problems immediately arose.

First, Blogger Importer imported all my drafts as published posts. Luckily, this is a known bug with a simple fix.

Secondly, WordPress slugs and Blogger slugs are often but not always identical. They are both based on the title, but Blogger removes short words and cuts off at a certain length. Unfortunately, while Blogger Importer records the original Blogger slug as metadata for each post, it does not actually set the WordPress slug (post_name for those familiar with WordPress internals) when it imports posts. I therefore had to write a Perl script that scans the posts and metadata looking for Blogger slugs, then sets the WordPress slug to the correct value if it does not already match the Blogger slug.

Finally, after I had imported posts and comments but before I was ready to throw the switch, new comments were posted on Blogger. In theory, Blogger Importer can import new posts and comments without touching existing ones. However, I had already gone through existing posts and added breaks, fixed image placement, and fixed a few conversion errors (mostly related to the use of angle brackets in posts), and Blogger Importer was extremely confused. In the end, I realized that what was happening was that it did not recognize posts and comments it had already imported, because the duplicate checks were too narrow. I managed to work around this by modifying import_posts() and import_comments() in blogger-importer.php so the post_exists() and comment_exists() relied solely on the timestamp when comparing posts and comments. I also had to increase the MAX_EXECUTION_TIME parameter from 20 to 60 seconds.

I had a lot of trouble finding a good WordPress theme for the site, and I’m still not really satisfied. I may end up switching to Delicate, which is just about as basic as it gets (short of Toolbox) yet manages not to stray on the wrong side of drab.

I hope you like the new site, or at least that you don’t hate it. Feel free to comment, and I’ll be happy to answer any technical questions regarding the transition and my setup. I’ll discuss my server configuration in more detail in a later post—if I get around to it…


Edit 2013-02-14: I added a couple of extra RewriteRules:

RewriteRule ^/search/label/(en|fr|no)/?$ /category/$1 [R,L]
RewriteRule ^/search/label/([^/]+)/?$ /tag/$1 [R,L]

These rewrite Blogger tag URLs to WordPress tag URLs, except for the en, fr and no, which are categories (Blogger Importer imports Blogger tags as WordPress categories; I used the Categories to Tags Converter to convert most of them back to tags).

I use Piwik analytics and have enabled 404 tracking, which should help me identify additional URLs that need to be rewritten.


Edit 2013-02-16: Additional RewriteRules, courtesy of SEO Ultimate‘s 404 Monitor:

RewriteCond %{QUERY_STRING} ^m=1$
RewriteRule ^(.*)$ $1? [R,L]
RewriteRule ^/([0-9]{4})_([0-9]{2})_([0-9]+)_archive.html$ /$1/$2/page/$3 [R,L]
RewriteRule ^/feeds/posts/default/-/(en|fr|no)/?$ /category/$1/feed/ [R,L]
RewriteRule ^/feeds/posts/default/-/([^/]+)/?$ /tag/$1/feed/ [R,L]
RewriteRule ^/feeds/posts/default/?$ /feed/ [R,L]

The first two lines strip a query string that shows up a lot—I’m not sure what it means to Blogger. The third line rewrites archive page URLs. The final three rewrite RSS feed URLs, including feeds for categories and tags.

LinkedIn endorsements

I keep getting email from LinkedIn telling me that someone in my network has endorsed me for this, that or the other skill. My initial reaction when these emails started arriving was surprise, because I knew about LinkedIn’s recommendation system but hadn’t heard about their new endorsement system. After a while, surprise turned to amusement. Allow me to explain why.

Unlike recommendations, where one LinkedIn user actively solicits a recommendation from another, endorsements are purely one-way. Any LinkedIn user can endorse any of their connections for any skill. There is no verification step, no filtering, except that each user can select whether or not to show endorsements on their profile page—but not, to my knowledge which endorsements to show.

Here’s how endorsements work: whenever you view one of your connections’ profile, LinkedIn lists a handful of skills and asks you whether that person has them. You can also type in any skill which isn’t among those LinkedIn preselected. There is a big yellow button that says “Endorse

Focus follows mouse in Gnome 3

There are plenty of Google hits on that topic, but so far I haven’t found one that gives the right answer. Here it is:

% dconf write /org/gnome/desktop/wm/preferences/focus-mode \
    "string 'mouse'"

To switch back:

% dconf write /org/gnome/desktop/wm/preferences/focus-mode \
    "string 'click'"

This works in Gnome 3.4, and I assume (but haven’t verified) that it also works in Gnome 3.6. However, given the recent developments in Gnome land, I will probably move away from Gnome altogether.

NTP and asymmetric routes

I run one of the servers in the global NTP pool. I noticed something interesting in the data from the pool monitoring system, which determines which enlisted servers to include in the pool:

The blue line is the server’s score; the green dots show the offset in milliseconds between the time reported by my server and the monitoring system’s own clock. NTP uses a fairly simple algorithm to compensate for network latency; basically, it computes the round-trip delay and divides by two. A non-zero offset can mean one of three things:

  1. The client is synchronizing with this server, but hasn’t quite adjusted yet.
  2. The client is synchronizing with a different source and the two servers don’t agree on what the time is.
  3. The client is synchronizing with a different source and the two servers agree on what the time is, but the path between the client and one of the servers is asymmetric.

As far as I know, the monitoring system doesn’t synchronize with any of the servers it monitors, so either the second or third case applies. However, the graph clearly shows that the offset measured by the monitoring system alternates between two values (approximately -4 ms and -8 ms), with allowance for jitter. The only possible explanations for this are:

  1. The monitoring system keeps switching between two time sources that are 4 ms apart and synchronizes very quickly every time it switches.
  2. My server’s clock keeps switching between two time sources that are 4 ms apart and synchronizes very quickly every time it switches.
  3. The path taken by requests from the monitoring system to my server and responses from my server to the monitoring system varies, and one path is shorter than the other(s).

The NTP software used by the pool is designed to prevent the first two options, leaving only the third. This is not surprising to anyone familiar with the principles of Internet routing, but I don’t think I’ve ever seen such a clear demonstration of those principles. This graph would be a perfect exercise for a network communications class: “given the above description of the NTP protocol, explain why the offset switches alternates between two different values.

The return of the FreeBSD desktop

I have a confession to make: I haven't used FreeBSD as a desktop OS for years. The reason is twofold:

  1. Since 2005, my work has required me to run Linux (Debian and Ubuntu at Linpro, RedHat at the University of Oslo) and, briefly, Windows at Kongsberg Maritime. I eventually stopped using stationary computers, resorting instead to a (company-provided) laptop running either Ubuntu, or Windows with Ubuntu in VirtualBox.
  2. More importantly, around the time I started at Linpro, it became increasingly difficult to maintain a FreeBSD desktop. The modularization of X.org and the increasing complexity of desktop environments mean that the number of packages required for a complete desktop system has grown from a bit over 100 to well over 600 (in addition to the kernel and base operating system, which is monolithic in FreeBSD). The FreeBSD ports system does not scale well, and the lack of a proper binary update procedure makes it almost impossible to keep that many packages up-to-date.

This is about to change. Thanks to what may very well be the most important innovations in recent FreeBSD history, I am now once again running a FreeBSD desktop.

I am referring, of course, to PKGNG and its companion, Poudrière.

(Actually, I've been running PC-BSD for a while, but I'm not a big fan of PBIs, and they don't really address the updating problem. This is about building a FreeBSD desktop from scratch, and keeping it up-to-date.)

Here is the procedure I followed:

I created a new VirtualBox VM with a 128 GB disk—this may not seem like much, but most of my storage requirements are met by the host system, which has two 2 TB disks, and an older file server with over 2 TB of storage, including ~40 GB of source code.

Instead of the default NAT configuration, I set up a bridged network interface with promiscuous mode enabled; I run my own DNS and DHCP servers (on a soekris, of course), so within my home network, the VM has its own fixed IP address and DNS name.

I then installed FreeBSD 9.0-RELEASE amd64 from the disc1 ISO. I did not select any packages, nor did I install the ports tree. However, as soon as the system booted, I downloaded and extracted an up-to-date ports tree using portsnap and built pkg and poudriere.

I had initially planned to rely entirely on the pkgbeta repository, but there are two problems with this: firstly, all the packages there are built with the default options, which in many cases make no sense at all; and secondly and most importantly, it did not at that time have a full set of Gnome 2 packages.

I therefore set up my own Poudrière. I ran a first build with only ports-mgmt/pkg and ports-mgmt/poudriere, then updgraded those two packages and started over with a slightly larger set of packages, including mail/postfix, security/sudo, shells/zsh and other essentials which don't take long to build. Finally, I added emulators/virtualbox-ose-additions, x11/xorg, x11/gnome2, and a number of desktop applications (e.g. Emacs) and development tools (e.g. Subversion) which I use. Then it was just a matter of

% sudo pkg install x11/xorg x11/gnome2 editors/emacs

Whenever I need an additional package, I add it to the package list and re-run Poudrière. I don't really need to do that—I could just as easily build it straight from the ports tree—but it ensures that I get a “clean” package that I can safely use on another VM or machine, should I ever need to, and that it is rebuilt when updated, along with all the other packages I use:

% sudo poudriere ports -u
% sudo poudriere bulk -f ~/poudriere-packages -j 83amd64 -k

I had some minor configuration trouble. First, X.org's autoconfiguration feature still doesn't work very well, at least in VirtualBox, so I used the xorg.conf from a previous post, with only minor modifications:

Section "InputDevice"
        Identifier      "Generic Keyboard"
        Driver          "kbd"
        Option          "XkbRules"      "xorg"
        Option          "XkbModel"      "pc105"
        Option          "XkbLayout"     "us"
EndSection

Section "InputDevice"
        Identifier      "VBox Mouse"
        Driver          "vboxmouse"
        Option          "CorePointer"
EndSection

Section "Device"
        Identifier      "VBox Video"
        Driver          "vboxvideo"
EndSection

Section "Monitor"
        Identifier      "VBox Monitor"
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Device          "VBox Video"
        Monitor         "VBox Monitor"
EndSection

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen          "Default Screen"
        InputDevice     "Generic Keyboard"
        InputDevice     "VBox Mouse"
EndSection

With the VirtualBox guest additions installed and this xorg.conf in place, everything works beautifully—mouse integration, clipboard integration and dynamic desktop resizing.

The second issue I had was with GDM's greeter, which did not display my user in the user list and would not let me type in my user name. The first part is understandable, as it only displays users who have logged in at least once already. The second is more surprising, but I did not have the energy to try to figure out why. Instead, I worked around it by disabling the user list:

% sudo -u gdm gconftool-2 --type bool \
    --set /apps/gdm/simple-greeter/disable_user_list true

Shared folders are still not implemented for FreeBSD guests, but I rarely need to transfer files between the host and the guest; when I do, I use FileZilla to SFTP them over, and there is always SMB for more complex use.

If I were running on real hardware or a weaker host (this is a four-core i7 with 16 GB RAM), I might use a subset of PC-BSD's tricked-out sysctl.conf, but I don't need audio and I haven't experienced any interactivity issues yet.

That's it—try it yourself, and share your experience below!

The return of the FreeBSD desktop

I have a confession to make: I haven’t used FreeBSD as a desktop OS for years. The reason is twofold:

  1. Since 2005, my work has required me to run Linux (Debian and Ubuntu at Linpro, RedHat at the University of Oslo) and, briefly, Windows at Kongsberg Maritime. I eventually stopped using stationary computers, resorting instead to a (company-provided) laptop running either Ubuntu, or Windows with Ubuntu in VirtualBox.
  2. More importantly, around the time I started at Linpro, it became increasingly difficult to maintain a FreeBSD desktop. The modularization of X.org and the increasing complexity of desktop environments mean that the number of packages required for a complete desktop system has grown from a bit over 100 to well over 600 (in addition to the kernel and base operating system, which is monolithic in FreeBSD). The FreeBSD ports system does not scale well, and the lack of a proper binary update procedure makes it almost impossible to keep that many packages up-to-date.

This is about to change. Thanks to what may very well be the most important innovations in recent FreeBSD history, I am now once again running a FreeBSD desktop.

I am referring, of course, to PKGNG and its companion, Poudrière.

(Actually, I’ve been running PC-BSD for a while, but I’m not a big fan of PBIs, and they don’t really address the updating problem. This is about building a FreeBSD desktop from scratch, and keeping it up-to-date.)

Here is the procedure I followed:

I created a new VirtualBox VM with a 128 GB disk—this may not seem like much, but most of my storage requirements are met by the host system, which has two 2 TB disks, and an older file server with over 2 TB of storage, including ~40 GB of source code.

Instead of the default NAT configuration, I set up a bridged network interface with promiscuous mode enabled; I run my own DNS and DHCP servers (on a soekris, of course), so within my home network, the VM has its own fixed IP address and DNS name.

I then installed FreeBSD 9.0-RELEASE amd64 from the disc1 ISO. I did not select any packages, nor did I install the ports tree. However, as soon as the system booted, I downloaded and extracted an up-to-date ports tree using portsnap and built pkg and poudriere.

I had initially planned to rely entirely on the pkgbeta repository, but there are two problems with this: firstly, all the packages there are built with the default options, which in many cases make no sense at all; and secondly and most importantly, it did not at that time have a full set of Gnome 2 packages.

I therefore set up my own Poudrière. I ran a first build with only ports-mgmt/pkg and ports-mgmt/poudriere, then updgraded those two packages and started over with a slightly larger set of packages, including mail/postfix, security/sudo, shells/zsh and other essentials which don’t take long to build. Finally, I added emulators/virtualbox-ose-additions, x11/xorg, x11/gnome2, and a number of desktop applications (e.g. Emacs) and development tools (e.g. Subversion) which I use. Then it was just a matter of

% sudo pkg install x11/xorg x11/gnome2 editors/emacs

Whenever I need an additional package, I add it to the package list and re-run Poudrière. I don’t really need to do that—I could just as easily build it straight from the ports tree—but it ensures that I get a “clean

Information is hard

Many of you are probably familiar with Information is Beautiful, a blog that attempts to demonstrate how complex information can be presented in a clear, transparent and visually pleasing manner. They usually succeed, but the latest entry is an absolute horror:

This is not beautiful information. This is a textbook example of information design gone bad. The symbology and colors (pale blue suits for men and pale pink dresses for women), which I find offensive, may be a matter of taste, but the following are matters of fact:

  • There is no clear explanation of what the numbers, the lengths of the bars or the number of silhouettes on each row mean. In fact, there is no explanation whatsoever of what the number of silhouettes or the bars on the right mean.
  • The infographic is supposed to show the gender distribution for various social web sites, but there is no way to actually compare the data for each gender, because the data sets are entirely disjoint.
  • There is no indication of how the data support the conclusion (“99 million more monthly female visitors

Bensinpriser

Petter Reinholdtsen skrev nylig om Bitfactorys bensinpris-app (iOS, Android) og mangelen på dokumentasjon av protokollen den bruker for å laste ned priser fra databasen deres.

Jeg har lenge lurt på hvorfor ikke Forbrukerombudet pålegger stasjoner å rapportere inn prisene slik at forbrukere kan sammenligne før de velger hvor de skal fylle. Det kan hende at de mener at det er nok konkurranse i markedet, eller at det ville være en urimelig kostnad. Sistnevnte er imidlertid ikke et holdbart argument iom. at alt er automatisert; det ville aldri være snakk om manuell innrapportering.

Problemet er at selv om stasjonene følger med på konkurrentenes priser og justerer sine egne priser deretter, så er det vanskelig for forbrukerne å dra nytte av denne konkurransen. Jeg passerer 8 bensinstasjoner på vei til jobb, men ikke alle har prisskilt som er synlige fra veien, så jeg ville måtte svinge innom alle sammen for å finne ut hva prisen er, og deretter kjøre tilbake til den billigste. Det nytter ikke å sjekke prisene på morgenen og fylle på kvelden, siden prisene endres flere ganger om dagen.

Resultatet er at jeg stort sett gir beng og fyller på Statoil på Bjerke annenhver mandag morgen, fordi rabatten jeg har der som regel overstiger prisdifferansen mellom denne stasjonen og det billigste alternativet.

(det er selvfølgelig mulig å få tilsvarende rabatter hos andre kjeder, men jeg kjenner ikke til noen avtaler som er like gode som Statoil-avtalen til NAF – selv ikke 365 Direkte, som hevder å være best i landet på bensin – og jeg vil helst ikke måtte sjonglere et halvt dusin rabattkort og kredittkort)

Mange i Oslo kan til og med risikere å oppleve at det, i snitt, koster mer å undersøke prisen før de fyller enn å bare fylle på nærmeste stasjon, fordi de må krysse bomringen for å komme til andre stasjoner i nærheten.

I en idéell verden hadde altså Forbrukerombudet hatt en kontinuerlig oppdatert oversikt over bensinpriser, på samme måte som Post- og teletilsynet har en kontinuerlig oppdatert prisoversikt for telefoni og bredbånd. Bitfactory kunne fortsatt tjent penger på å selge en app som er bedre enn den Forbrukerombudet selv eventuelt tilbød, og ville dessuten spare utgiftene de har med å vedlikeholde databasen.

On testing

Last fall, I wrote a completely new configuration parser for OpenPAM Lycopsida. Although the new parser was far more robust than the one it replaced, it was large, unwieldy, and suffered from a number of issues relating to whitespace handling, which stemmed from reusing some old code which unfortunately was thoroughly documented and therefore could not be easily modified. So I decided to rewrite it again, from scratch this time.

Then I did what I should have done last fall but didn't: I wrote some unit tests. And of the first dozen or so tests I came up with, three failed, revealing two different bugs—one of them fairly serious.

There's a lesson in here somewhere...

On testing

Last fall, I wrote a completely new configuration parser for OpenPAM Lycopsida. Although the new parser was far more robust than the one it replaced, it was large, unwieldy, and suffered from a number of issues relating to whitespace handling, which stemmed from reusing some old code which unfortunately was thoroughly documented and therefore could not be easily modified. So I decided to rewrite it again, from scratch this time.

Then I did what I should have done last fall but didn’t: I wrote some unit tests. And of the first dozen or so tests I came up with, three failed, revealing two different bugs—one of them fairly serious.

There’s a lesson in here somewhere…

Downtime

I haven't been able to read email sent to [email protected] or [email protected] for five days, due to a series of unfortunate incidents involving dodgy power supplies and the fragility of ZFS boot in FreeBSD. Work and other duties prevented me from addressing the issue in a more timely manner, but I am now regaining control. Luckily, neither my ~30 GB IMAP spool nor any other data was lost, nor did my backup MX bounce any mail. My IMAP server is now back up with a small UFS SU+J boot / root partition instead of ZFS. I am still unable to read email, but that should be fixed within 24 hours.

I also uncovered an annoying but luckily not fatal bug in the Cyrus IMAP server. When TLS is configured, the IMAP daemon stores state for each TLS session in a DB file. If that file is corrupted, the server will start, but it will refuse any incoming IMAP or LMTP connections, and will instead spit out a stream of completely unhelpful error messages. The only recourse is to delete the TLS session state database; I set up an rc script to do that at boot time, so hopefully this won't bite me again.

Downtime

I haven’t been able to read email sent to [email protected] or [email protected] for five days, due to a series of unfortunate incidents involving dodgy power supplies and the fragility of ZFS boot in FreeBSD. Work and other duties prevented me from addressing the issue in a more timely manner, but I am now regaining control. Luckily, neither my ~30 GB IMAP spool nor any other data was lost, nor did my backup MX bounce any mail. My IMAP server is now back up with a small UFS SU+J boot / root partition instead of ZFS. I am still unable to read email, but that should be fixed within 24 hours.

I also uncovered an annoying but luckily not fatal bug in the Cyrus IMAP server. When TLS is configured, the IMAP daemon stores state for each TLS session in a DB file. If that file is corrupted, the server will start, but it will refuse any incoming IMAP or LMTP connections, and will instead spit out a stream of completely unhelpful error messages. The only recourse is to delete the TLS session state database; I set up an rc script to do that at boot time, so hopefully this won’t bite me again.

ZFS-to-ZFS backups

ZFS has a couple of very useful functions, zfs send and zfs receive, which allow you to serialize a complete ZFS dataset and recreate it in a different location. They can also be used to serialize a delta between two snapshots and apply that delta to a previously created copy of the dataset. You see where I'm going with this... That's right, incremental backups of a ZFS dataset or even an entire pool to a different ZFS dataset or pool.

Why would you want to perform incremental ZFS-to-ZFS backups instead of just adding redundancy to the pool, or cloning a snapshot? Because—provided the ZFS pool and filesystem versions match—it allows you to duplicate your dataset or pool on removable media (which you can store off-site), or even on a different machine across the network. This technique is far more efficient than rsync, because there is no need to compare the source and destination: ZFS already knows exactly what has changed. It also preserves the filesystem hierarchy and dataset properties.

In my case, I need to duplicate a pool onto removable media because I am replacing a server that only takes PATA disks with another that only takes SATA disks, which precludes just moving the disks over and progressively replacing them with new ones. Using this technique, when the time comes, I can slide the new server into the rack, hook up the backup disk, and restore just the parts I want to keep.

Of course, like a good little hacker, I wrote a script, which you can find here, to automate this.

The script takes two arguments: the source dataset and the destination dataset. Either of these can be the root of a ZFS pool or a dataset within a pool; they can even be datasets within the same pool, provided they do not overlap. The script selects the latest snapshot of the destination dataset (it uses a naming scheme which ensures that lexical order corresponds to chronological order), verifies that the source dataset has a snapshot with the same name, takes a new snapshot of the source dataset, and streams the difference between the old and new snapshots from the source dataset to the destination dataset. Finally, it deletes the old snapshot to allow ZFS to reclaim the space occupied by old data.

You can use this script with multiple backup disks, since it will only delete the snapshot that was actually used for the current disk. If you have one disk for each day of the week, for instance, it will delete last Monday's snapshot once it has completed this Monday's backup, but leave the other six in place. Likewise, if you decide to keep Sunday's disk for a month instead of reusing it next Sunday, the script will leave the snapshot in place until you run it again with the same disk.

The script does not currently support over-the-network backups, but it should be fairly easy to implement.

More Advanced Format drives: Samsung SpinPoint F4 EcoGreen and Seagate Barracuda Green

I've acquired a couple more 2 TB Advanced Format drives: a Seagate Barracuda Green (ST2000DL003) and a Samsung SpinPoint F4 EcoGreen (HD204UI, no data sheet available online).

I was extremely impressed with the Samsung HD204UI. It's the first AF drive I've seen with decent performance. In fact, it's the fastest disk I've tested so far—its unaligned writes are faster than the non-AF Hitachi I used as a reference last time, and its aligned writes are twice as fast.

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096       43984    2979    2979
  131072    1024     512    4096      127047    1031    1031

   65536    2048       0    8192       14764    4438    8877
   65536    2048     512    8192       12453    5262   10524
   65536    2048    1024    8192       12460    5259   10518

   32768    4096       0   16384        4609    7109   28436
   32768    4096     512   16384        7829    4185   16740
   32768    4096    1024   16384        8413    3894   15579
   32768    4096    2048   16384        8211    3990   15961

   16384    8192       0   32768        3952    4145   33165
   16384    8192     512   32768        9050    1810   14481
   16384    8192    1024   32768        9317    1758   14067
   16384    8192    2048   32768        9315    1758   14069
   16384    8192    4096   32768        3996    4099   32793

The Seagate ST2000DL003, on the other hand, is so slow it's not even funny. It's actually the slowest of all the drives I've tested: its performance on aligned random writes is half that of the Western Digital WD20EARS. It's three times as fast on unaligned writes, but three times nothing (100 kBps) is still nothing (300 kBps) compared to the Samsung HD204UI (15 MBps). Here are the numbers:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096     2419280      54      54
  131072    1024     512    4096     2199286      59      59

   65536    2048       0    8192     1283667      51     102
   65536    2048     512    8192      985184      66     133
   65536    2048    1024    8192      995423      65     131

   32768    4096       0   16384       45980     712    2850
   32768    4096     512   16384      345291      94     379
   32768    4096    1024   16384      432533      75     303
   32768    4096    2048   16384      429781      76     304

   16384    8192       0   32768       34192     479    3833
   16384    8192     512   32768      166440      98     787
   16384    8192    1024   32768      210147      77     623
   16384    8192    2048   32768      207356      79     632
   16384    8192    4096   32768       34221     478    3830

This time, I also ran sequential write tests—basically, dding eight gigabytes' worth of zeroes to the disk in 128 kB blocks, which is the optimal I/O size for FreeBSD. This time, the results are pretty close: the Samsung HD204UI gets slightly less than 90 MBps, and the Seagate ST2000DL003 gets slightly less than 80 MBps.

OpenBSD IPSec backdoor allegations: update

I'm sure I don't need to remind anyone what this is about...

The latest news: Theo now says that it is probable that NetSec was indeed contracted to insert backdoor code into OpenBSD, but after a month of review and changelog archeology, there is still no sign that they succeeded or even attempted to push tainted code into the tree.

The audit (which is still ongoing) did uncover one serious bug, but there is no reason to believe that it was planted deliberately. This relates to CBC mode, an encryption protocol in which each block of plaintext is combined with the ciphertext of the previous block before encryption to make it harder to attack ciphertext blocks individually.

If I understand Theo's message correctly,

  • It used to be common practice to use the last ciphertext block from one message as IV for the next message. This seemed like a good idea at the time, because the alternative is to generate a random IV for each new message, which requires a strong, fast PRNG, and strong, fast PRNGs didn't grow on trees back when this scheme was devised. By reusing the last ciphertext block from the previous message, a costly random IV was only required for the very first message.
  • This practice was discovered to be a bad idea because in n - 1 out of n cases (where n is the block size in bytes), the last plaintext block of any message encrypted with a block cipher contains somewhat predictable padding.
  • The flawed IV logic was replicated in several parts of the OpenBSD source tree, and the fix was implemented in some of them, but not all.
  • The person who implemented this flawed logic was at that time a NetSec employee, but he had been involved in the development of OpenBSD's IPSec stack for years before he was hired, and, as previously mentioned, he was only following common practice.
  • The same person implemented the obvious fix (generating a new, random IV for every message) once the attack was discovered.
  • The person responsible for those parts of the tree in which the fix was not implemented is one of the people fingered by Perry, but his tenure started after Perry had left and ended before the attack was discovered.
  • Anyone with any amount of experience in a large F/OSS project, or any large software development effort for that matter, can tell you that this kind of oversight is the rule rather than the exception. Although there is no evidence that he did not intentionally “forget” to fix his code, it is far more likely that he simply did not realize that the fix that had already been committed did not extend to his own code, or that he wasn't paying attention, and nobody else noticed.

My bounty still stands, and I will even relax the requirements a bit: you are not required to show that OpenBSD is still exploitable, only that it was exploitable on December 11, 2010 (the date of Perry's email to Theo).

4k drive update

Just to let you know what the current status is wrt. 4k drives:

It looks like the consensus in the industry (meaning everyone except Western Digital) is to announce dual sector sizes, i.e. 512-byte logical sectors on top of 4096-byte physical sectors.

Ivan Voras has taken the initiative to organize a 4k BoF at BSDCan, although judging from the (private) email exchange on the subject, it's quite possible that a decision will be made before then. Currently, it looks like we're moving towards having the low-level driver report a 512-byte sector size and 4096-byte stripe width (and, if necessary, an appropriate offset) to GEOM. This preserves backward compatibility, but announces to GEOM consumers that it is a good idea to do I/O in 4096-byte blocks and align data structures on 4096-byte boundaries. All that remains is then to make sure that those GEOM consumers we care about (particularly ZFS) take advantage of this information.

The situation for WD “Advanced Format” drives is a bit more complex, because they announce 512-byte logical sectors. The only solution I can see is to add a quirk system to the ada driver (and possibly to ata as well, if we still care about it) similar to the ones we have for SCSI and USB devices, and match the model number. I believe /WD\d+[A-Z]+RS/ should match all existing Advanced Format drives with no false positives.