Author Archives: Dag-Erling Smørgrav

DNS in FreeBSD 10

Yesterday, I wrote about the local caching resolver we now have in FreeBSD 10. I’ve fielded quite a few questions about it (in email and on IRC), and I realized that although this has been discussed and planned for a long time, most people outside the 50 or so developers who attended one or both of the last two Cambridge summits (201208 and 201308) were not aware of it, and may not understand the motivation.

There are two parts to this. The first is that BIND is a support headache with frequent security advisories and a lifecycle that aligns poorly with our release schedule, so we end up having to support FreeBSD releases containing a discontinued version of BIND. The second part is the rapidly increasing adoption of DNSSEC, which requires a caching DNSSEC-aware resolver both for performance reasons (DNSSEC validation is time-consuming) and to avoid having to implement DNSSEC validation in the libc resolver.

We could have solved the DNSSEC issue by configuring BIND as a local caching resolver, but for the reasons mentioned above, we really want to remove BIND from the base system; hence the adoption of a lightweight caching resolver. An additional benefit of importing LDNS (which is a prerequisite for Unbound) is that OpenSSH can now validate SSHFP records.

Note that the dns/unbound port is not going away, and that users who want to run Unbound as a caching resolver for an entire network rather than just a single machine have the option of either moving their configuration into /var/unbound/unbound.conf, or running the base and port versions side-by-side. This should not be a problem as long as the port version doesn’t try to listen on 127.0.0.1 or ::1.

I’d like to add that since my previous post on the subject, and with the help of readers, developers and users, I have identified and corrected several issues with the initial commit

  • /etc/unbound is now a symlink to /var/unbound. My original intention was to have the configuration files in /etc/unbound and the root anchor, unbound-control keys etc. in /var/unbound, but the daemon needs to access both locations at run-time, not just on start-up, so they must all be inside the chroot. Running the daemon un-chrooted is, of course, out of the question.
  • The init script ordering has been amended so the local_unbound service now starts before most (hopefully all) services that need functioning DNS.
  • resolvconf(8) is now blocked from updating /etc/resolv.conf to avoid failing over from the DNSSEC-aware local resolver to a potentially non-DNSSEC-aware remote resolver in the event of a request returning an invalid record.
  • The configure command line and date / time are no longer included in the binary.

Finally, I just flipped the switch so that BIND is now disabled by default and the LDNS utilities are enabled. The BIND_UTILS and LDNS_UTILS build options are mutually exclusive; in hindsight, I should probably have built and installed the new host(1) as ldns-host(1) so both options could have been enabled at the same time. We don’t yet have a dig(1) wrapper for drill(1), so host(1) is the only actual conflict.

Local caching resolver in FreeBSD 10

As of a few hours ago, all it takes to set up a local caching resolver in FreeBSD 10 is:

# echo local_unbound_enable=yes >>/etc/rc.conf
# service local_unbound start

Yes, it really is that simple—and it works fine with DHCP, too. Hold my beer and watch this:

# pgrep -lf dhclient
1316 dhclient: vtnet0
1265 dhclient: vtnet0 [priv]
# cat /etc/resolv.conf
# Generated by resolvconf
search example.com
nameserver 192.0.2.53
# time host www.freebsd.org
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .
        0.02 real         0.00 user         0.01 sys

As you can see, we’re running DHCP on a VirtIO network interface. Let’s work our magic:

# echo local_unbound_enable=yes >>/etc/rc.conf
# service local_unbound start
Performing initial setup.
Extracting forwarders from /etc/resolv.conf.
/var/unbound/forward.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf created
original /etc/resolv.conf saved as /etc/resolv.conf.20130923.075319
Starting local_unbound.

And presto:

# pgrep -lf unbound
3799 /usr/sbin/unbound -c/var/unbound/unbound.conf
# cat /var/unbound/unbound.conf 
# Generated by local-unbound-setup
server:
        username: unbound
        directory: /var/unbound
        chroot: /var/unbound
        pidfile: /var/run/local_unbound.pid
        auto-trust-anchor-file: /var/unbound/root.key

include: /var/unbound/forward.conf
# cat /var/unbound/forward.conf
# Generated by local-unbound-setup
forward-zone:
        name: .
        forward-addr: 192.0.2.53
# cat /etc/resolv.conf
# Generated by resolvconf
search example.com
# nameserver 192.0.2.53

nameserver 127.0.0.1
options edns0

We can see the cache at work; the first request takes significantly longer than before, but the second is served from cache:

# time host www.freebsd.org
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .
        0.07 real         0.01 user         0.00 sys
# time host www.freebsd.org
www.freebsd.org is an alias for wfe0.ysv.freebsd.org.
wfe0.ysv.freebsd.org has address 8.8.178.110
wfe0.ysv.freebsd.org has IPv6 address 2001:1900:2254:206a::50:0
wfe0.ysv.freebsd.org mail is handled by 0 .
        0.01 real         0.00 user         0.00 sys

Finally, let’s see how this interacts with DHCP:

# resolvconf -u
# cat /etc/resolv.conf
# Generated by resolvconf
search example.com
nameserver 127.0.0.1
nameserver 192.0.2.53
options edns0

# cat /var/unbound/forward.conf 
# Generated by resolvconf

forward-zone:
        name: "example.com"
        forward-addr: 192.0.2.53

forward-zone:
        name: "."
        forward-addr: 192.0.2.53

Note that resolvconf(8) re-added the 192.0.2.53 entry. It doesn’t really matter, as long as 127.0.0.1 comes first.

[ETA: it does matter—see Jakob Schlyter's comment below and my reply.]

[ETA: see my followup about the motivation for importing Unbound.]

Growing a VirtualBox disk with ZFS on it

I have a VirtualBox VM on a Windows host with a 32 GB disk. That disk is partitioned with GPT and has four partitions: a boot partition, a swap partition, a smallish UFS root partition, and a ZFS partition. I need more space in the latter, so let’s grow it.

The first step is to shut down the VM and resize the virtual disk. This cannot be done in the GUI—we have to use the command-line utility:

C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9>"\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo "CrashBSD 9.vdi"
UUID:                 4a088148-72ef-4737-aae6-0a39e05aee06
Accessible:           yes
Logical size:         32768 MBytes
Current size on disk: 14484 MBytes
Type:                 normal (base)
Storage format:       VDI
Format variant:       dynamic default
In use by VMs:        CrashBSD 9 (UUID: 06bbe99d-9118-4c11-b29b-4ffd175ad06c)
Location:             C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9\CrashBSD 9.vdi

C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9>"\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "CrashBSD 9.vdi" --resize 65536
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9>"\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo "CrashBSD 9.vdi"
UUID:                 4a088148-72ef-4737-aae6-0a39e05aee06
Accessible:           yes
Logical size:         65536 MBytes
Current size on disk: 14485 MBytes
Type:                 normal (base)
Storage format:       VDI
Format variant:       dynamic default
In use by VMs:        CrashBSD 9 (UUID: 06bbe99d-9118-4c11-b29b-4ffd175ad06c)
Location:             C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9\CrashBSD 9.vdi

Next, we boot the VM into single-user mode. It will repeatedly complain about the secondary GPT table, which is supposed to be located at the end of the disk but is now in the middle, since we doubled the size of the disk:

GEOM: ada0: the secondary GPT header is not in the last LBA.
# gpart list ada0
Geom name: ada0
modified: false
state: CORRUPT
fwheads: 16
fwsectors: 63
last: 67108830
first: 34
entries: 128
scheme: GPT
Providers:
[...]

Thankfully, this is trivial to fix. In fact, this exact use case is mentioned in the gpart(8) man page:

# gpart ada0 recover
ada0 recovered.
# gpart list ada0
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 134217694
first: 34
entries: 128
scheme: GPT
Providers:
[...]

Next, we resize the ZFS partition to fill all available space:

# gpart list ada0
[...]
4. Name: ada0p4
   Mediasize: 25769803776 (24G)
[...]
# gpart resize -i 4 ada0
ada0p4 resized
# gpart list ada0
[...]
4. Name: ada0p4
   Mediasize: 60129442304 (56G)
[...]

Let’s see what our pool looks like now:

# zpool import crash
# zfs list crash
NAME    USED  AVAIL  REFER  MOUNTPOINT
crash  14.2G  9.31G    31K  /

Hmm, no cigar. The pool hasn’t grown because the underlying vdev hasn’t automatically expanded to fill the resized partition. That’s easy to fix, though:

# zpool online -e crash ada0p4

And there we go:

# zfs list crash
NAME    USED  AVAIL  REFER  MOUNTPOINT
crash  14.2G  40.8G    31K  /

Challenges in Identity Management and Authentication

This was my presentation at the 2012 EuroBSDCon in Warsaw, Poland. I’ve been meaning to write more extensively on this subject, but never got around to it. I just watched through the video twice, and it was a lot less cringe-inducing than I expected (especially when you consider that I was sick and sleep-deprived when I gave it).

Towards the end, I got a question about Apple’s security framework. In my answer, I referred to it as CDDL. That was a slip of the tongue; I was referring to CDSA, which is actually an Open Group specification which Apple implemented and open-sourced. Furthermore, CDSA does not to everything I said it does. However, Apple built their Security Services Framework (described in their Authentication, Authorization and Permissions Guide and various other documents) on top of CDSA; so the combination of CDSA and what Apple added on top does everything from key management to authentication and authorization.

My presentation at the 2013 EuroBSDCon in St Julians, Malta will continue where I left off last year, outlining a concrete solution based on the principles set forth in the second part of last year’s presentation (starting at 32:06).

pkgng without ports: addenda

Two things I forgot to mention in my previous post:

  1. In order to use OpenPAM from svn instead of the version that comes with FreeBSD, you need to copy security/pam_mod_misc.h and pam_debug_log.c into the OpenPAM source tree and adjust the Makefiles accordingly, otherwise FreeBSD’s service modules won’t run and you won’t be able to log in. I don’t plan to include this code in OpenPAM; I’d rather overhaul FreeBSD’s modules so they no longer need it.
  2. What I actually wanted to do, but didn’t because I needed a solution there and then, was patch automake itself to add a pkgng target so gmake pkgng creates a package with no additional input required (except possibly a +DESC file).

Creating pkgng packages without ports

Lately, I’ve been working on expanding the scope of OpenPAM to more than just a PAM library. Specifically, I’ve added support (in a separate library) for the OATH HOTP and TOTP one-time password algorithms. In the long term, I also intend to implement PSKC and OCRA, the ultimate goal being full compliance with the OATH client and server certification profiles. Part of the reason I’m doing this is that my employer needs it, which is why the University of Oslo holds the copyright on most of the OATH code, but it is also something I’ve been wanting to do for a long time, and which I believe will greatly benefit FreeBSD.

This is a large undertaking, though. I’m not comfortable rolling a new OpenPAM release with the OATH code at this time—and I probably won’t be for quite a while. I’ve created a “nooath

Managing your own pkgng repository

[edit 2013-08-05: fixed a typo in the two command lines used to create the repo definition files, spotted by swills@]

Say you have your own poudriere and your own pkgng repo. You’ve set up Apache to point at your poudriere’s package directory:

<VirtualHost *>
  ServerName pkg.des.no
  ServerAdmin [email protected]
  DocumentRoot /poudriere/data/packages
  <Directory "/poudriere/data">
    Options +Indexes +SymLinksIfOwnerMatch
    IndexOptions +FancyIndexing +FoldersFirst
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

The 91amd64-default and 91i386-default directories are so named by poudriere because they contain the output of the 91amd64 and 91i386 jails, respectively, based on the default ports tree. These are details which you don’t necessarily want your clients to know (or need to know), so you create symlinks which match your clients’ ABIs:

# cd /poudriere/data/packages
# ln -s 91amd64-default freebsd:9:x86:64
# ln -s 91i386-default freebsd:9:x86:32

All you need to do on the client side now is:

# cat >/usr/local/etc/pkg.conf <<EOF
packagesite: http://pkgng.tsdutv.usit.no/freebsd:9:x86:64
EOF

Now, let’s think about this for a while. Every time you install a new machine, you have to copy or type in that pkg.conf, and while this is a pretty minimal example, your real pkg.conf could be much larger: you could have multiple repos, multiple servers with failover, etc. How about we fetch it from a central location?

# fetch -o/usr/local/etc/ http://pkg.des.no/pkg.conf

But what if it changes? Well, why not use the package system itself to distribute and maintain it?

We want to distribute our pkg.conf as a package, and since we want pkg to update it when it changes, we need to place it in a repo. We can’t stick it in the FreeBSD ports tree, and while it is possible to sneak it into the local copy of the ports tree that poudriere builds from, it’s not very convenient. So what we do is create an additional pkgng repo with only one package, which contains two pkg.conf files: one for our real pkgng repo, and one for the repo that contains our configuration package.

First, we create the contents of our package:

% mkdir des-repos
% cd des-repos
% mkdir -p usr/local/etc/pkg/repos
% cat >usr/local/etc/pkg/repos/des-packages.conf <<EOF
des-packages:
  url: http://pkg.des.no/${ABI}
EOF
% cat >usr/local/etc/pkg/repos/des-repos.conf <<EOF
des-repos:
  url: http://pkg.des.no/repos
EOF

Now we need a manifest:

% cat >+MANIFEST <<EOF
name: des-repos
version: 20130715
origin: local/des-repos
comment: Repository definitions for pkg.des.no.
arch: 
www: http://pkg.des.no/
maintainer: [email protected]
prefix: /usr/local
desc: Repository definitions for pkg.des.no.
categories: local, ports-mgmt
deps:
  pkg: { name: pkg, origin: ports-mgmt/pkg, version: 1.1 }
files:
  /usr/local/etc/pkg/repos/des-packages.conf: { uname: root, gname: wheel, perm: 0644 }
  /usr/local/etc/pkg/repos/des-repos.conf: { uname: root, gname: wheel, perm: 0644 }
EOF

Note that arch is intentionally left blank, as this package is architecture-neutral.

Once we have contents and a manifest, we can create the package file:

% pkg create -r $PWD -m $PWD
% tar tf des-repos-20130715.txz 
+COMPACT_MANIFEST
+MANIFEST
/usr/local/etc/pkg/repos/des-packages.conf
/usr/local/etc/pkg/repos/des-repos.conf

All that remains (on the server) is to create the repo:

# mkdir /poudriere/data/packages/repos
# cp des-repos-20130715.txz /poudriere/data/packages/repos
# pkg repo /poudriere/data/packages/repos
# cd /poudriere/data/packages
# ln -s repos/des-repos-20130715.txz des-repos.txz

Then, on each client (presumably including the server itself):

# rm /var/db/pkg/repo*sqlite
# rm /usr/local/etc/pkg.conf
# pkg add http://pkg.des.no/des-repos.txz
# pkg update

Tada!

Benchmark: WD Red NAS

My wife is in the market for large, cheap drives with decent performance to store sequencing data, so I ordered and tested a 2 TB Western Digital Red NAS (WD20EFRX—no link because wdc.com is broken at the moment). The Red series seems to be a halfway point between the WD Green and WD Black series: like the Green series, they have 4096-byte sectors and IntelliPower (i.e. variable rpm), but they are designed for 24×7 operation and seem to have far more consistent performance—although not quite on par with the Black series.

The big news is that this is the first Advanced Format disk I’ve seen that correctly reports its physical sector size:

protocol              ATA/ATAPI-9 SATA 3.x
device model          WDC WD20EFRX-68AX9N0
firmware revision     80.00A80
serial number         WD-WMC301592199
WWN                   50014ee6adf1fbaf
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       3907029168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

As shown below, random-access performance is decent, but not mind-blowing—with the important caveat that I tested it on a machine that only has SATA I. I will update the numbers if and when I get the chance to test it on a machine with a SATA II or SATA III controller.

   count    size  offset    step        msec     tps    kBps

   32768    4096       0   16384       10222    3205   12822
   32768    4096     512   16384       33900     966    3866
   32768    4096    1024   16384       35417     925    3700
   32768    4096    2048   16384       36207     905    3620

   16384    8192       0   32768        8298    1974   15794
   16384    8192     512   32768       31238     524    4195
   16384    8192    1024   32768       31666     517    4139
   16384    8192    2048   32768       31547     519    4154
   16384    8192    4096   32768        8037    2038   16307

    8192   16384       0   65536        6471    1265   20252
    8192   16384     512   65536       27815     294    4712
    8192   16384    1024   65536       27201     301    4818
    8192   16384    2048   65536       27607     296    4747
    8192   16384    4096   65536        6722    1218   19497
    8192   16384    8192   65536        6396    1280   20489

    4096   32768       0  131072        5199     787   25210
    4096   32768     512  131072       22564     181    5808
    4096   32768    1024  131072       23349     175    5613
    4096   32768    2048  131072       20816     196    6296
    4096   32768    4096  131072        5540     739   23655
    4096   32768    8192  131072        5307     771   24693
    4096   32768   16384  131072        5303     772   24716

Sequential performance is also pretty decent:

# dd if=/dev/zero of=/dev/ada2 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 8.881374 secs (120898164 bytes/sec)

Hurtigruten mener seg hevet over norsk lov

Hurtigruten har i flere år, med ujevne mellomrom, sendt reklame til min Gmail-adresse. Så langt i år har jeg mottatt fire såkalte nyhetsbrev fra dem.

Jeg har aldri reist med Hurtigruten. Jeg har heller aldri bedt dem om noe pristilbud, prospekt e.l. som kunne oppfattes som et ønske om å motta reklame. Jeg har aldri kontaktet dem – bortsett fra de gangene jeg har bedt dem om å slutte å sende meg reklame.

Dette er et klart brudd på Markedsføringslovens §15:

 I næringsvirksomhet er det forbudt, uten mottakerens forutgående samtykke, å rette markedsføringshenvendelser til fysiske personer ved elektroniske kommunikasjonsmetoder som tillater individuell kommunikasjon, som for eksempel elektronisk post, telefaks eller automatisert oppringningssystem (talemaskin). [...] Krav om forhåndssamtykke etter første ledd gjelder heller ikke markedsføring ved elektronisk post i eksisterende kundeforhold der den næringsdrivende avtaleparten har mottatt kundens elektroniske adresse i forbindelse med salg.

Hurtigruten mener tydeligvis at de er hevet over norsk lov.

Jeg klaget dem inn til Forbrukerombudet for drøye to måneder siden, men har ikke fått noe svar. Klagen min er heller ikke journalført, hvilket i seg selv er et brudd på Offentlighetslovens §10 og dertil hørende forskrift 2008.10.17 nr 1119. Sic transit gloria mundi; Forbrukerombudet pleide å være flinke til å følge opp spam-klager, men for rundt halvannet eller to år siden sluttet de å behandle dem «på grunn av stor saksmengde».

Nineteen-ninety-six

1996. The Spice Girls rock (pop?) the world with Wannabe. Will Smith kicks alien butt in Independence day. DVDs become commercially available. Scientists clone the first mammal. Ebay opens. Three important standards are either released or reshaped into their current form: MIME, Unicode and IPv6. 17 years later, a shocking amount of software still does not support these standards.

sigh

Windows Backup slowdown

My Windows 7 desktop is set up to back up to a Drobo B800i (over iSCSI) every night at 04:00, using Windows Backup. Even though it only uses about 700 GB of its 2 TB mirror, and only backs up a small fraction of that, backup jobs routinely took 15 hours or more. It could have copied the entire disk in half that time!

I set about hunting for a solution. One suggestion that turned up repeatedly in Google searches was to turn off the Background Intelligent Transfer Service. BITS is basically a download manager designed to only run when there is little or no other network traffic; among other thing, it is used by Windows Update to download patches. I couldn’t understand how this could help, but I had no better ideas and nothing to lose, so I stopped BITS. The next backup job completed in 45 minutes.

Patch Tuesday came along, and I rebooted the computer. Since I had only stopped BITS and not disabled it, it started again when the machine booted. Backup jobs slowed down again. This time, I disabled BITS, and I was back to sub-hour backups.

This makes absolutely no sense. BITS wasn’t even downloading anything; as far as I know, the only program or service I have running that actually uses it is Windows Update. BITS was slowing down backups just by being there. I don’t remember having this issue when I ran backups to an eSATA drive, so there must be some network-related interaction between BITS and iSCSI, but I have no idea what.

What Google knows about me

Based om my Google Ads profile:

Personal details
Gender: male No prize for guessing; it’s in my Google+ profile.
Age: 35-44
Languages: unknown I’m surprised they didn’t figure this one out. English, French, Norwegian.
Interests
Action & Adventure Films Vaguely
Air Travel I’ve booked two flights in the last 24 months. They don’t seem to have noticed that I’ve spent a lot of time recently researching ferries.
Banking Who doesn’t use Internet banking these days?
Bicycles & Accessories I own a bike. I’ve ridden it twice in the last six years.
Computer & Video Games Yes, definitely.
Consumer Electronics Depends on how you define the term. Computers, computer parts and peripherals, yes.
East Asian Music No
Fashion & Style You have hundreds of photos of me, and you never noticed that I always wear the exact same clothes?
Fiat Try Audi.
Food & Drink Yes, but not online.
Hair Care You have hundreds of photos of me, and you never noticed that I shave my head?
Hygiene & Toiletries I’ve used the same shower gel and deodorant for years. They’re both no-brand products sold in pharmacies.
Make-Up & Cosmetics You know I’m male. You know I’m straight. Do the math.
Olympics My initial reaction was “huh?

Backing up your VMs

A few weeks ago, I finally got my Drobo (a B800i with eight 2 TB disks) set up correctly so I can back up my Windows 7 computer to it. The only data I really care about on that computer are my VirtualBox VMs—so imagine my surprise when I discovered today that they weren’t being backed up! It turns out that with the default settings (“let Windows choose

On testing, part II

About a year ago, I blogged about writing unit tests for OpenPAM to weed out bugs in the configuration file parser, and mentioned a specific bug which my unit tests caught, and which I wouldn’t have noticed without them.

Yesterday, this came back to bite me.

The patch in question fixed the following edge case:

hello "" world

This is supposed to be parsed as three words (“hello

Not too proud to beg

I’ve added two items to the FreeBSD donations wantlist.

The first, and most important, is hosting for a new source tinderbox  / continuous integration cluster. The source tinderbox currently run on three servers graciously donated, maintained and hosted by Sentex which can handle approximately two full builds per day. The University of Oslo recently replaced its old HPC cluster, and I’ve secured 28 1U nodes (including rack-mount kits but no power cords) from the old one which I hope to be able to use for a new source tinderbox cluster with a completely new, more efficient implementation, but I have nowhere to host them. As noted in the wantlist entry, once installed, the cluster will not require large amounts of external bandwidth—just enough to keep an svn mirror regularly updated and to serve the web frontend—but it will need a dedicated gigabit switch or partition for internal communication. The hosting center needs to be within driving distance of Oslo, Norway.

The second item is a request for parts to upgrade my main FreeBSD development box: an Intel H77-based motherboard, an Intel Core i7-3770S CPU and 32 GB of PC3-12800 RAM. I already have a case, a sufficiently powerful PSU and plenty of disks. I’d also like to replace my old Chieftec hot-swap frames with a pair of trayless four-disk frames such as this one or preferably this one which has one large fan instead of two small ones (and is therefore presumably quieter) but is not available in Norway.

You can also use the button in the sidebar to make a donation which will go towards hardware, software and hosting costs related to my FreeBSD and Open Source work.

Transition

If you can see this, I have successfully moved my blog from Blogger to a self-hosted WordPress instance.

This was not a trivial endeavor. I have had this blog since 2006, and there are links to it all over the net. I did not want to break those links, nor did I want them to point to an abandoned Blogger instance. In particular, I wanted readers to still be able to comment on old posts.

I had already set up Blogger to redirect maycontaintracesofbolts.blogspot.com to blog.des.no, so if I changed blog.des.no to point to my WordPress instance and made sure the slugs matched, most of the links should still work.

The first step was to configure WordPress to use the same style of permalinks as Blogger, i.e. year/month/slug.html. There was a catch, though: Blogger views posts as individual pages, whereas WordPress views them as directories, and is more comfortable with year/month/slug/ (note the final slash). Therefore, instead of trying to get WordPress to mimic Blogger more closely, I used the following mod_rewrite hack to convert Blogger-style URLs in incoming requests to WordPress-style URLs:

RewriteEngine On
RewriteRule ^/([0-9]{4}/[0-9]{2}/.*)\.html$ /$1/ [R,L]

Next, I used the Blogger Importer plugin to import posts and comments from Blogger. Several problems immediately arose.

First, Blogger Importer imported all my drafts as published posts. Luckily, this is a known bug with a simple fix.

Secondly, WordPress slugs and Blogger slugs are often but not always identical. They are both based on the title, but Blogger removes short words and cuts off at a certain length. Unfortunately, while Blogger Importer records the original Blogger slug as metadata for each post, it does not actually set the WordPress slug (post_name for those familiar with WordPress internals) when it imports posts. I therefore had to write a Perl script that scans the posts and metadata looking for Blogger slugs, then sets the WordPress slug to the correct value if it does not already match the Blogger slug.

Finally, after I had imported posts and comments but before I was ready to throw the switch, new comments were posted on Blogger. In theory, Blogger Importer can import new posts and comments without touching existing ones. However, I had already gone through existing posts and added breaks, fixed image placement, and fixed a few conversion errors (mostly related to the use of angle brackets in posts), and Blogger Importer was extremely confused. In the end, I realized that what was happening was that it did not recognize posts and comments it had already imported, because the duplicate checks were too narrow. I managed to work around this by modifying import_posts() and import_comments() in blogger-importer.php so the post_exists() and comment_exists() relied solely on the timestamp when comparing posts and comments. I also had to increase the MAX_EXECUTION_TIME parameter from 20 to 60 seconds.

I had a lot of trouble finding a good WordPress theme for the site, and I’m still not really satisfied. I may end up switching to Delicate, which is just about as basic as it gets (short of Toolbox) yet manages not to stray on the wrong side of drab.

I hope you like the new site, or at least that you don’t hate it. Feel free to comment, and I’ll be happy to answer any technical questions regarding the transition and my setup. I’ll discuss my server configuration in more detail in a later post—if I get around to it…


Edit 2013-02-14: I added a couple of extra RewriteRules:

RewriteRule ^/search/label/(en|fr|no)/?$ /category/$1 [R,L]
RewriteRule ^/search/label/([^/]+)/?$ /tag/$1 [R,L]

These rewrite Blogger tag URLs to WordPress tag URLs, except for the en, fr and no, which are categories (Blogger Importer imports Blogger tags as WordPress categories; I used the Categories to Tags Converter to convert most of them back to tags).

I use Piwik analytics and have enabled 404 tracking, which should help me identify additional URLs that need to be rewritten.


Edit 2013-02-16: Additional RewriteRules, courtesy of SEO Ultimate‘s 404 Monitor:

RewriteCond %{QUERY_STRING} ^m=1$
RewriteRule ^(.*)$ $1? [R,L]
RewriteRule ^/([0-9]{4})_([0-9]{2})_([0-9]+)_archive.html$ /$1/$2/page/$3 [R,L]
RewriteRule ^/feeds/posts/default/-/(en|fr|no)/?$ /category/$1/feed/ [R,L]
RewriteRule ^/feeds/posts/default/-/([^/]+)/?$ /tag/$1/feed/ [R,L]
RewriteRule ^/feeds/posts/default/?$ /feed/ [R,L]

The first two lines strip a query string that shows up a lot—I’m not sure what it means to Blogger. The third line rewrites archive page URLs. The final three rewrite RSS feed URLs, including feeds for categories and tags.

LinkedIn endorsements

I keep getting email from LinkedIn telling me that someone in my network has endorsed me for this, that or the other skill. My initial reaction when these emails started arriving was surprise, because I knew about LinkedIn’s recommendation system but hadn’t heard about their new endorsement system. After a while, surprise turned to amusement. Allow me to explain why.

Unlike recommendations, where one LinkedIn user actively solicits a recommendation from another, endorsements are purely one-way. Any LinkedIn user can endorse any of their connections for any skill. There is no verification step, no filtering, except that each user can select whether or not to show endorsements on their profile page—but not, to my knowledge which endorsements to show.

Here’s how endorsements work: whenever you view one of your connections’ profile, LinkedIn lists a handful of skills and asks you whether that person has them. You can also type in any skill which isn’t among those LinkedIn preselected. There is a big yellow button that says “Endorse