Category Archives: hellug

Concatenating Video Files with MPlayer

I found out how to concatenate the parts of a video to a single file with MPlayer. It’s relatively easy, so this is just a mini-post to save it for posterity:

$ cat video_part1.avi video_part2.avi ... > temp.avi
$ mencoder -forceidx -oac copy -ovc copy \
    temp.avi -o final.avi && \
    rm temp.avi

Filed under: Computers, Free software, FreeBSD, Linux, Open source, Software Tagged: Computers, Free software, FreeBSD, hellug, Linux, Open source, Software

Unit Testing Uncovers Bugs

As part of the ‘utility’ library in one of the projects we are using at work, I wrote two small wrappers around strtol() and strtoul(). These two functions support a much more useful error reporting mechanism than the plain atoi() and atol() functions, but getting the error checking right in all the places they are called is a bit boring and cumbersome. This is probably part of the reason why there are still programs out there that use atoi() and atol().

For example here’s how I usually check for errors in calls to the strtol() and strtoul() functions:

char *endp;
long x;

endp = NULL;
errno = 0;
x = strtol(str, &endp, base);
if (errno != 0 || (endp != NULL && *endp != '\0' &&
    (isdigit(*endp) != 0 || isspace(*endp) == 0)))
        /* Return 'endp' if possible. */
        return -1;
/* At this point 'x' contains the parsed value. */

This is a lot of code for parsing a single long value. For one or two input strings it may be ok to repeat the code in the places where the numeric parsing code is needed. For more than a couple of input strings it really feels boring to repeat this code again and again.

When I set out to write the wrapper code for strtol() and strtoul() my goal was to make it very easy to parse input strings. A typical call to the parsing function should be a single line of code; it should be very clear if the parsing attempt succeeded or failed; it should also be possible to get both the parsing success or failure and the numeric value we just parsed; it should also be possible to get hold of the last character we managed to parse, so that strings like "100 200 300" can be parsed efficiently without having to manually find where the textual representation of the first number ends or the second one starts.

That's quite a list of goals for a single function, but the function call style I envisioned looked something like this:

long value;
char *endp = NULL;

if (parselong("0x12345678", &endp, 16, &value) != 0) {
        err(1, "parse error");

The return value of parselong() makes it very clear if the parsing attempt succeeded or failed. A return value of zero means success. Any other return value means failure.

The parsed value is returned through the &value pointer. If the parsing attempt has failed parselong() can leave the value unmodified to avoid inflicting spurious side-effects to its calling code because of a failed attempt to parse an input string.

If the parsing attempt has succeeded, &endp may be set to point right after the last character that was successfully parsed. This is actually part of the documented interface of strtol() and strtoul(), so it comes for free by wrapping these functions.

Finally, parsing a long value is a single function call. It is a lot easier to call the parsing function without having to repeat all the error checking boilerplate at each calling site. It's even easy to "chain" multiple parsing attempts using a style similar to:

long value1, value2, value3;

if (parselong("0x12345678", NULL, 16, &value1) != 0 ||
    parselong("0xdeadbeef", NULL, 16, &value2) != 0 ||
    parselong("0xf00fc0de", NULL, 16, &value3) != 0)
        err(1, "parse error");

Not that this is a good style of reporting errors, but it is possible, just because it's now easy to parse a value and check if it was parsed correctly with a single line of code.

The Unit Tests Fail on Linux

Several months passed after I wrote the initial parselong() and parseulong() functions. In the meantime I had to port the program using them to other platforms. The initial target platform was FreeBSD.

This is a bug that lurked for a few months in the initial code of parselong() until I had to port the function to another platform and started writing unit tests to verify that it works the way I expected it to work on all possible systems. In retrospect I should have started by writing the unit tests, but that's something I can say now because I finally got around to doing it and they did serve a very useful purpose.

When I had to port my 'utility' functions to work on several Linux versions too, I wrote a collection of unit tests for parselong() and parseulong(). The testing framework I used was CUnit because of the way it nicely integrates with plain ANSI C code.

One of the test functions I wrote was supposed to check for failures returned by parselong() for invalid input strings. The bulk of the test function was something like this:

#include "CUnit/Basic.h"

        long value = TEST_VALUE_ULONG_MAGIC;

        CU_ASSERT_EQUAL(parselong("xxx", NULL, 0, &value), -1);

        CU_ASSERT_EQUAL(parselong("+", NULL, 0, &value), -1);

        CU_ASSERT_EQUAL(parselong("-", NULL, 0, &value), -1);
        CU_PASS("parselong() failures for invalid values look ok");

Running the unit tests on FreeBSD seemed to work fine. After all the initial version of the parselong() function had been manually tested with the same input strings earlier.

When I tried running the same test cases on Linux though, they failed. Apparently parselong() was not detecting that strtol() failed to parse the input string "xxx" or any other input strings from the ones tested in the test_parselong_failures() function!

The Bug Uncovered

Adding a couple of debugging printf() calls to parselong() itself showed that on Linux parselong() was returning zero for invalid input strings when strtol() could parse no character at all from the input string.

The initial version of the error checking code for strtol() was similar to:

char *endp;
long x;

endp = NULL;
errno = 0;
x = strtol(str, &endp, base);
if (errno != 0 || (endp != NULL && endp != str && *endp != '\0' &&
    (isdigit(*endp) != 0 || isspace(*endp) == 0)))
        /* Return 'endp' if possible. */
        return -1;
/* At this point 'x' contains the parsed value. */

The highlighted part (endp != str) of the error checking code assumes that strtol() will move the 'endp' pointer at least one character after the start of the input string. Apparently on Linux this is not the case. The strtol() function of Linux does not move 'endp' at all if it cannot parse even a single character of the input string. This seems to be the correct behavior for strtol(), but it was hidden for a while, lurking in the original parselong() code, until I ran the unit tests of the function on Debian GNU/Linux.

The CUnit driver program that I used to run the test cases failed on Linux with error messages like:

  1. test_parselong.c:63  - CU_ASSERT_EQUAL(parselong("xxx", NULL, 0, &value),-1)
  2. test_parselong.c:64  - CU_ASSERT_EQUAL(value, TEST_VALUE_ULONG_MAGIC)
  3. test_parselong.c:66  - CU_ASSERT_EQUAL(parselong("+", NULL, 0, &value), -1)
  4. test_parselong.c:67  - CU_ASSERT_EQUAL(value, TEST_VALUE_ULONG_MAGIC)

The culprit for these test case failures was the assumption that Linux would set errno to a non-zero value for an invalid input string... Apparently, it doesn't. The following small program prints different output on BSD vs. Linux:

$ cat -n strtest.c
     1  #include <errno.h>
     2  #include <limits.h>
     3  #include <stdio.h>
     4  #include <stdlib.h>
     6  int
     7  main(void)
     8  {
     9          long value;
    10          const char *input = "xxx";
    11          char *endp = NULL;
    13          errno = 0;
    14          value = strtol(input, &endp, 0);
    15          printf("str = %p = \"%s\"\n", input, input);
    16          printf("endp = %p \"%s\"\n", endp, endp ? endp : "(null)");
    17          if (endp != NULL) {
    18                  printf("endp[0] = '%c' (%d 0%03o #x%02x)\n",
    19                    *endp, *endp, *endp, *endp);
    20          }
    21          printf("errno = %d\n", errno);
    22          printf("value = %ld 0%lo #x%lx\n", value, value, value);
    23          return EXIT_SUCCESS;
    24  }

On FreeBSD the output of this program includes an errno value of EINVAL:

freebsd$ cc strtest.c
freebsd$ ./a.out
str = 0x8048604 = "xxx"
endp = 0x8048604 "xxx"
endp[0] = 'x' (120 0170 #x78)
errno = 22
value = 0 00 #x0
freebsd$ fgrep 22 /usr/include/sys/errno.h
#define EINVAL          22              /* Invalid argument */

On a recent update of Debian GNU/Linux "testing" the output is slightly different:

debian$ cc strtest.c
debian$ ./a.out
str = 0x8048630 = "xxx"
endp = 0x8048630 "xxx"
endp[0] = 'x' (120 0170 #x78)
errno = 0
value = 0 00 #x0

This means that the only indication we have that the Linux version of strtol() failed to parse some of the input text is the value of 'endp': it's the same as the input string. The error-checking code of the original parselong() wrapper was:

        x = strtol(str, &endp, base);
        if (errno != 0 || (endp != NULL && endp != str && *endp != '\0' &&
            (isdigit(*endp) != 0 || isspace(*endp) == 0)))

But on Linux both of the following are true:

  • errno is not set to a non-zero value.
  • If strtol() could not parse even one input character, endp == str.

This caused parselong() to bypass the error checking code, and try to return a 'valid' result even tough the Linux strtol() version has failed. Hence the failure of the unit tests.

Removing the (endp != str) conditional expression means that the error checking code works equally well on Linux and BSD. The BSD version of strtol() returns a non-zero errno value, triggerring the first part of the error checking code. The Linux version returns an endp pointer that is non-null and fails the '\0' check later on. The new parselong() function is slightly shorter and it passes the unit tests on both BSD and Linux.


There is something thrilling about fixing bugs by removing code. This bug was one of the few cases I've come across during the last couple of months where removing code was an improvement. There's probably a joke about "writing too much code" and the bug-resolving debt each line of new code introduces. I think I'll leave that for another time though.

The most important conclusion of today's bug hunting session was that Unit Testing really does work and it pays back in real, quite tangible ways. Had I not spent a bit of time to think about what the parselong() and parseulong() functions are supposed to do, when they are supposed to fail and how they are allowed to fail, I would not spent the time to write test cases for them. Had I not written the test cases, I wouldn't notice there is a failing test case on Linux. Had I not seen that I wouldn't realize some times the two functions were returning completely bogus results on Linux systems.

The central place the unit testing code has in this story is an important and serious lesson for me:


Filed under: Computers, FreeBSD, GNU/Linux, Linux, Programming, Software Tagged: Computers, FreeBSD, GNU/Linux, hellug, Linux, Programming, Software, testing

Mercurial Clones without a Working Copy

Mercurial repository clones can have two parts:

  1. An .hg/ subdirectory, where all the repository metadata is stored
  2. A "working copy" area, where checked out files may live

The .hg/ subdirectory stores the repository metadata of the specific clone, including the history of all changesets stored in the specific clone, clone-specific hooks and scripts, information about local tags and bookmarks, and so on. This is the only part of a Mercurial repository that is actually mandatory for a functional repository.

The "working copy" area is everything under the clone that is not under the toplevel .hg/ subdirectory of the particular clone. The working area of each Mercurial repository may contain a snapshot of the files stored in the repository: either a clean snapshot, checked out from one of the changesets stored in the repository itself, or a locally modified version of a changeset.

One important detail that may not be apparent from the descriptions above is that:

Even if you have already checked out a particular version, you can delete everything except the .hg/ subdirectory and the Mercurial repository will still function normally.

Clones Without a Working Copy

An example is a good way to demonstrate how a clone still functions as a Mercurial repository without a working copy. Let's assume that you have a tiny repository at /tmp/hgdemo that contains revisions of just a small hello.c program:

% pwd
% hg root
% hg log --style compact
1[tip]   c48ee3a9fd78   2010-01-11 08:33 +0200   keramida
  Use EXIT_SUCCESS instead of hard-coded zero.

0   041227edc91b   2010-01-11 08:32 +0200   keramida
  Add hello.c

% hg manifest tip

You can check-out a copy of the latest file revision of hello.c with the "hg checkout" command:

% hg checkout --clean tip
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
% cat -n hello.c
     1  #include <stdio.h>
     2  #include <stdlib.h>
     4  int
     5  main(void)
     6  {
     7      printf("Hello world\n");
     8      return EXIT_SUCCESS;
     9  }

The repository does not need a checkout to function though. The fact that your working copy has been updated to a particular revision is independent of the way the repository machinery under .hg/ works. So you can remove the source of hello.c and still use the repository to browse the history of the project:

% rm -f hello.c
% hg log --style compact
1[tip]   c48ee3a9fd78   2010-01-11 08:33 +0200   keramida
  Use EXIT_SUCCESS instead of hard-coded zero.

0   041227edc91b   2010-01-11 08:32 +0200   keramida
  Add hello.c


With a clone like this it is still possible to use any Mercurial command that does not require a working copy, e.g. "hg diff" to look at the differences between two arbitrary revisions:

% hg diff -r 0:1
diff -r 041227edc91b -r c48ee3a9fd78 hello.c
--- a/hello.c   Mon Jan 11 08:32:59 2010 +0200
+++ b/hello.c   Mon Jan 11 08:33:28 2010 +0200
@@ -1,8 +1,9 @@
 #include <stdio.h>
+#include <stdlib.h>
     printf("Hello world\n");
-    return 0;
+    return EXIT_SUCCESS;

You can even checkout the "null" revision (a magic revision name which Mercurial treats as "not any revision stored in this repository"):

% hg checkout --clean null
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
% hg identify --id --branch
000000000000 default

When a Mercurial clone has checked out the null revision all the tracked files of the working copy are removed. If the clone does not already contain build-time artifacts you should only see the .hg/ subdirectory when you look at the clone:

% find . -maxdepth 2 -exec /bin/ls -1 -dF {} +

The disk space such clone requires is limited by the size of the history metadata.

Why Would You Want Such a Clone

For a small repository like the one shown in this example, it seems pretty useless to be able to have a Mercurial clone without a working copy. You don't really gain much by deleting the source of a small 9-line C program. The space savings of doing that are quite insignificant.

If you are, however, hosting clones of large repositories in a web server somewhere, stripping the working copy of Mercurial clones may be very handy indeed and it may save you a large part of the disk space you would need to keep working copies around. By "large repository" I mean something like a single clone with several hundreds or thousands of files, or a clone whose working copy requires tens or hundreds of megabytes of data.

The OpenSolaris onnv-gate repository is one of the large repositories that use Mercurial. My own Mercurial-based mirror of the FreeBSD head branch is another example for which I readily have size data. Size information for these two repositories is shown in the table below:

  FreeBSD head/ branch
since 2008-01-01
onnv-gate repository
Tracked files 41.807 44.784
Changesets 15.513 11.462
Size of .hg store 238 MB 292 MB
Size of working copy 385 MB 543 MB

Both of these Mercurial repositories have a moderately large number of files. It's also important that the size of the working copy exceeds the size of the .hg/ repository store in both cases. In the onnv-gate repository of OpenSolaris the working copy needs almost twice as much as the entire history of the project. That's a lot of disk space to carry around in all your local clones of onnv-gate!

If all you are looking for is a local mirror of the project sources — so that you can look at the history of a project, browse the diffs committed over time, search for interesting commit information (e.g. "when was bug 6801336 fixed in OpenSolaris?") — carrying around a full working copy is probably a waste of space. Updating the files of the working copy after every pull operation from the upstream master-repository is a waste of time too.

Posted in Computers, Free software, FreeBSD, Mercurial, Open source, Programming, SCM, Software Tagged: Computers, Free software, FreeBSD, hellug, Mercurial, Open source, Programming, SCM, Software


Earlier tonight, on December 7 2009, a friend and me booked our flight tickets for FOSDEM 2010. I am really excited that I am going to attend another open source & free software conference. It has been a while since I had a chance to meet with other BSD people. The last time was in Milan, in EuroBSDCon 2006. It will certainly be tons of fun to meet in person with other free and open source fans, contributors and developers!


FOSDEM is an open conference, organized every year by volunteers to promote the widespread use of Free Software and Open Source Software. It takes place in the beautiful city of Brussels (Belgium). FOSDEM meetings are recognized as “The best Free Software and Open Source events in Europe”.

This year’s FOSDEM will take place on February 6-7 of 2010. The web page of the conference is already online at Updates about the organization of the conference, transportation tips, accommodation options, the dev rooms and talks available this year, and any other bits that may be useful to attendees are often posted there by the organizing team. So if you are planning to attend, add this link to your bookmarks and keep up with the news until we meet in Brussels.

More Updates

That’s all for now. I will be posting more details about the conference and my trip to Belgium as they become available.

Posted in Computers, Conferences, Free software, FreeBSD, GNU/Linux, Open source, Programming, Software Tagged: Computers, Conferences, Free software, FreeBSD, GNU/Linux, hellug, Open source, Programming, Software

Be Careful With SHELL=/usr/local/bin/bash

One of the things I often do on FreeBSD machines is to install shells/mksh or shells/bash and work most of the time with a bourne-compatible shell.

The default /bin/csh shell is mostly ok for short interactive sessions, but I can't stand its command-syntax for semi-complex looping, iteration or other combined commands. So install bash or mksh and I launch one of them from my login prompt, using something like:

csh# env SHELL=/usr/local/bin/bash /usr/local/bin/bash -l

There's a minor catch with SHELL being set to /usr/local/bin/bash though. The default binary of bash is dynamically linked, and it depends on from the devel/gettext package. This means that if you happen to run a package update command that rebuilds gettext from source, there is a small period between the time the old gettext is uninstalled and the new version is installed that the following are all true:

  • Your current SHELL points to bash
  • The bash binary needs to run, so it (temporarily) fails with a runtime linker error
  • The "configure" script of the gettext sources thinks that configure-time checks can use your current SHELL
  • Boom!...

So you cannot upgrade gettext, and your current shell needs it to run. Any other packages that depend on gettext cannot be upgraded either. Not a very nice corner to paint yourself into...

There are, however, at least three options to recover from a mess like this:

  1. Run a shell that is statically linked version of bash. The shells/bash port of FreeBSD can build a statically linked version of the shell when WITH_STATIC_BASH is set at build-time.
  2. Use /bin/csh or shells/mksh as a temporary shell to reinstall the broken packages, e.g. gettext and any other package that depends on it.
  3. Set CONFIG_SHELL to a shell that works even without gettext (the /bin/sh shell of FreeBSD should work fine for this), or to one that has minimal library dependencies (the /usr/local/bin/mksh shell only depends on on FreeBSD, so it should work fine too).

Posted in Computers, Free software, FreeBSD, Software Tagged: Computers, Free software, FreeBSD, hellug, Software

Using the Condensed DejaVu Font-Family Variant by Default

The DejaVu font family is a very popular font collection for Linux and BSD systems. The font package of DejaVu includes a condensed variant; a variation of the same basic font theme that sports narrower characters.

The difference between the two font variants is very easy to spot when they are displayed side by side. The following image shows a small part of a Firefox window, displaying news articles as part of a Google Reader session:

DejaVu Font Family Variants

The window snapshot on the left of this image shows the normal variant of the DejaVu Sans font family. The right-hand snapshot shows the condensed variant of the font family.

I usually to prefer the condensed variant for the display of text on a computer monitor. The narrower characters, with a height that is slightly larger than the width of each glyph, are more æsthetically pleasing for my eyes. Naturally, this is only a matter of personal preference; the normal variant may look and feel more pleasing to some other person. If your own preference leans towards the normal variant of the font, this article may not be very interesting to you, so it is probably ok if you stop reading now.

If you are one of those people who like the condensed variant of the DejaVu font family more than the normal variant though, by all means, keep reading. The “hack” I am going to describe uses a configuration tweak of the fontconfig package to forcibly replace all instances of the DejaVu Sans and the DejaVu Serif fonts with their condensed version.

The Main Problem with Firefox and Condensed Fonts

An interaction between fontconfig and the way some programs select font variants means that Firefox, OpenOffice and a few other programs cannot display the condensed variant in their GTK+ font selection dialog. Firefox, for example, shows only “DejaVu Sans”:

Firefox Font Selection Dialog

As a result, it is impossible to use the font selection dialog of Firefox to configure the condensed font variant as the default font for web content. This kept annoying me for a while, but not enough to actually do something about it. I always thought it would be much better if we could select either font variant, but kept saying to myself that “this may eventually be fixed”. Since this is a long standing bug that has not been fixed in Firefox or the other programs that exhibit the same misbehavior, I decided this morning to forcefully substitute all instances of DejaVu Sans with DejaVu Sans Condensed and all instances of DejaVu Serif with DejaVu Serif Condensed in my FreeBSD/Gnome desktop.

What Could Work but is Not a Good Idea

One way to do this is, of course, by manually replacing the TrueType font files of the non-condensed font variants with their condensed counterparts (e.g. by logging in as the system administrator and overwriting the non-condensed font files). I didn’t want to go that way. Manually modifying the installed versions of files registered into the package database is a bad idea and very ugly hack, because it will stop working the next time the same package is installed. So I chose to read a bit more about fontconfig and see if I could do the same without all the smelly hackery of overwriting font files.

Using Fontconfig Might be a Better Idea

Fontconfig is a library that enables system-wide and per-user configuration, customization and application access to font files. The personal fontconfig configuration of each user is stored in a file called .fonts.conf, in the home directory of each user. The format of this file is defined by a relatively "simple" XML schema. Some of the options supported by the fontconfig schema are described in the fontconfig manual. The quotes around "simple" are there because once you see a few examples of fontconfig tweaks, it is not very hard to come up with similar configuration tweaks, but the precise format and syntax of all the options supported by the syntax is, alas, not very intuitive.

When the .fonts.conf exists in your home directory it has the following general format:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">

Font options that apply to your personal fontconfig setup go inside the <fontconfig>...</fontconfig> element. There are many sorts of options that can be placed inside this XML element, but it is not the intent of this article to describe all of them. For a full list of the options, you should read the fontconfig manual.

There are only three fontconfig options that we are interested in to install the condensed font tweak: <match>, <test> and <edit>. Using these three, we can forcefully replace all uses of the DejaVu fonts with their condensed versions, by adding the following XML snippet to ~/.fonts.conf:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">

  <match target="pattern">
    <test name="family" qual="any">
      <string>DejaVu Sans</string>
    <edit mode="assign" name="family">
      <string>DejaVu Sans Condensed</string>

  <match target="pattern">
    <test name="family" qual="any">
      <string>DejaVu Serif</string>
    <edit mode="assign" name="family">
      <string>DejaVu Serif Condensed</string>

XML is a very verbose and chatty format, but what these two small snippers of XML configuration do should be easy to understand:

  • When an application asks for a font whose family matches "DejaVu Sans", return a font from the "DejaVu Sans Condensed" variant.
  • When an application asks for a font whose family matches "DejaVu Serif", return a font from the "DejaVu Serif Condensed" variant.

That's it. Now every time an application asks for a font of the DejaVu family, fontconfig will always return a condensed variant of the font. The font-variant replacement is done transparently by fontconfig, so you don't have to configure each application separately, or to install application-specific hacks that will work for one application but fail or be invisible to all other programs running on your desktop.

The downside of this forceful font-variant replacement is that it is now impossible to select the non-condensed variants of the DejaVu fonts. The good thing is that, at least in my case, this is "ok" and I can certainly live with it, because I always prefer the condensed variant for this particular font family.

Posted in Computers, Free software, FreeBSD, GNOME, GNU/Linux, Linux, Open source, Software Tagged: Computers, Free software, FreeBSD, GNOME, GNU/Linux, hellug, Linux, Open source, Software

fts(3) or Avoiding to Reinvent the Wheel

One of the C programs I was working on this weekend had to find all files that satisfy a certain predicate and add them to a list of “pending work”. The first thing that comes to mind is probably a custom opendir(), readdir(), closedir() hack. This is probably ok when one only has these structures, but it also a bit cumbersome. There are various sorts of DIR and dirent structures and recursing down a large path requires manually keeping track of a lot of state.

A small program that uses opendir() and its friends to traverse a hierarchy of files and print their names may look like this:

% cat -n opendir-sample.c
     1  #include <sys/types.h>
     3  #include <sys/stat.h>
     5  #include <assert.h>
     6  #include <dirent.h>
     7  #include <limits.h>
     8  #include <stdio.h>
     9  #include <string.h>
    11  static int      ptree(char *curpath, char * const path);
    13  int
    14  main(int argc, char * const argv[])
    15  {
    16          int k;
    17          int rval;
    19          for (rval = 0, k = 1; k < argc; k++)
    20                  if (ptree(NULL, argv[k]) != 0)
    21                          rval = 1;
    22          return rval;
    23  }
    25  static int
    26  ptree(char *curpath, char * const path)
    27  {
    28          char ep[PATH_MAX];
    29          char p[PATH_MAX];
    30          DIR *dirp;
    31          struct dirent entry;
    32          struct dirent *endp;
    33          struct stat st;
    35          if (curpath != NULL)
    36                  snprintf(ep, sizeof(ep), "%s/%s", curpath, path);
    37          else
    38                  snprintf(ep, sizeof(ep), "%s", path);
    39          if (stat(ep, &st) == -1)
    40                  return -1;
    41          if ((dirp = opendir(ep)) == NULL)
    42                  return -1;
    43          for (;;) {
    44                  endp = NULL;
    45                  if (readdir_r(dirp, &entry, &endp) == -1) {
    46                          closedir(dirp);
    47                          return -1;
    48                  }
    49                  if (endp == NULL)
    50                          break;
    51                  assert(endp == &entry);
    52                  if (strcmp(entry.d_name, ".") == 0 ||
    53                      strcmp(entry.d_name, "..") == 0)
    54                          continue;
    55                  if (curpath != NULL)
    56                          snprintf(ep, sizeof(ep), "%s/%s/%s", curpath,
    57                              path, entry.d_name);
    58                  else
    59                          snprintf(ep, sizeof(ep), "%s/%s", path,
    60                              entry.d_name);
    61                  if (stat(ep, &st) == -1) {
    62                          closedir(dirp);
    63                          return -1;
    64                  }
    65                  if (S_ISREG(st.st_mode) || S_ISDIR(st.st_mode)) {
    66                          printf("%c %s\n", S_ISDIR(st.st_mode) ? 'd' : 'f', ep);
    67                  }
    68                  if (S_ISDIR(st.st_mode) == 0)
    69                          continue;
    70                  if (curpath != NULL)
    71                          snprintf(p, sizeof(p), "%s/%s", curpath, path);
    72                  else
    73                          snprintf(p, sizeof(p), "%s", path);
    74                  snprintf(ep, sizeof(ep), "%s", entry.d_name);
    75                  ptree(p, ep);
    76          }
    77          closedir(dirp);
    78          return 0;
    79  }

With more than 80 lines, this looks a bit too complex for the simple task it does. It has to keep a lot of temporary state information around in the two ep[] and p[] buffers, and all the manual work of setting updating and maintaining this internal state is adding so much noise around the actual printf() statement at line 68 that it is almost too hard to understand what this particular bit of code is supposed to do.

The program still "works", in a way, so if you compile and run it, the expected results come up:

keramida@kobe:/home/keramida$ cc -O2 opendir-sample.c
keramida@kobe:/home/keramida$ ./a.out /tmp
d /tmp/.snap
d /tmp/.X11-unix
d /tmp/.XIM-unix
d /tmp/.ICE-unix
d /tmp/.font-unix
f /tmp/aprtdTjbX
f /tmp/aprEdWP4d
d /tmp/fam-gdm
f /tmp/.X0-lock
d /tmp/fam-keramida
d /tmp/.esd-1000
d /tmp/screens
d /tmp/screens/S-root
d /tmp/screens/S-keramida
d /tmp/emacs1000
f /tmp/a
f /tmp/b
f /tmp/kot
f /tmp/logsort
keramida@kobe:/home/keramida$ ./a.out /tmp /etc/defaults
d /tmp/.snap
d /tmp/.X11-unix
d /tmp/.XIM-unix
d /tmp/.ICE-unix
d /tmp/.font-unix
f /tmp/aprtdTjbX
f /tmp/aprEdWP4d
d /tmp/fam-gdm
f /tmp/.X0-lock
d /tmp/fam-keramida
d /tmp/.esd-1000
d /tmp/screens
d /tmp/screens/S-root
d /tmp/screens/S-keramida
d /tmp/emacs1000
f /tmp/a
f /tmp/b
f /tmp/kot
f /tmp/logsort
f /etc/defaults/rc.conf
f /etc/defaults/bluetooth.device.conf
f /etc/defaults/devfs.rules
f /etc/defaults/periodic.conf

But this program looks "ugly". Fortunately, the BSDs and Linux provide a more elegant interface for traversing file hierarchies: the fts(3) family of functions. A similar program that uses fts(3) to traverse the filesystem hierarchies rooted at the arguments of main() is:

% cat -n fts-sample.c
     1  #include <sys/types.h>
     3  #include <sys/stat.h>
     5  #include <err.h>
     6  #include <fts.h>
     7  #include <stdio.h>
     9  static int      ptree(char * const argv[]);
    11  int
    12  main(int argc, char * const argv[])
    13  {
    14          int rc;
    16          if ((rc = ptree(argv + 1)) != 0)
    17                  rc = 1;
    18          return rc;
    19  }
    21  static int
    22  ptree(char * const argv[])
    23  {
    24          FTS *ftsp;
    25          FTSENT *p, *chp;
    26          int fts_options = FTS_COMFOLLOW | FTS_LOGICAL | FTS_NOCHDIR;
    27          int rval = 0;
    29          if ((ftsp = fts_open(argv, fts_options, NULL)) == NULL) {
    30                  warn("fts_open");
    31                  return -1;
    32          }
    33          /* Initialize ftsp with as many argv[] parts as possible. */
    34          chp = fts_children(ftsp, 0);
    35          if (chp == NULL) {
    36                  return 0;               /* no files to traverse */
    37          }
    38          while ((p = fts_read(ftsp)) != NULL) {
    39                  switch (p->fts_info) {
    40                  case FTS_D:
    41                          printf("d %s\n", p->fts_path);
    42                          break;
    43                  case FTS_F:
    44                          printf("f %s\n", p->fts_path);
    45                          break;
    46                  default:
    47                          break;
    48                  }
    49          }
    50          fts_close(ftsp);
    51          return 0;
    52  }

This version is not particularly smaller; it's only 34-35% smaller in LOC. It is, however, far more elegant and a lot easier to read:

  • By using a higher level interface, the program is shorter and easier to understand.
  • By using simpler constructs in the fts_read() loop, it very obvious what the program does for each file type (file vs. directory).
  • The FTS_COMFOLLOW flag sets up things for following symbolic links in one simple place (something entirely missing from the opendir version).
  • There are no obvious bugs about copying half of a pathname, or forgetting to recurse in some cases, or forgetting to print some directory because of a complex interaction between superfluous bits of code. Simpler is also less prone to bugs in this case.

So the next time you are about to build a filesystem traversal toolset from scratch, you can avoid all the pain (and bugs): use fts(3)! :-)

Posted in Computers, FreeBSD, GNU/Linux, Linux, NetBSD, Open source, OpenBSD, Programming, Software Tagged: Computers, FreeBSD, GNU/Linux, hellug, Linux, NetBSD, Open source, OpenBSD, Programming, Software

What keramida said…

As pleasures go, it is a strange yet somewhat refined one to see a project one has started pick up speed. My fellow translators at the Greek documentation team of FreeBSD have been busy lately, and the result of our collective work is a fairly large number of commits to the “doc-el” repository.

There are now at least four translators actively working on a chapter of their own: Manolis Kiagias, Vaggelis Typaldos, Kyriakos Kentrotis and me. Changesets flow between our repository clones almost every day, and I often find myself pulling patches from two or three places at the same time.

This morning I picked up patches from both Kyriakos and Manolis. Manolis had already integrated with Vaggelis, so pulling from him I also got the translations of Vaggelis. In the meantime, my nightly cron job had finished importing a new snapshot from the official CVS tree, so today’s history graph looks scary:

Greek FreeBSD Translations: Commit History of 28 June 2009

The “surface complexity” of a change history like this may seem scary, but to me it is nothing of the sort. It is, in fact, quite the opposite: something to be proud and happy about, because it shows a lively team, working steadily towards our common goal—a fully translated doc/ tree with a translated, accessible version of the FreeBSD Handbook for Greek users.

It really makes me very happy to see an effort started several years ago gain momentum. My own personal commits are far less than those of the other translators now, and I often find myself in the role of a “patch integrator” instead of actively translating new text. But this is ok, because now we have more people working on the translations so we still get many improvements every day :-)

Posted in Computers, Free software, FreeBSD, Mercurial, Open source, Software Tagged: Computers, Free software, FreeBSD, hellug, Mercurial, Open source, Software

What keramida said…

Looking at the search terms that people used to reach this weblog, I noticed that one of the most popular posts of all time is the “Contributing to FreeBSD” post of Feb 2009.

Search terms for this weblog

This is fantastic! I didn’t realize readers of this weblog would like the particular post so much, but I am extremely pleased you did!

Posted in Computers, Free software, FreeBSD, Open source, Software Tagged: Computers, Free software, FreeBSD, hellug, Open source, Software