Category Archives: netbsd

New release: ELF Toolchain v0.6.1

I am pleased to announce the availability of version 0.6.1 of the software being developed by the ElfToolChain project.

This new release supports additional operating systems (DragonFly BSD, Minix and OpenBSD), in addition to many bug fixes and documentation improvements.

This release also marks the start of a new "stable" branch, for the convenience of downstream projects interested in using our code.

Comments welcome.

A build system for the Elftoolchain project

Summary

This post describes the cross-platform build and test system being written for the Elftoolchain project.

The need for a cross-platform build and test system


In the upcoming v1.0 release of the Elftoolchain project we plan to support 6 operating system families---prominent *BSD OSes such as FreeBSD and NetBSD, Ubuntu GNU/Linux and Minix. We may additionally support an Illumos-derived operating system.

For each supported OS, we may need to test our sources on multiple released OS versions. We may need to build and test on multiple architectures (e.g., ARM, MIPS, i386 and X64/AMD64), depending on the target operating system.

Apart from mainline development, we would need to support maintenance branches of our source tree on these OS instances.

Clearly, some kind of automation is needed to help manage this combinatorial increase in support load.

Design goals


In brief, the design goals are:
  • The system should support builds of our source tree, on the target operating systems and machine architectures of our interest.
  • The system should support builds on non-native architectures (relative to the build host).
  • The system should allow a source tree that is in-development to be built and tested, prior to a check-in.
  • The system should be deployable with the minimum of software dependencies, and should be easy to configure.
  • The system should be able to run entirely on a relatively power and resource constrained system such as a laptop, i.e., without needing a beefy build box, or architecture-specific hardware.

Related Projects

Continuous integration systems such as Buildbot, Bitten and Hudson are commonly used to manage automated builds. While these tools are featureful, their large resource requirements, and the additional dependencies needed to run them (a Java/Python runtime, along with other dependencies) make these tools difficult to use in the Elftoolchain context.

The QEMU and VirtualBox programs are popular machine emulators. When running on X86/X64 hardware, these programs support the emulation of i386 and x86_64/amd64 CPUs. Additionally, QEMU can emulate non-native architectures using dynamic translation techniques. The GXemul project is a BSD-licensed machine emulator, similar to QEMU.

The Design


In the current design, the build system comprises of two major parts:

  • A simple daemon---a portable C program built using libevent, that runs inside the target OS in the machine emulator. This 'slave' component connects to a 'master/despatcher' component that runs on the build host and executes commands issued to it.
  • A 'master' component that is responsible for managing the build process at the top-level: starting up the relevant machine emulators, waiting for the OS inside to boot and for the 'slave' inside to connect back, transferring the source tree of interest into the slave, running the build/test cycle, collecting output files and output status, and shutting down the emulator cleanly.

Note that the actual build (& test) within a source tree is controlled using BSD make.

The protocol between the 'slave' and the 'master' components is spartan: it supports the execution of arbitrary shell script fragments on the slave with redirection of input and output, and supports simple data transfer between the 'slave' and the 'master'. In order to maintain responsiveness, the protocol between the 'slave' and the 'master' is asynchronous. Multiple 'slaves' could be connected to the 'master' concurrently.

The 'master' would be controlled by a set of configuration files and shell scripts.

The Implementation

The implementation (a work-in-progress) may be found in the tools/build-automation directory of the Elftoolchain project's source tree.

The implementation is being written as a literate program.

Comments welcome.

fts(3) or Avoiding to Reinvent the Wheel

One of the C programs I was working on this weekend had to find all files that satisfy a certain predicate and add them to a list of “pending work”. The first thing that comes to mind is probably a custom opendir(), readdir(), closedir() hack. This is probably ok when one only has these structures, but it also a bit cumbersome. There are various sorts of DIR and dirent structures and recursing down a large path requires manually keeping track of a lot of state.

A small program that uses opendir() and its friends to traverse a hierarchy of files and print their names may look like this:

% cat -n opendir-sample.c
     1  #include <sys/types.h>
     2
     3  #include <sys/stat.h>
     4
     5  #include <assert.h>
     6  #include <dirent.h>
     7  #include <limits.h>
     8  #include <stdio.h>
     9  #include <string.h>
    10
    11  static int      ptree(char *curpath, char * const path);
    12
    13  int
    14  main(int argc, char * const argv[])
    15  {
    16          int k;
    17          int rval;
    18
    19          for (rval = 0, k = 1; k < argc; k++)
    20                  if (ptree(NULL, argv[k]) != 0)
    21                          rval = 1;
    22          return rval;
    23  }
    24
    25  static int
    26  ptree(char *curpath, char * const path)
    27  {
    28          char ep[PATH_MAX];
    29          char p[PATH_MAX];
    30          DIR *dirp;
    31          struct dirent entry;
    32          struct dirent *endp;
    33          struct stat st;
    34
    35          if (curpath != NULL)
    36                  snprintf(ep, sizeof(ep), "%s/%s", curpath, path);
    37          else
    38                  snprintf(ep, sizeof(ep), "%s", path);
    39          if (stat(ep, &st) == -1)
    40                  return -1;
    41          if ((dirp = opendir(ep)) == NULL)
    42                  return -1;
    43          for (;;) {
    44                  endp = NULL;
    45                  if (readdir_r(dirp, &entry, &endp) == -1) {
    46                          closedir(dirp);
    47                          return -1;
    48                  }
    49                  if (endp == NULL)
    50                          break;
    51                  assert(endp == &entry);
    52                  if (strcmp(entry.d_name, ".") == 0 ||
    53                      strcmp(entry.d_name, "..") == 0)
    54                          continue;
    55                  if (curpath != NULL)
    56                          snprintf(ep, sizeof(ep), "%s/%s/%s", curpath,
    57                              path, entry.d_name);
    58                  else
    59                          snprintf(ep, sizeof(ep), "%s/%s", path,
    60                              entry.d_name);
    61                  if (stat(ep, &st) == -1) {
    62                          closedir(dirp);
    63                          return -1;
    64                  }
    65                  if (S_ISREG(st.st_mode) || S_ISDIR(st.st_mode)) {
    66                          printf("%c %s\n", S_ISDIR(st.st_mode) ? 'd' : 'f', ep);
    67                  }
    68                  if (S_ISDIR(st.st_mode) == 0)
    69                          continue;
    70                  if (curpath != NULL)
    71                          snprintf(p, sizeof(p), "%s/%s", curpath, path);
    72                  else
    73                          snprintf(p, sizeof(p), "%s", path);
    74                  snprintf(ep, sizeof(ep), "%s", entry.d_name);
    75                  ptree(p, ep);
    76          }
    77          closedir(dirp);
    78          return 0;
    79  }

With more than 80 lines, this looks a bit too complex for the simple task it does. It has to keep a lot of temporary state information around in the two ep[] and p[] buffers, and all the manual work of setting updating and maintaining this internal state is adding so much noise around the actual printf() statement at line 68 that it is almost too hard to understand what this particular bit of code is supposed to do.

The program still "works", in a way, so if you compile and run it, the expected results come up:

keramida@kobe:/home/keramida$ cc -O2 opendir-sample.c
keramida@kobe:/home/keramida$ ./a.out /tmp
d /tmp/.snap
d /tmp/.X11-unix
d /tmp/.XIM-unix
d /tmp/.ICE-unix
d /tmp/.font-unix
f /tmp/aprtdTjbX
f /tmp/aprEdWP4d
d /tmp/fam-gdm
f /tmp/.X0-lock
d /tmp/fam-keramida
d /tmp/.esd-1000
d /tmp/screens
d /tmp/screens/S-root
d /tmp/screens/S-keramida
d /tmp/emacs1000
f /tmp/a
f /tmp/b
f /tmp/kot
f /tmp/logsort
keramida@kobe:/home/keramida$ ./a.out /tmp /etc/defaults
d /tmp/.snap
d /tmp/.X11-unix
d /tmp/.XIM-unix
d /tmp/.ICE-unix
d /tmp/.font-unix
f /tmp/aprtdTjbX
f /tmp/aprEdWP4d
d /tmp/fam-gdm
f /tmp/.X0-lock
d /tmp/fam-keramida
d /tmp/.esd-1000
d /tmp/screens
d /tmp/screens/S-root
d /tmp/screens/S-keramida
d /tmp/emacs1000
f /tmp/a
f /tmp/b
f /tmp/kot
f /tmp/logsort
f /etc/defaults/rc.conf
f /etc/defaults/bluetooth.device.conf
f /etc/defaults/devfs.rules
f /etc/defaults/periodic.conf

But this program looks "ugly". Fortunately, the BSDs and Linux provide a more elegant interface for traversing file hierarchies: the fts(3) family of functions. A similar program that uses fts(3) to traverse the filesystem hierarchies rooted at the arguments of main() is:

% cat -n fts-sample.c
     1  #include <sys/types.h>
     2
     3  #include <sys/stat.h>
     4
     5  #include <err.h>
     6  #include <fts.h>
     7  #include <stdio.h>
     8
     9  static int      ptree(char * const argv[]);
    10
    11  int
    12  main(int argc, char * const argv[])
    13  {
    14          int rc;
    15
    16          if ((rc = ptree(argv + 1)) != 0)
    17                  rc = 1;
    18          return rc;
    19  }
    20
    21  static int
    22  ptree(char * const argv[])
    23  {
    24          FTS *ftsp;
    25          FTSENT *p, *chp;
    26          int fts_options = FTS_COMFOLLOW | FTS_LOGICAL | FTS_NOCHDIR;
    27          int rval = 0;
    28
    29          if ((ftsp = fts_open(argv, fts_options, NULL)) == NULL) {
    30                  warn("fts_open");
    31                  return -1;
    32          }
    33          /* Initialize ftsp with as many argv[] parts as possible. */
    34          chp = fts_children(ftsp, 0);
    35          if (chp == NULL) {
    36                  return 0;               /* no files to traverse */
    37          }
    38          while ((p = fts_read(ftsp)) != NULL) {
    39                  switch (p->fts_info) {
    40                  case FTS_D:
    41                          printf("d %s\n", p->fts_path);
    42                          break;
    43                  case FTS_F:
    44                          printf("f %s\n", p->fts_path);
    45                          break;
    46                  default:
    47                          break;
    48                  }
    49          }
    50          fts_close(ftsp);
    51          return 0;
    52  }

This version is not particularly smaller; it's only 34-35% smaller in LOC. It is, however, far more elegant and a lot easier to read:

  • By using a higher level interface, the program is shorter and easier to understand.
  • By using simpler constructs in the fts_read() loop, it very obvious what the program does for each file type (file vs. directory).
  • The FTS_COMFOLLOW flag sets up things for following symbolic links in one simple place (something entirely missing from the opendir version).
  • There are no obvious bugs about copying half of a pathname, or forgetting to recurse in some cases, or forgetting to print some directory because of a complex interaction between superfluous bits of code. Simpler is also less prone to bugs in this case.

So the next time you are about to build a filesystem traversal toolset from scratch, you can avoid all the pain (and bugs): use fts(3)! :-)


Posted in Computers, FreeBSD, GNU/Linux, Linux, NetBSD, Open source, OpenBSD, Programming, Software Tagged: Computers, FreeBSD, GNU/Linux, hellug, Linux, NetBSD, Open source, OpenBSD, Programming, Software

Getting online with GPRS in India

In order to get online using GPRS you would need:

  1. A device capable of handling GPRS; usually a cell phone or a GPRS modem. I've used a Nokia 3110c with some success. This particular cellphone model offers a USB port using which it may be connected to a host computer with a compatible USB cable.
  2. A subscription to GPRS service. Contact your cell service provider on the way to enable this on your cell phone. The examples below describe the configuration needed for using AirTel's GPRS service.
  3. The appropriate configuration for your OS. This post describes how to use GPRS using FreeBSD and NetBSD.

FreeBSD Configuration

  1. Ensure that the relevant drivers are present in the kernel. The Nokia 3110c is supported by the stock umodem(4) driver. Use kldload(8) to load this into your kernel if it is not already present:
    # kldload umodem
    
  2. On connecting the cellphone to the computer, you should see a message similar to the following on the console (or in /var/log/messages):
    ... kernel: ucom0: Nokia Nokia 3110c, rev x.yy/a.bb, addr 2, iclass 2/2
    ... kernel: ucom0: data interface 2, has CM over data, has break
    ... kernel: ucom0: status change notification available
    
    If all goes well, a new device entry such as /dev/cuaU0 should be present under /dev.
  3. Add the following to /etc/ppp/ppp.conf (note the leading whitespace on most lines):
    airtel:
     set device /dev/cuaU0
     enable dns
     set phone "*99***1#"
     set authname "airtel"
     set authkey "airtel"
     set dial "ABORT BUSY ABORT NO\\sCARRIER TIMEOUT 5 \
      \"\" AT OK-AT-OK ATQ0V1E1S0=0&C1&D2+FCLASS=0 OK \
      AT+CGDCONT=1,\\\"IP\\\",\\\"airtelgprs.com\\\" OK \\dATDT\\T \
      TIMEOUT 40 CONNECT"
      add default HISADDR
      set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.0 0.0.0.0
    

    For the curious, the ppp(8) manual page fully describes the syntax and semantics of the configuration commands above.

    Note that the contents of the authname and authkey configuration items do not seem to matter. You could put the name of your pet dog there if you wish.

  4. Invoke the ppp(8) daemon in the usual way:
    % ppp airtel
    ppp> dial
    ... the prompt changes as PPP setup progresses
    PPP>
    

    And you should be done.

NetBSD Configuration

In NetBSD as in FreeBSD, this particular cellphone model is supported by the umodem(4) driver. Once recognized, the modem will be accessible using a path name such as /dev/ttyU0.

The following recipe uses the pppd(8) daemon. This recipe requires the NetBSD kernel to have been compiled with in-kernel PPP support.

  1. Create a file /etc/ppp/peers/airtel with the following contents:
    ttyU0 115200
    connect '/usr/sbin/chat -v -f /etc/ppp/chat-airtel -T "*99***1#"'
    defaultroute
    usepeerdns
    ipcp-accept-local
    ipcp-accept-remote
    user "airtel"
    password "airtel"
    
  2. Create a file /etc/ppp/chat-airtel with the following contents:
    ABORT "BUSY"
    ABORT "NO CARRIER"
    ABORT "ERROR"
    ABORT "NO ANSWER"
    "" "AT"
    OK "ATQ0V1E1S0=0&C1&D2+FCLASS=0"
    OK AT+CGDCONT=1,\"IP\",\"airtelgprs.com\"
    OK \dATDT\T
    TIMEOUT 40
    CONNECT
    
  3. Start ppp using:
    # pppd call airtel
    

    Monitor /var/log/messages for error messages.

  4. Once the connection is setup, IP addresses for the ISP's DNS servers would be placed in /etc/ppp/resolv.conf. Add these to /etc/resolv.conf if needed.

Connection performance

In my experience, connection speeds vary widely from a few bytes/second to a peak of a few KB/sec. Connection latencies also tend to be variable, ranging from a few hundred milliseconds upwards. There are frequent long pauses, a minute or more in duration, during which no traffic flows (though the connection remains up).

Thus GPRS based connectivity is best used for batch tasks such as downloading and sending email.

Getting online with GPRS in India

In order to get online using GPRS you would need:

  1. A device capable of handling GPRS; usually a cell phone or a GPRS modem. I've used a Nokia 3110c with some success. This particular cellphone model offers a USB port using which it may be connected to a host computer with a compatible USB cable.
  2. A subscription to GPRS service. Contact your cell service provider on the way to enable this on your cell phone. The examples below describe the configuration needed for using AirTel's GPRS service.
  3. The appropriate configuration for your OS. This post describes how to use GPRS using FreeBSD and NetBSD.

FreeBSD Configuration

  1. Ensure that the relevant drivers are present in the kernel. The Nokia 3110c is supported by the stock umodem(4) driver. Use kldload(8) to load this into your kernel if it is not already present:
    # kldload umodem
    
  2. On connecting the cellphone to the computer, you should see a message similar to the following on the console (or in /var/log/messages):
    ... kernel: ucom0: Nokia Nokia 3110c, rev x.yy/a.bb, addr 2, iclass 2/2
    ... kernel: ucom0: data interface 2, has CM over data, has break
    ... kernel: ucom0: status change notification available
    
    If all goes well, a new device entry such as /dev/cuaU0 should be present under /dev.
  3. Add the following to /etc/ppp/ppp.conf (note the leading whitespace on most lines):
    airtel:
     set device /dev/cuaU0
     enable dns
     set phone "*99***1#"
     set authname "airtel"
     set authkey "airtel"
     set dial "ABORT BUSY ABORT NO\\sCARRIER TIMEOUT 5 \
      \"\" AT OK-AT-OK ATQ0V1E1S0=0&C1&D2+FCLASS=0 OK \
      AT+CGDCONT=1,\\\"IP\\\",\\\"airtelgprs.com\\\" OK \\dATDT\\T \
      TIMEOUT 40 CONNECT"
      add default HISADDR
      set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.0 0.0.0.0
    

    For the curious, the ppp(8) manual page fully describes the syntax and semantics of the configuration commands above.

    Note that the contents of the authname and authkey configuration items do not seem to matter. You could put the name of your pet dog there if you wish.

  4. Invoke the ppp(8) daemon in the usual way:
    % ppp airtel
    ppp> dial
    ... the prompt changes as PPP setup progresses
    PPP>
    

    And you should be done.

NetBSD Configuration

In NetBSD as in FreeBSD, this particular cellphone model is supported by the umodem(4) driver. Once recognized, the modem will be accessible using a path name such as /dev/ttyU0.

The following recipe uses the pppd(8) daemon. This recipe requires the NetBSD kernel to have been compiled with in-kernel PPP support.

  1. Create a file /etc/ppp/peers/airtel with the following contents:
    ttyU0 115200
    connect '/usr/sbin/chat -v -f /etc/ppp/chat-airtel -T "*99***1#"'
    defaultroute
    usepeerdns
    ipcp-accept-local
    ipcp-accept-remote
    user "airtel"
    password "airtel"
    
  2. Create a file /etc/ppp/chat-airtel with the following contents:
    ABORT "BUSY"
    ABORT "NO CARRIER"
    ABORT "ERROR"
    ABORT "NO ANSWER"
    "" "AT"
    OK "ATQ0V1E1S0=0&C1&D2+FCLASS=0"
    OK AT+CGDCONT=1,\"IP\",\"airtelgprs.com\"
    OK \dATDT\T
    TIMEOUT 40
    CONNECT
    
  3. Start ppp using:
    # pppd call airtel
    

    Monitor /var/log/messages for error messages.

  4. Once the connection is setup, IP addresses for the ISP's DNS servers would be placed in /etc/ppp/resolv.conf. Add these to /etc/resolv.conf if needed.

Connection performance

In my experience, connection speeds vary widely from a few bytes/second to a peak of a few KB/sec. Connection latencies also tend to be variable, ranging from a few hundred milliseconds upwards. There are frequent long pauses, a minute or more in duration, during which no traffic flows (though the connection remains up).

Thus GPRS based connectivity is best used for batch tasks such as downloading and sending email.