Category Archives: Userland

Jumstart/JET for FreeBSD (brainstorming)

There are some HOWTOs out there in the net which describe some automatic network based install via PXE-booting a machine from a server which has a specific FreeBSD release in the PXE-booting area and a non-interactive config for sysinstall to install this FreeBSD version on the machine which PXE-boots this.

The setup of this is completely manual and only allows to netboot one FreeBSD version. The server-side setup for the clients is also completely manual (and only allows to install one client at a time, it seems). This is not very user-friendly, and far away from the power of Jumpstart/JET for Solaris where you create a template (maybe from another template with automatic value (IP, name, MAC) replacement) and can specify different OS releases for different clients and then just run a command to generate a good config for this.

I thought a little bit how it could be done and decided to write down all the stuff (so far 160 lines, 830 words) to not forget some details. All in all I think this could be done (at least a sensible subset) in a week or two (fulltime) if you have the hardware, motivation, and time. As always, the problems are within the details, so I may be off with my estimation a little bit (also depends upon the knowledge-level (shell, tftp, dhcpd, install–software) of the person doing this).

Unfortunately I do not know if I have the hardware at home to do something like this. I have some unused harddisks which could be used in a machine which is used temporary as a test-install-client (normally I use this machines as my Desktop… if I do not use my little Netbook instead, as I do not do much at home currently), but I’ve never checked if this machine is PXE-booting-capable (VIA KT133 chipset with a 3Com 3c905C-TX Fast Etherlink XL). I also do not have the time to do this (with the current rate of free time I would expect to need about a year), except maybe someone would call my boss and negotiate something.

I can not remember any request to have something like this on the freebsd-current, freebsd-arch or freebsd-hackers list since I read them (and that is since about at least 3.0-RELEASE). Is this because nearly nobody is interested in something like this, or are the current possibilities enough for your needs? Do you work at a place where this would be welcome (= directly used when it would be done)? If you use a simple solution to make a net-install, what is your experience with this (pros/cons)?

All internal services migrated to IPv6

In the last days I migrated all my internal services to IPv6.

All my jails have an IPv4 and an IPv6 address now. All Apaches (I have one for my picture gallery, one for webmail, and one for internal management) now listen on the internal IPv6 address too. Squid is updated from 2.x to 3.1 (the most recent version in the Ports Collection) and I added some IPv6 ACLs. The internal Postfix is configured to handle IPv6 too (it is delivering everything via an authenticated and encrypted channel to a machine with a static IPv4 address for final delivery). My MySQL does not need an IPv6 address, as it is only listening to requests via IPC (the socket is hardlinked between jails). All ssh daemons are configured to listen to IPv6 too. The IMAP and CUPS server was picking the new IPv6 addresses automatically. I also updated Samba to handle IPv6, but due to lack of a Windows machine which prefers IPv6 over IPv4 for CIFS access (at least I think my Windows XP netbook only tries IPv4 connections) I can not really test this.

Only my Wii is a little bit behind, and I have not checked if my Sony-TV will DTRT (but for this I first have to get some time to have a look if I have to update my DD-WRT firmware on the little WLAN-router which is “extending the cable

Weather station readout with FreeBSD

A while ago a wind turbine was installed not far away from my place. It is far enough to not disturb us, and it is near enough to notice that it turns a lot (IIRC I have seen it only once not turning).

This triggered a question. How much energy would such a device (smaller of course) produce at my place?

The answer depends upon several factors. The wind speed, the wind direction and the wind-speed-to-power-output curve of the device. If you do not take a device which rotates around the horizontal axis but the vertical axis, the wind direction can be taken out of the question (probably not completely, but to answer my question this simplification should be ok). The output-power curve depends upon the device, and I hope it is easy to get it from the vendors. The remaining open question it the wind speed at my place. Is there enough wind with enough speed?

To answer this question I bought a weather station with an anemometer (wind speed sensor). I searched a little bit until I decided to buy a specific one (actually I bought three of them, some coworkers got interested too but they found only much more expensive ones, so soon there will be three more weather stations in use in Belgium, France and Germany). The main point is, I can connect it to an USB port of a PC and there is some software for Linux to read out the data. It also comes with some other outdoor-sensors (temperature, rain, wind direction, humidity, …) and an indoor-control-unit with some internal sensors (temperature, humidity). The user interface is mainly the touchscreen of the control-unit. There is also some Windows software, which is needed to program the interval in which the measurements are taken and saved in the control-unit.

It seems the weather station is produced by Fine Offset Electronics Co.,Ltd and sold within different brands in different locations. The Linux software can read all of them, as the vendor and product IDs are not changed.

Porting the software was easy, it uses libusb and I just had to correct a little problem for the non-portable functions which are used (I asked about them on usb@ and the response was that they just got implemented upon my request and will be committed to HEAD soon). I made a little patch for the software to only use them when available (if you have not loaded the USB HID driver, you do not need to care about them) and committed it to the Ports Collection as astro/fowsr.

Now I just need to attach the outside sensors at the place where I would put the vertical axis wind turbine, install some toolkit which takes a series of measurements and displays them as a nice graph (while keeping all data values) and write some glue code to feed the output of fowsr to it. After a year I can then calculate how much power a given wind turbine would have produced during the year and calculate the return of investment for it.

The Linux software also references several weather sites, for some of them you can get even an iGoogle widget so that you can view the data from wherever you want (as long as you have a suitable internet connection). I think this is also something I will have a look at later.

Note to users in Europe, the device also comes with a DCF77 receiver. As the time is distributed in UTC+1 (or +2, depending on the daylight saving time), you should adjust the timezone setting accordingly to this, not to plain UTC (so for me the timezone should be ‘0’ for the same timezone).


Periodic scrubbing of ZFS pools

I noticed that we do not have some automatic way of scrubbing a ZFS pool periodically. A quick poll on fs@ revealed, that there is interest in something like this. So I took a little bit of time to write a periodic daily script which checks if the last scrub is X days ago and scrubs a pool accordingly. The script has options to scrub all pools, or just a specific subset. It also allows to specify a time-interval between scrubs for each pool with different levels of fall-back (if no pool-specific interval is set, the default interval is used, which is set to 30 days if no other default interval is specified).

The discussion about this is happening over at fs@, so go there and have a look for the CFT (with a link to the WIP of the script) and the discussion if you are interested.

So far there are some minor details to sort out (and a little bit of documentation to write) before I can commit it… probably next week.


Direct, indirect and explicit dependencies in progams/ports

The discussion about direct and indirect dependencies is coming up again on the FreeBSD mailinglists. Seems I should make some blog post about it, maybe it makes this topic more findable than my postings in the mailinglists.

Some definitions:

  • A direct dependency from A to B is when program/port A uses symbols from library/port B.
  • An indirect dependency from A to C is when program/port A uses symbols from library/port B but no symbols from library/port C, and library/port B uses symbols from library/port C.
  • An explicit dependency from A to C is when it is a direct or indirect dependency A to C, and when the compiler-time-linker added an explicit reference to C to the program/lib of A.

Ideally we have no indirect dependencies in the explicit dependencies, only direct dependencies. Unfortunately in reality we also have indirect dependencies there. This has at least two causes:

  1. libtool (at least 1.x) does not (or was not) come with a hint on FreeBSD, which tells that the run-time-linker is recursively resolving dependencies.
  2. Some pkg-config setups list indirect dependencies as explicit dependencies (IIRC it depends if Requires.private and/or Libs.private is used in the .pc file or not; if it is used, there should be no indirect dependency appear from this software, but I am not 100% sure about this).

Three years ago I wrote /usr/ports/Tools/scripts/, it looks at the files of a given port (it needs to be installed), and prints out explicit dependencies. Because of the indirect dependencies which could be listed there, this list is not a list of ports which are real dependencies from a source code point of view, but it reflects the link-time reality. If a port C shows up there, the port which is checked needs to be rebuild in case the ABI of library/port C changes.


Cheap process monitoring (no additional software required)

I have an old system (only the hardware, it runs –current) which reboots itself from time to time (mostly during the daily periodic(8) run, but also during a lot of compiling (portupgrade)). There is no obvious reason (no panic) why it is doing this. It could be that there is some hardware defect, or something else. It is not important enough to get a high enough priority that I try hard to analyze the problem with this machine. The annoying part is, that sometimes after a restart apache does not start. So if this happens, the solution is to login and start the webserver. If the webserver would start each time, nearly nobody would detect the reboot (root gets an EMail on each reboot via an @reboot crontab entry).

My pragmatic solution (for services started via a good rc.d script which has a working status command) is a crontab entry which checks periodically if it is running and which restarts the service if not. As an example for apache and an interval of 10 minutes:

*/10 * * * *    /usr/local/etc/rc.d/apache22 status >/dev/null 2>&1 || /usr/local/etc/rc.d/apache22 restart

For the use case of this service/machine, this is enough. In case of a problem with the service, a mail with the restart output would arrive each time it runs, else only after a reboot for which the service did not restart.

One-Time-Passwords for Horde/IMP?

I search a way to use one-time-passwords for Horde/IMP on FreeBSD. I do not want to use PAM (local users on the machine). Currently I use the authentication via IMAP4 (link between the IMAP4-server and postfix via MySQL, to have the same PW for sending and receiving), and I expect that not all users of Horde/IMP will use OTP if available, so the problem case is not that easy. I can imagine a solution which tries to authenticate via OTP first, and if it succeeds gets a password for the login to the IMAP4 server. If the OTP-auth fails, it could try the entered password for the login to the IMAP4 server. Migrating existing users to a new solution can be done by telling them to enter the password from the machine of the person doing the migration. The solution needs to automatically login to the IMAP4 server, entering a password for the IMAP4 server after the OTP-login to Horde is not an option.

Oh, yes, sending the passwords over SSL is not an option (that is already the only way to login there). The goals are to have

  • an easy to remember password for an OTP app on the mobile to generate the real password
  • the password expire fast, so that a stolen password does not cause much harm
  • not the same login-password for different services (mail-pw != jabber-pw != user-pw)


One-Time-Passwords for XMPP/Jabber?

I search a way to use one-time-passwords for jabber/XMPP (ejabberd) on FreeBSD. I do not want to use PAM (local users on the machine). Currently I use the internal authentication, and I expect that not all users of the jabber server will use OTP if available, so the problem case is not that easy (migrating existing users to a new solution can be done by changing the password myself and then telling them to change their password, but there needs to be a way to let them change the non-OTP password).

I assume that OTP is not foreseen in the XMPP protocol, so where could I ask to have something like that considered as an extension (if such a place exists at all)?

Oh, yes, sending the passwords over SSL is not an option (that is already the only way to login there). The goals are to have

  • an easy to remember password for an OTP app on the mobile to generate the real password
  • the password expire fast, so that a stolen password does not cause much harm
  • not the same login-password for different services (mail-pw != jabber-pw != user-pw)


Google’s new RE engine

I stumbled over Google’s new RE engine. Unfortunately it is not handling backreferences, so it is not a drop-in replacement for the regular expressions code in FreeBSD. It has a POSIX mode, but this only seems to be enough for the egrep syntax. For people which need backreferences, they refer to the Google Chrome’s RE engine irregexp which in turn references a paper from 2007 which is titled Regular Expression Matching Can Be Simple And Fast.

The techniques in the paper can not be applied to the irregexp engine, but maybe could help to speed up awk, egrep and similar programs.

I think it would be interesting to compare those recent developments to what we have in FreeBSD, and if they are faster, to see if it is possible to improve the FreeBSD implementation based upon them (either by writing new code, or by importing existing code, depending on the corresponding license and the language the code is written in).

Maybe a candidate for the GSoC?