Monthly Archives: April 2010

Cheap process monitoring (no additional software required)

I have an old system (only the hardware, it runs –current) which reboots itself from time to time (mostly during the daily periodic(8) run, but also during a lot of compiling (portupgrade)). There is no obvious reason (no panic) why it is doing this. It could be that there is some hardware defect, or something else. It is not important enough to get a high enough priority that I try hard to analyze the problem with this machine. The annoying part is, that sometimes after a restart apache does not start. So if this happens, the solution is to login and start the webserver. If the webserver would start each time, nearly nobody would detect the reboot (root gets an EMail on each reboot via an @reboot crontab entry).

My pragmatic solution (for services started via a good rc.d script which has a working status command) is a crontab entry which checks periodically if it is running and which restarts the service if not. As an example for apache and an interval of 10 minutes:

*/10 * * * *    /usr/local/etc/rc.d/apache22 status >/dev/null 2>&1 || /usr/local/etc/rc.d/apache22 restart

For the use case of this service/machine, this is enough. In case of a problem with the service, a mail with the restart output would arrive each time it runs, else only after a reboot for which the service did not restart.

Share/Bookmark

I am a blogger, at last

So, when you become a FreeBSD committer you get a number of privileges. The most obvious one is the ability to commit to the FreeBSD source tree (this should tell you something about my sense of humor), which at the beginning makes you feel a little clumsy. I mean: you’re about to send your first [...]

Qt 4.7 imported in area51

…just to make room for another one. :) Yeah, it can be funny, and useful, to write about development stuff! Here’s the first news: Qt 4.7 Technology Preview has just landed on KDE’s area51. If you’re thinking about helping us, you’re welcome. And you can follow the instructions below. Head to http://wiki.FreeBSD.org/KDE4/QT4 and have some [...]

Interesting projects in the GSoC

I counted 18 projects which are given to FreeBSD in this years GSoC. For 3 of them I have some comments.

Very interesting to me is the project which is named Collective limits on set of processes (a.k.a. jobs). This looks a bit like the Solaris contract/project IDs. If this project results in something which allows the userland to query which PID belongs to which set, than this allows some nice improvement for start scripts. For example at work on Solaris each application is a mix of several projects (apache = “name:web� project, tomcat = “name:app� project, Oracle DB = “name:ora� project). Our management framework (written by a co-worker) allows to easily do something with those projects, a “show� displays the prstat (similar to top) info just for processes which belong to the project, a “kill� sends a kill-signal to all processes of the project, and so on. We could do something similar with our start scripts by declaring a namespace (FreeBSD:base:XXX / FreeBSD:ports:XXX?) and maybe number space (depending on the implementation) as reserved and use it to see if processes which belong to a particular script are still running or kill them or whatever.

The other two projects I want to comment upon here are Complete libpkg and create new pkg tools and Complete Package support in the pkg_install tools and cleanup. Both projects reference libpkg in their description. I hope the mentors of both projects pay some attention to what is going on in the other project to not cause dependencies/clashes between the students.

That I do not mention other projects does not mean that they are not interesting or similar, it is just that I do not have to say something valuable about them…

Share/Bookmark

HOWTO mentor in the GSoC (initial communication with the student)

Every mentor in the GSoC has a different way of handling students. Here is what I do.

The student introduced himself to me as requested by our soc-admins in the initial mail to our students. He looked up in which timezone I am (public info) and presented his timezone (and rough location) to me. That is nice. He also offered different communication channels (basically EMail and IM).

I confirmed what he looked up, and presented what I did in the past GSoC in which I participated so that he has an idea if am new to the game or not. I told him that quick/short questions are better asked via IM, while long explanations or questions are better handled via EMail. I also gave him a rough overview when he can expect quick answers from me and when I am not available.

Following are some questions I asked him, so that I get an impression about what to expect and that I can plan a bit (some of those may already be told in student application, but I prefer to have everything in one place):

  • From when to when do you intent to spend how much time for the GSoC?
  • Any holidays / non-availability planned during the GSoC?
  • Any university-stuff (exams/lessons/…) during this time (the uni has higher priority than the GSoC for Google)?
  • Anything else in parallel of the GSoC (some paid work, taking care about ill (grand-)parents, …)?
  • At what level of knowledge do you see yourself regarding computer-science/programming/OS-concepts (relative to other students and relative to the topic)?
  • How do you want to start about the project (where do you want to start, what do you intent to do… just a quick overview… a bit more than saying “I add Xâ€?, but not as far as copy&paste of code examples)?

More important than that (IMO), is to give an idea what is expected from the student:

  • you have FreeBSD–current installed (on a real PC or in a virtual machine)
  • you give me a report about the status each week (“did nothingâ€? is also a valid report, it gives me the info that you are still alive and did not lose interest in the GSoC)
  • if your schedule changes in a significant way, give me a little notification (e.g. “I can not do anything next weekâ€?)
  • if you spend more than 30 minutes with a problem, prepare an email with the problem description; if this preparation did not solve your problem, send me the mail (if you solve the problem 5 minutes later, no problem, I prefer to get a mail too much than to have you stuck with something for an incredible amount of time)

A mentor does not know everything, off course, so the student should be subscribed to hackers@ and current@, and if there is a specific list which matches good to the project he is working on, then to this mailing list too. This allows the mentor to tell the student to send a mail with the questions to one of those lists without much preparation to receive all answers.

Another helpful resource is the FreeBSD kernel cross-reference. For some people my doxygen generated docs of parts of the FreeBSD kernel may be helpful (put unfortunately not a lot of doxygen-markup is within our source code).

I also told that he shall prepare himself that I will ask him to send a reference to a patch of his work long enough before the GSoC ends to an appropriate mailing list, and that comments from there regarding changes he must or shall do are not something bad, but a way to improve the result and/or his skills.

Share/Bookmark

Mentoring again in the GSoC

Seems that I will actively mentor again in this Google Summer of Code (as opposed to just review the submissions from students and/or acting as a fall-back mentor).

The project I will mentor is the “Make optional kernel subsystems register themselves via sysctl�-one from the FreeBSD ideas page.

The student already got into contact with me and it looks like he is motivated (he is already subscribed to several FreeBSD mailinglists, which is not a requirement we have in our GSoC docs).

Share/Bookmark

DVB-S Live TV on FreeBSD with MythTV 0.23 and webcamd

It's not true anymore that FreeBSD does not support any DVB-S devices. Thanks to the work of Hans Petter Selasky on video4bsd there are now DVB-S/2 devices for USB that just work.

img/articles/mythtv-0.23-DVB-1_small.jpg

The work on MythTV to get this running only took me one evening and was just because nobody compiled mythtv with v4l support lately. It also helped a lot that Jürgen Lock already played with the same device and found and fixes a few things.

So what do you need to do now if you want to build your PVR on FreeBSD?

First you need a USB device for DVB-C/DVB-T/DVB-S/DVB-S2 that is supported by webcamd. I took an Pinnacle PCTV Sat HDTV Pro USB 452e that supports DVB-S2 because I talked to Jürgen Lock and knew he had success with it. I don't know if there is already a list with all devices that work but you could have a look at the Makefile of webcamd in svn to see what drivers and cards should be supported.

Now that you have a supported card follow the instructions on the video4bsd page and build webcamd from the svn repository because the current version in ports is too old already.

At last get the latest mythtv 0.23 port from the call for testers that enables v4l support. This will get committed when they have done their release.

Finally start mythtv-setup and configure your v4l device, scan for some channels and watch live-tv with your brand new PVR solution on FreeBSD!

DVB-S Live TV on FreeBSD with MythTV 0.23 and webcamd

It's not true anymore that FreeBSD does not support any DVB-S devices. Thanks to the work of Hans Petter Selasky on video4bsd there are now DVB-S/2 devices for USB that just work.



The work on MythTV to get this running only took me one evening and was just because nobody compiled mythtv with v4l support lately. It also helped a lot that Jürgen Lock already played with the same device and found and fixes a few things.

So what do you need to do now if you want to build your PVR on FreeBSD?

First you need a USB device for DVB-C/DVB-T/DVB-S/DVB-S2 that is supported by webcamd. I took an Pinnacle PCTV Sat HDTV Pro USB 452e that supports DVB-S2 because I talked to Jürgen Lock and knew he had success with it. I don't know if there is already a list with all devices that work but you could have a look at the Makefile of webcamd in svn to see what drivers and cards should be supported.

Now that you have a supported card follow the instructions on the video4bsd page and build webcamd from the svn repository because the current version in ports is too old already.

At last get the latest mythtv 0.23 port from the call for testers that enables v4l support. This will get committed when they have done their release.
Finally start mythtv-setup and configure your v4l device, scan for some channels and watch live-tv with your brand new PVR solution on FreeBSD!

One-Time-Passwords for Horde/IMP?

I search a way to use one-time-passwords for Horde/IMP on FreeBSD. I do not want to use PAM (local users on the machine). Currently I use the authentication via IMAP4 (link between the IMAP4-server and postfix via MySQL, to have the same PW for sending and receiving), and I expect that not all users of Horde/IMP will use OTP if available, so the problem case is not that easy. I can imagine a solution which tries to authenticate via OTP first, and if it succeeds gets a password for the login to the IMAP4 server. If the OTP-auth fails, it could try the entered password for the login to the IMAP4 server. Migrating existing users to a new solution can be done by telling them to enter the password from the machine of the person doing the migration. The solution needs to automatically login to the IMAP4 server, entering a password for the IMAP4 server after the OTP-login to Horde is not an option.

Oh, yes, sending the passwords over SSL is not an option (that is already the only way to login there). The goals are to have

  • an easy to remember password for an OTP app on the mobile to generate the real password
  • the password expire fast, so that a stolen password does not cause much harm
  • not the same login-password for different services (mail-pw != jabber-pw != user-pw)

Share/Bookmark

One-Time-Passwords for XMPP/Jabber?

I search a way to use one-time-passwords for jabber/XMPP (ejabberd) on FreeBSD. I do not want to use PAM (local users on the machine). Currently I use the internal authentication, and I expect that not all users of the jabber server will use OTP if available, so the problem case is not that easy (migrating existing users to a new solution can be done by changing the password myself and then telling them to change their password, but there needs to be a way to let them change the non-OTP password).

I assume that OTP is not foreseen in the XMPP protocol, so where could I ask to have something like that considered as an extension (if such a place exists at all)?

Oh, yes, sending the passwords over SSL is not an option (that is already the only way to login there). The goals are to have

  • an easy to remember password for an OTP app on the mobile to generate the real password
  • the password expire fast, so that a stolen password does not cause much harm
  • not the same login-password for different services (mail-pw != jabber-pw != user-pw)

Share/Bookmark

Sting – Symphonica in Rosso

Well.. WELL!!! I just bought tickets for Denise and myself to go to Sting on the day before my birthday (15 okt). I like sting very much, I know a lot of songs from him.. and I am pleased about it already.. So if you are going to Arnhem on the 15th of oct and will visit Sting, I would be glad to know, perhaps we can even say hi.. I have standing places in the arena itself, so jumping is a requirement! Denise is already excited as well, or rather she told me “subtle” that she thought that it might be a very good concert :-) :-) so this is really cool!!

Sting – Symphonica in Rosso

Well.. WELL!!! I just bought tickets for Denise and myself to go to Sting on the day before my birthday (15 okt). I like sting very much, I know a lot of songs from him.. and I am pleased about it already.. So if you are going to Arnhem on the 15th of oct and will visit Sting, I would be glad to know, perhaps we can even say hi.. I have standing places in the arena itself, so jumping is a requirement! Denise is already excited as well, or rather she told me “subtle” that she thought that it might be a very good concert

Using multiple CPUs with gzip

gzip is ofcourse an ubiqutous compression program used in unix-like systems everywhere. Unfortunately, like most old utilities, it is completely sequential and single-threaded, which is a shame in a world where it is now becoming hard to find a desktop CPU without at least 4 CPU cores on it.

There is a quick and dirty hack around it: gzip -2 < bigfile.vmdk | gzip -9 > bigfile.vmdk.2gz

Read more...

Using multiple CPUs with gzip

gzip is ofcourse an ubiqutous compression program used in unix-like systems everywhere. Unfortunately, like most old utilities, it is completely sequential and single-threaded, which is a shame in a world where it is now becoming hard to find a desktop CPU without at least 4 CPU cores on it.

There is a quick and dirty hack around it: gzip -2 < bigfile.vmdk | gzip -9 > bigfile.vmdk.2gz

Read more...