Category Archives: Software

Gaining space on Android after ART->Dalvik switch (root access required)

I (still) use a Nexus S phone. I am using Cyanogenmod on it. After an article in a computer magazine I decided to give the ART-runtime a try instead of the default Dalvic-runtime. Unfortunately I do not have enough free space free (and all what I can is moved to the USB storage already) to really use the ART-runtime.

After switching back to the Dalvic-runtime, I had only 2/3 of the previously available space free. After a little bit of looking around I found /data/dalvik-cache. I deleted with a file manager the content of the directory (you will get some “app crashed/died” messages) and rebooted the phone (this is not the same as formatting the cache partition in the recovery system).

During boot it populated the directory again and now I have more than 4/3 of free space on the internal storage.

Share/Save

Lin­ux­u­la­tor explained: How to create Linux binaries on FreeBSD

There may by cases where you want to generate a Linux binary on a FreeBSD machine. This is not a problem with the linuxulator, but not with the default linux_base port.

As you may know, the linux_base port is designed to deliver an integrated experience with FreeBSD native programs. As such some parts of the native FreeBSD infrastructure is used. If you would try to use a Linux–compiler to generate Linux–binaries, you would run into the problem that by default the FreeBSD includes are used.

Prerequisites

To have a fully featured and non-integrated Linux environment on your FreeBSD system either mount an existing (and compatible) Linux installation somewhere into your FreeBSD system, or install a linux_dist port. This can be done additionally to an already installed linux_base port.

Preparation

When you have a complete Linux environment available, you need to mount the FreeBSD devfs to /path/to/complete_linux/dev, linprocfs to /path/to/complete_linux/proc and linsysfs to /path/to/complete_linux/sys to have a complete setup.

Use it

Now you just need to chroot into this  /path/to/complete_linux and you configure/make/install or whatever you need to do to generate your desired Linux binary.

Bookmark/FavoritesShare/Save

Powerful Regular Expressions Combined with Lisp in Emacs

Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).

Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.

An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:

----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
     time     |usr sys idl wai hiq siq| recv  send| read  writ|util
16-05 08:36:15|  2   3  96   0   0   0|  66B  178B|   0     0 |   0
16-05 08:36:16| 42  14  37   0   0   7|  92M 1268k|   0     0 |   0
16-05 08:36:17| 45  11  36   0   0   7|  76M 1135k|   0     0 |   0
16-05 08:36:18| 27  55   8   0   0  11|  67M  754k|   0    99M|79.6
16-05 08:36:19| 29  41  16   5   0  10| 113M 2079k|4096B   63M|59.6
16-05 08:36:20| 28  48  12   4   0   8|  58M  397k|   0    95M|76.0
16-05 08:36:21| 38  37  14   1   0  10| 114M 2620k|4096B   52M|23.2
16-05 08:36:22| 37  54   0   1   0   8|  76M 1506k|8192B   76M|33.6

So if you want to graph one of the columns, it's useful to convert all the numbers in the same unit. Bytes would be nice in this case.

Separating all columns with '|' characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. 'C-x r t' can do that in Emacs, and you end up with something like this:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66B| 178B|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1268k|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1135k|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 754k|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2079k|4096B|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 397k|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2620k|4096B|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1506k|8192B|  76M|33.6|

The leading and trailing '|' characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.

What we would like to do is convert text like "408B" to just "408", text like "1268k" to the value of (1268 * 1024), and finally text like "67M" to the value of (67 * 1024 * 1024). The first part is easy:

M-x replace-regexp RET \([0-9]+\)B RET \1 RET

This should just strip the "B" suffix from byte values.

For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like "replace \1 with the value of (expression \1)". This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the "real" substitution text.

So if we match all numeric values that are suffixed by 'k', we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:

\,(* 1024 (string-to-number \1))

The full Emacs command would then become:

M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET

This, and the byte suffix removal, yield now the following text in our Emacs buffer:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 772096|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2128896|4096|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 406528|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2682880|4096|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1542144|8192|  76M|33.6|

Note: Some of the columns are indeed not aligned very well. We'll fix that later. On to the megabyte conversion:

M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET

Which produces a version that has no suffixes at all:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  96468992|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  79691776|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  70254592| 772096|   0 |  103809024|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 118489088|2128896|4096|  66060288|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  60817408| 406528|   0 |  99614720|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 119537664|2682880|4096|  54525952|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  79691776|1542144|8192|  79691776|33.6|

Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type "C-c C-c" with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:

| time           | cpu | cpu | cpu | cpu | cpu | cpu |      eth0 |    eth0 |  disk |      disk | sda- |
| time           | usr | sys | idl | wai | hiq | siq |      recv |    send |  read |      writ | util |
| 16-05 08:36:15 |   2 |   3 |  96 |   0 |   0 |   0 |        66 |     178 |     0 |         0 |    0 |
| 16-05 08:36:16 |  42 |  14 |  37 |   0 |   0 |   7 |  96468992 | 1298432 |     0 |         0 |    0 |
| 16-05 08:36:17 |  45 |  11 |  36 |   0 |   0 |   7 |  79691776 | 1162240 |     0 |         0 |    0 |
| 16-05 08:36:18 |  27 |  55 |   8 |   0 |   0 |  11 |  70254592 |  772096 |     0 | 103809024 | 79.6 |
| 16-05 08:36:19 |  29 |  41 |  16 |   5 |   0 |  10 | 118489088 | 2128896 |  4096 |  66060288 | 59.6 |
| 16-05 08:36:20 |  28 |  48 |  12 |   4 |   0 |   8 |  60817408 |  406528 |     0 |  99614720 | 76.0 |
| 16-05 08:36:21 |  38 |  37 |  14 |   1 |   0 |  10 | 119537664 | 2682880 |  4096 |  54525952 | 23.2 |
| 16-05 08:36:22 |  37 |  54 |   0 |   1 |   0 |   8 |  79691776 | 1542144 |  8192 |  79691776 | 33.6 |

The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2 ... \9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.

Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.


Filed under: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Lisp, Open source, Programming, Software Tagged: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Lisp, Open source, Programming, Software

Fixing Shifted-Arrow Keys in 256-Color Terminals on Linux

The terminfo entry for “xterm-256color" that ships by default as part of ncurses-base on Debian Linux and its derivatives is a bit annoying. In particular, shifted up-arrow key presses work fine in some programs, but fail in others. It's a bit of a gamble if Shift-Up works in joe, pico, vim, emacs, mutt, slrn, or what have you.

THis afternoon I got bored enough of losing my selected region in Emacs, because I forgot that I was typing in a terminal launched by a Linux desktop. SO I thought "what the heck... let's give the FreeBSD termcap entry for xterm-256color a try":

keramida> scp bsd:/etc/termcap /tmp/termcap-bsd
keramida> captoinfo -e $(                                  \
  echo $( grep '^xterm' termcap | sed -e 's/[:|].*//' ) |  \
  sed -e 's/ /,/g'                                         \
  ) /tmp/termcap  > /tmp/terminfo.src
keramida> tic /tmp/terminfo.src

Restarted my terminal, and quite unsurprisingly, the problem of Shift-Up keys was gone.

The broken xterm-256color terminfo entry from /lib/terminfo/x/xterm-256color is now shadowed by ~/.terminfo/x/xterm-256color, and I can happily keep typing without having to worry about losing mental state because of this annoying little misfeature of Linux terminfo entries.

The official terminfo database sources[1], also work fine. So now I think some extra digging is required to see what ncurses-base ships with. There's definitely something broken in the terminfo entry of ncurses-base, but it will be nice to know which terminal capabilities the Linux package botched.

Notes:
[1] http://invisible-island.net/ncurses/ncurses.faq.html#which_terminfo


Filed under: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Linux, Open source, Software Tagged: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Linux, Open source, Software

Status crypto cards HOWTO: problems with the card reader (support could be better)

After hours (spread over weeks) I come to the conclusion that there is a lot of potential to improve the documentation of card readers (but I doubt the card reader vendors will do it) and of the pcsc documentation. It is not easy to arrive at a point where you understand everything. The compatibility list does not help much, as the card readers are partly past their end of life and the models which replace them are not listed. Respectively the one I bought does not support all the features I need. I even ported the driver to FreeBSD (not committed, I wanted to test everything first) and a lot of stuff works, but one critical part is that I can not store a certificate on the crypto card as the card reader or the driver  does not support extended APDUs (needed to transfer more than 255 bytes to the card reader).

Well, the status so far:

  • I have a HOWTO what to install to use crypto cards in FreeBSD
  • I have a HOWOT what to install / configure in Windows
  • I have a HOWTO regarding creating keys on a openpgp v2 card and how to use this key with ssh on FreeBSD (or any other unix-like OS which can run pcsc)
  • I have a card reader which does not support extended APDUs
  • I want to make sure what I write in the HOWTOs is also suitable for the use with Windows / PuTTY
  • it seems Windows needs a certificate and not only a key when using the Windows CAPI (using the vendor supplied card reader driver) in PuTTY-CSC (works at work with a USB token)
  • the pcsc pkcs11 Windows DLL is not suitable yet for use on Windows 8 64bit
  • I contacted the card reader vendor if the card reader or the driver is the problem regarding the extended APDUs
  • I found problems in gpg4win / pcsc on Windows 8
  • I have send some money to the developers of gpg4win to support their work (if you use gnupg on Windows, try to send a few units of money to them, the work stagnated as they need to spend their time for paid work)

So either I need a new card reader, or have to wait for an update of the linux driver of the vendor… which probably means it may be a lot faster to buy a new card reader. When looking for one with at least a PIN pad, I either do not find anything which is listed as supported by pcsc on the vendor pages (it is incredible how hard it is to navigate the websites of some companies… a lot of buzzwords but no way to get to the real products), or they only list updated models where I do not know if they will work.

When I have something which works with FreeBSD and Windows, I will publish all the HOWTOs here at once.

Bookmark/FavoritesShare/Save

Mutt-like Scrolling for Gnus

Mutt scrolls the index of email folders up or down, one line at a time, with the press of a single key: ‘<’ or ‘>’. This is a very convenient way to skim through email folder listings, so I wrote a small bit of Emacs Lisp to do the same in Gnus tonight.

;;;
;; Scrolling like mutt for group, summary, and article buffers.
;;
;; Being able to scroll the current buffer view by one line with a
;; single key, rather than having to guess a random number and recenter
;; with `C-u NUM C-l' is _very_ convenient.  Mutt binds scrolling by one
;; line to '<' and '>', and it's something I often miss when working
;; with Gnus buffers.  Thanks to the practically infinite customizability
;; of Gnus, this doesn't have to be an annoyance anymore.

(defun keramida-mutt-like-scrolling ()
  "Set up '<' and '>' keys to scroll down/up one line, like mutt."
  ;; mutt-like scrolling of summary buffers with '<' and '>' keys.
  (local-set-key (kbd ">") 'scroll-up-line)
  (local-set-key (kbd "<") 'scroll-down-line))

(add-hook 'gnus-group-mode-hook 'keramida-mutt-like-scrolling)
(add-hook 'gnus-summary-mode-hook 'keramida-mutt-like-scrolling)
(add-hook 'gnus-article-prepare-hook 'keramida-mutt-like-scrolling)

This is now the latest addition to my ~/.gnus startup code, and we're one step closer to making Gnus behave like my favorite old-time mailer.


Filed under: Computers, Emacs, Email, Free software, FreeBSD, GNU/Linux, Gnus, Linux, Open source, Programming, Software Tagged: Computers, Emacs, Email, Free software, FreeBSD, GNU/Linux, Gnus, Linux, Open source, Programming, Software

OpenPGP crypto cards ordered

I wrote in a previous blog post that I want to switch to crypto cards for use with ssh and GnuPG. After some research I settled on the OpenPGP cryto cards. I ordered them from kernelconcepts. As soon as they arrive (and I have some free time), I will start to use them and write down how to work with them with FreeBSD.

Bookmark/FavoritesShare/Save

Calculating the target size of H264 videos

From time to time I convert videos to H264. When I do this I want to get the best quality out of a give filesize. This means I create VBR videos. The question here is, how big the target video file shall be.

After searching around a little bit in the net I found a formula which is supposed to give a hint about the target filesize. Naturally this depends on the encoder (or even the encoder-version), the encoder settings and even the video.

The formula is at least a good hint for my use, so I wrote a script which calculates several filesize values for a given video (based upon the output of mediainfo, which the scripts expects in a file with the filename as an argument to the script). It calculates a CBR and a VBR value for a given video based upon the width, height and duration. It should work on all system with a POSIX compatible shell.

Example output for a video from my HD-ready cam, original filesize 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 2
Per second: 6451200.000 bps / 6300 Kibps
Total CBR: 1148313600 bytes / 1121400 KiB / 1095 MiB
Total VBR: 861235200 bytes / 841050 KiB / 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 3
Per second: 9676800.000 bps / 9450 Kibps
Total CBR: 1722470400 bytes / 1682100 KiB / 1642 MiB
Total VBR: 1291852800 bytes / 1261575 KiB / 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 4
Per second: 12902400.000 bps / 12600 Kibps
Total CBR: 2296627200 bytes / 2242800 KiB / 2190 MiB
Total VBR: 1722470400 bytes / 1682100 KiB / 1642 MiB

There are 3 sections, the difference is the “motion” value. It is a kind of multiplicator depending on the amount of motion in the video. For the videos I made myself (family videos, and even some videos of volley ball games), the first section seems to be just fine. So I reduced the original MP4 file to about 50% (not visible here is the audio size, normally I copy the original audio unmodified).

For the curious ones, the formula is

width_in_pixels * height_in_pixels * fps * motion_value * 0.07

for the bps value. The CBR value is

bps * playtime_in_seconds / 8

and the VBR value is

3 / 4 * CBR_value.

Bookmark/FavoritesShare/Save

Which crypto card to use with FreeBSD (ssh/gpg)

The recent security incident triggered a discussion how to secure ssh/gpg keys.

One way I want to focus on here (because it is the way I want to use at home), is to store the keys on a crypto card. I did some research for suitable crypto cards and found one which is called Feitian PKI Smartcard, and one which is called OpenPGP card. The OpenPGP card also exists in a USB version (basically a small version of the card is already integrated into a small USB card reader).

The Feitian card is reported to be able to handle RSA keys upto 2048 bits. They do not seem to handle DSA (or ECDSA) keys. The smartcard quick starter guide they have  (the Tuning smartcard file system part) tells how to change the parameters of the card to store upto 9 keys on it.

The spec of the OpenPGP card tells that it supports RSA keys upto 3072 bits, but there are reports that it is able to handle RSA keys upto 4096 bits (you need to have at least GPG 2.0.18 to handle that big keys on the crypto card). It looks to me like the card is not handle DSA (or ECDSA) cards. There are only slots for upto 3 keys on it.

If I go this way, I would also need a card reader. It seems a class 3 one (hardware PIN pad and display) would be the most “future-proof” way to go ahead. I found a Reiner SCT cyberJack secoder card reader, which is believed to be supported by OpenSC and seems to be a good balance between cost and features of the Reiner SCT card readers.

If anyone reading this can suggest a better crypto card (keys upto 4096 bits, more than 3 slots, and/or DSA/ECDSA  support), or a better card reader, or has any practical experience with any of those components on FreeBSD, please add a comment.

Bookmark/FavoritesShare/Save

Book review: FreeBSD Device Drivers

In mid-April a woman from the marketing department of No Starch Press contacted me and asked if I am interested to do a public review of the FreeBSD Device Drivers book by Joseph Kong (no link to a book shop, go and have a look in your preferred one). Just this simple question, no strings attached.

I had my nose in some device drivers in the past, but I never wrote one, and never had a look at the big picture. I was interested to know how everything fits together, so this made me a good victim for a review (novice enough to learn something new and to have a look if enough is explained, and experienced enough to understand what is going on in the FreeBSD kernel).

Some minutes after I agreed to review it (but with a little notice that I do not know how long I need to review it), I had the PDF version of the book. That was faster than I expected (maybe I am too old-school and used to have paper versions of books in my hands).

Let the review begin… but bear with me, this is the first time I do a real public review of a book (instead of a technical review for an author). And as this is my very own personal opinion, I will not allow comments here. This page is all about my opinion while reading the book, questions I have while reading the book shall serve as a hint about the quality of the book and they should be answered in the book, not here.

In short, the book is not perfect, but it is a good book. There is room for improvement, but on a very high level. If you want to write a device driver for FreeBSD, this book is a must. I suggest to read it completely, even chapters which do not belong to the type of driver you want to write (specially the case studies of real drivers). The reason is that each chapter has some notes which may not only apply to the chapter in question, but to all kinds of device drivers. The long review follows now.

The first chapter is titled “Building and running modules”. The author begins with description of the usual device driver types (NIC driver, pseudo-device, …) and how they can be added to the kernel (statically linked in or as a module). The first code example is a small and easy kernel module, so that we do not have to reboot the system we use to develop a driver (except we make a fault during driver development which causes the machine to panic or hang). Every part of the example is well explained. This is followed by an overview about character devices (e.g. disks) and a simple character-device driver (so far a pseudo-device, as we do not have real hardware we access) which is not only as-well explained as the module-example, but there is also a note where the code was simplified and what should be done instead.

After reading this chapter you should be able to write your own kernel module in 5 minutes (well, after 5 minutes it will not be able to do a lot — just a “hello world” – but at least you can already load/unload/execute some code into/from/in the kernel).

I have not tried any example myself, but I compiled a lot of modules and drivers I modified in the past and remember to have seen the described parts.

The second chapter explains how to allocate and free memory in the kernel. There is the possibility to allocate maybe-contiguous memory (the normal case, when your hardware does not do DMA or does not have the requirement that the memory region it makes DMA from/too needs to be contiguous), and really contiguous. For the size argument of the freeing of the the contiguous memory there is the sentence “Generally, size should be equal the amount allocated.”. Immediately I wanted to know what happens if you specify a different size (as a non-native english speaker I understand this sentence in a way that I am allowed to specify a different size and as such are able to free only parts of the allocated memory). Unfortunately this is not answered. I had a look into the source, the kernel frees memory pages, so the size argument (and addr argument) will be rounded to include a full page. This means theoretically I am able to free parts of the allocated memory, but this is a source-maintenance nightmare (needs knowledge about the machine specific page boundaries and you need to make sure that you do the absolutely correct size calculations).  To me this looks more like as long as nobody is pointing a gun at my head and tells me to use a different size, specifying the same size as made during the allocation of this memory region is the way to go.

After reading this chapter you should know how to kill the system by allocating all the RAM in the kernel.

Again, I did not try to compile the examples in this chapter, but the difference of the memory allocation in the kernel compared with memory allocation in the userland is not that big.

The third chapter explains the device communication and control interfaces (ioctl/sysctl) of a driver. The ioctl part teached me some parts I always wanted to know when I touched some ioctls, but never bothered to find out before. Unfortunately this makes me a little bit nervous about the way ioctls are handled in the FreeBSD linuxulator, but this is not urgent ATM (and can probably be handled by a commend in the right place). The sysctl part takes a little bit longer to follow through, but there is also more to learn about it. If you just modify an existing driver with an existing sysctl interface, it probably just comes down to copy&paste with little modifications, but if you need to make more complex changes or want to add a sysctl interface to a driver, this part of the book is a good way to understand what is possible and how everything fits together. Personally I would have wished for a more detailed guide when to pick the ioctl interface and when the sysctl interface than what was written in the conclusion of the chapter, but it is probably not that easy to come up with a good list which fits most drivers.

After reading this chapter you should be able to get data in and out of the kernel in 10 minutes.

As before, I did not compile the examples in this chapter. I already added ioctls and sysctls in various places in the FreeBSD kernel.

Chapter 4 is about thread synchronization – mutexes, shared/exclusive locks, reader/writer locks and condition variables. For me this chapter is not as good as the previous ones. While I got a good explanation of everything, I missed a nice overview table which compares the various methods of thread synchronization. Brendan Gregg did a nice table to give an overview of DTrace variable types and when to use them. Something like this would have been nice in this chapter too. Apart from this I got all the info I need (but hey, I already wrote a NFS client for an experimental computer with more than 200000 CPUs in 1998, so I’m familiar with such synchronization primitives).

Delayed execution is explained in chapter 5. Most of the information presented there was new to me. While there where not much examples presented (there will be some in a later chapter), I got a good overview about what exists. This time there was even an overview when to use which type of delayed execution infrastructure. I would have preferred to have this overview in the beginning of the chapter, but that is maybe some kind of personal preference.

In chapter 6 a complete device driver is dissected. It is the virtual null modem terminal driver. The chapter provides real-world examples of event-handlers, callouts and taskqueues which where not demonstrated in chapter five. At the same time the chapter serves as a description of the functions a TTY driver needs to have.

Automated device detection with Newbus and the corresponding resource allocation (I/O ports, device memory and interrupts) are explained in chapter 7. It is easy… if you have a real device to play with. Unfortunately the chapter missed a paragraph or two about the suspend and resume methods. If you think about it, it is not hard to come up with what they are supposed to do, but a little explicit description of what they shall do, in what state the hardware should be put and what to assume when being called would have been nice.

Chapter 8 is about interrupts. It is easy to add an interrupt handler (or to remove one), the hard part is to generate an interrupt. The example code uses the parallel port, and the chapter also contains a little explanation how to generate an interrupt… if you are not afraid to touch real hardware (the parallel port) with a resistor.

In chapter 9 the lpt(4) driver is explained, as most of the topics discussed so far are used inside. The explanation how everything is used is good, but what I miss sometimes is why they are used. The most prominent (and only) example here for me is why are callouts used to catch stray interrupts? That callouts are a good way of handling this is clear to me, the big question is why can there be stray interrupts. Can this happen only for the parallel port (respectively a limited amount of devices), or does every driver for real interrupt driven hardware need to come with something like this? I assume this is something specific to the device, but a little explanation regarding this would have been nice.

Accessing I/O ports and I/O memory for devices are explained in chapter 10 based upon a driver for a LED device (turn on and off 2 LEDs on an ISA bus). All the functions to read and write data are well explained, just the part about the memory barrier is a little bit short. It is not clear why the CPU reordering of memory accesses matter to what looks like function calls. Those function calls may be macros, but this is not explained in the text. Some little examples when to use the barriers instead of an abstract description would also have been nice at this point.

Chapter 11 is similar to chapter 10, just that a PCI bus driver is discussed instead of an ISA bus driver. The differences are not that big, but important.

In chapter 12 it is explained how to do DMA in a driver. This part is not easy to understand. I would have wanted to have more examples and explanations of the DMA tag and DMA map parts. I am also surprised to see different supported architectures for the flags BUS_DMA_COHERENT and BUS_DMA_NOCACHE for different functions. Either this means FreeBSD is not coherent in those parts, or it is a bug in the book, or it is supposed to be like this and the reasons are not explained in the book. As there is no explicit note about this, it probably leads to confusion of readers which pay enough attention here. It would also have been nice to have an explanation when to use those flags which are only implemented on a subset of the architectures FreeBSD supports. Anyway, the explanations give enough information to understand what is going on and to be able to have a look at other device drivers for real-live examples and to get a deeper understanding of this topic.

Disk drivers and block I/O (bio) requests are described in chapter 13. With this chapter I have a little problem. The author used the word “undefined” in several places where I as a non-native speaker would have used “not set” or “set to 0″. The word “undefined” implies for me that there may be garbage inside, whereas from a technical point of view I can not imagine that some random value in those places would have the desired result. In my opinion each such place is obvious, so I do not expect that an experienced programmer would lose time/hairs/sanity over it, but inexperienced programmers which try to assemble the corresponding structures on the (uninitialized) heap (for whatever reason), may struggle with this.

Chapter 14 is about the CAM layer. While the previous chapter showed how to write a driver for a disk device, chapter 14 gave an overview about how to an HBA to the CAM layer. It is just an overview, it looks like CAM needs a book on its own to be fully described. The simple (and most important) cases are described, with the hardware-specific parts being an exercise for the person writing the device driver. I have the impression it gives enough details to let someone with hardware (or protocol), and more importantly documentation for this device, start writing a driver.

It would have been nice if chapter 13 and 14 would have had a little schematic which describes at which level of the kernel-subsystems the corresponding driver sits. And while I am at it, a schematic with all the driver components discussed in this book at the beginning as an overview, or in the end as an annex, would be great too.

An overview of USB drivers is given in chapter 15 with the USB printer driver as an example for the explanation of the USB driver interfaces. If USB would not be as complex as it is, it would be a nice chapter to start driver-writing experiments (due to the availability of various USB devices). Well… bad luck for curious people. BTW, the author gives pointers to the official USB docs, so if you are really curious, feel free to go ahead. :)

Chapter 16 is the first part about network drivers. It deals with ifnet (e.g. stuff needed for ifconfig), ifmedia (simplified: which kind of cable and speed is supported), mbufs and MSI(-X). As in other chapters before, a little overview and a little picture in the beginning would have been nice.

Finally, in chapter 17, the packet reception and transmission of network drivers is described. Large example code is broken up into several pieces here, for more easy discussion of related information.

One thing I miss after reaching the end of the book is a discussion of sound drivers. And this is surely not the only type of drivers which is not discussed, I can come up with crypto, firewire, gpio, watchdog, smb and iic devices within a few seconds. While I think that it is much more easy to understand all those drivers now after reading the book, it would have been nice to have at least a little overview of other driver types and maybe even a short description of their driver methods.

Conclusion: As I wrote already in the beginning, the book is not perfect, but it is good. While I have not written a device driver for FreeBSD, the book provided enough insight to be able to write one and to understand existing drivers. I really hope there will be a second edition which addresses the minor issues I had while reading it to make it a perfect book.

Bookmark/FavoritesShare/Save

Free DLNA server which works good with my Sony BRAVIA TV

In several previous posts I wrote about my quest for the right source format to stream video to my Sony BRAVIA TV (build in 2009). The last week-end I finally found something which satisfies me.

What I found was serviio, a free UPnP-AV (DLNA) server. It is written in java and runs on Windows, Linux and FreeBSD (it is not listed on the website, but we have an not-so-up-to-date version in the ports tree). If necessary it transcodes the input to an appropriate format for the DLNA renderer (in my case the TV).

I tested it with my slow Netbook, so that I was able to see with which input format it will just remux the input container to a MPEG transport stream, and which input format would be really re-encoded to a format the TV understands.

The bottom line of the tests is, that I just need to use a supported container (like MKV or MP4 or AVI) with H.264-encoded video (e.g. encoded by x264) and AC3 audio.

The TV is able to chose between several audio streams, but I have not tested if serviio is able to serve files with multiple audio streams (my wife has a different mother language than me, so it is interesting for us to have multiple audio streams for a movie), and I do not know if DLNA supports something like this.

Now I just have to replace minidlna (which only works good with my TV for MP3s and Pictures) with serviio on my FreeBSD file server and we can forget about the disk-juggling.

Bookmark/FavoritesShare/Save

Alexander Leidinger

A while ago I committed the linuxulator D-Trace probes I talked about earlier. I waited a little bit for this announcement to make sure I have not broken anything. Nobody complained so far, so I assume nothing obviously bad crept in.

The >500 probes I committed do not cover the entire linuxulator, but are a good start. Adding new ones is straight forward, if someone is interested in a junior–kernel–hacker task, this would be one. Just ask me (or ask on emulation@), and I can guide you through it.

Share

Alexander Leidinger

Seems I forgot to announce that the linux_base-c6 is in the Ports Collection now. Well, it is not a replacement for the current default linux base, the linuxulator infrastructure ports are missing and we need to check if the kernel supports enough of 2.6.18 that nothing breaks.

TODO:

To my knowledge, nobody is working on anything of this. Anyone is welcome to have a look and provide patches.

Share

Alexander Leidinger

In case you have not noticed yet, KDTRACE_HOOKS is now in the GENERIC kernel in FreeBSD-current. This means you just need to load the DTrace modules and can use DTrace with the GENERIC kernel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you invest reading it.

Share

Alexander Leidinger

This weekend I made some progress in the linuxulator:

  • I MFCed the reporting of some linux-syscalls to 9-stable and 8-stable.
  • I updated my linuxulator-dtrace patch to a recent -current. I already compiled it on i386 and arundel@ has it compiled on amd64. I counted more than 500 new DTrace probes. Now that DTrace rescans for SDT probes when a kernel module is loaded, there is no kernel panic anymore when the linux module is loaded after the DTrace modules and you want to use DTrace. I try to commit this at a morning of a day where I can fix things during the day in case some problems show up which I did not notice during my testing.
  • I created a PR for portmgr@ to repocopy a new linux_base port.
  • I set the expiration date of linux_base-fc4 (only used by 7.x and upstream way past its EoL) and all dependent ports. It is set to the EoL of the last 7.x release, which can not use a later linux_base port. I also added a comment which explains that the date is the EoL of the last 7.x release.

Share

Alexander Leidinger

I just updated to a recent -current and tried the new nullfs. Sockets (e.g. the MySQL one) work now with nullfs. No need to have e.g. jails on the same FS and hardlink the socket to not need to use TCP in MySQL (or an IP at all for the jail).

Great work!

Share

New CentOS linux_base for testing soonish

It seems my HOWTO create a new linux_base port was not too bad. There is now a PR for a CentOS 6 based linux_base port. I had a quick look at it and it seems that it is nearly usable to include into the Ports Collection (the SRPMs need to be added, but that can be done within some minutes).

When FreeBSD 8.3 is released and the Ports Collection open for sweeping commits again, I will ask portmgr to do a repo-copy for the new port and commit it. This is just the linux_base port, not the complete infrastructure which is needed to completely replace the current default linuxulator userland. This is just a start. The process of switching to a more recent linux_base port is a long process, and in this case depends upon enough support in the supported FreeBSD releases.

Attention: Anyone installing the port from the PR should be aware that using it is a highly experimental task. You need to change the linuxulator to impersonate himself as a linux 2.6.18 kernel (described in the pkg-message of the port), and the code in FreeBSD is far from supporting this. Anyone who wants to try it is welcome, but you have to run FreeBSD-current as of at least the last weekend, and watch out for kernel messages about unsupported syscalls. Reports to [email protected] please, not here on the webpage.

Share

New opportunities in the linuxulator

Last weekend I committed some dummy-syscalls to the linuxulator in FreeBSD-current. I also added some comments to syscalls.master which should give a hint which linux kernel had them for the first time (if the linux man–page I looked this up in is correct). So if someone wants to experiment with a higher compat.linux.osrelease than 2.6.16 (as it is needed for a CentOS based linux_base), he should now get some kernel messages about unimplemented syscalls instead of a silent failure.

There may be some low-hanging fruits in there, but I did not really verify this by checking what the dummy syscalls are supposed to do in linux and if we can easily map this to existing FreeBSD features. In case someone has a look, please send an email to emulation@FreeBSD.org.

Share

How to fix FreeBSD for corporations (and user with smaller installations), the tools-viewpoint

There is a huge discussion going on on hackers@ about how FreeBSD is not suitable for large installations (anymore?). As of this writing, the discussion seems to get some discussion-clusters. We have some sub-topics which could lead to some good improvements.

One subtopic is the release engineering. Some changes like a more guided approach of what should be merged to which branch, the frequency of releases and maybe some kind of long-term-branch(es). There is some discussion to get maybe some joined-funding in some way from interested parties to pay someone to take care about long-term-branch(es).

Another subtopic is the way bugs are handled in our old bugtracking software and how patches go unnoticed there.

And both of them are connected (parts more, parts less) by what can be done in a volunteer project.

To me it looks like the proposals “just