Author Archives: FreeBSD

Patching screens EDID information

Quite a long time ago (September 2012), the x11/nvidia-driver port was updated to 304.43, which brought-in some major changes. The most noticeable one from my point of view was the inability to use my two Samsung SyncMaster BX2240 monitors at a resolution better than 800x600 (they have a native resolution of 1920x1080), providing a stunning 1600x600 desktop experience. With no time to dig-in this at that time, I reverted to the previous nvidia-driver and planned to have a look at it later.

I finally took the time to search for a solution. I post it here for two reasons: a) I might need this information at some point in the future and am more likely to find it that way and b) I might not be the only one here which faced this situation.

The first step was to modify my Xorg configuration to be a bit more verbose about the process of choosing a screen resolution when X starts to see if something was different with both drivers. This can be achieved by editing the Monitor section:

Section "Monitor"
    ...
    Option "ModeDebug" "TRUE"
EndSection

With the new driver, no section for Validating Mode "1920x1080", but when trying to validate other modes, the driver reported that it would be happier with EDID information from the screens:

217 (II) Feb 07 09:52:30 NVIDIA(GPU-0):   Validating Mode "640x350":
218 (II) Feb 07 09:52:30 NVIDIA(GPU-0):     640 x 350 @ 85 Hz
219 (II) Feb 07 09:52:30 NVIDIA(GPU-0):     Mode Source: X Server
220 (II) Feb 07 09:52:30 NVIDIA(GPU-0):       Pixel Clock      : 31.50 MHz
221 (II) Feb 07 09:52:30 NVIDIA(GPU-0):       HRes, HSyncStart :  640,  672
222 (II) Feb 07 09:52:30 NVIDIA(GPU-0):       HSyncEnd, HTotal :  736,  832
223 (II) Feb 07 09:52:30 NVIDIA(GPU-0):       VRes, VSyncStart :  350,  382
224 (II) Feb 07 09:52:30 NVIDIA(GPU-0):       VSyncEnd, VTotal :  385,  445
225 (II) Feb 07 09:52:30 NVIDIA(GPU-0):       H/V Polarity     : +/-
226 (WW) Feb 07 09:52:30 NVIDIA(GPU-0):     Mode is rejected: Only EDID-provided modes are allowed on
227 (WW) Feb 07 09:52:30 NVIDIA(GPU-0):     DFP-0.

I enabled EDID which was previously disabled because the screen reported invalid information:

Section "Device"
    ...
    #Option "IgnoreEDIDChecksum" "DFP-0, DFP-1"
EndSection

With the new driver, a Validating Mode "1920x1080" section appeared but with the very same Mode is rejected: Only EDID-provided modes are allowed on DFP-0. The old driver provided a bit more of information actually:

1822 (II) Feb 07 09:54:10 NVIDIA(GPU-0):   Validating Mode "1920x1080":
1823 (II) Feb 07 09:54:10 NVIDIA(GPU-0):     1920 x 1080 @ 50 Hz
1824 (II) Feb 07 09:54:10 NVIDIA(GPU-0):     Mode Source: EDID
1825 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Pixel Clock      : 148.50 MHz
1826 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       HRes, HSyncStart : 1920, 2448
1827 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       HSyncEnd, HTotal : 2492, 2640
1828 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       VRes, VSyncStart : 1080, 1084
1829 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       VSyncEnd, VTotal : 1089, 1125
1830 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       H/V Polarity     : +/+
1831 (WW) Feb 07 09:54:10 NVIDIA(GPU-0): The EDID for Samsung SMBX2240 (DFP-1) contradicts itself: mode
1832 (WW) Feb 07 09:54:10 NVIDIA(GPU-0):     "1920x1080" is specified in the EDID; however, the EDID's
1833 (WW) Feb 07 09:54:10 NVIDIA(GPU-0):     valid VertRefresh range (56.000-75.000 Hz) would exclude
1834 (WW) Feb 07 09:54:10 NVIDIA(GPU-0):     this mode's VertRefresh (50.0 Hz); ignoring VertRefresh
1835 (WW) Feb 07 09:54:10 NVIDIA(GPU-0):     check for mode "1920x1080".
1836 (II) Feb 07 09:54:10 NVIDIA(GPU-0):     Viewport                 1920x1080+360+22
1837 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Horizontal Taps        0
1838 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Vertical Taps          0
1839 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Base SuperSample       x4
1840 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Base Depth             32
1841 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Distributed Rendering  1
1842 (II) Feb 07 09:54:10 NVIDIA(GPU-0):       Overlay Depth          32
1843 (II) Feb 07 09:54:10 NVIDIA(GPU-0):     Mode is valid.

Okay, since all this seems to be related to EDID, and because some major changes where introduced in version 302.xx of the NVidia driver, let's try to patch it! I could not find a tool to dump the EDID directly from FreeBSD, however when Xorg starts, it prints a hex dump of it so it is actually quite trivial to get it from the log file (don't want to code? Clone!).

sysutils/edid-decode can be used to easily locate relevant information from the EDID dump (monitor range), and spot inconsistencies:

Extracted contents:
header:          00 ff ff ff ff ff ff 00
serial number:   4c 2d 84 06 32 32 42 43 0e 15
version:         01 03
basic params:    80 30 1b 78 2a
chroma info:     78 f1 a6 55 48 9b 26 12 50 54
established:     bf ef 80
standard:        71 4f 81 00 81 40 81 80 95 00 b3 00 a9 40 95 0f
descriptor 1:    02 3a 80 18 71 38 2d 40 58 2c 45 00 dd 0c 11 00 00 1e
descriptor 2:    00 00 00 fd 00 38 4b 1e 51 11 00 0a 20 20 20 20 20 20
descriptor 3:    00 00 00 fc 00 53 4d 42 58 32 32 34 30 0a 20 20 20 20
descriptor 4:    00 00 00 ff 00 48 39 58 42 34 30 31 30 32 34 0a 20 20
extensions:      01
checksum:        b0

Manufacturer: SAM Model 684 Serial Number 1128411698
Made week 14 of 2011
EDID version: 1.3
Digital display
Maximum image size: 48 cm x 27 cm
Gamma: 2.20
DPMS levels: Off
Supported color formats: RGB 4:4:4, YCrCb 4:2:2
First detailed timing is preferred timing
Established timings supported:
720x400@70Hz
640x480@60Hz
640x480@67Hz
640x480@72Hz
640x480@75Hz
800x600@56Hz
800x600@60Hz
800x600@72Hz
800x600@75Hz
832x624@75Hz
1024x768@60Hz
1024x768@70Hz
1024x768@75Hz
1280x1024@75Hz
1152x870@75Hz
Standard timings supported:
1152x864@75Hz
1280x800@60Hz
1280x960@60Hz
1280x1024@60Hz
1440x900@60Hz
1680x1050@60Hz
1600x1200@60Hz
1440x900@75Hz
Detailed mode: Clock 148.500 MHz, 477 mm x 268 mm
1920 2008 2052 2200 hborder 0
1080 1084 1089 1125 vborder 0
+hsync +vsync
Monitor ranges (GTF): 56-75Hz V, 30-81kHz H, max dotclock 170MHz
Monitor name: SMBX2240
Serial number: H9XB401024
Has 1 extension blocks
Checksum: 0xb0 (valid)

CEA extension block
Extension version: 1
0 8-byte timing descriptors
Detailed mode: Clock 148.500 MHz, 477 mm x 268 mm
1920 2448 2492 2640 hborder 0
1080 1084 1089 1125 vborder 0
+hsync +vsync
Detailed mode: Clock 74.250 MHz, 477 mm x 268 mm
1280 1390 1430 1650 hborder 0
720  725  730  750 vborder 0
+hsync +vsync
Detailed mode: Clock 74.250 MHz, 277 mm x 286 mm
1280 1720 1760 1980 hborder 0
720  725  730  750 vborder 0
+hsync +vsync
Detailed mode: Clock 27.000 MHz, 477 mm x 286 mm
720  732  796  864 hborder 0
576  581  586  625 vborder 0
-hsync -vsync
Detailed mode: Clock 27.000 MHz, 477 mm x 286 mm
720  736  798  858 hborder 0
480  489  495  525 vborder 0
-hsync -vsync
Checksum: 0x0 (should be 0x99)

EDID block does not conform at all!
Block has broken checksum

After changing the lower limit of the vertical refresh rate from 56 Hz (0x38) to 50 Hz (0x32) and adjusting the checksums, it's time to see if the situation is improved. Once more we have to tweak xorg.conf to tell Xorg to use our dumps instead of querying the screens for EDID information:

Section "Device"
    ...
    Option "CustomEDID" "DFP-0:/usr/local/etc/X11/EDID-BX2240-FIXED; DFP-1:/usr/local/etc/X11/EDID-BX2240-FIXED"
EndSection

And suddenly, it worked!

February 11th, 2013 edit

I wrote an e-mail to the screen manufacturer (Samsung) asking for assistance. I will publish their response on this blog as soon as I get one. The sent message is basically:

Hi,

I own two BX2240 screens. These screens provide invalid EDID information (I wrote an article [1] about that) that makes them unusable with the prorietary NVidia driver version 302.xx and up. It is possible to workaround the problem at the OS level (I provide some instructions in my article) however it would be better to put valid information back into the screen so that they work out of the box.

However, you don't provide any information about this on your website. Can you please send me a technical notice providing information on flashing EDID information if it's possible, and if not, the location of the EEPROM containing the wrong EDID information so that I can replace it and continue to use my screens in proper conditions?

Please find attached to this message the fixed EDID information I use at the moment and want to flash back into my screens.

Kind regards,
Romain Tartière

References:

  1. http://romain.blogreen.org/blog/2013/02/patching-screens-edid-information/

TeXLive 2012 on FreeBSD

Two months ago (August 2nd, 2012), I updated the freebsd-texlive project to provide TeXLive 2012 to the FreeBSD community.

More recently (October 1st, 2012), I made major changes in the freebsd-texlive project to fix the long-standing problem of TeXLive shipping distfiles with no version in the filename and replacing them in place. After setting up a new mirror with renamed distfiles (sponsored by NFrance, a French hosting company using the FreeBSD operating system. Many thanks to them for providing hosting and bandwidth!), updating the program that creates and updates ports, updating the updating tools, switching to the new mirror and updating all ~2250 ports; all the caveats around this choice from the maintainers of the TeXLive distribution should be gone!

This change will however not be without consequences for users. Distfiles version numbers where created by using the date at which a file appeared or where updated on the upstream mirrors, but because of the mirroring delays, the file mtime could be earlier by more than one day. As a consequence, a file updated on 17th May, 2012 could have version number 20120518 (getting the real package version number in the distfile is a PITA). Because having the version in the distfile filename require consistent dates, the new ports use the date from the upstream distfile mtime, and many ports had their version going backwards… PORTEPOCH have so been bumped, and port management tools will want to update all these ports which are in fact unchanged.

It's the first time such a bad situation happen in the freebsd-texlive repository. I hope this will be the last and apologies to users who will be worried with this massive update.

Happy TeXing!

Book Review: FreeBSD Device Drivers

Joseph Kong strikes again! When I saw Designing BSD Rootkits I was quite excited because:

  1. it was on some topics I had interests in but had not as much time as I would have liked to to dig in;
  2. it was focussed on my main operating system.

I bought the book and although it was rather small (~140 pages) enjoyed learning quite a large amount of things about the FreeBSD kernel.

So, when I was contacted by No Stash Press to know if I would be interested in a review copy of their upcoming book FreeBSD Device Drivers by the same author in exchange of writing what I though about it, I had no hesitation and accepted the deal.

Book cover

Just like the preceding book, this one is based on examples, each chapter focussing on a different aspect of device drivers, and going progressively further and further in the details of the kernel, explaining how things work and why they do work that way so that the reader can better understand the involved mechanisms, even if he is not an Operating Systems guru.

Precious information about the FreeBSD kernel (like what you have in The Design and Implementation of the FreeBSD Operating-System but in a more up-to-date and friendlier fashion) are spread all along the book, and — apart from the long C listings — the style is quite pleasant to read.

To sum-up, this book is definitively a must have for anybody interested in how are designed FreeBSD device drivers, not mentioning those who are interested in writing their very own ones for the FreeBSD operating system!

Many thanks Joseph for this awesome book!

Updating a ZFS Mirror

A few days ago, while I was on the phone, my machine experienced a kernel panic. The backtrace pointed a problem somewhere in the swap management code. I was on a hurry at that time and rebooted the machine without taking the time dig in the problem deeper.

On the next day, I eventually realised that an hard disk was logically missing on the system and the ZFS mirror it was belonging to was working in a degraded mode. This disk holding a swap partition, the panic quite makes sense: some data was stored there and could not be paged-out anymore.

# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   294G  68,2G   226G    23%  1.25x  DEGRADED  -
tank  1,81T   300G  1,52T    16%  1.06x  ONLINE  -
# zpool status data
  pool: data
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
  scan: none requested
config:

NAME                                            STATE     READ WRITE CKSUM
data                                            DEGRADED     0     0     0
  mirror-0                                      DEGRADED     0     0     0
    gptid/36711e52-a69e-11de-8adf-0018f38af467  ONLINE       0     0     0
    15152536002702365387                        UNAVAIL      0     0     0  was /dev/gptid/602da1ae-c474-11de-960d-0008a14dbca1

errors: No known data errors

This disk already had problems in the past and I even had to improve sysutils/smartmontools periodic script to take into account SMART attributes that have been failing at some point in the past but have recovered since that time on that disk. This time, the disk is really dead, so no other choice than changing it. Hopefully, I have a brunch of spare heard disks on the shelve, so I took two 500 GB disks to replace the two 320 GB of the degraded ZFS pool.

I first replaced the broken disk with a new one, identified it using geom(8) and partitioned using basically the same settings I used when installing FreeBSD on full-ZFS:

# geom disk list ada1
Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 500107862016 (465G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: ST3500418AS
   ident: (null)
   fwsectors: 63
   fwheads: 16

# geom part create -s GPT ada1
ada1 created
# geom part add -s 128 -t freebsd-boot ada1
ada1p1 added
# geom part add -s 4G -t freebsd-swap ada1
ada1p2 added
# geom part add -t freebsd-zfs ada1
ada1p3 added
# geom part show ada1
=>       34  976773101  ada1  GPT  (465G)
         34        128     1  freebsd-boot  (64k)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  968384365     3  freebsd-zfs  (461G)

# geom part bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
bootcode written to ada1

I then replaced the unavailable ZFS partition with the new one:

# zpool replace data 15152536002702365387 ada1p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'data', you may need to update
boot code on newly attached disk 'ada1p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

… waited a few moments for resilvering to finish:

% zpool status data
  pool: data
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Jan  2 19:26:44 2012
    330M scanned out of 68,2G at 4,07M/s, 4h44m to go
    329M resilvered, 0,47% done
config:

	NAME                                            STATE     READ WRITE CKSUM
	data                                            DEGRADED     0     0     0
	  mirror-0                                      DEGRADED     0     0     0
	    gptid/36711e52-a69e-11de-8adf-0018f38af467  ONLINE       0     0     0
	    replacing-1                                 UNAVAIL      0     0     0
	      15152536002702365387                      UNAVAIL      0     0     0  was /dev/gptid/602da1ae-c474-11de-960d-0008a14dbca1
	      ada1p3                                    ONLINE       0     0     0  (resilvering)

errors: No known data errors

… then shut the system down, replaced the working 320 GB disk by a new 500 GB one and booted into FreeBSD again:

% zpool status data
  pool: data
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
  scan: resilvered 68,2G in 1h58m with 0 errors on Mon Jan  2 21:25:42 2012
config:

	NAME                     STATE     READ WRITE CKSUM
	data                     DEGRADED     0     0     0
	  mirror-0               DEGRADED     0     0     0
	  3635039039460500206    UNAVAIL      0     0     0  was /dev/gptid/36711e52-a69e-11de-8adf-0018f38af467
	    ada1p3               ONLINE       0     0     0

errors: No known data errors

Same story with the other disk:

geom part create -s GPT ada0
ada0 created
# geom part add -s 128 -t freebsd-boot ada0
ada0p1 added
# geom part add -s 4G -t freebsd-swap ada0
ada0p2 added
# geom part add -t freebsd-zfs ada0
ada0p3 added
# geom part show ada0
=>       34  976773101  ada0  GPT  (465G)
         34        128     1  freebsd-boot  (64k)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  968384365     3  freebsd-zfs  (461G)

# geom part bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
bootcode written to ada0
# zpool replace data 3635039039460500206 ada0p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'data', you may need to update
boot code on newly attached disk 'ada0p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

When done, I saw that the available space on the data zpool was still the same:

# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   294G  65,3G   229G    22%  1.19x  ONLINE  -
tank  1,81T   300G  1,52T    16%  1.08x  ONLINE  -

This is due to the autoexpand property set to off by default and that should be set to on before replacing disks if this feature is desired.

# zpool set autoexpand=on data

Hopefully, it is possible to use the zpool(8)'s online command to make ZFS take into account the extra space available for the pools:

# zpool online -e data ada0p3
# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   460G  65,3G   395G    14%  1.20x  ONLINE  -
tank  1,81T   300G  1,52T    16%  1.06x  ONLINE  -

Typesetting Korean with TeXLive

Today, I received a request for adding support for ko.teX to the FreeBSD TeXLive ports. I created 3 ports for this purpose: print/kotex-util, print/kotex-macros and kotex-fonts-all.

I did some basic testing locally, but I don't read Korean, thus I could not test beyond the (minimalistic) examples. I am therefore looking for FreeBSD TeXLive users who are able to write Korean to help testing this package. If you are one of them, please give it a try and report any success or failures on the issue page.

To install these experimental ports, the easiest way is using devel/subversion:

% svn co http://freebsd-texlive.googlecode.com/svn/branches/ports/print/kotex-util/
% cd kotex-util
% make
% sudo make install
% cd ..
% svn co http://freebsd-texlive.googlecode.com/svn/branches/ports/print/kotex-macros/
% cd kotex-macros
% make
% sudo make install
% cd ..
% svn co http://freebsd-texlive.googlecode.com/svn/branches/ports/print/kotex-fonts-all/
% cd kotex-fonts-all
% make
% sudo make install
% cd ..

Thanks!

TeXLive 2010 on FreeBSD

I definitively have a lack of time at the moment. Since this blog last entry is about Hey guys, I have just pushed TeXLive 2009 in the freebsd-texlive project, I guess I have to take the time to write this short post:

One month ago, I pushed TeXLive 2010 to the freebsd-texlive repository. Moreover, because portshaker is now in the FreeBSD ports tree, installing bleeding-edge TeXLive on FreeBSD has never been so easy!

TeXLive 2009 on FreeBSD

I recently updated the freebsd-texlive project to make TeXLive 2009 (the latest verison) available as ports for FreeBSD.

I used a script for update TeXLive 2008 on a regular basis, but since these new ports feature pkg-plist files, I had to setup an update jail. So updates for FreeBSD TeXLive ports will be generated daily at 5:30 CET.

I am also thinking about pushing the portshaker(8) utility in the FreeBSD ports tree in order to make it easier to track this project along with others (e.g. BSD#).

Feedback is welcomed!

ZFS: unsupported ZFS version 14 (should be 13)

How I got there?

After updating to the latest development version of GNOME thanks' to MarcusCom ports I wanted to log out and back in but X refused to restart because of the in-heavy-development Nouveau video driver. I run in this problem every once a while and a full reboot solve this problem. However, the system did not boot:

ZFS: unsupported ZFS version 14 (should be 13)
ZFS: unsupported ZFS version 14 (should be 13)
NO ZFS pools located? can't boot

Okay, I recall running the following to update my full-ZFS system from ZFS 13 to 14 a while ago:

zpool upgrade -a

I completly forgot about updating the GPT bootloader accordingly and thus it was unable to use the version of ZFS on my disks.

The FreeBSD 8.0-STABLE livefs

The FreeBSD projects provides snapshots of -STABLE version of the Operating System (I am running FreeBSD 8.0-STABLE) and the livefs disk contains the fixit environement which would allow me to solve my problems.

I so used my laptop to look for such an ISO image and burn it, unfortunately, the bootp and dns servers are run by the machine that was down. Hopefully, I had a web browser oppenned on this system and it had the address of google's servers in it's cache. I could search for my ISP DNS and add them to /etc/resolv.conf. I was then able to download the image and burn it.

Fixing

By default, the livefs will not have ZFS support unless you ask for it before starting. In other word, at the boot menu, Escape to loader prompt and type:

OK load zfs
OK boot

When the sysinstall(8) menu appears, select Fixit / CDROM/DVD. At the Fixit# prompt, you will have to search for your zfs pools, mount your filesystems, locate the new version of the bootloader and install it. This is basically a transcript of what I did (I previously described my setup in this blog):

Fixit# zpool import -f data # at this point I had to leave and return in the Fixit environement
                            # because by ZFS root was hiding the livefs root.
Fixit# zfs set mountpoint=/mnt data
Fixit# gpart bootcode -p /mnt/boot/gptzfsboot -i 1 ad10
Fixit# gpart bootcode -p /mnt/boot/gptzfsboot -i 1 ad12
Fixit# zfs set mountpoint=legacy data
Fixit# zfs umount -a
Fixit# logout

Note to myself: in the future do not forget to update the bootloader when updating my ZFS filesystems.

The state of Mono 2.6 on FreeBSD

Long time no post... Well, I an writing a serie of articles about NFC and it takes me a lot of time, not mentioning I still do many other things in parallel. If you are interested in this area, stay tuned ;-)

So apart NFC, what's new with Mono on FreeBSD? First, you wight have seen that mono 2.6 has been released for a while and is still not available in FreeBSD ports. The main reasons are:

  • a thread concurrency problem which is still not solved but an ugly workaround have been found;
  • a bug (well, a weird implementation) of the new built-in soft-mode debugger that makes it totally useless on amd64. A workaround have been found and in the meantime, the code has been updated upstream and will be part of mono-2.6.2 which will probably be released at the end of the month;
  • the moonlight tools (e.g. the smcs.exe compiler) are still part of mono (read the code source is in mono-2.6.tar.bz2) but installed by moonlight (read the Makefiles with install targets are in moonlight-2.0.tar.bz2). This leads to some juggling which is in progress (read: I can build and install it on my system, Firefox segfaults when I browse a website with Silverlight contents, and configure fails in Tinderbox because it cannot find bison, which is however installed according to the logs).

I will not mention here the mono-basic-2.6.tar.bz2 source tarball that exists besides the moonlight-2.0.tar.bz2 tarball in the moon(light) source directory and the mono-basic-2.6.tar.bz2 source tarball in the mono-basic directory having different dates & checksums… Lot of fun to come I guess.

Once these various problems have been fixed, you can expect mono-2.6 to reach the ports, promptly followed by the last version of the MonoDevelop IDE which is quite lovely.

Last but not least, I am now a ports committer and flz@ is my mentor.

Unbreaking Mono on FreeBSD 6.4

A few days ago ports/140916: lang/mono (2.4.2.3) installation fails was opened. The reported problem had already been reported a few time but with insufficient feedback so far and I was not able to diagnose the problem, not mentioning providing a fix. But this time, the reporter could spot that the problem was happening for him only with FreeBSD-6.4. I did the test on a fresh FreeBSD-6.4 install and could trigger the error:

arthur# portsnap fetch
[...]
arthur# portsnap extract
[...]
arthur# cd /usr/ports/lang/mono
arthur# make -V PKGNAME
mono-2.4.2.3_1
arthur# make
[...]
if test -w ../mcs; then :; else chmod -R +w ../mcs; fi
cd ../mcs && gmake NO_DIR_CHECK=1 PROFILES='net_1_1 net_2_0 net_3_5 net_2_1' CC='cc' all-profiles
gmake[3]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake profile-do--net_1_1--all profile-do--net_2_0--all profile-do--net_3_5--all profile-do--net_2_1--all
gmake[4]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake PROFILE=basic all
gmake[5]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[6]: *** [build/deps/basic-profile-check.exe] Error 1
gmake[6]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
*** The compiler 'false' doesn't appear to be usable.
*** Trying the 'monolite' directory.
gmake[7]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[8]: *** [build/deps/basic-profile-check.exe] Error 138
gmake[8]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
*** The contents of your 'monolite' directory may be out-of-date
*** You may want to try 'make get-monolite-latest'
gmake[8]: *** [do-profile-check-monolite] Error 1
gmake[8]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[7]: *** [do-profile-check] Error 2
gmake[7]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[6]: *** [do-profile-check-monolite] Error 2
gmake[6]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[5]: *** [do-profile-check] Error 2
gmake[5]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[4]: *** [profile-do--basic--all] Error 2
gmake[4]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[3]: *** [profiles-do--all] Error 2
gmake[3]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs'
gmake[2]: *** [all-local] Error 2
gmake[2]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/runtime'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3'
gmake: *** [all] Error 2
*** Error code 1

Stop in /usr/ports/lang/mono.
*** Error code 1

Stop in /usr/ports/lang/mono.

A first look leads us on a wrong way (lane?): *** The compiler 'false' doesn't appear to be usable. This message is actually not an error message: the mono C# compiler (mcs(1)) is written in C# so some bootstrapping may be required if no version of mono is present on the system. The mono port adds EXTERNAL_MCS=false to MAKE_ARGS so that the mono port will be build exactly the same way regardless of an available mcs(1) in the path.

Hopefully, I tested on an old machine and could see that something was running for a couple seconds before failing. Ctrl+T in tcsh(1) told me more:

load: 0.08  cmd: mono 10910 [ksesigwait] 0.13u 0.10s 0% 24492k
load: 0.08  cmd: mono 10910 [ksesigwait] 0.13u 0.10s 0% 24964k

The good news is that some mono application is running. But wait, what is this ksesigwait state? A first look at the FreeBSD Kernel Cross Reference reveals that this state is defined in kern/kern_kse.c; the man page is kse(2); Wikipedia states KSEs were mandatory at introduction; made optional at kernel build time in the 7.0 release and removed from the 8.0 release with a compatibility library.

The second good news is that a 22MB mono.core file is written in the mcs directory.

arthur# gdb -q /usr/ports/lang/mono/work/mono-2.4.2.3/mono/mini/mono mono.core
Core was generated by `mono'.
Program terminated with signal 10, Bus error.
Reading symbols from /usr/local/lib/libgthread-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libgthread-2.0.so.0
Reading symbols from /usr/local/lib/libglib-2.0.so.0...done.
Loaded symbols for /usr/local/lib/libglib-2.0.so.0
Reading symbols from /usr/local/lib/libintl.so.8...done.
Loaded symbols for /usr/local/lib/libintl.so.8
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /usr/local/lib/libpcre.so.0...done.
Loaded symbols for /usr/local/lib/libpcre.so.0
Reading symbols from /lib/libm.so.4...done.
Loaded symbols for /lib/libm.so.4
Reading symbols from /lib/libpthread.so.2...done.
Loaded symbols for /lib/libpthread.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x2856bf2a in signalcontext () from /lib/libc.so.6
[New Thread 0x8319c00 (sleeping)]
[New Thread 0x8319800 (sleeping)]
[New Thread 0x8319600 (LWP 100098)]
[New Thread 0x82fd000 (runnable)]
[New LWP 100058]
(gdb) thread apply all bt

Thread 5 (LWP 100058):
#0  0x2856bf2a in signalcontext () from /lib/libc.so.6
#1  0x28527fc4 in pthread_mutexattr_init () from /lib/libpthread.so.2
#2  0x28303480 in ?? ()

Thread 4 (Thread 0x82fd000 (runnable)):
#0  0x285eb0ac in __vfprintf () from /lib/libc.so.6
#1  0x2855ff90 in vasprintf () from /lib/libc.so.6
[...]
#33814 0x081464b8 in mono_runtime_invoke (method=0x82fd81c, obj=0x0, params=0xbfbfe510, exc=0x0) at object.c:2401
#33815 0x0814785b in mono_runtime_exec_main (method=0x82fd81c, args=0x28657f60, exc=0x0) at object.c:3301
#33816 0x0814736c in mono_runtime_run_main (method=0x82fd81c, argc=2, argv=0xbfbfe74c, exc=0x0) at object.c:3089
#33817 0x080cdf7d in mono_jit_exec (domain=0x28653ee0, assembly=0x833c880, argc=3, argv=0xbfbfe748) at driver.c:924
#33818 0x080ce0e8 in main_thread_handler (user_data=0xbfbfe690) at driver.c:972
#33819 0x080cfc3a in mono_main (argc=6, argv=0xbfbfe73c) at driver.c:1647
#33820 0x08058f90 in main (argc=6, argv=0xbfbfe73c) at main.c:34

Thread 3 (Thread 0x8319600 (LWP 100098)):
#0  0x2852f5f3 in pthread_testcancel () from /lib/libpthread.so.2
#1  0x28527fc4 in pthread_mutexattr_init () from /lib/libpthread.so.2
#2  0x28303480 in ?? ()

Thread 2 (Thread 0x8319800 (sleeping)):
#0  0x28528097 in pthread_mutexattr_init () from /lib/libpthread.so.2
#1  0x28521ade in _nanosleep () from /lib/libpthread.so.2
#2  0x28521c42 in nanosleep () from /lib/libpthread.so.2
#3  0x081e32ca in collection_thread (unused=0x0) at collection.c:34
#4  0x28520449 in pthread_create () from /lib/libpthread.so.2
#5  0x285deecb in _ctx_start () from /lib/libc.so.6

Thread 1 (Thread 0x8319c00 (sleeping)):
#0  0x28528097 in pthread_mutexattr_init () from /lib/libpthread.so.2
#1  0x2852822b in pthread_mutexattr_init () from /lib/libpthread.so.2
#2  0x2852c839 in _pthread_cond_wait () from /lib/libpthread.so.2
#3  0x2852cd82 in pthread_cond_wait () from /lib/libpthread.so.2
#4  0x081e86d8 in _wapi_handle_timedwait_signal_handle (handle=0x1d05, timeout=0x0, alertable=0, poll=0) at handles.c:1605
#5  0x081e846e in _wapi_handle_wait_signal_handle (handle=0x1d05, alertable=0) at handles.c:1548
#6  0x08205f2e in WaitForSingleObjectEx (handle=0x1d05, timeout=4294967295, alertable=0) at wait.c:205
#7  0x081608a0 in finalizer_thread (unused=0x0) at gc.c:1061
#8  0x081806f0 in start_wrapper (data=0x8326380) at threads.c:623
#9  0x08200367 in thread_start_routine (args=0x832c230) at threads.c:286
#10 0x08223adf in GC_start_routine (arg=0x2865aec0) at pthread_support.c:1382
#11 0x28520449 in pthread_create () from /lib/libpthread.so.2
#12 0x285deecb in _ctx_start () from /lib/libc.so.6
#0  0x2852f5f3 in pthread_testcancel () from /lib/libpthread.so.2

Yes, you read it well, the thread 4 backtrace is 33820 frame long! Looks like we have a stack overflow... Since the problem occurs on FreeBSD 6 (where KSE is enabled by default) and not on FreeBSD 7 (where KSE is disabled by default) nor FreeBSD 8 (No KSE at all), it is likely to be related to the thread implementation.

Let's launched one more time gmake(1) and hit Ctrl+Z to suspend all when mono was filling-in it's stack in order to have some hints about the exact command being executed. Reducing it to a simple test is trivial, e.g.:

arthur# pwd
/usr/ports/lang/mono/work/mono-2.4.2.3/mcs
arthur# env MONO_PATH="$PWD/class/lib/monolite" ../mono/mini/mono --config ../runtime/etc/mono/config ./class/lib/monolite/mcs.exe
Bus error (core dumped)
arthur# 

Now we know what is being run. Let's get some info about this binary file:

arthur# ldd ../mono/mini/mono
../mono/mini/mono:
	libgthread-2.0.so.0 => /usr/local/lib/libgthread-2.0.so.0 (0x28304000)
	libglib-2.0.so.0 => /usr/local/lib/libglib-2.0.so.0 (0x28309000)
	libintl.so.8 => /usr/local/lib/libintl.so.8 (0x283c9000)
	libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x283d2000)
	libpcre.so.0 => /usr/local/lib/libpcre.so.0 (0x284c7000)
	libm.so.4 => /lib/libm.so.4 (0x284fc000)
	libpthread.so.2 => /lib/libpthread.so.2 (0x28512000)
	libc.so.6 => /lib/libc.so.6 (0x28537000)

pthread(3) list all thread libraries available on the system:

  • POSIX Threads Library (libpthread, -lpthread)
  • 1:1 Threading Library (libthr, -lthr)
  • Reentrant C Library (libc_r, -lc_r)

On my FreeBSD 8.0-STABLE machine, ldd(1) reports (mono-2.6 instead of 2.4):

marvin# ldd `which mono`
/usr/local/bin/mono:
	libgthread-2.0.so.0 => /usr/local/lib/libgthread-2.0.so.0 (0x8008b4000)
	libglib-2.0.so.0 => /usr/local/lib/libglib-2.0.so.0 (0x8009b8000)
	libicui18n.so.38 => /usr/local/lib/libicui18n.so.38 (0x800b74000)
	libintl.so.8 => /usr/local/lib/libintl.so.8 (0x800dca000)
	libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x800ed3000)
	libpcre.so.0 => /usr/local/lib/libpcre.so.0 (0x8010cd000)
	libm.so.5 => /lib/libm.so.5 (0x8011fd000)
	libthr.so.3 => /lib/libthr.so.3 (0x80131c000)
	libc.so.7 => /lib/libc.so.7 (0x801434000)
	libicuuc.so.38 => /usr/local/lib/libicuuc.so.38 (0x80166e000)
	libicudata.so.38 => /usr/local/lib/libicudata.so.38 (0x80189f000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x80248e000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x802699000)

So let's try to use libthr on FreeBSD 6. This can be achieved using libmap.conf(5): instead of /lib/libpthread.so(.2) we want to use /usr/lib/libthr.so(.2):

arthur# cat >> /etc/libmap.conf << EOT
libpthread.so.2  libthr.so.2
libpthread.so    libthr.so
EOT

then...

arthur# make
[...]
arthur# make tests
[...]
363 test(s) passed. 0 test(s) did not pass.
[...]
arthur# echo $?
0

\o/

OpenRD console access on FreeBSD.

This morning, I received the Open-RD I bought a few days ago and started playing with my new device. One of the first thing I wanted to access was (of course) the system console. While the default setup provides a SSH daemon and even GDM and a full desktop, my goal is to have FreeBSD on this device and move the services I run at home from my personal computer to this low-consumption computer (and I don't intend to switch to GNU/Linux).

Since the console is usually available on the serial port, I looked for some cable to connect the OpenRD to my computer. Unfortunately, I didn't have the required cable. I then spent about two hour with various serial cables, cables looking like serial cables but which where not serial cables, parallel SUB-D25 connectors, a serial SUB-D25 cross-over cable and a soldering iron to build a null-modem SUB-D9 serial cable. Unfortunately, when I connected my computer to the OpenRD, I was still not able to access the console using cu(1).

After reading a bit of documentation, I realised I missed the fact that the console was available on USB and not via the serial port. However, FreeBSD did not recognised the chip the OpenRD uses to provide serial over USB support.

Using usbconfig(8), I could get the vendor ID and product ID the uftdi(4) driver was lacking. I patches the source code and a few minutes later could enjoy a set of new devices when I plug the OpenRD on my computer:

% ls /dev/cuaU*
/dev/cuaU0
/dev/cuaU0.init
/dev/cuaU0.lock
/dev/cuaU1
/dev/cuaU1.init
/dev/cuaU1.lock

I can now access the OpenRD console using cu(1):

% cu -l /dev/cuaU1 -s 115200

\o/ I can boot a FreeBSD 8.0-STABLE arm kernel!

I filled-in the usb/140951 problem report in the FreeBSD GNATS with the patch. If you own an OpenRD and want to have access to the console, the patch is just waiting for you.

Introducing ZFS support in portshaker(8)

portshaker(8) is a tool designed for merging partial ports trees into the FreeBSD ports tree. In other words, it implements some kind of overlay for the FreeBSD ports.

When merging port trees, portshaker(8) first clones the upstream FreeBSD ports tree to the target location. For this purpose, portshaker(8) used to rely on rsync(1) because the target ports tree was supposed to be quite near to the source ports tree. With an UFS file-system, it took my computer circa 10 minutes to merge the 390 Mib of files.

I recently switched to full ZFS, and this gave the clone process a boost: less than four minutes where required to perform the clone action for the first time, and thanks to caching, a single minute was enought to clone a second time.

But ZFS provides cloning capabilities, thus, I added support for the ZFS file-system into portshaker(8). Enabling this feature is as simple as adding the following to /usr/local/etc/portshaker.conf:

use_zfs="yes"

There is however a prerequisite for this to work: the target ports tree's parent directory and the source directory have to be ZFS filesystems. For example, on my computer, the FreeBSD upstream ports are stored in /var/cache/portshaker/freebsd and I merge my ports to the usual /usr/ports directory. All these directories are ZFS filesystems:

romain@marvin ~ % zfs list /usr/ports /var/cache/portshaker /var/cache/portshaker/freebsd
NAME                                USED  AVAIL  REFER  MOUNTPOINT
data/usr/ports                     57,1M  80,9G   402M  /usr/ports
data/var/cache/portshaker           514M  80,9G  48,6M  /var/cache/portshaker
data/var/cache/portshaker/freebsd   427M  80,9G   390M  /var/cache/portshaker/freebsd

When merging, portshaker(8) snapshots the FreeBSD ports and clone them to the target ports tree. This requires only a few seconds to achieve.

Banshee: Like Frankenstein! It rips!

A while ago (I mean when it landed in the FreeBSD ports tree), Banshee reacted when a music CD-ROM was in a CD-Drive: it crashed. Well, not so cool in fact, you had to double-check it was empty before launching the player...

More recently, I was happy to find out that it was not the case anymore: Banshee was simply unable to detect the CD-ROM anymore. Still not so cool: you were still unable to listen to an audio CD, not mentioning ripping it.

For some reason I cannot really figure out (I have spent quite some time in fixing Banshee for FreeBSD and giving it some love), I took a look at this problem.

Obviously, the first rusty part of the chain of tools involved with audio CD was MusicBrainz and more precisely musicbrainz-sharp (a client library for .NET / Mono). FreeBSD having it's own ioctls for managing CD drives, and musicbrainz-sharp having 3 concrete classes for managing such devices called DiscLinux, DiscWin32 and DiscWin32NT, a piece was definitively missing. With a great deal of inspiration, it was called DiscFreeBSD and provides bindings for almost everything referenced in sys/cdio.h. When instanced, this class reads the CD in the drive and computes an ID used to fetch information about the audio CD from the Internet.

Unfortunately, I could not test the library on a system one which it is supposed to work (read GNU/Linux) so I asked a friend of mine, Baptiste (Yes! It works!) [edit: 2009-11-10:12:04 he just told me he has a a-bit-better website where he is playing now...], to tell me how it goes and to help me make the library behave the same on GNU/Linux and FreeBSD. We quickly had some good results:

Banshee listing an audio CD tracks
Yay! Tracks from Sopor Aeternus' album Les fleurs du Mal are being listed!

Unfortunately, it was not possible to listen to the CD nor rip it... Not so cool once more.

And here takes place the magic of GStreamer!

cd /usr/ports/audio/gstreamer-plugins-cdparanoia && make install

Now, Banshee can play and RIP audio CDs with a single mouse clic!

Banshee playing an audio CD
Playing Cinema Strange's The Astonished Eyes Of Evening...

Banshee ripping an audio CD
And ripping it (progress in the bottom left corner)...

All this will be soon in your ports tree!

For those who can't wait, patches for musicbrainz-shap are in the project's trac: Add support for FreeBSD in MusicBrainzSharp.

Packing structures with Mono or how to fix alignment problems for P/Invoke

This is basically a[nother] note for myself.

I have just spent a while wondering why the size of some data structures in C# was wrong while writing bindings to the CD-ROM control features of the FreeBSD libc.

Considering the following code snippet:

struct msf {
	public byte unused;
	public byte minute;
	public byte second;
	public byte frame;
};
[StructLayout (LayoutKind.Explicit)]
struct msf_lba {
	[FieldOffset (0)] public msf msf;
	[FieldOffset (0)] public int lba;
	[MarshalAs (UnmanagedType.ByValArray, SizeConst = 4)]
	[FieldOffset (0)] public byte [] addr;
};
struct cd_toc_entry {
	byte unused1;
	public byte control_addr_type;
	public byte track;
	byte unused2;
	public msf_lba addr;
};
struct cd_sub_channel_media_catalog {
	public byte data_format;
	public int mc_valid;
	[MarshalAs (UnmanagedType.ByValArray, SizeConst = 15)]
	public byte [] mc_number;
};

We can compute the size of each structure (MSDN page about types size):

msf
A msf structure contains 4 members each of 1 byte long so the whole structure is 4 bytes;
msf_lba
This is not more than the union of a msf structure (4 bytes), an int (4 bytes) and an array of 4 byte (4 bytes), so it's 4 bytes long too;
cd_toc_entry
A cd_toc_entry structure is composed of an unused byte (1 byte), 2 data byte (1 byte + 1 byte), another unused byte (1 byte) and a msf_lba structure (4 bytes) so it is 1 + 1 + 1 + 1 + 4 = 8 bytes long;
cd_sub_channel_media_catalog
A cd_sub_channel_media_catalog structure starts with 1 byte (1 byte), followed by an int (4 bytes), and an array of 15 byte (15 bytes). The whole structure is therefore 1 + 4 + 15 = 20 bytes long.

These results can be checked using Marshal.SizeOf:

Marshal.SizeOf (typeof (msf)) = 4
Marshal.SizeOf (typeof (msf_lba)) = 4
Marshal.SizeOf (typeof (cd_toc_entry)) = 8
Marshal.SizeOf (typeof (cd_sub_channel_media_catalog)) = 24

So the cd_sub_channel_media_catalog structure it 4 bytes long more that expected. This really sounds like alignment problems in C. A quick search on the Interop with Native Libraries page on Novell's wiki does not give any info about such a problem. Switching the two first fields order fix the size (but of course breaks the workability) so it is definitively an alignment problem that takes place here.

After some search, I could find the StructLayoutAttribute.Pack Field (actually I could not find this piece of information using the MSDN search engine, I had to search on the web similar problems and find this one with a reference to Pack to find this page in the Microsoft .NET documentation).

Finally, the solution is to write this:

[StructLayout (LayoutKind.Sequential, Pack = 1)]
struct cd_sub_channel_media_catalog {
	public byte data_format;
	public int mc_valid;
	[MarshalAs (UnmanagedType.ByValArray, SizeConst = 15)]
	public byte [] mc_number;
};

The MusicBrainzSharp port to FreeBSD can go on!