Monthly Archives: February 2010

ZFS: unsupported ZFS version 14 (should be 13)

How I got there?

After updating to the latest development version of GNOME thanks' to MarcusCom ports I wanted to log out and back in but X refused to restart because of the in-heavy-development Nouveau video driver. I run in this problem every once a while and a full reboot solve this problem. However, the system did not boot:

ZFS: unsupported ZFS version 14 (should be 13)
ZFS: unsupported ZFS version 14 (should be 13)
NO ZFS pools located? can't boot

Okay, I recall running the following to update my full-ZFS system from ZFS 13 to 14 a while ago:

zpool upgrade -a

I completly forgot about updating the GPT bootloader accordingly and thus it was unable to use the version of ZFS on my disks.

The FreeBSD 8.0-STABLE livefs

The FreeBSD projects provides snapshots of -STABLE version of the Operating System (I am running FreeBSD 8.0-STABLE) and the livefs disk contains the fixit environement which would allow me to solve my problems.

I so used my laptop to look for such an ISO image and burn it, unfortunately, the bootp and dns servers are run by the machine that was down. Hopefully, I had a web browser oppenned on this system and it had the address of google's servers in it's cache. I could search for my ISP DNS and add them to /etc/resolv.conf. I was then able to download the image and burn it.

Fixing

By default, the livefs will not have ZFS support unless you ask for it before starting. In other word, at the boot menu, Escape to loader prompt and type:

OK load zfs
OK boot

When the sysinstall(8) menu appears, select Fixit / CDROM/DVD. At the Fixit# prompt, you will have to search for your zfs pools, mount your filesystems, locate the new version of the bootloader and install it. This is basically a transcript of what I did (I previously described my setup in this blog):

Fixit# zpool import -f data # at this point I had to leave and return in the Fixit environement
                            # because by ZFS root was hiding the livefs root.
Fixit# zfs set mountpoint=/mnt data
Fixit# gpart bootcode -p /mnt/boot/gptzfsboot -i 1 ad10
Fixit# gpart bootcode -p /mnt/boot/gptzfsboot -i 1 ad12
Fixit# zfs set mountpoint=legacy data
Fixit# zfs umount -a
Fixit# logout

Note to myself: in the future do not forget to update the bootloader when updating my ZFS filesystems.

Emprisonner une debian dans un FreeBSD

Parfois on a besoin de tester des choses sous linux, parfois on a besoin d'utiliser des applications linux-only, où pour tout un tas d'autre raison on a besoin de faire tourner un linux.

Pour ça c'est vrai que l'on peut virtualiser dans un virtualbox ou un qemu, c'est bien mais c'est quand même coûteux en terme de ressources pour le host.

Sous FreeBSD nous disposons du linuxulator c'est la couche d'emulation permettant de faire tourner des applications Linux sous FreeBSD.

Ni une ni deux je me dis qu'il est possible de faire de petites choses sympathiques avec ça : une jail linux-only.

Il faut d'abord préparer le terrain :

# mkdir /home/jails/debian
# mkdir /home/jails/debian/dev
# mkdir /home/jails/debian/proc
# mkdir /home/jails/debian/sys
# kldload linux
# kldload linprocfs
# kldload linsysfs
# kldload lindev
# mount -t devfs none /home/jails/debian/dev
# mount -t linprocfs none /home/jails/debian/proc
# mount -t linsysfs none /home/jails/debian/sys

Nous allons donc utiliser /home/jails/debian comme racine de la debian que nous allons installer.

Nous chargeons tous les pilotes nécessaires (notez que lindev est apparu en FreeBSD 9 et a été MFCed en 8-STABLE il n'est absolument pas obligatoire)

On pourrait faire l'installation avec debootstrap, mais comme je suis un feignant j'ai préféré utiliser un template openvz tout fait :

# fetch http://download.openvz.org/template/precreated/debian-5.0-x86.tar.gz

Puis le depacker dans ma jail :

# tar xvfp debian-5.0-x86.tar.gz -C debian --exclude dev* --exclude proc* --exclude sys*

Pour démarrer correctement la jail il faut qu'au minimum un service tourne dans la jail (je n'ai pas réussit à faire une jail linux-only persistente). Par défaut le script de démarrage des jails essaye de lancer /etc/rc que nous allons créer et de lancer /etc/rc.shutdown pour s'arrêter.

# echo "/etc/init.d/cron start" > /home/jails/debian/etc/rc
# chmod 755 /home/jails/debian/etc/rc
# echo "/etc/init.d/cron stop" > /home/jails/debian/etc/rc.shutdown
# chmod 755 /home/jails/debian/etc/rc.shutdown

dans /etc/rc.conf on règle le lancement de la jail

jail_debian_rootdir=/home/jails/debian
jail_debian_hostname="debian"
jail_debian_ip="192.168.1.3"
jail_debian_interface="nfe0"
jail_debian_devfs_enable="YES"
jail_debian_devfs_ruleset="devfsrules_jail"
jail_debian_flags="-n debian"

on démarre la jail :

# /etc/rc.d/jail start debian

et magie :

#jls
   JID  IP Address      Hostname                      Path
    15  192.168.1.3     debian                        /home/jails/debian
#jexec debian uname -a
Linux debian 2.6.16 FreeBSD 8.0-STABLE #3: Sun Jan 10 20:39:38 CET 2010 i686 GNU/Linux
#jexec debian cat /etc/debian_version
5.0.4

Vous voila avec une belle debian emprisonnée dans un freebsd.

Attention, tout ne fonctionne pas parfaitement : sysklogd ne marche pas à cause des accès /dev par exemple, mais c'est 99% fonctionnel quand même.

Une jail 32bits sur un host 64bits

J'ai un eeepc 1000H qui fonctionne à merveille sous FreeBSD tout y est reconnu, juste un petite compilation maison pour le wifi, bref c'est pas le sujet de ce post.

Le truc c'est qu'un eeepc bah c'est lent pour la compilation, le binaire fournis par freebsd ne sont pas vraiment compilés avec les options que je souhaite, mais il s'avère que j'ai un beau Q6660 qui lui est plutôt très bien pour la compilation.

D'un autre côté je mijote toujours ma version FreeBSD de pkgin, du coup je me dis pourquoi pas compiler sur mon Q6600 les packages binaires depuis les ports et les installer via pkgin sur le 1000H, bah oui en voila une idée qu'elle est bonne.

De ce pas je me configure une jail 32bits (de la doc pour ça il y en a partout je vous laisse chercher :)) je me connecte dessus :

$ jexec eeepc /bin/sh

Je commence à compiler des ports, mais rapidement ça brotch de partout. Normal il me compile la moitié des choses en pensant avoir affaire à un amd64 :

$ uname -a
FreeBSD eeepc 8.0-STABLE FreeBSD 8.0-STABLE #3: Sun Jan 10 20:39:38 CET 2010     [email protected]:/usr/obj/usr/src/sys/GALWAY  amd64

C'est pas du tout, mais alors pas du tout la cible que je cherche, moi je veux build de l'i386, en suivant la branche RELEASE du kernel (pour les drivers venant des ports).

Réglons les problèmes un par un.

Tout d'abord faisons fonctionner freebsd-update afin de pouvoir avoir le dernier niveau de patch de la branche RELEASE

$ freebsd-update fetch
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 8.0-STABLE from update5.FreeBSD.org... failed.
Fetching metadata signature for 8.0-STABLE from update4.FreeBSD.org... failed.
Fetching metadata signature for 8.0-STABLE from update2.FreeBSD.org... failed.
No mirrors remaining, giving up.

C'est pas gagné pourtant j'ai bien installé un jail 8.0-RELEASE. Il me faut donc le lui faire savoir. Pour cela, il suffit de modifier le /etc/login.conf de la jail et remplacer :

:setenv=MAIL=/var/mail/$,BLOCKSIZE=K,FTP_PASSIVE_MODE=YES:\

par

:setenv=MAIL=/var/mail/$,BLOCKSIZE=K,FTP_PASSIVE_MODE=YES,UNAME_m=i386,UNAME_p=i386,UNAME_r=8.0-RELEASE-p2,OSVERSION=800107,UNAME_v=FreeBSD 8.0-RELEASE-p2:\

Une recontruction de la db est nécessaire :

$ cap_mkdb /etc/login.conf

On peut maintenant quitter l'environnement et se reconnecter à la jail, par contre pour forcer la prise en compte de login.conf il faut forcer l'utilisateur dans la ligne jexec du coup :

$ jexec -U root eeepc /bin/sh
$ uname -a
FreeBSD eeepc 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 i386

Parfait, maintenant relancez freebsd-update vous verrez qu'il fonctionne correctement.

Second problème les ports, là il en faut un peu plus.

En effet les ports vont chercher l'architecture via le sysctl :

  • hw.machine
  • hw.machine_arch

Là notre astuce ne fonctionne plus, de plus il est impossible de les changer depuis une jail. Du coup en cherchant un peu dans /usr/ports/Mk/*.mk on se rend vite compte que l'on peut contourner le problème via /etc/make.conf. Pour ça, il suffit de rajourer les variables suivantes :

MACHINE=i386
MACHINE_ARCH=i386

Désormais je peux tout compiler tranquilement depuis ma jail et utiliser les packages sous mon eeepc. Afin de gagner en simplicité j'utilise ma version maison de pkgin (oui un jour elle sera intégrée upstream), en générant un INDEX.bz2 de la liste de mes packages et en mettant ça à dispo sur un serveur web par exemple

PS: pour générer l'INDEX.bz2, j'ai fait un petit prog en C: pkg_index, il n'est pas aussi complet que l'INDEX.bz2 normal mais beaucoup plus simple à utiliser et suffisant pour pkgin.

$ ./pkg_index /usr/ports/packages

PS2: pour créer mes packages j'utilise portmaster c'est moins lourd qu'une tinderbox

$ portmaster -ag

PC-BSD 8.0 Released

PC-BSD 8.0 has been released. PC-BSD is a successful desktop operating system based on FreeBSD that focuses on providing an easy to use desktop system for casual computer users. A list of new features/updates since the last version can be found here.

Improved Conference Captions from Amazon Mechanical Turk (2)

After my initial experiments last month, I applied to the FreeBSD Foundation for funds to pay for additional human editing of the YouTube machine generated transcripts. The screenshot on the left shows an example HIT (Human Intelligence Task) available on Amazon Mechanical Turk.The task description on the left is based on a template I created with three variables: $VIDEO_URL, $VIDEO_TITLE, and $CAPTIONS_URL. New HITs are then created by uploading a CSV file with three columns for each of those variables, e.g.
VIDEO_URL,VIDEO_TITLE,CAPTIONS_URLhttp://www.youtube.com/watch?v=mMmbjJI5su0,"BSD v. GPL, Jason Dixon, NYCBSDCon 2008",http://people.FreeBSD.org/~murray/improved-captions-bsdvsgpl.sbvhttp://www.youtube.com/watch?v=Pe8LdJpBGJ4,"Isolating Cluster Jobs for Performance and Predictability, Brooks Davis (DCBSDCon 2009",http://people.FreeBSD.org/~murray/improved-captions-isolatingcluster.sbv
Using this method I created 12 HITs for the first pass of editing for which I offered between $9 and $14 per video. A slightly modified template with the same three variables was used to pay ~$7 per video for a second pass to further improve the transcripts improved in the first pass. The template has gotten more detailed over the past month in response to all of the minor ways that workers submitted less than perfect transcripts. The actual SBV file format used by YouTube captions is not formally specified anywhere as far as I can tell, but the 60 character maximum width and simple format can be verified in submitted transcripts with a few emacs macros.The transcript files have been checked into the FreeBSD Doc CVS Repository. The full list of videos with human-edited English language transcripts is:

Improved Conference Captions from Amazon Mechanical Turk (2)


After my initial experiments last month, I applied to the FreeBSD Foundation for funds to pay for additional human editing of the YouTube machine generated transcripts. The screenshot on the left shows an example HIT (Human Intelligence Task) available on Amazon Mechanical Turk.

The task description on the left is based on a template I created with three variables: $VIDEO_URL, $VIDEO_TITLE, and $CAPTIONS_URL. New HITs are then created by uploading a CSV file with three columns for each of those variables, e.g.

VIDEO_URL,VIDEO_TITLE,CAPTIONS_URL
http://www.youtube.com/watch?v=mMmbjJI5su0,"BSD v. GPL, Jason Dixon, NYCBSDCon 2008",http://people.FreeBSD.org/~murray/improved-captions-bsdvsgpl.sbv
http://www.youtube.com/watch?v=Pe8LdJpBGJ4,"Isolating Cluster Jobs for Performance and Predictability, Brooks Davis (DCBSDCon 2009",http://people.FreeBSD.org/~murray/improved-captions-isolatingcluster.sbv


Using this method I created 12 HITs for the first pass of editing for which I offered between $9 and $14 per video. A slightly modified template with the same three variables was used to pay ~$7 per video for a second pass to further improve the transcripts improved in the first pass.

The template has gotten more detailed over the past month in response to all of the minor ways that workers submitted less than perfect transcripts. The actual SBV file format used by YouTube captions is not formally specified anywhere as far as I can tell, but the 60 character maximum width and simple format can be verified in submitted transcripts with a few emacs macros.

The transcript files have been checked into the FreeBSD Doc CVS Repository. The full list of videos with human-edited English language transcripts is:

MongoDB and durability

It looks like my post about MongoDB got a lot more popular than usual, and also provoked a sort-of official response from the MongoDB developer(s). It is fair to metion them together to allow people finding one part of the story to find the other. Since my original post talks about multiple issues and the comments wander through various topics I want to summarize the part of the discussion about durability here.

Read more...

MongoDB and durability

It looks like my post about MongoDB got a lot more popular than usual, and also provoked a sort-of official response from the MongoDB developer(s). It is fair to metion them together to allow people finding one part of the story to find the other. Since my original post talks about multiple issues and the comments wander through various topics I want to summarize the part of the discussion about durability here.

Read more...

HAST Project is Complete!

Late yesterday, Paweł Jakub Dawidek committed HAST to HEAD, marking the completion of this Foundation sponsored project. We asked Pawel to write a few words about the project. He says:

HAST is ready!

I'm very happy to report to FreeBSD users that the HAST project I was working on for the last three months is ready for testing and already committed to the HEAD branch.

I'll describe what HAST does in few words. HAST allows for synchronous block-level replication of any storage media (called GEOM providers, using FreeBSD nomenclature) over a TCP/IP network for fast failure recovery. HAST provides storage using the GEOM infrastructure, meaning it is file system and application independent and can be combined with any existing GEOM class. In case of a primary node failure, the cluster will automatically switch to the secondary node, check and mount the UFS file system or import the ZFS pool, and continue to work without missing a single bit of data.

I must admit the project was quite challenging, not only from the technical point of view, but also because it was sponsored by the FreeBSD Foundation. The FreeBSD Foundation has a great reputation and is known to select the projects it funds very carefully. I felt strong pressure that should I fail, the FreeBSD Foundation's reputation might be hurt. Of course, not a single dollar would be spent on a failed project, but the FreeBSD community's expectations were very high and I really wanted to do a good job.

During the work a number of people contacted me privately offering help, explaining how important HAST is for FreeBSD and giving me the motivation to soldier on.

I hope that HAST will meet the community's expectations and I myself am looking forward to using it :)

Once again, I'd like to thank the HAST sponsors: the FreeBSD Foundation, OMCnet Internet Service GmbH, and TransIP BV.

Writing FreeBSD NIC driver

>I've been through writing NIC driver 2.5 times. 0.5 was porting ADM5120 switch driver by Ruslan Ermilov and Vsevolod Lobko from NetBSD. The usual routine for this kind of thing is "take existing driver and rewrite it", e.g. copy selected parts or remove unnecessary ones. So I decided that it would be nice to skip "remove" part of procedure next time.

All cards I had to deal with ("both" wouldn't be that impressive here) had similar design save for registers layout and some quirks. I believe that vast majority of NICs have the same design to some extent: there are circular RX/TX rings of more or less similar structure, interrupt status/mask register, media settings registers, you name it. Not a rocket science.

So I took if_arge driver from Atheros AR71XX SoC and replaced hardware-dependent parts with FIXME comments. Also string "ARGE" was replaced to "ADAPTER" and "arge" to "adapter" so simple s/adapter/xyz/g and s/ADAPTER/XYZ/g would give us a half-baked source base for if_xyz driver.

It's yet to be tested whether this approach would be of any good. I'm planning to try it in next few days :) Meanwhile you can check sources here.

Writing FreeBSD NIC driver

I've been through writing NIC driver 2.5 times. 0.5 was porting ADM5120 switch driver by Ruslan Ermilov and Vsevolod Lobko from NetBSD. The usual routine for this kind of thing is "take existing driver and rewrite it", e.g. copy selected parts or remove unnecessary ones. So I decided that it would be nice to skip "remove" part of procedure next time. All cards I had to deal with ("both" wouldn't be that impressive here) had similar design save for registers layout and some quirks. I believe that vast majority of NICs have the same design to some extent: there are circular RX/TX rings of more or less similar structure, interrupt status/mask register, media settings registers, you name it. Not a rocket science. So I took if_arge driver from Atheros AR71XX SoC and replaced hardware-dependent parts with FIXME comments. Also string "ARGE" was replaced to "ADAPTER" and "arge" to "adapter" so simple s/adapter/xyz/g and s/ADAPTER/XYZ/g would give us a half-baked source base for if_xyz driver. It's yet to be tested whether this approach would be of any good. I'm planning to try it in next few days :) Meanwhile you can check sources here.

Writing FreeBSD NIC driver

I've been through writing NIC driver 2.5 times. 0.5 was porting ADM5120 switch driver by Ruslan Ermilov and Vsevolod Lobko from NetBSD. The usual routine for this kind of thing is "take existing driver and rewrite it", e.g. copy selected parts or remove unnecessary ones. So I decided that it would be nice to skip "remove" part of procedure next time.

All cards I had to deal with ("both" wouldn't be that impressive here) had similar design save for registers layout and some quirks. I believe that vast majority of NICs have the same design to some extent: there are circular RX/TX rings of more or less similar structure, interrupt status/mask register, media settings registers, you name it. Not a rocket science.

So I took if_arge driver from Atheros AR71XX SoC and replaced hardware-dependent parts with FIXME comments. Also string "ARGE" was replaced to "ADAPTER" and "arge" to "adapter" so simple s/adapter/xyz/g and s/ADAPTER/XYZ/g would give us a half-baked source base for if_xyz driver.

It's yet to be tested whether this approach would be of any good. I'm planning to try it in next few days :) Meanwhile you can check sources here.

The state of Mono 2.6 on FreeBSD

Long time no post... Well, I an writing a serie of articles about NFC and it takes me a lot of time, not mentioning I still do many other things in parallel. If you are interested in this area, stay tuned ;-)

So apart NFC, what's new with Mono on FreeBSD? First, you wight have seen that mono 2.6 has been released for a while and is still not available in FreeBSD ports. The main reasons are:

  • a thread concurrency problem which is still not solved but an ugly workaround have been found;
  • a bug (well, a weird implementation) of the new built-in soft-mode debugger that makes it totally useless on amd64. A workaround have been found and in the meantime, the code has been updated upstream and will be part of mono-2.6.2 which will probably be released at the end of the month;
  • the moonlight tools (e.g. the smcs.exe compiler) are still part of mono (read the code source is in mono-2.6.tar.bz2) but installed by moonlight (read the Makefiles with install targets are in moonlight-2.0.tar.bz2). This leads to some juggling which is in progress (read: I can build and install it on my system, Firefox segfaults when I browse a website with Silverlight contents, and configure fails in Tinderbox because it cannot find bison, which is however installed according to the logs).

I will not mention here the mono-basic-2.6.tar.bz2 source tarball that exists besides the moonlight-2.0.tar.bz2 tarball in the moon(light) source directory and the mono-basic-2.6.tar.bz2 source tarball in the mono-basic directory having different dates & checksums… Lot of fun to come I guess.

Once these various problems have been fixed, you can expect mono-2.6 to reach the ports, promptly followed by the last version of the MonoDevelop IDE which is quite lovely.

Last but not least, I am now a ports committer and flz@ is my mentor.

Preliminary Arduino Port for FreeBSD

One of the things that has been on my TODO list for quite some time was to port the Arduino IDE over to FreeBSD. Fortunately, Warren Block took the time to sit down work on a port and he is please to announce that a preliminary version of it is ready for testing.

I’ll be testing it out over the next few days and I encourage you to do the same. As always, any feedback or patches will be much appreciated. If all goes well, I will be committing it to the tree in the very near future.

The port can be found on GitHub: http://github.com/wblock/Arduino-port-for-FreeBSD

Call for Help Xorg Team need Fresh Blood!

Howdy All,

How you all know is Robert Noland our X guy but he lose most of his time
for his new job and x11 is to many for one people. Robert is dealing
most time with x stuff on the src site and we need now some people to
help him on the ports side. Beat@ and I have been started to help him,
we’ve setup a SVN [1] and small wiki page [2] with all needed infomations.
If you have intrested to help us a bit please mail me back or join
us via irc EFnet/#freebsd-xorg.

[1]
http://trillian.chruetertee.ch/ports/browser/branches/xorg-dev

[2]:
http://wiki.freebsd.org/ModularXorg/7.5

Many Thanks

- Martin

ports feature freeze now in effect

In preparation for 7.3-RELEASE, the ports tree is now in feature freeze.

Normal upgrade, new ports, and changes that only affect other branches are allowed without prior approval but with the extra Feature safe: yes tag in the commit message. Any commit that is sweeping, i.e. touches a large number of ports, infrastructural changes, commits to ports with unusually high number of dependent ports, and any other commit that requires the rebuilding of many packages is not allowed without prior explicit approval from portmgr after that date.

When in doubt, please do not hesitate to contact portmgr.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.