Author Archives: Bapt

pkg(8) passes coverity scans

At FOSDEM phk@ reminded me to always on regular basis make static analysis of the code via all possible tools available.

We did but on unregular basis and only paid attention to very critical reports And not all reports.

That is now fixed, I relaunched a few scan via coverity and I'm happy to say that the latest scan on master claims 0 defects!

Meaning that all known defects have been fixed.

I was also planning to use lint(1) as well, unfortunatly on FreeBSD lint is not supporting C99...

If I'm brave enough I may synchronise lint(1) with NetBSD which seems to have added C99 support to that tool. Or maybe someone will volunteer to do it? :)

pkg(8) is now the only package management tool

The ports tree has been modified to only support pkg(8) as package management system for all supported version of FreeBSD.

if you were still using pkg_install (pkg_* tools) you will have to upgrade your system.

The simplest way is

cd /usr/ports/ports-mgmt/pkg
make install

then run


You will have lots of warning, don’t be scared, they are expected, pkg_*  databases used to get easily mangled. pkg2ng is most of the time able to deal
with it.

If however you encounter a problem then please report to [email protected]

A tag has been applied to the ports tree if you need to get the latest ports tree before the EOL of pkg_install:

A branch has been created if some committers want to provides updates on the for pkg_install users:

Please note that this branch is not officially maintained and that we strongly recommend that you do migrate to pkg(8)

The ports tree is now stage only

The ports tree is now fully staged (only 2% has been left unstaged, marked as broken and will be removed from the ports tree if no PR to stage them are pending in bugzilla).

I would like to thank every committer and maintainers for their work on staging!
It allowed us to convert more than 23k packages to support stage in only 11 months!

Staging is a very important state, it allows us to right now be able to run quality testing scripts on the packages (which already allowed to fix tons of hidden problems) and it allows use to be able to build packages as a regular user!

It also opens the gates to new features that users have been requesting for many years:

  • flavors
  • multiple packages

Expect those features to happen in the near future.

2014Q3 Branched

The 2014Q3 branch has just been branched and the package builder has been
updated to use that branch. This means that the next update on the quarterly
packages will be on the 2014Q3 branch.

What happened during the last 3 months:
- 177 different committers have participated
- 9918 commits happened
- diffstat says: 23646 files changed, 554070 insertions(+), 577210 deletions(-)

What does that means for users:
- default Java is now 1.7
- massive conversion to stagedir (93% of the ports are now properly staged)
- massive improvement of the usage of libtool (which reduces a lot overlinking)
- new USES: mono, objc, drupal, gecko, cpe, gssapi, makeinfo
- new Keywords for plist: @sample, @shell
- LibreOffice has been updated to 4.2.5
- Firefox has been updated to 30.0
- Firefox-esr has been updated to 24.6
- Default postgresql has moved from 9.0 to 9.2
- nginx has been updated to 1.6.0
- Default lua is 5.2
- subversion has been split into multiple ports for each features
- On FreeBSD 9-STABLE and 10-STABLE the default xorg 1.12.4 (for default binary
packages it is still 1.7.7)
- Improved QA checking in the infrastructure
- Info files are handle correctly even if base has been built WITHOUT_INFO
- Ancient emacs version has been cleaned out

fossil in prison

Since my last post about how I do host fossil I have been asked write about the new setup I do have

The jail content

I have created a minimal jail:

$ find /usr/local/jails/fossil -print

/bin/sh is necessary to get the exec.start jail argument to work /var/tmp is necessary to get fossil to open his temporary files (I created it with 1777 credential) /data is a empty directory where the fossil files will be stored

Jail configuration

The configuration file is the following:

fossil {
	path = "/usr/local/jails/fossil";
	host.hostname = "";
	exec.start = "/bin/fossil server -P 8084 --localhost --files *.json,*.html,*.js,*.css,*.txt --notfound /index.html /data &";
	exec.system_jail_user = "true";
	exec.jail_user = "www";
	exec.consolelog = "/var/log/jails/fossil.log" ;

More about fossil itself

In /data I created an index.html which is an almost empty html with a bit of Javascript.

When loading the javascript will request a list.txt file.

This file contain the list of repositories I want to show publically (one per line).

For each of them the javascript will use the json interface of fossil (meaning your fossil has to be built with json) and gather the name and the description of the repo to print them on the index.

Starting/Stopping the service

2 simple command are necessary to manage the service:

Starting up:

# jail -c fossil


# jail -r fossil

The service is only listening on the localhost, it is up to you to create your reverse proxy, in my case I do use nginx with the following config:

server {
	listen       [::]:443 ssl;
	listen       443 ssl;
	ssl_certificate     ssl/fossil.crt;
	ssl_certificate_key ssl/fossil.key;

	location / {
		client_max_body_size 10M;
		proxy_buffering off;
		proxy_set_header HTTPS on;
		proxy_set_header   Host             $host;
		proxy_set_header   X-Real-IP        $remote_addr;
		proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

Ports 2014Q2 branched

I am pleased to announce that we have created the 2014Q2 branch of the ports

Because the first 2014Q1 branch was experimental you might not have heard of it

January 2014 saw the release of the first quarterly branch, intended at
providing a stable and high-quality ports tree. Those stable branches are a
snapshot of the head ports tree taken every 3 months and currently supported
for three months, during which they receive security fixes as well as build and
runtime fixes.

Packages are built on regular basis on that branch (weekly) and published as
usual via (/quarterly instead of the usual /latest).

They are signed the same way the /latest branch is.

While packages for 2014Q1 were only built for 10 (i386 and amd64) 2014Q2 will be
built for both FreeBSD 9 and 10 (i386 and amd64).

The first build of 2014Q2 will started this morning (wednesday at 1 am UTC) and should
hit your closest mirrors very soon.

On behalf of the port management team

The Ports Management Team 2013-10-03 14:05:32

For the sake of being able to have clean working binary packages, pkgng has had to use a dirty hack: origin is used has an identifier instead of the package name.

Why: How to determine than ImageMagick-nox11 and ImageMagick are the same package? and if I don’t know they are the same package how to upgrade them properly? same goes how the package manager can know that mysql-client-5.2.1 is not an upgrade of mysql-client-5.0.1?

How can pkgng determine which subversion is the right version to use: ‘pkg install subversion’ will propose all possible subversion, in that case the higher version is probably the default version, but in perl case, the higher version is probably not the one you want.

Some packages are totally different project but have the same package name… How can pkgng differenciate them?

So please stop using LATEST_LINK to differenciate the different port but directly make _unique_ package names, pkgng will switch back to package name as soon as possible (that will also solve the ugly and stupid pkg set -o).

There is multiple possibility to make sure to properly handle multiple versions of packages:

  1. suffix everything with part of the version (like tcl)
  2. suffix everything but the default with part of the version
  3. have only 2 or 3 different version: project-legacy project project-devel

Please stop renaming the packages based on options! Options are now registered with the package with both pkgng and pkg_install, please stop adding suffix base on options!

Think about the end user, he will try to install a package, it should be as transparent for him as possible.

I is really important to change that as quick as possible. This page on the wiki reference all the port with conflicting names.

The Ports Management Team 2013-10-03 14:05:12

You may has notice that staging has hit the ports tree, staging is something really important, all packages system are using that feature for eons, sometime called DESTDIR sometime called FAKEDIR.

Staging is consistent in adding a new step while building packages: install everything into ${STAGEDIR}. Then we can directly create packages out of that directory without having to install into /. What the implementation does is:

With pkg_install (legacy package tools):

  • Create a package from the stage directory
  • Install that package.

With pkgng:

  • Create a package from that stage directory


  • Install to / (pkgng can consider the stage directory as if it is a package)

What it means is, that it is the end of the bad plist, only things in plist are really installed, this is the end of the broken packages that break the system because they fail in the middle of make install as a package will only be installed once we are sure the build process properly pass.

It allows right now to build and create a package without the need to be root.
It gives us new targets:

  • make makeplist (to create the pkg-plist)
  • make check-orphans (to print what is in the stage directory but not in pkg-plist)

It reduces code duplication: in the end the installation is being done via a package installation meaning that PKG-MESSAGE is automatically printed, all pre/post installation scripts are executed.

It reduces patches: no need anymore to patch upstream Makefiles to avoid installing the DOCS just do not list them in pkg-plist or conditionnaly list them in the pkg-plist.

This also allows lots of new features to come:

  • Allow to create sub-packages
  • Allow to create debuginfo packages
  • Allow to do a lot of sanity check in the staging area to improve our QA

To convert your ports refer to:

Along with stage you have noticed that MAN* and MLINKS has gone, and now all the manpages should be added to the plist. BTW MANCOMPRESSED is also not necessary now.

There is 3 reason behind making it go and not being replaced:

  1. The initial goal of those macros was to be able to compress the right manpages and to handle the different hardlinks/symlinks between those pages. Because it was directly installed in / rather than using a stage directory, it was necessary to at a point list them. and to properly track the different links in MLINKS (which wasn’t done properly in most ports btw :) ). With stage, we have a new compress-man which does it properly on its own without the need to get the list of the manpages, without the need of an explicit declaration of the links. This syntax to handle localized man pages was also terrific :) and if you have a port mixing manpages in “non regular” and “regular” localtion, it was totally messing
  2. Consistency, a port is basically: some metadata, a plist and some operations to perform. for metadata all metadata are in the Makefile, all operations are in the Makefile but plist can be a mix of pkg-plist and Makefile, this is inconsistent, it makes sometime hard to figure out why a file has been added to the plist etc. Also this makes us having tons of targets define to handle those extras entries and results in highly inefficient make(1) usage.
  3. Stage is also a first step to go to sub-packages, sub-packages will basically be: 1 port able to produce N packages, to be able to do this we will use multiple plist, having the files properly defined already in plist will help that. Having files defined in macros on Makefile will make it hard todetermine which one should go in which plist.

Last thing I would like to add about it: I don’t see the difference personnaly between listing N lines of manpages into Makefile MAN* macros where btw you have to manually define the categories where to put them and adding those N lines
directly into the plist where make makeplist and/or make check-orphans can help you getting the exact lines automatically?

pkg 1.1

After almost a year of development pkg 1.1 has reached the ports tree, actually pkg 1.1.1 has 1.1 was too buggy :(

What happened in 1 year of development (I'll focus on use visible features)


The multi-repository support was experimental in pkg 1.0 and to be honest it was not really usable. With pkg 1.1 the support has been greatly improved and it is now the default behaviour (you can't deactivate).

To define repository you just have to create a simple configuration file in /usr/local/etc/pkg/repos/myrepo.conf

  url: http://myurl
  pubkey: /usr/local/etc/pkg/repos/myrepo.key
  mirror_type: SRV

Meaning you can provide a package to autosetup a repository creating a package containing like this one:

$ tar tf myrepo-1.0.txz

Host this file somewhere and say to the use to do the following

$ pkg add http://yourhost/myrepo-1.0.txz

Now you can see that the repository is configured properly pkg -vv should show you in the last lines:

                enabled: yes
                    url: http://myurl
                    key: /usr/local/etc/pkg/repos/myrepo.key
                enabled: yes
            mirror_type: SRV

The user can also choose to make sure a given package will always be updated from 'myrepo'

$ pkg install -r myrepo mypackage
$ pkg annotate -A mypackage reposiroty myrepo

Now the package 'mypackage' will only be updated from 'myrepo'

pkg lock/unlock

If a use want to prevent a package from being updated anyway he can just lock it:

$ pkg lock mypackage

To unlock it just update use the following command:

$ pkg unlock mypackage

ssh transport

If your server has pkg 1.1+ installed then you do not need so set up a HTTP server or a FTP server, pkg can now use ssh to share the packages

packagesite: ssh://user@host:/path

Or in the repository configuration:

url: ssh://user@host:/path

Do not forget to restrict on the server the directory where files can be retrieved by adding the following line on the server pkg.conf:



This allows to add any key/value annotation to a given package once installed, if you recreate the package after that, the annotation will be added to the manifest and then a new reinstallation will keep the annotation.


pkg now supports 2 kind of plugins: commands (to add new subcommand to pkg) and hooks (which will be executed in the middle of any process of pkg).

I'll write another post dedicated to plugins later.

explained reinstallation

As pkg is able to determine that a package needs to be reinstalled because the remote one has been compiled with new options or the required shared libraries for the package has changed, pkg now explains why a package will be reinstalled.


We have stabilized the public API, so now bindings, and program using libpkg are more than welcome :) Lots of cleanup has occurred in the code, and lots of code optimisation. New pkg_printf(3) function to help printing a preparing strings with pkg informations. We are more and more adding some regressions test using the ATF framework. The catalog has changed and is now a simple yaml files which gives us more flexibility and allow simples incremental update. pkg audit can now directly read the vuxml native format.

Way more things but I'll let you discover :)

Thanks to all people that has been involved in the new release (coders, testers, doc writers, etc.)

Stash for svn

When hacking on the ports tree or on the sources, you often have unfinished patches you are testing step by step.

I'm also hacking on something unfinished and then some other area needs some fixes with a higher priority and in the same time some people are asking for some testing/review on their own patches. So I need to quickly interrupt what I was working on get back to a clean tree, and switch from patches to patches.

While doing this is easy with git, fossil or mercurial it is more complicated with svn. The feature in particular I use on fossil/git for that is the stash feature.

So I wrote my own stash for svn, and because of mobility I was willing to be able to share my patches across boxes, so I have made stash able to be under a vcs itself, with support for git, hg, fossil and svn as a vcs for the stash.

How does it works: The stash command will discover that .svn of the working copy you are working on and will create a patches subdirectory.

Now imagine that directory itself contains a .hg, .fslckout, a .git or a .svn then stash will know it is being under vcs control.

$ stash save <name> [-u] [files...]

This will create a .svn/patches/.patch file using svn diff --git on the specified files (if none is provided it will diff the current directory the stash command is being run on).

Once the diff created it will rollback the tree (on specified files or current directory) to the clean state before any modification.)

By default it will not overwrite a patch with the same name except is -u is provided on the command line.

If the stash directory is under a vcs control then a add/commit (or just commit in case of update) will be performed in the stash directory.

$ stash ls

List all the available patches.

$ stash show <name>

Print on stdout the content of a given patch

$ stash apply <name>

Apply a specified patch on the working copy using svn patch --strip 1 from the root of the working copy.

$ stash rm <name>

Remove/destroy a patch from the stash directory. In case the stash directory is under vcs control then the proper rm command followed by the needed commit will be performed.

$ stash pop <name>

It is the equivalent of stash apply followed by stash rm this is useful when your patch is finished and you want to commit it directly.

$ stash push <name>

Push (scp) the patch on a remote site (currently freefall is hard coded :)

$ stash sync <name>

This command is only useful if the stash directory is under vcs control, it performs the necessary pull/push mechanism depending on the VCS used.

I use fossil to maintain the stash script And here is for example my repository of patches for the ports tree

No git svn won't have worked in my case for multiple reason: 1/ want something flexible which can also only work with svn 2/ the ports tree can work properly with git svn (properties setting adding new files etc will not work as one expect) 3/ I want to use fossil for the stash, other might prefer svn or hg.

Disclaimer: hg and git support hasn't been tested yet, patches welcome to fix them if needed. if you want to add support for your own favorite vcs just them me the patches I'll integrate them.

Conversion to new options: done

It took a year but at least it is done! Huge thanks to all people involved in the new Option Framework.

So now what we have:

  • A consisten way to set options in make.conf (OPTIONS_SET, OPTIONS_UNSET, ${UNIQUENAME}_SET, ${UNIQUENAME}_UNSET )
  • 0-1 options
  • only 1 options
  • 0 or N options
  • 1 or N options
  • helpers to sanely create slaves (OPTIONS_SLAVE aand OPTIONS_EXCLUDE)

What still needs to be done:

  • Stop popping the dialog only for global options
  • Fix the very old bug about OPTIONSFILE name dansing (PKGNAMEPREFIX changing)

With the previous done, my 2 main priorities for FreeBSD now are:

  • Get pkg 1.1.0 released
  • Introduce the next big and long awaited change to the ports tree: stage directory

New options framework is in, what next?

The new options framework has been committed one week ago, I would have expecting things to have been smoother but lots of fixes has been needed to have it correctly working.

In one week everything at least every major issues reported has been fixed, and now everything is working again as expected (if not please report it to me asap so I can fix it)

So from a maintainer point of view what does it brings:

  • mutually exclusive options (only 1 among N options) aka single
  • optional mutually exclusive options (could be done better)
  • grouped options (at least 1 among N options) aka multi
  • optional grouped options (could be done better same as single)

From a port developer point of view:

  • the code is cleaner and easier to review modified.
  • the code is easier to extend, I know some people already trying to extend it.

From a user perspective:

  • options are consistent: you can set global or per port options which will be pass over the whole ports tree in a consistent way, which means yes you can drop all X11 on all ports supporting it in one single setting!
  • options are simpler, no more WITH_BLA or WITHOUT_BLA just a simple BLA options which you need to enable/disable

How to enable/disable an options? simple in make.conf:

OPTIONS_SET= BLA # enable BLA for the whole ports tree
OPTIONS_UNSET=	BLA # disable BLA for the whole ports tree
foo_SET= BLA # enable BLA just for foo
foo_UNSET= BLA # diable BLA just for foo

the user settings have an order or priority:

  • OPTIONS_SET is applied
  • OPTIONS_UNSET is applied
  • ${UNIQUENAME}_SET is applied
  • ${UNIQUENAME}_UNSET is applied
  • all of the previous is overriden by make config

Note that you can disable this automatic make config by adding NO_DIALOG to your /etc/make.conf.

All of this means that ports-mgmt/portconf is not needed anymore for a centalisation of your configurations.

Short term goal: to avoid having the ports tree infrastructure keeping for ever old code like it was the case in the past for other manjor improvements, please update your ports to the newoption framework as soon as possible.

Long term goal:

  • please convert any of the knobs you are using the new options framework, it will be more consistent for users.
  • please try to reuse options name for example if your ports do support GTK2 use GTK2 no GTK, it is more consistent for users.
  • we now have share options, please fix the description for it to be the more user friendly possible, and please overwrite the description in your port each time it makes sense!

The options framework was an important step in the cleanup of the ports infrastructure, next step in my concern will be to provide stage directory support aka: a ports will install everything in a stage directory and sync the files following the plist to the system or directly create the package out of this stage directory. Every sane package building system are using that kind of thing but the FreeBSD ports this is necessary as it also open the way to tons of new possibilities. Stay tune, this is coming soon!

Permission de sortie : 30 minutes

Le Besoin

  • Un firewall qui interdit tout sauf le port 443 local de la machine.
  • Un système permettant d'ouvrir pour 30 minutes seulement l'accès a un port du firewall donné

Les ingrédients

  • FreeBSD 8.0
  • PF
  • AT
  • /bin/sh
  • sudo



scrub in all
set skip on lo0
pass out all
block in all
table <allowips> persist
pass in quick inet proto tcp port https keep state
pass in quick inet proto tcp from <allowips> to port 8080 keep state

Voila tout simple tout beau, en gros on bloque tout sauf le https. On autorise toutes les ips qui sont contenu dans la table allowips

Problème : comme rajouter des adresses IPs dans la table allowips et surtout comment les supprimer au bout de 30 minutes

CGI shell

Nous créons donc un script cgi qui après une authentification "ultra basique" ajoutera l'adresse IP du demandeur dans la table allowips et schedulera la supression de celle ci

if [ "$REQUEST_METHOD" = "POST" ];then
	query="dd count=$CONTENT_LENGTH bs=1 2> /dev/null"
	echo "Bad request"

# Ceci n'est pas super clean, il y a mieux à faire mais flemme :)
eval $query

if [ -z $passwd ]; then
	echo "Bad request"

decodedpasswd=$(echo $passwd | awk '"'BEGIN{for(i=0;i<10;i++)hex[i]=i;hex["A"]=hex["a"]=10;hex["B"]=hex["b"]=11;hex["C"]=hex["c"]=12;hex["D"]=hex["d"]=13;hex["E"]=hex["e"]=14;hex["F"]=hex["f"]=15;}{gsub(/\+/," ");i=$0;while(match(i,/%../)){;if(RSTART>1);printf"%s",substr(i,1,RSTART-1);printf"%c",hex[substr(i,RSTART+1,1)]*16+hex[substr(i,RSTART+2,1)];i=substr(i,RSTART+RLENGTH);}print i;}'"')

if [ -f /usr/local/etc/authpasswd ]; then
	read encryptpasswd < /usr/local/etc/authpasswd
	echo "No password file"

cryptpasswd=$(echo $decodedpasswd | /sbin/sha256)
if [ "$cryptpasswd" = "$encryptpasswd" ]; then
	/usr/local/bin/sudo /sbin/pfctl -t allowips -T add $REMOTE_ADDR
	echo "/usr/local/bin/sudo /sbin/pfctl -t allowips -T delete $REMOTE_ADDR" | /usr/bin/at now + 30 minute
	echo "Acces granted for 30 minutes"
	echo "Acces refused"

C'est pas terrible mais suffisant : le mot de passe de référence est stocké dans /usr/local/etc/authpasswd et est en sha256

Il faut maintenant autoriser l'utilisateur www à pouvoir utiliser at :

$ echo "www" > /var/at/at.allow

Ensuite rajouter les entrées sudo qui vont bien dans /usr/local/etc/sudoers

%www	ALL=NOPASSWD:	/sbin/pfctl -t allowips *

Il vous reste à faire un fichier html qui appelle le script CGI via la méthode POST en lui fournissant le mot de passe donné dans une variable "passwd". Mettre le tout à dispo via un serveur web supportant le CGI.

Après on me demande pourquoi j'aime les Unix...

EDIT : alors on me dit dans l'oreillette que pfctl -t allowips -T expire 1800 lancé toutes les minutes dans une crontab ferait le même boulot mais je préfère ma solution à coup de at puisque at est géré par cron sous FreeBSD et que ce n'est exécuté qu'au besoin et non toutes les minutes

Fresh blood needed for LibreOffice

For some time now, I have been maintaining pretty much alone the LibreOffice port for FreeBSD, trying to provide all the features LibrOffice has to our users.

I managed to port the 3.3.x series of LibreOffice to FreeBSD some time ago and tried to keep updating it as fast as possible as soon as new releases were out.

At the beginning I was thinking about maintaining 2 concurrent of LibreOffice at the same time: legacy and "normal" to reflect the upstream support. But this take too much time, I'll kill libreoffice-legacy in the next weeks.

The problem is that it is really time consuming, not that it is really complex, the LibreOffice upstream is really nice and really responsive, other maintainers from the BSD community, has also been really helpful.

Some time ago, I decided to create office@ to help maintaining every office related ports maintained on FreeBSD, it worked quite well, sunpoet@ taking good care for examples of the different hunspell/hyphen/mythes dictionnaries/thesaurus, pfg and maho taking care of Apache OpenOffice and all of us working on the third party libraries/fonts/etc needed by all those big office project.

But I'm still missing help on LibreOffice itself, I just committed 3.5.2 some days ago, and 3.5.3 is almost there.

We also still have known issues on 3.5.2:

  • Doesn't build on recent -HEAD (problem with clang 3.1): all the fixes are in the upstream git, but need to be tracked and backported.
  • Doesn't work propertly with base lpd.
  • Doesn't build using the WITH_DEBUG option
  • Doesn't build with gcc from ports (gcc from base is too old for libreoffice)

All the libreoffice work is done on redports svn

if you have an account and are willing to help tell me so that you can have access to the office svn on redports.

Home made pkgng repositories

By popular demand I have been requested a tutorial about how to maintain your own pkgng repositories.

Currently the best solution for that is to use poudriere (ports-mgmt/poudriere).

Let say you want to maintain 2 repositories: 8.2-RELEASE i386 and 9.0-RELEASE amd64.

For that you will need an amd64 (to support both amd64 and i386 box with 9.0 binary support and 8.2 binary support, a default 9.0-RELEASE amd64 should be enough :)

poudriere only depends on zfs and sh, so you won't need much, just a zfs pool available.

First: install poudriere:

$ make -C /usr/ports/ports-mgmt/poudriere install clean

To be able to build the repository the host would also need to have pkgng installed (no need to convert it to pkgng anyway.)

$ make -C /usr/ports/ports-mgmt/pkg install clean

Now configure your poudriere, defining some configuration to /usr/local/etc/poudriere.conf:


This should be enough (see poudriere.conf.sample for more informations)

Create the default ports tree:

$ poudriere ports -c

Create the two jails (in fact it will be chroot :))

$ poudriere jails -c -j 82i386 -v 8.2-RELEASE -a i386
$ poudriere jails -c -j 90amd64 -v 9.0-RELEASE -a amd64

Make them pkgng aware

$ mkdir /usr/local/etc/poudriere.d
$ echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/82i386-make.conf
$ echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/90amd64-make.conf

Add the list of packages you want to build:

$ cat ~/mylist1
$ cat ~/mylist2

If you want special options just add them to you different make.conf in poudriere.d

to share options between all the jails managed by poudriere, add them to /usr/local/etc/poudriere.d/make.conf

You can now create your packages:

$ poudriere bulk -f ~/mylist1 -j 82i386
$ poudriere bulk -f ~/mylist2 -j 90amd64

When finished you will get 2 pkgng repositories in /poudrieredata/packages/82i386-default and /poudrieredata/packages/90amd64-default

You can now provide them through your webserver.

On you client boxes just:

$ echo "packagesite: http://yoururl/82i386-default" >> /usr/local/etc/pkg.conf


$ echo "packagesite: http://yoururl/90amd64-default" >> /usr/local/etc/pkg.conf

If your client box already has packages from an old installation, first convert it to pkgng

$ fetch http://yoururl/90amd64-default/Latest/pkg.txz
$ tar xf ./pkg.txz -s ",/.*/,,g" "*/pkg-static"
$ ./pkg-static add ./pkg.txz
$ pkg2ng

Now you can forget about pkg_install and pkg2ng

Just use pkgng (for example):

$ pkg update
$ pkg upgrade
$ pkg install firefox

To update your repository:

$ poudriere ports -u # this update your default ports tree
$ poudriere bulk -f ~/mylist1 -j 82i386 -k
$ poudriere bulk -f ~/mylist2 -j 90amd64 -k

bonus poudriere will rebuild only what has changed and what is impacted by this change.

Once it is done simply upgrade your client box:

$ pkg update
$ pkg upgrade

More informations on poudriere here

Full binary upgrade with pkgng 1.0-beta7

My laptop (a small eeepc 1000H) is running FreeBSD 8.2-RELEASE for a while now, I didn't upgraded it to 9 because it would have taken eons to rebuild all the packages I use on this laptop.

Of course this laptop has been converted to pkgng long time ago, and I'm happily using it in a full binary way: building packages and preparing pkgng repositories on a build box using poudriere (ports-mgmt/poudriere).

In the git version of pkgng (what would become soon 1.0-beta7) one of the new feature is the ability to reinstall the whole set of packages. This allow us to now perform a full binary upgrade of the system.

Here is how I performed an upgrade from 8.2-RELEASE to 9.0-RELEASE without building anything on that small laptop:

Create a 9.0-RELEASE repository

First you need to prepare a set of packages for 9.0-RELEASE on poudriere:

# create the new jail:
$ poudriere jail -c -j eeepc9 -a i386 -v 9.0-RELEASE
# copy the make.conf from the old to the new jail:
$ cp /usr/local/etc/poudriere.d/eeepc-make.conf /usr/local/etc/poudriere.d/eeepc9-make.conf
# build the set of packages:
$ poudriere bulk -f ~/bulkeeepc -j eeepc9 -k

Now that your repository of packages is ready just expose it in your http server and you are done with that part.

Upgrading the laptop

Update to the lastest 8.2-RELEASE

You need to upgrade your 8.2-RELEASE to the latest version available if not already done (it contains a fix for freebsd-update which is necessary to perform the upgrade to 9.0)

$ freebsd-update fetch
$ freebsd-update install

Upgrade to 9.0-RELEASE

You can now upgrade to 9.0:

$ freebsd-update -r 9.0-RELEASE upgrade
$ freebsd-update install
$ reboot
$ freebsd-update install

Once you are here your system now upgraded to 9.0-RELEASE but still have some old shared object files, it is the moment when you should perform the upgrade of the packages.

Edit your /usr/local/etc/pkg.conf file and adjust the url of the PACKAGESITE to the new eeepc9 repository

$ pkg update
$ pkg upgrade -fy

you can now remove the old shared object:

$ freebsd-update install

In case you would have forgotten to perform the package upgrade, pkg-static is there to save your life, you can still do it now:

$ pkg-static update
$ pkg-static upgrade -fy

Aujourd’hui j’ai un an

Il y a un an jour pour jour que je faisais mon premier commit dans les ports FreeBSD sous le mentorat jadawin@ et tabthorpe@ merci à eux :).

Ça passe vite... :)

Le bilan de cette première année est plutôt positif je pense malgré quelques ratés. Plein de bonnes choses mais aussi quelques trucs que je pensais faire que je n'ai pas encore réussi à pousser.

Je vais commencer par les échecs et finir par les réussites c'est toujours mieux de finir par les bonnes choses :)

Pas Bien

Il y a un moment déjà que je bossai sur un framework d'options pour les ports qui viserai à remplacer l'actuel en offrant plus de fonctionnalités: groupe d'options, options incompatibles, options dépendantes d'autres options etc. Dans ce cahier des charges, il fallait aussi que le nouveau framework puisse remplacer le vieux de manière transparente.

La première étape je l'ai atteinte assez facilement, la complication est venue quand il a fallu faire la seconde. Du coup un an après je n'ai toujours pas pu pousser mon framework et il reste toujours le vieux dans l'infrastructure. J'espère pouvoir changer ça rapidement, j'ai une idée à creuser sous le coude :).

Un autre loupé pour le moment c'est le fakeroot ou pkgdir. J'aimerai bien pouvoir construire des packages sans avoir à les installer. J'aimerai aussi que l'on ne puisse pas installer de packages depuis les ports dont le PLIST est foireux. Je me suis donc lancé dans la création d'un patch fakeroot (en fait j'ai piqué dans un premier temps ce que nos voisins de midnightbsd avaient fait quand ils ont forké notre système de ports). Au final j'ai obtenu quelque chose qui marche dans 80% des cas. C'est pas mal, mais c'est complexe et je n'aime pas ne pas proposer de choses simples. Donc il n'a pour le moment pas été plus loin. En revanche j'ai récemment eu de nouvelles idées à ce sujet et devrait apporter de bonnes choses :). Wait and See.

Moyen pas bien ou moyen bien au choix

Passons des loupés de cette année aux choses mitigées: libreoffice. J'ai fait le port de la version 3.3 avec succès. J'avais 2 motivations pour ça: réutiliser au maximum ce qui est déjà dans les ports (ajouter les ports nécessaires si besoin) et pouvoir tourner sur pointyhat. Dans les deux cas c'est réussit.

En revanche, du côté de 3.4 c'est une autre paire de manches. Pour des raison qui m'échappent pour le moment je n'arrive pas à le compiler :( alors que la version 3.5 (git) compile sans trop de difficultés je galère sur la version 3.4. En même temps je ne consacre pas beaucoup de temps à libreoffice du coup je ne suis pas de près les évolutions et découvre les problèmes sur le tard. J'ai créé il y a peu une équipe office@ pour en prendre soin, en espérant que à plusieurs personnes, on arrive à toujours rester synchronisés avec l'upstream. Pour le moment la mayonnaise à l'air de prendre.


Les réussites maintenant:

Je suis portmgr@ maintenant, ce qui veut dire que mes démarches pour essayer d'améliorer les choses ont été bien acceptées :). Je ne m'attendais vraiment pas à devenir portmgr et encore moins aussi rapidement. Comme quoi s'investir est payant :)

Les campagnes de "deprecation". Les ports sont tout sauf dictatoriaux, du coup tout le monde peut y mettre ce qu'il veut ou presque. Mais le problème c'est que très peu de gens pensent à enlever les ports morts. Ceux, dont il n'y a plus de distfile publiquement dispo, dont le projet est totalement abandonné, etc. Tout le monde laisse traîner ça dans ports@. Moi je n'aime pas laisser traîner les vieux trucs partout comme ça, j'ai donc lancé 2 campagnes de "deprecation" la première a déjà viré environ 500 ports, la seconde devrait en virer 200 dans quelques jours, je pense que c'est une réussite et je vais le renouveler régulièrement. :)

Enfin last but not least, PKGNG :). Il y a un peu plus d'un an je tentai de porter pkgin sur FreeBSD. Je pense avec un certain succès dans le sens où ça a fonctionné pour de vrai.

Mais j'étais déçu par le côté trop pkgsrc (logique en même temps :)) de l'outil et par les tours de magie qu'il aurait fallu réaliser pour le faire fonctionner vraiment correctement sur FreeBSD. La faute n'est pas à pkgin mais plutôt aux différences entre les ports et pkgsrc.

Bref de là je me dit qu'il faudrait tout refaire "from scratch", en profiter pour avoir une architecture autour d'une bibliothèque et tout et tout. En gros je pensais partir sur une n-ième libpkg-qui-n-aboutira-jamais(c)(tm), pas très motivant :). Heureusement je n'étais pas seul à m'intéresser au sujet. Deux personnes qui ne sont pas motivées à l'idée de faire un projet qui n'aboutira pas et qui si même si il aboutissait ne serait jamais accepté ... Bah c'est assez con pour se lancer dedans :). Julien Laffaye et moi nous lançons donc dans le projet (aidé, surtout au début, de Philippe Pépiot), de longs mois plus tard, et toujours pas convaincu de ne pas faire ça pour rien, on décide de parler du projet pour le faire connaître. En effet nous avions quelque chose de viable et espérions ramener ainsi du monde sur le projet. Ce fut une très bonne idée ce mail :), dans la foulée nous avons été invité à participer au BSDCan pour présenter pkgng, et participer au groupe de travail sur pkgng. Le baptême du feu :). Préparé à recevoir des volées de critiques sur le pourquoi de SQLite, ou de tel autre choix, nous pensions devoir défendre notre bout de gras.

Bilan: pkgng a été accepté sans trop de discussions (grosse surprise :), et est maintenant clairement vu que le remplaçant des pkg_* tools. Dans quelques jours, on devrait sortir l'alpha2 avec beaucoup d'évolutions. Mais pas de teasing pour le moment :).

PS: pkgng c'est ici que ça se passe :)

En attendant pkgng…

pkgng c'est cool, ça va être tout beau tout ça, mais voila c'est pas encore fini, il y a beaucoup de boulot à prévoir dessus encore pour que ça rentre en production.

En attendant des babasses FreeBSD on en veut toujours en production.

Voici donc comment je gère mes machines FreeBSD en full binaires.

La machine de build

Tout d'abord il vous faut une machine de build. (Je n'aime pas prendre les packages officiels, parce que je veux des versions plus récentes ou des set d'options différents, parce que avoir un mirroir avec 22000 packages ça ne me sert à rien sinon bouffer de l'espace disque.)

Faire les packages à la main n'est pas non plus très industriel, ni très sérieux.

Utiliser tinderbox me donne des boutons.

J'ai donc réutilisé un outil maison poudriere.

J'y ai rajouté une fonction bulk.

Petit rappel pour commencer, poudriere est un outil très simple ne nécessitant rien qui ne soit pas dans base. Son but est de permettre le test et la génération de packages pour FreeBSD.

Avec cette fonctionnalité de bulk, l'utilisateur donne une liste de packages à manger à poudriere et celui-ci va tous les générer, ainsi que leurs dépendances. Il va présenter le tout sous la forme d'un répertoire de packages identique à ceux officiels.

Commençons donc par créer nos environnements de build en considérant poudriere comme déjà installé.

Voici le fichier de configuration de mon poudriere:


poudriere utilise zfs il suffit de lui donner un zpool il se débrouille avec. Ici nous avons choisi "data". poudriere utilise les jails, il a donc besoin de connaître un host ftp pour récupérer les éléments nécessaires à la construction automatique des jails, une adresse IP et une interface réseau sur laquelle rajouter en alias l'adresse IP donnée.

Il a besoin de connaître l'arbre des ports sur lequel il va travailler, et où il va générer ses données: packages et logs.

Enfin quelques informations comme le WRKDIRPREFIX seront nécessaires si l'on veut tout construire en RAM (mdmfs et tmpfs possibles).

poudriere est maintenant utilisable, préparons donc les machines de build:

# poudriere createjail -v 8.2-RELEASE -n 82amd64 -a amd64 -s

Voila c'est tout ! Vous disposez désormais d'une machine de build pour FreeBSD 8.2-RELEASE pour l'architecture amd64. Vous pouvez en rajouter autant que vous voulez bien sûr.

Le bulk

Il s'agit maintenant de construire ses premiers packages. poudriere souhaite une liste, alors allons y:

# cat ~/listpkgs

C'est déjà pas mal pour tester. Il est bien sûr possible de spécifier des options de compilation via: /usr/local/etc/poudriere.d/make.conf pour les options globales et/ou /usr/local/etc/poudriere.d/82amd64-make.conf pour les options spécifiques à la jail 82amd64.

# cat /usr/local/etc/poudriere.d/82amd64-make.conf

C'est prêt on peut lancer le build:

# poudriere bulk -f ~/listpkgs -j 82amd64

Si l'option -j n'est pas donnée alors il fera sur le boulot sur chacune des jails disponibles.

Ce que fait poudriere là c'est construire dans la jail vierge (merci les snapshots zfs) tous les packages demandés, en ayant pris soin de supprimer tout résidu d'un précédent bulk. Puis il va générer l'INDEX et le compresser en bzip2.

Cette étape peut donc être très longue.

Une fois terminée les résultats se trouveront dans : /usr/local/pdata/packages/bulk-82amd64

Le Répertoire étant utilisable tel quel.

Le client maintenant

On part donc d'une installation toute fraiche de freebsd sans l'arbre des ports

# export PACKAGESITE=http://monjoliserver/avec/mes/packages/
# pkg_add -r portmaster

Quelques options pour le portmaster.rc

# cat /usr/local/etc/portmaster.rc

Comme portmaster recherche encore quelques informations sur l'arbre des ports et que l'on ne souhaite pas ce comportement :

# mkdir /usr/ports/Mk
# touch /usr/ports/Mk/

portmaster est maintenant utilisable:

# portmaster www/newsbeuter
===>>> The following actions will be taken if you choose to proceed:
	Install www/newsbeuter
	Install devel/gettext
	Install converters/libiconv
	Install devel/pkg-config
	Install devel/stfl
	Install ftp/curl
	Install security/ca_root_nss
	Install textproc/libxml2
	Install databases/sqlite3

===>>> Proceed? y/n [y]

===>>> The following actions were performed:
	Installation of converters/libiconv (libiconv-1.13.1_1)
	Installation of devel/gettext (gettext-
	Installation of devel/pkg-config (pkg-config-0.25_1)
	Installation of devel/stfl (stfl-0.21_1)
	Installation of security/ca_root_nss (ca_root_nss-3.12.9)
	Installation of ftp/curl (curl-7.21.3_1)
	Installation of textproc/libxml2 (libxml2-2.7.8_1)
	Installation of databases/sqlite3 (sqlite3-3.7.5)
	Installation of www/newsbeuter (newsbeuter-2.4)

Voila retour/docs/patchs bienvenus, as usual :)

Nano jails

A la demande populaire voici un micro howto sur comment créer des nanos jails(8).

Pour cela je vais vous monter comment tourne mon blog (à base de cblog).

Quelques petites commandes pour bien montrer de quoi on parle

$ jls -j cblog
JID  IP Address      Hostname                      Path
1       cblog                         /zdata/jails/cblog/
$ du -hs /zdata/jails/cblog/
2,3M    /zdata/jails/cblog

c'est tout rien de plus.

Il y a même des choses en trop par exemple cette jail n'a pas besoin de réseau pour fonctionner normalement.

En fait une jail n'a pas besoin de grand chose pour fonctionner et depuis que l'on dispose des jails persistantes elles ont même besoin de rien du tout.

Examinons un peu le contenu de cette jail :


Nous y retrouvons donc le strict nécessaire pour faire tourner cblog.

pour en arriver là il faut préparer le fs. (je suis utilisateur de zfs :))

$ zfs create /zdata/jails/cblog
$ mkdir /zdata/jails/cblog/bin
$ mkdir /zdata/jails/cblog/lib
$ mkdir /zdata/jails/cblog/libexec

Je dépose ensuite le binaire cblog.fcgi (je l'ai pris dans le paquet officiel)

$ cp /tmp/cblog-0.1.15/libexec/cblog.fcgi /zdata/jails/cblog/bin

Je regarde ce que demande ce binaire:

$ ldd /zdata/jails/cblog/bin/cblog.fcgi
/zdata/jails/cblog/bin/cblog.fcgi: => /usr/local/lib/ (0x80066b000) => /lib/ (0x800775000) => /lib/ (0x80088a000)

Je rajoute donc ce qui va bien dans ma future jail:

$ cp /lib/ /zdata/jails/cblog/lib
$ cp /lib/ /zdata/jails/cblog/lib
$ cp /usr/local/lib/ /zdata/jails/cblog/lib

J'ai vérifié les dépendances de la libfcgi et elle ne ramène rien d'autre.

Le paquet cblog est compilé avec comme prefix /usr/local du coup il cherche son fichier de conf dans /usr/local/etc/cblog.conf (vu de la jail)

$ mkdir /zdata/jails/cblog/usr/local/etc

Je prévoie de mettre les données que le binaire va manipuler dans /data (vu de la jail)

$ mkdir /zdata/jails/cblog/data

Je prépare le fichier de conf de mon cblog pour lui dire de trouver ses données dans /data (je zappe cette partie ce n'est pas le but de ce post). Le fichier de conf est déposé dans /zdata/jails/cblog/usr/local/etc sous le nom cblog.conf comme attendu.

C'est là que ça devient intéressant. Il est impossible de faire tourner une jail sans son runtime link-editor.

$ cp /libexec/ /zdata/jails/cblog/libexec/

Ici j'ai pris celui du système hôte car les binaire sont ceux du même format que ceux de l'hôte. Dans le cas d'un jail linux par exemple, il faudra prendre l'équivalent au format linux.

Nous pouvons maintenant démarrer la jail en question.

Ne surtout pas passer par /etc/rc.d/jail qui est complètement dépassé maintenant.

$ jail -c persist name=cblog path=/zdata/jails/cblog/ host.hostname=cblog ip4.addr=

Si vous ne voulez pas d'adresse ip ni de hostname

$ jail -c persist name=cblog path=/zdata/jails/cblog/

jls vous montrera la jail qui tourne.

Démarrons le cblog pour qu'il écoute sur une socket unix.

$ jexec cblog /sbin/cblog.fcgi unix:/cblog.sock

Il suffit de donner le chemin /zdata/jails/cblog/cblog.sock à nginx par exemple dans sa config fastcgi pour que votre cblog soit accessible au plus grand nombre.

Pour stopper la jail:

$ jail -r cblog

/me est puni

Voila ça m'apprendra, à vouloir faire des trucs, j'ai été puni depuis fin juillet.

Maintenant je dois pousser moi-même mes modifications, il va donc falloir que je fasse attention.

Heureusement, pendant quelque temps on ne tape pas que sur moi si je casse tout, mais on tape aussi et surtout sur mes mentors : jadawin et tabthorpe :)


Après 1 mois de ce nouveau statut, j'ai accumulé beaucoup de bave contre github, il me faut l'exprimer, il parait que c'est thérapeutique :

De plus en plus de développeurs utilisent github, c'est pas mal en soit: du git, une interface moderne kikooweb3.0.

Mais ça serait bien de NE PAS UTILISER LES TAGS COMME FICHIER RELEASE!!!! Bordel, github fait de la grosse merde avec ça !!!

  • il utilise des noms à la con pour l'archive.

    Si le projet toto dispose d'un tag 0.1 la logique voudrait que l'archive toto-0.1.tar.gz correspondant à ce tag contienne un répertoire toto-0.1, logique quoi. Bah non pas chez github.

    Déjà l'archive s'appelle legrosrobert-toto-0.1-hashalacon.tar.gz, génial non ? Mais admettons bon alors soyons logique si je décompresse ça je devrais avoir un répertoire legrosrobert-toto-0.1-hashalacon contenant les sources ? Bah non toujours pas !! Là, à la place, j'ai legrosrobert-toto-hashalacon ... J'adore.

    Tout pour faire chier le maintainer.

  • Des redirections moisis :

    Pour plusieurs raisons, le système de ports n'accepte pas de suivre les redirections (302) pour télécharger les distfiles. par exemple : éviter les boucles infinie de redirection pour les utilisateurs.

    Si vous utilisez les tags pour les distfiles, il n'y a pas le choix que de suivre une redirection 302, il est donc impossible de faire un ports qui utilise directement les archives via les tags github. Alors que si vous regardez bien, il y a une section pour ajouter les fichiers statiques à la main, même qu'il y a plus d'excuse pour ne pas l'utiliser, flash n'est plus nécessaire pour envoyer les fichiers. depuis cette section il est possible d'avoir un lien direct pour télécharger le fichier, donc ports compliant.

Donc si vous êtes utilisateurs de github, s'il vous plaît fournissez une vraie archive tout ce qu'il y a de plus classique distribuée via un système qui autorise un téléchargement par lien direct.


Pour revenir à ma punition, voila les projets que j'aimerai mener à bien concernant FreeBSD:

  • Fédérer tous les efforts actuellements éparpillés pour la modernisation de pkg_install: réécriture, support de la mise à jour propre, utilisation de libarchive, extensibilité, etc.
  • J'ai commencé un projet que j'aimerai mener à bien (pour faciliter les tests sur les ports avant modification): poudriere. Il s'agit d'une sorte de tinderbox mais en plus léger (moins de fonctionnalité) et beaucoup plus simple. Mais ça fera l'objet d'un autre post.
  • Continuer et faire passer mon patch "fakeroot" de manière a pouvoir passer par les outils pkg_install pour l'installation du ports et ainsi nettoyer les ports de beaucoup de hack.
  • Si le fakeroot passe, implémenter la notions de FLAVORS comme le fait OpenBSD, cad 1 ports capable de produire plusieurs packages.

Dans tous les cas je tiens à remercier particulièrement jadawin et tabthorpe pour leur patience et les conseils qu'ils me donnent lors de la revue/validation de mes commits : "pas trop usant ? :)"