Category Archives: Ports Collection

Alexander Leidinger » FreeBSD 2012-06-15 20:18:07

FreeBSD is on its way to move from CVS to SVN  for the version control system for the Ports Collection. The decision was made to keep the complete history, so the complete CVS repository has to be converted to SVN.

As CVS has no way to record a copy or move of files inside the repository, we copied the CVS files inside the repository in case we wanted to copy or move a file (the so called “repocopy�). While this allows to see the full history of a file, the drawback is that you do not really know when a file was copied/moved if you are not strict at recording this info after doing a copy. Guess what, we where not.

Now with the move to SVN which has a build-in way for copies/moves, it would be nice if we could record this info. In an internal discussion someone told its not possible to detect a repocopy reliably.

Well, I thought otherwise and an hour later my mail went out how to detect one. The longest time was needed to write how to do it, not to come up with a solution. I do not know if someone picked up this algorithm and implemented something for the cvs2svn converter, but I decided to publish the algorithm here if someone needs a similar functionality somewhere else. Note, the following is tailored to the structure of the Ports Collection. This allows to speed up some things (no need to do all steps on all files). If you want to use this in a generic repository where the structure is not as regular as in our Ports Collection, you have to run this algorithm on all files.

It also detects commits where multiple files where committed at once in one commit (sweeping commits).

Preparation

  • check only category/name/Makefile
  • generate a hash of each commitlog+committer
  • if you are memory-limited use ha/sh/ed/dirs/cvs-rev and store pathname in the list cvs-rev (pathname = “category-nameâ€?) as storage
  • store the hash also in pathname/cvs-rev

If you have only one item in ha/sh/ed/dirs/cvs-rev in the end, there was no repocopy and no sweeping commit, you can delete this ha/sh/ed/dirs/cvs-rev.

If you have more than … let’s say … 10 (subject to tuning) pathnames in ha/sh/ed/dirs/cvs-rev you found a sweeping commit and you can delete the ha/sh/ed/dirs/cvs-rev.

The meat

The remaining ha/sh/ed/dirs/cvs-rev are probably repocopies. Take one ha/sh/ed/dirs/cvs-rev and for each pathname (there may be more than 2 pathnames) in there have a look at pathname/. Take the first cvs-rev of each and check if they have the same hash. Continue with the next rev-number for each until you found a cvs-rev which does not contain the same hash. If the number of cvs-revs since the beginning is >= … let’s say … 3 (subject to tuning), you have a candidate for a repocopy. If it is >=  … 10 (subject to tuning), you have a very good indicator for a repocopy. You have to proceed until you have only one pathname left.

You may detect multiple repocopies like A->B->C->D or A->B + A->D + A->C here.

Write out the repocopy candidate to a list and delete the ha/sh/ed/dirs/cvs-rev for each cvs-rev in a detected sequence.

This finds repocopy candidates for category/name/Makefile. To detect the correct repocopy-date (there are maybe cases where another file was changed after the Makefile but before the repocopy), you now have to look at all the files for a given repocopy-pair and check if there is a matching commit after the Makefile-commit-date. If you want to be 100% sure, you compare the complete commit-history of all files for a given repocopy-pair.

Share

Related Posts:

Alexander Leidinger » FreeBSD 2012-05-18 15:30:06

Seems I forgot to announce that the linux_base-c6 is in the Ports Collection now. Well, it is not a replacement for the current default linux base, the linuxulator infrastructure ports are missing and we need to check if the kernel supports enough of 2.6.18 that nothing breaks.

TODO:

To my knowledge, nobody is working on anything of this. Anyone is welcome to have a look and provide patches.

Share

Related Posts:

New CentOS linux_base for testing soonish

It seems my HOWTO create a new linux_base port was not too bad. There is now a PR for a CentOS 6 based linux_base port. I had a quick look at it and it seems that it is nearly usable to include into the Ports Collection (the SRPMs need to be added, but that can be done within some minutes).

When FreeBSD 8.3 is released and the Ports Collection open for sweeping commits again, I will ask portmgr to do a repo-copy for the new port and commit it. This is just the linux_base port, not the complete infrastructure which is needed to completely replace the current default linuxulator userland. This is just a start. The process of switching to a more recent linux_base port is a long process, and in this case depends upon enough support in the supported FreeBSD releases.

Attention: Anyone installing the port from the PR should be aware that using it is a highly experimental task. You need to change the linuxulator to impersonate himself as a linux 2.6.18 kernel (described in the pkg-message of the port), and the code in FreeBSD is far from supporting this. Anyone who wants to try it is welcome, but you have to run FreeBSD-current as of at least the last weekend, and watch out for kernel messages about unsupported syscalls. Reports to [email protected] please, not here on the webpage.

Related Posts:

Share

HOWTO add linux-infrastructure ports for a new linux_base port

In my last blog-post I described how to create a new linux_base port. This blog-post is about the other Linux–ports which make up the Linux–infrastructure in the FreeBSD Ports Collection for a given Linux-release.

What are linux-infrastructure ports?

A linux_base port contains as much as possible and at the same time as little as possible to make up a useful Linux-compatibility-experience in FreeBSD. I know, this is not a descriptive explanation. And it is not on purpose. There are no fixed rules what has to be inside or what not. It “matured� into the current shape. A practical example is, that there is no GUI-stuff in the linux_base. While you need the GUI parts like GTK or QT for software like Skype and acroread, you do not need them for headless game servers. While you may need various libraries for game servers, you may not need those for Skype or acroread. As such some standard parts are in separate ports which are named linux–LINUX_DIST_SUFFIX-NAME. For GTK and the Fedora 10 release this results in linux-f10-gtk2. Such generic ports which depend upon a specific Linux-release make up the Linux-infrastructure in the FreeBSD Ports Collection. Those ports are referenced in port-Makefiles via the USE_LINUX_APPS variable, e.g. USE_LINUX_APPS=gtk2.

If you created a new linux_base port, you need most standard infrastructure ports in a version for the Linux-release used in the linux_base port, to have the Linux-application ports in the FreeBSD Ports Collection working (if you are unlucky, some ports do not play well with the Linux-release you have chosen, but this is out of the scope of this HOWTO).

Updating Mk/bsd.linux-apps.mk

 First we need to set the LINUX_DIST_SUFFIX variable to a value suitable to the new Linux-release. This is done in the conditional which checks the OVERRIDE_LINUX_NONBASE_PORTS variable for valid values. Add an appropriate conditional, and do not forget to add the new valid value to the IGNORE line in the last else branch of the conditional.

The next step is to check the _LINUX_APPS_ALL and _LINUX_26_APPS variables. If there are some infrastructure ports which are not available for the new Linux-release, the conditional which checks the availability of a given infrastructure port for a given Linux-release needs to be modified. If at a later step you notice that there are some additional infrastructure ports necessary for the new Linux-release, _LINUX_APPS_ALL and the check-logic needs to be modified too (e.g. add a new variable for your Linux-release, add the content of the variable to _LINUX_APPS_ALL, and change the check to do the right thing).

After that two tedious parts need to be done.

For each infrastructure port there is a set of variables. The name_PORT variable contains the location of the port in the Ports Collection. Typically you do not have to change it (if you really want to change it, do not do it, fix the naming of the infrastructure port instead), because we use a naming convention here which includes the LINUX_DIST_SUFFIX. The name_DETECT variable is an internal variable, do not change it (if you create a new infrastructure port, copy it from somewhere else and make sure the name in value of the variable matches the port name in the name of the variable). Then there are several name_suffix_FILE variables. Leave the existing ones alone, and add a new one with the correct suffix for your new Linux-release. The value of the variable needs to be an important file which is installed by the infrastructure port in question. FYI: The content of the name_suffix_FILE variables are used to set the name_DETECT variables, depending on the Linux-relase the name_DETECT variables are used to check if the port is already installed. Ideally the name_suffix_FILE variable points to a library in the port. The name_DEPENDS variable lists dependencies of this infrastructure port. If the dependencies changed in your Linux-release, you need to add a conditional to change the dependency if LINUX_DIST_SUFFIX is set to your Linux-release.

Normally this is all what needs to be done in PORTSDIR/Mk/bsd.linux-apps.mk, the rest of the file is code to check dependencies and some correctness checks.

The second tedious part is to actually create all those infrastructure ports. Normally you can copy an existing infrastructure port, rename it, adjust the PORTNAME, PORTVERSION, PORTREVISION, MASTER_SITES, PKGNAMEPREFIX, DISTFILES, CONFLICTS (also in all other Linux-release versions of this infrastructure port), LINUX_DIST_VER, RPMVERSION (if set/neccesary) and SRC_DISTFILE variables, generate the distfile checksums (make makesum), and fix the plist. I suggest to script parts of this work (as of this writing Freshports counts 68 ports where the portname starts with linux-f10-).

Adding new infrastructure ports, or removing infrastructure ports for a given Linux-release

If your Linux-release does not come with a package for an existing infrastructure port, just do not create a corresponding name_suffix_FILE line. You still need to do the right thing regarding dependencies of ports which depend upon this non-existing infrastructure port (if your Linux-release comes with packages for them).

To add a new infrastructure port, copy an existing block, rename the variables, set them correctly, add a new variable for your Linux-release in the first _LINUX_APPS_ALL section, add the content of this variable to _LINUX_APPS_ALL, and change the check-logic as described above.

Final words

If you have something which installs and deinstalls correctly, feel free to provide it on [email protected] for review/testing. If you have questions during the porting, feel also free to send a mail there.

Share

HOWTO create a new linux_base port

FreeBSD is in need of a new linux_base port. It is on my TODO list since a long time, but I do not get the time to create one. I still do not have the time to work on a new one, but when you read this, I managed to get the time to create a HOWTO which describes what needs to be done to create a new linux_base port.

I will not describe how to create a new linux_base port from scratch, I will just describe how you can copy the last one and update it to something newer based upon the existing infrastructure for RPM packages.

Specific questions which come up during porting a new Linux release should be asked on [email protected],  there are more people which can answer questions than here in my blog. I will add useful information to this HOWTO if necessary.

In the easy case most of the work is searching the right RPMs and their dependencies to use, and to create the plist.

Why do we need a new linux_base port?

The current linux_base port is based upon Fedora 10, which is end of life since December 2009. Even Fedora 13 is already end of life. Fedora 16 is supposed to be released this year. From a support point of view, Fedora 15 or maybe even Fedora 16 would be a good target for the next linux_base port. Other alternatives would be to use an extended lifetime release of another RPM based distribution, like for example CentOS 6 (which seems to be based upon Fedora 12 with backports from Fedora 13 and 14). Using a Linux release which is told to be supported for at least 10 years, sounds nice from a FreeBSD point of view (only minor changes to the linux ports in such a case, instead of creating a complete new linux_base each N+2 releases like with Fedora), but it also means additional work if you want to create the first linux_base port for it.

The mysteries you have to conquer if you want to create a new linux_base port

What we do not know is, if Fedora 15/16, CentOS 6, or any other Linux release will work in a supported FreeBSD release. There are two ways to find this out.

The first one is to take an existing Linux system, chroot into it (either via NFS or after making a copy into a directory of a FreeBSD system), and to run a lot of programs (acroread, skype, shells, scripts, …). The LTP testsuite is not that much useful here, as it will test mostly kernel features, but we do not know which kernel features are mandatory for a given userland of a Linux release.

The second way of testing if a given Linux release works on FreeBSD is to actually create a new linux_base port for it and test it without chrooting.

The first way is faster, if you are only interested in testing if something works. The second way provides an easy to setup testbed for FreeBSD kernel developers to fix the Linuxulator so that it works with the new linux_base port. Both ways have their merits, but it is up to the person doing the work to decide which way to go.

The meat: HOWTO create a new linux_base port

First off, you need a system (or a jail) without any linux_base port installed. After that you can create a new linux_base port (= lbN), by just making a copy of the latest one (= lbO). In lbN you need to add lbO as a CONFLICT, and in all other existing linux_base ports, you need to add lbN as a conflict.

Change the PORTNAME, PORTVERSION, reset the PORTREVISION in lbN, and set LINUX_DIST_VER  to the new Linux-release version in the lbN Makefile (this is used in PORTSDIR/Mk/bsd.linux-rpm.mk and PORTSDIR/Mk/bsd.linux-apps.mk).

If you do not stay with Fedora, there is some more work to do before you can have a look at chosing RPMs for installation. You need to have a look at PORTSDIR/Mk/bsd.linux-rpm.mk and add some cases for the new LINUX_DIST you want to use. Do not forget to set LINUX_DIST in the lbN Makefile to the name of the distribution you use. You also need to augment the LINUX_DIST_VER check in PORTSDIR/Mk/bsd.linux-rpm.mk with some LINUX_DIST conditionals. If you are lucky, the directory structure for downloads is similar to the Fedora structure, and there is not a lot to do here.

When this is done, you can have a look at the BIN_DISTFILES variable in the lbN Makefile. Try to find similar RPMs for the new Linux release you want to port. Some may not be available, and it may also be the case that different ones are needed instead. I suggest to first work with the ones which are available (make makesum, test install and create plist). After that you need to find out what the replacement RPMs for non-existing ones are. You are on your own here. Search around the net, and/or have a look at the dependencies in the RPMs of lbO to determine if something was added as a dependency of something else or not (if not, forget about it ATM). When you managed to find replacement RPMs, you can now have a look at the dependencies of the RPMs in lbN. Do not add blindly all dependencies, not all are needed in FreeBSD (the linux_base ports are not supposed to create an environment which you can chroot into, they are supposed to augment the FreeBSD system to be able to run Linux programs in ports like they where FreeBSD native programs). What you need in the linux_base ports are libraries, config and data files which do not exist in FreeBSD or have a different syntax than in FreeBSD (those config or data files which are just in a different place, can be symlinked), and basic shell commands (which commands are needed or not… well… good question, in the past we made decisions what to include based upon problem reports from users). Now for the things which are not available and where not added as a dependency. Those are things which are either used during install, or where useful to have in the past. Find out by what it was replaced and have a look if this replacement can easily be used instead. If it can be used, add it. If not, well… bad luck, we (the FreeBSD community) will see how to handle this somehow.

If you think that you have all you need in BIN_DISTFILES, please update SRC_DISTFILES accordingly and generate the distfile via  make –DPACKAGE_BUILDING makesum to have the checksums of the sources (for legal reasons we need them on our mirrors).

The next step is to have a look at REMOVE_DIRS, REMOVE_FILES and ADD_DIRS if something needs to be modified. Most of them are there to fall back to the corresponding FreeBSD directories/files, or because they are not needed at all (REMOVE_*). Do not remove directories from ADD_DIRS, they are created here to fix some edge conditions (I do not remember exactly why we had to add them, and I do not take the time ATM to search in the CVS history).

If you are lucky, this is all (make sure the plist is correct). If you are not lucky and you need to make some modifications to files, have a look at the do-build target in the Makefile, this is the place where some changes are done to create a nice user experience.

If you arrive here while creating a new linux_base port, lean back and feel a bit proud. You managed to create a new linux_base port. It is not very well tested at this moment, and it is far from everything which needs to be done to have the complete Linux infrastructure for a given Linux release, but the most important part is done. Please notify [email protected] and call for testers.

What is missing?

The full Linuxulator infrastructure for the FreeBSD Ports Collection has some more ports around a linux_base port. Most of the infrastructure for this is handled in Mk/bsd.linux-apps.mk.

UPDATE: I got some time to write how to update the Linux-infrastructure ports.

Share

VDR ports docs

After a quick discussion with nox@ I made a copy&paste of his “VDR is committed now�-mail into the FreeBSD wiki. I also re-styled some small parts of it to fit better into the wiki. It is not perfect, but already usable. Now interested people can go and improve the docs there.

Thanks to Juergen for all his work in this area!

Share

Converting from Courier IMAP to dovecot is easy

I have used Courier IMAP at home since a long time. As I want to update a dovecot 1.2 setup to dovecot 2.x, I decided to first have a look at dovecot 2.x at home.

Switching from Courier IMAP to dovecot is really easy. I just configured the correct path to the maildir, setup a passdb/userdb, and it was working.

The important part was the correct transfer of the passwords. I used already an userdb in Courier IMAP with MD5 passwords. For each user it has imappw=XXX with XXX similar to $1$abc.

This can be converted into a dovecot passdb/userdb line very easily:

username:{MD5-CRYPT}$1$abc::UID:GID::HOMEDIR::userdb_mail=maildir:~/path/to/maildir

The corresponding passdb/userdb settings for dovecot are:

passdb {
   args = scheme=MD5-CRYPT username_format=%u /usr/local/etc/dovecot/dovecot.pws
   driver = passwd-file
}
userdb {
   args = username_format=%u /usr/local/etc/dovecot/dovecot.pws
   driver = passwd-file
}

Compared to when I had a look the last time, dovecot is also able to use OTP as an authentication mechanism now. Unfortunately I did not find any documentation how to configure/use it.

Share

Non-default linux base ports deprecated

Yesterday I deprecated the non-default Fedora based Linux base ports. This means fc6, f7, f8 and f9 will vanish soon (I decided for one month of expiry time). This is because all of them are End of Life upstream since a long time (= no security updates).

The fc4 and f10 ones are still available — even if they are End of Life too — because FreeBSD 7.x can not use something newer than the fc4 one, and we have not tested yet a more recent Linux distribution.

Probably the most easy way to update the Linux base ports to something newer is to stay with Fedora (we have a lot of ports-infrastructure for it already). Unfortunately it is not known if something newer works without problems (missing epoll/inotify support could be a roadblock here in case it is extensively used in a more recent version).

I want to get some time to have a look if a more recent Fedora version is suitable for the use as a Linux base in FreeBSD 8.x+, but I do not have an estimate when I can start and how long it may take. In case someone already tested a more recent Fedora version feel free to share your experience.

Share

HOWTO: creating your own updated linux RPM for the FreeBSD linuxulator

Background info

The FreeBSD linux compatibility environment currently uses RPMs from Fedora 10. Unfortunately Fedora 10 is end of life since a while. For one of the RPMs (the pango one) we where aware of a security vulnerability. As we do not know if it is feasible to update the linuxulator ports to something more recent, I decided to setup a VM with Fedora 10 and generate a new RPM for the linux-f10-pango port. Thanks to Luchesar V. ILIEV for explaining me how to do this.

Setup of the VM

I used VirtualBox 4.0.4 on a Solaris 10 x86 machine. I configured a fixed size disk of 16 GB and kept the default network setup (after installing the guest tools / kernel modules I switched to virtio, as I was not able to do anything useful besides a ping) and RAM size. The CD/DVD drive was configured to use the image of the full Fedora 10 DVD for i386 systems.

Setup of Fedora 10

Booting the VM from the DVD leads to the graphical Fedora 10 install software (after chosing to install a new system on the console). There I accepted all the defaults, except for the software to install. I deselected the Office and Productivity group and selected the Software Development group. When I was asked if I want to install some additional RPMs I had a look at the complete list and installed some I thought are necessary. I do not remember anymore which ones I chose, but everything which looks related to RPM building is a good candidate.

After a while the install will be finished and you can boot into the new system (eject the DVD from the drive before reboot). After reboot chose to install the Guest Additions in the menu of the VM. This should mount the ISO image in the VM. As root execute the file for Linux. This will build some kernel modules for better integration (e.g. seamless integration of the mouse between your desktop and the VM). At this point I rebooted and configured virtio as the NIC. I also had to configure the network settings by hand, as the GUI tool did not safe all the settings correctly.

Update and install of required RPMs

After the VM was up and the network configured, I updated the entire system (chose System Update in the menu). To update the pango port, I had to install the libthai-devel RPM. I had the RPM for it (and all the files I need to build a new pango RPM) already downloaded, so I did a “yum install /path/to/rpm�. At this point I was ready to create the RPM build environment.

The RPM build environment

As a normal user I executed the command rpmdev-setuptree which creates the directory rpmbuild and populates it with some directories. Now you just need to find a suitable .spec file and put it into rpmbuild/SPECS, put the sources (and maybe patches referenced in the .spec file) into rpmbuild/SOURCES, and you are ready to go (I patched pango.spec for a more recent pango version, basically just changing the version numbers). If you want to have a custom packager and vendor attribute in the RPM, you can add a line for each to ~/.rpmmacros, e.g. %packager yournamehere and %vendor whateverisappropriate. I used my @FreeBSD.org EMail address as the packager, and FreeBSD as the vendor.

Building a RPM

I used rpmbuild –ba –target i386-redhat-linux-gnu –clean rpmbuild/SPECS/pango.spec to build the new pango RPM. If everything is OK, the resulting RPMs (a source RPM, a devel RPM, a debuginfo RPM and the RPM for the binaries) are in rpmbuild/RPMS and rpmbuild/SRPMS. For a FreeBSD port we just need the source RPM (to comply to the (L)GPL) and the RPM for the binaries.

Additional info

The i386-redhat-linux-gnu string which is used for the –target option of the rpmbuild command is what seems to be used to build the Fedora 10 RPMs. After building pango, the RPM has i686-pc-linux-gnu in some filenames instead (the default value for this setup). The binaries seem to be compiled for i386, so there should be no problem even for old systems.

Share

A new linux-f10-pango port is ready

In the last days I took (and even had) the time to install a VM with Fedora 10, updated all the packages after installation, and created a new linux-f10-pango port (v 1.28.3). I did this because the port has a security vulnerability according to our VuXML DB and there where more and more reports in the last months from users which had a problem with this.

During the update of the port I noticed that the port does not contain a FORBIDDEN entry, just portaudit complains about it because there is an entry in the VuXML. That is not nice. I was told that the ports slush will be lifted soon (I need to bump some PORTREVISIONs), this means that I can commit the update probably tomorrow, just in time when the new RPM should hit the FreeBSD distribution infrastructure (MASTER_SITE_LOCAL is updated once a day from a specific folder in our home directories).

Thanks to Luchesar V. ILIEV for the nice writeup of what to install in Fedora 10 to be able to build RPMs, and the description of how to build your own RPM.

Share

The FreeBSD-linuxulator explained (for users)

After another mail where I explained a little bit of the linuxulator behavior, it is time to try to make an easy text which I can reference in future answers. If someone wants to add parts of this explanation to the FreeBSD handbook, go ahead.

Linux emulation? No, “native� execution (sort of)!

First, the linuxulator is not an emulation. It is “just� a binary interface which is a little bit different from the FreeBSD-“native�-one. This means that the binary files in FreeBSD and Linux are both files which comply to the ELF specification.

When the FreeBSD kernel loads an ELF file, it looks if it is a FreeBSD ELF file or a Linux ELF file (or some other flavor it knows about). Based upon this it looks up appropriate actions in a table for this binary (it can also differentiate between 64-bit and 32-bit, and probably other things too).

The FreeBSD-table is always compiled in (for a better big picture: at least on an AMD/Intel 64-bit platform there is also the possibility to include a 32-bit version of this table additionally, to be able to execute 32-bit programs on 64-bit systems), and other ones like the Linux one can be loaded additionally into the kernel (or build statically in the kernel, if desired).

Those tables contain some parameters and pointers which allow to execute the binary. If a program is making a system call, the kernel will look up the correct function inside this table. It will do this for FreeBSD binaries, and for Linux binaries. This means that there is no emulation/simulation (overhead) going on… at least ideally. Some behavior is a little bit differently between Linux and FreeBSD, so that a little bit of translation/house-keeping has to go on for some Linux system calls for the underlying FreeBSD kernel functions.

This means that a lot of Linux stuff in FreeBSD is handled at the same speed as if this Linux program would be a FreeBSD program.

Linux file/directory tricks

When the kernel detects a Linux program, it is also playing some tricks with files and directories (also a property of the above mentioned table in the kernel, so theoretically the kernel could play tricks for FreeBSD programs too).

If you look up for a file or directory /A, the kernel will first look for /compat/linux/A, and if it does not find it, it will look for /A. This is important! For example if you have an empty /compat/linux/home, any application which wants to display the contents of /home will show /compat/linux/home. As it is empty, you see nothing. If this application does not allow you to enter a directory manually via the keyboard, you have lost (ok, you can remove /compat/linux/home or fill it with what you want to have). If you can enter a directory via the keyboard, you could enter /home/yourlogin, this would first let the kernel look for /compat/linux/home/yourlogin, and as it can not find it then have a look for /home/yourlogin (which we assume is there), and as such would display the contents of your home directory.

This implies several things:

  • you can hide FreeBSD directory contents from Linux programs while still being able to access the content
  • “badlyâ€? programmed Linux applications (more correctly: Linux programs which make assumptions which do not hold in FreeBSD) can prevent you from accessing FreeBSD files, or files which are the same in Linux and FreeBSD (like /etc/group which is not available in /compat/linux in the linux_base ports, so that the FreeBSD one is read)
  • you can have different files for Linux than for FreeBSD

The Linux userland

The linux_base port in FreeBSD is coming from a plain installation of Linux packages. The difference is that some files are deleted, either because we can not use them in the linuxulator, or because they exist already in the FreeBSD tree at the same place and we want that the Linux programs use the FreeBSD file (/etc/group and /etc/passwd come to mind). The installation also marks binary programs as Linux programs, so that the kernel knows which kernel-table to consult for system calls and such (this is not really necessary for all binary programs, but it is harder to script the correct detection logic, than to just “brand� all binary programs).

Additionally some configurations are made to (hopefully) make it do the right thing out of the box. The complete setup of the linux_base ports is done to let Linux programs integrate into FreeBSD. This means if you start acroread or skype, you do not want to have to have to configure some things in /compat/linux/etc/ first to have your fonts look the same and your user IDs resolved to names (this does not work if you use LDAP or kerberos or other directory services for the user/group ID management, you need to configure this yourself). All this should just work and the application windows shall just pop up on your screen so that you can do what you want to do. Some linux_base ports also do not work on all FreeBSD releases. This can be because some kernel features which this linux_base ports depends upon is not available (yet) in FreeBSD. Because of this you should not choice a linux_base port yourself. Just go and install the program from the Ports Collection and let it install the correct linux_base port automatically (a different FreeBSD release may have a different default linux_base port).

A note of caution, there are instructions out there which tell how to install more recent linux_base ports into FreeBSD releases which do not have them as default. You do this on your own risk, it may or may not work. It depends upon which programs you use and at which version those programs are (or more technically, which kernel features they depend upon). If it does not work for you, you just have two possibilities: revert back and forget about it, or update your FreeBSD version to a more recent one (but it could be the case, that even the most recent development version of FreeBSD does not have support for what you need).

Linux libraries and “ELF file OS ABI invalid�-error messages

Due to the above explained fact about file/directory tricks by the kernel, you have to be careful with (additional) Linux libraries. When a Linux program needs some libraries, several directories (specified in /compat/linux/etc/ld.so.conf) are searched. Let us assume that the /compat/linux/etc/ld.so.conf specifies to search in /A, /B and /C. This means the FreeBSD kernel first gets a request to open /A/libXYZ. Because of this he first tries /compat/linux/A/libXYZ, and if it does not exist he tries /A/libXYZ. When this fails too, the Linux runtime linker tries the next directory in the config, so that the kernel looks now for /compat/linux/B/libXYZ and if it does not exist for /B/libXYZ.

Now assume that libXYZ is in /compat/linux/C/ as a Linux library, and in /B as a FreeBSD library. This means that the kernel will first find the FreeBSD library /B/libXYZ. The Linux binary which needs it can not do anything with this FreeBSD library (which depends upon the FreeBSD syscall table and FreeBSD symbols from e.g. libc), and the Linux runtime linker will bail out because of this (actually he sees that the lin is not of the required type by reading the ELF header of it). Unfortunately the Linux runtime linker will not continue to search for another library with the same name in another directory (at least this was the case last time I checked and modified the order in which the Linux runtime linker searches for libraries… this has been a while, so he may be smarter now) and you will see the above error message (if you started the linux program in a terminal).

The bottom line of all this is: the error message about ELF file OS ABI invalid just means that the Linux program was not able to find the correct Linux library and got a FreeBSD library instead. Go, install the corresponding Linux library, and make sure the Linux program can find it instead of the FreeBSD library (do not forget to run “/compat/linux/sbin/ldconfig –r /compat/linux� if you make changes by hand instead of using a port, else your changes may not be taken into account).

Constraints regarding chroot into /compat/linux

The linux_base ports are designed to have a nice install-and-start experience. The drawback of this is, that there is not a full Linux system in /compat/linux, so doing a chroot into /compat/linux will cause trouble (depending on what you want to do). If you want to chroot into the linux system on your FreeBSD machine, you better install a linux_dist port. A linux_dist port can be installed in parallel to a linux_base port. Both of them are independent and as such you need to redo/copy configuration changes you want to have in both environments.

Share

Weather station readout with FreeBSD

A while ago a wind turbine was installed not far away from my place. It is far enough to not disturb us, and it is near enough to notice that it turns a lot (IIRC I have seen it only once not turning).

This triggered a question. How much energy would such a device (smaller of course) produce at my place?

The answer depends upon several factors. The wind speed, the wind direction and the wind-speed-to-power-output curve of the device. If you do not take a device which rotates around the horizontal axis but the vertical axis, the wind direction can be taken out of the question (probably not completely, but to answer my question this simplification should be ok). The output-power curve depends upon the device, and I hope it is easy to get it from the vendors. The remaining open question it the wind speed at my place. Is there enough wind with enough speed?

To answer this question I bought a weather station with an anemometer (wind speed sensor). I searched a little bit until I decided to buy a specific one (actually I bought three of them, some coworkers got interested too but they found only much more expensive ones, so soon there will be three more weather stations in use in Belgium, France and Germany). The main point is, I can connect it to an USB port of a PC and there is some software for Linux to read out the data. It also comes with some other outdoor-sensors (temperature, rain, wind direction, humidity, …) and an indoor-control-unit with some internal sensors (temperature, humidity). The user interface is mainly the touchscreen of the control-unit. There is also some Windows software, which is needed to program the interval in which the measurements are taken and saved in the control-unit.

It seems the weather station is produced by Fine Offset Electronics Co.,Ltd and sold within different brands in different locations. The Linux software can read all of them, as the vendor and product IDs are not changed.

Porting the software was easy, it uses libusb and I just had to correct a little problem for the non-portable functions which are used (I asked about them on usb@ and the response was that they just got implemented upon my request and will be committed to HEAD soon). I made a little patch for the software to only use them when available (if you have not loaded the USB HID driver, you do not need to care about them) and committed it to the Ports Collection as astro/fowsr.

Now I just need to attach the outside sensors at the place where I would put the vertical axis wind turbine, install some toolkit which takes a series of measurements and displays them as a nice graph (while keeping all data values) and write some glue code to feed the output of fowsr to it. After a year I can then calculate how much power a given wind turbine would have produced during the year and calculate the return of investment for it.

The Linux software also references several weather sites, for some of them you can get even an iGoogle widget so that you can view the data from wherever you want (as long as you have a suitable internet connection). I think this is also something I will have a look at later.

Note to users in Europe, the device also comes with a DCF77 receiver. As the time is distributed in UTC+1 (or +2, depending on the daylight saving time), you should adjust the timezone setting accordingly to this, not to plain UTC (so for me the timezone should be ‘0’ for the same timezone).

Share

Direct, indirect and explicit dependencies in progams/ports

The discussion about direct and indirect dependencies is coming up again on the FreeBSD mailinglists. Seems I should make some blog post about it, maybe it makes this topic more findable than my postings in the mailinglists.

Some definitions:

  • A direct dependency from A to B is when program/port A uses symbols from library/port B.
  • An indirect dependency from A to C is when program/port A uses symbols from library/port B but no symbols from library/port C, and library/port B uses symbols from library/port C.
  • An explicit dependency from A to C is when it is a direct or indirect dependency A to C, and when the compiler-time-linker added an explicit reference to C to the program/lib of A.

Ideally we have no indirect dependencies in the explicit dependencies, only direct dependencies. Unfortunately in reality we also have indirect dependencies there. This has at least two causes:

  1. libtool (at least 1.x) does not (or was not) come with a hint on FreeBSD, which tells that the run-time-linker is recursively resolving dependencies.
  2. Some pkg-config setups list indirect dependencies as explicit dependencies (IIRC it depends if Requires.private and/or Libs.private is used in the .pc file or not; if it is used, there should be no indirect dependency appear from this software, but I am not 100% sure about this).

Three years ago I wrote /usr/ports/Tools/scripts/explicit_lib_depends.sh, it looks at the files of a given port (it needs to be installed), and prints out explicit dependencies. Because of the indirect dependencies which could be listed there, this list is not a list of ports which are real dependencies from a source code point of view, but it reflects the link-time reality. If a port C shows up there, the port which is checked needs to be rebuild in case the ABI of library/port C changes.

Share/Bookmark

LAME updated in the FreeBSD ports collection

After all the big-impact commits (Gnome/gettext/KDE/X11/…) have settled now, I took the time to update audio/lame (I identified more than 100 ports with an (implicit) dependency on lame, 45 of them needed a portrevision bump; if I forgot/overlooked some, bump the revision yourself or notify me please). That is the first update of my ports where miwi@ did not beat me in committing an update since a year (he has implicit approval to do anything he wants with my ports).

I can be happy that he is/was this fast (and that we have such a productive and efficient committer), or I can be sad that I do not have the time anymore to be faster than I am with such things… or both. Hmmm… I think I will go the happy way. ;-)

Share/Bookmark

Cheap process monitoring (no additional software required)

I have an old system (only the hardware, it runs –current) which reboots itself from time to time (mostly during the daily periodic(8) run, but also during a lot of compiling (portupgrade)). There is no obvious reason (no panic) why it is doing this. It could be that there is some hardware defect, or something else. It is not important enough to get a high enough priority that I try hard to analyze the problem with this machine. The annoying part is, that sometimes after a restart apache does not start. So if this happens, the solution is to login and start the webserver. If the webserver would start each time, nearly nobody would detect the reboot (root gets an EMail on each reboot via an @reboot crontab entry).

My pragmatic solution (for services started via a good rc.d script which has a working status command) is a crontab entry which checks periodically if it is running and which restarts the service if not. As an example for apache and an interval of 10 minutes:

*/10 * * * *    /usr/local/etc/rc.d/apache22 status >/dev/null 2>&1 || /usr/local/etc/rc.d/apache22 restart

For the use case of this service/machine, this is enough. In case of a problem with the service, a mail with the restart output would arrive each time it runs, else only after a reboot for which the service did not restart.

Share/Bookmark

Debugging lang/mono — 2nd round

Today I had again some energy to look at why mono fails to build on FreeBSD–current.

I decided to do a debug-build of mono. This did not work initially, I had to produce some patches. :(

Does this mean nobody is doing debug builds of mono on FreeBSD?

I have to say, this experience with lang/mono is completely unsatisfying.

Ok, bottom line, either the debug build seems to prevent a race condition in most cases (I had a lot less lockups for each of the two builds I did).

Whatever it is, I do not care ATM (if the configure stuff is looking at the architecture of the system, it may be the case that the i386-portbld-freebsdX does not enable some important stuff which would be enabled when run with i486-portbld-freebsdX or better). Here are the patches I used in case someone is interested (warning, copy&paste converted tabs to spaces, you also have to apply the map.c (a generated file… maybe a touch of the right file would allow to apply this patch in the normal patch stage) related stuff when the build fails, else there is some parser error in mono):

--- mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs.orig       2010-01-29 11:34:00.592323482 +0100
+++ mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs    2010-01-29 11:34:18.540607357 +0100
@@ -57,7 +57,7 @@ namespace Mono.Unix {
 int r = Native.Syscall.waitpid (pid, out status,
 Native.WaitOptions.WNOHANG | Native.WaitOptions.WUNTRACED);
 UnixMarshal.ThrowExceptionForLastErrorIf (r);
-                       return r;
+                       return status;
 }

 public int ExitCode {

--- mono/io-layer/processes.c.orig    2010-01-29 11:36:08.904331535 +0100
+++ mono/io-layer/processes.c 2010-01-29 11:42:21.819159544 +0100
@@ -160,7 +160,7 @@ static gboolean waitfor_pid (gpointer te
 ret = waitpid (process->id, &status, WNOHANG);
 } while (errno == EINTR);

-       if (ret <= 0) {
+       if (ret == 0 || (ret < 0 && errno != ECHILD)) {
 /* Process not ready for wait */
 #ifdef DEBUG
 g_message ("%s: Process %d not ready for waiting for: %s",
@@ -169,6 +169,17 @@ static gboolean waitfor_pid (gpointer te

 return (FALSE);
 }
+
+       if (ret < 0 && errno == ECHILD) {
+#ifdef DEBUG
+               g_message ("%s: Process %d does not exist (anymore)", __func__,
+                          process->id);
+#endif
+               /* Faking the return status. I do not know if it is correct
+                * to assume a successful exit.
+                */
+               status = 0;
+       }

 #ifdef DEBUG
 g_message ("%s: Process %d finished", __func__, ret);

--- mono/metadata/mempool.c.orig      2010-01-29 11:58:16.871052861 +0100
+++ mono/metadata/mempool.c   2010-01-29 12:30:45.143367454 +0100
@@ -212,12 +212,14 @@ mono_backtrace (int size)

         EnterCriticalSection (&mempool_tracing_lock);
         g_print ("Allocating %d bytes\n", size);
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         symbols = backtrace (array, BACKTRACE_DEPTH);
         names = backtrace_symbols (array, symbols);
         for (i = 1; i < symbols; ++i) {
                 g_print ("\t%s\n", names [i]);
         }
         free (names);
+#endif
         LeaveCriticalSection (&mempool_tracing_lock);
 }

--- mono/metadata/metadata.c.orig     2010-01-29 11:59:38.552316989 +0100
+++ mono/metadata/metadata.c  2010-01-29 12:00:43.957337476 +0100
@@ -3673,12 +3673,16 @@ mono_backtrace (int limit)
         void *array[limit];
         char **names;
         int i;
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         backtrace (array, limit);
         names = backtrace_symbols (array, limit);
         for (i =0; i < limit; ++i) {
                 g_print ("\t%s\n", names [i]);
         }
         g_free (names);
+#else
+       g_print ("No backtrace available.\n");
+#endif
 }
 #endif

--- support/map.c.orig        2010-01-29 12:05:22.374653708 +0100
+++ support/map.c 2010-01-29 12:10:29.024412452 +0100
@@ -216,7 +216,7 @@
 #define _cnm_dump(to_t, from) do {} while (0)
 #endif /* def _CNM_DUMP */

-#ifdef DEBUG
+#if defined(DEBUG) && !defined(__FreeBSD__)
 #define _cnm_return_val_if_overflow(to_t,from,val)  G_STMT_START {   \
         int     uns = _cnm_integral_type_is_unsigned (to_t);             \
         gint64  min = (gint64)  _cnm_integral_type_min (to_t);           \

Mono build problems on FreeBSD-current

I try to build mono on FreeBSD–current (it is a dependency of some GNOME program). Unfortunately this does not work correctly.

What I see are hangs of the build. If I stop the build when it hangs and restart it, it will continue and succeed to process the build steps a little bit further, but then it hangs again.

If I ktrace the hanging process, I see that there is a call to wait returning with the error message that the child does not exist. Then there is a call to nanosleep.

It looks to me like this process missed some SIGCLD (or is waiting for something which did not exist at all), and a loop is waiting for a child to exit. This loop probably has no proper condition for the fact that there is no such child (anymore). As such it will stay forever in this loop.

So I grepped a litte bit around in mono and found the following code in <mono-src-dir>/mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs:

public void WaitForExit ()
{
    int status;
    int r;
    do {
        r = Native.Syscall.waitpid (pid, out status, (Native.WaitOptions) 0);
    } while (UnixMarshal.ShouldRetrySyscall (r));
    UnixMarshal.ThrowExceptionForLastErrorIf (r);
}

This does look a little bit as it could be related to the problem I see, but ShouldRetrySyscall only returns true if the errno is EINTR. So this looks correct. :-(

I looked a little bit more at this file and it looks like either I do not understand the semantic of this language, or GetProcessStatus does return the returnvalue of the waitpid call instead of the status (which is not what it shall return to my understanding). If I am correct, it can not really detect the status of a process. It would be very bad if such a fundamental thing went unnoticed in mono…  which does not put a good light on the unit-tests (if any) or the general testing of mono. For this reason I hope I am wrong.

I did not stop there, as this part does not look like it is the problem. I found the following in mono/io-layer/processes.c:

static gboolean waitfor_pid (gpointer test, gpointer user_data)
{
...
    do {
        ret = waitpid (process->id, &status, WNOHANG);
    } while (errno == EINTR);

    if (ret <= 0) {
        /* Process not ready for wait */
#ifdef DEBUG
        g_message ("%s: Process %d not ready for waiting for: %s",
                   __func__, process->id, g_strerror (errno));
#endif

        return (FALSE);
    }

#ifdef DEBUG
    g_message ("%s: Process %d finished", __func__, ret);
#endif

    process->waited = TRUE;
...
}

And here we have the problem, I think. I changed the (ret <= 0) to  (ret == 0 || (ret < 0 && errno != ECHILD)). This will not really give the correct status, but at least it should not block anymore and I should be able to see the difference during the build.

And now after testing, I see a difference, but the problem is still there. The wait with ECHILD is gone in the loop, but there is still some loop with a semaphore operation:

62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)
62960 mono     RET   nanosleep 0
62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)

OK, there is more going on. I think someone with more knowledge about mono should have a look at this (do not only look at this semop thing, but also look why it loses a child).

Some updates (kernel/ports)

There was not much to tell in the last months. I was busy with moving and the pregnancy of my wife (ok, she was more busy with this than I was…).

So the recent updates are, that I took some time to commit some of my patches to SVN. Most of them are in my SVN user area in various branches. The interesting ones may be deskjail and linuxaio. The first one allows to run your desktop in a jail. The second one gives async I/O for the linuxulator.

There’s also some other stuff. Feel free to have a look.

It also seems that the we may see the Fedora 8 infrastructure landing in the ports collection “soon”. I have the impression that Boris just waits for the complete unfreeze of the ports collection. The last patch I’ve reviewed looked very good. There are some loose ends, like switching it on as the default linux base for FreeBSD–current for example, but those are things which I prefer to do later than in the same commit. First let it be there for a while and let curious users test it a little bit more. If everything is ok, we can switch the default linux base to F8 in –current.

Share/Bookmark

Interesting stuff upcomming (multimedia, Wiimote)

For a long time I didn’t wrote something in my blog. This happens from time to time to a lot of people I think… :)

Ok, so some interesting news on the FreeBSD front: I’m porting NMM (version 1.0.0).Back in the days when I was writting my diploma thesis, those people where working on it already. I always wanted to port it, as it was cool to see it in action (one system picked up a football match and distributed it to a lot of PCs in the local subnet (AFAIR multicast) and even handhelds in real-time (with automatic downsizing to the output device), and was also distributing it to the Uni-Aula (AFAIR TCP stream)). The box was not powerful at all, and you where able to do a lot of processing on any machine in the network, while the client didn’t know where which processing happened. They also present it for several years on the CEBIT in Germany. They have videos showing it in action. You can do a lot more cool things with this. Think about a network aware multimedia center. You can have your movies / MP3s / whatever on several machines in your network, and the output is displayed on a not so powerful machine without any trace in the GUI that there are other machines involved. And if you want to play around, you can even see/hear the same stuff synchronized at the same time in multiple rooms and even on your handheld device.

This is scheduled to be used in the new KDE multimedia infrastructure, and even in some Bang&Olufsen products.

So far it compiles after a little bit of patching, but there are some strange things to solve before I can even try to use it. I posted a message to the development forum, let’s wait and see what they have to tell.

And on a related area, I also got a Wiimote (Wii remote controller) working in FreeBSD. I think this will be a nice mouse replacement for a multimedia center. I have a discussion on the FreeBSD bluetooth mailinglist regarding this.

Share/Bookmark

Removing superflous calls to basename/dirname in bsd.port.mk

I just submitted my patch to remove calls (exec()ing) to basename/dirname in bsd.port.mk to GNATS. I didn’t time if this improves anything, but I assume doing a regex replacement in a make-variable uses less resources than doing a fork()&exec() or a system() call.

I don’t think this gives a huge improvement at all, it’s maybe even below the measurement threshold, but hey, why using external tools if the internal features are enough to do what you want? At least this patch is recorded now somewhere…

Share/Bookmark