Category Archives: Userland

Alexander Leidinger » FreeBSD 2013-06-23 07:48:27

There may by cases where you want to generate a Linux binary on a FreeBSD machine. This is not a problem with the linuxulator, but not with the default linux_base port.

As you may know, the linux_base port is designed to deliver an integrated experience with FreeBSD native programs. As such some parts of the native FreeBSD infrastructure is used. If you would try to use a Linux–compiler to generate Linux–binaries, you would run into the problem that by default the FreeBSD includes are used.

Prerequisites

To have a fully featured and non-integrated Linux environment on your FreeBSD system either mount an existing (and compatible) Linux installation somewhere into your FreeBSD system, or install a linux_dist port. This can be done additionally to an already installed linux_base port.

Preparation

When you have a complete Linux environment available, you need to mount the FreeBSD devfs to /path/to/complete_linux/dev, linprocfs to /path/to/complete_linux/proc and linsysfs to /path/to/complete_linux/sys to have a complete setup.

Use it

Now you just need to chroot into this  /path/to/complete_linux and you configure/make/install or whatever you need to do to generate your desired Linux binary.

Share

Alexander Leidinger » FreeBSD 2012-12-23 17:22:43

From time to time I convert videos to H264. When I do this I want to get the best quality out of a give filesize. This means I create VBR videos. The question here is, how big the target video file shall be.

After searching around a little bit in the net I found a formula which is supposed to give a hint about the target filesize. Naturally this depends on the encoder (or even the encoder-version), the encoder settings and even the video.

The formula is at least a good hint for my use, so I wrote a script which calculates several filesize values for a given video (based upon the output of mediainfo, which the scripts expects in a file with the filename as an argument to the script). It calculates a CBR and a VBR value for a given video based upon the width, height and duration. It should work on all system with a POSIX compatible shell.

Example output for a video from my HD-ready cam, original filesize 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 2
Per second: 6451200.000 bps / 6300 Kibps
Total CBR: 1148313600 bytes / 1121400 KiB / 1095 MiB
Total VBR: 861235200 bytes / 841050 KiB / 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 3
Per second: 9676800.000 bps / 9450 Kibps
Total CBR: 1722470400 bytes / 1682100 KiB / 1642 MiB
Total VBR: 1291852800 bytes / 1261575 KiB / 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 4
Per second: 12902400.000 bps / 12600 Kibps
Total CBR: 2296627200 bytes / 2242800 KiB / 2190 MiB
Total VBR: 1722470400 bytes / 1682100 KiB / 1642 MiB

There are 3 sections, the difference is the “motion� value. It is a kind of multiplicator depending on the amount of motion in the video. For the videos I made myself (family videos, and even some videos of volley ball games), the first section seems to be just fine. So I reduced the original MP4 file to about 50% (not visible here is the audio size, normally I copy the original audio unmodified).

For the curious ones, the formula is

width_in_pixels * height_in_pixels * fps * motion_value * 0.07

for the bps value. The CBR value is

bps * playtime_in_seconds / 8

and the VBR value is

3 / 4 * CBR_value.

Share

Alexander Leidinger » FreeBSD 2012-11-25 07:37:52

The recent security incident triggered a discussion how to secure ssh/gpg keys.

One way I want to focus on here (because it is the way I want to use at home), is to store the keys on a crypto card. I did some research for suitable crypto cards and found one which is called Feitian PKI Smartcard, and one which is called OpenPGP card. The OpenPGP card also exists in a USB version (basically a small version of the card is already integrated into a small USB card reader).

The Feitian card is reported to be able to handle RSA keys upto 2048 bits. They do not seem to handle DSA (or ECDSA) keys. The smartcard quick starter guide they have  (the Tuning smartcard file system part) tells how to change the parameters of the card to store upto 9 keys on it.

The spec of the OpenPGP card tells that it supports RSA keys upto 3072 bits, but there are reports that it is able to handle RSA keys upto 4096 bits (you need to have at least GPG 2.0.18 to handle that big keys on the crypto card). It looks to me like the card is not handle DSA (or ECDSA) cards. There are only slots for upto 3 keys on it.

If I go this way, I would also need a card reader. It seems a class 3 one (hardware PIN pad and display) would be the most “future-proof� way to go ahead. I found a Reiner SCT cyberJack secoder card reader, which is believed to be supported by OpenSC and seems to be a good balance between cost and features of the Reiner SCT card readers.

If anyone reading this can suggest a better crypto card (keys upto 4096 bits, more than 3 slots, and/or DSA/ECDSA  support), or a better card reader, or has any practical experience with any of those components on FreeBSD, please add a comment.

Share

Alexander Leidinger » FreeBSD 2012-05-31 16:25:32

In several previous posts I wrote about my quest for the right source format to stream video to my Sony BRAVIA TV (build in 2009). The last week-end I finally found something which satisfies me.

What I found was serviio, a free UPnP-AV (DLNA) server. It is written in java and runs on Windows, Linux and FreeBSD (it is not listed on the website, but we have an not-so-up-to-date version in the ports tree). If necessary it transcodes the input to an appropriate format for the DLNA renderer (in my case the TV).

I tested it with my slow Netbook, so that I was able to see with which input format it will just remux the input container to a MPEG transport stream, and which input format would be really re-encoded to a format the TV understands.

The bottom line of the tests is, that I just need to use a supported container (like MKV or MP4 or AVI) with H.264-encoded video (e.g. encoded by x264) and AC3 audio.

The TV is able to chose between several audio streams, but I have not tested if serviio is able to serve files with multiple audio streams (my wife has a different mother language than me, so it is interesting for us to have multiple audio streams for a movie), and I do not know if DLNA supports something like this.

Now I just have to replace minidlna (which only works good with my TV for MP3s and Pictures) with serviio on my FreeBSD file server and we can forget about the disk-juggling.

Share

Alexander Leidinger

This weekend I made some progress in the linuxulator:

  • I MFCed the reporting of some linux-syscalls to 9-stable and 8-stable.
  • I updated my linuxulator-dtrace patch to a recent -current. I already compiled it on i386 and arundel@ has it compiled on amd64. I counted more than 500 new DTrace probes. Now that DTrace rescans for SDT probes when a kernel module is loaded, there is no kernel panic anymore when the linux module is loaded after the DTrace modules and you want to use DTrace. I try to commit this at a morning of a day where I can fix things during the day in case some problems show up which I did not notice during my testing.
  • I created a PR for portmgr@ to repocopy a new linux_base port.
  • I set the expiration date of linux_base-fc4 (only used by 7.x and upstream way past its EoL) and all dependent ports. It is set to the EoL of the last 7.x release, which can not use a later linux_base port. I also added a comment which explains that the date is the EoL of the last 7.x release.

Share

Alexander Leidinger

I just updated to a recent -current and tried the new nullfs. Sockets (e.g. the MySQL one) work now with nullfs. No need to have e.g. jails on the same FS and hardlink the socket to not need to use TCP in MySQL (or an IP at all for the jail).

Great work!

Share

(Free)BSD inside Android

Today I was looking into the OpenSource licenses which are displayed for Android (2.3.4). There are several files which come with a BSD license.

During looking at it, I noticed that the libm has the copyright of several FreeBSD people. I did not had an in-deep look if this is because they took the FreeBSD libm, or if this is because parts of the FreeBSD libm where adopted by other BSD projects.

What I noticed is, that some special characters are not displayed correctly. For example the name Dag-Erling Smørgrav looks mangled in the display of the license inside the phone (I hope it is displayed better in my blog). His name is not the only problem case, there are also other characters which are not rendered as expected.

This does not really look professional.

Share

(Free)BSD inside Android

Today I was looking into the OpenSource licenses which are displayed for Android (2.3.4). There are several files which come with a BSD license.

During looking at it, I noticed that the libm has the copyright of several FreeBSD people. I did not had an in-deep look if this is because they took the FreeBSD libm, or if this is because parts of the FreeBSD libm where adopted by other BSD projects.

What I noticed is, that some special characters are not displayed correctly. For example the name Dag-Erling Smørgrav looks mangled in the display of the license inside the phone (I hope it is displayed better in my blog). His name is not the only problem case, there are also other characters which are not rendered as expected.

This does not really look professional.

Share

HOWTO add linux-infrastructure ports for a new linux_base port

In my last blog-post I described how to create a new linux_base port. This blog-post is about the other Linux–ports which make up the Linux–infrastructure in the FreeBSD Ports Collection for a given Linux-release.

What are linux-infrastructure ports?

A linux_base port contains as much as possible and at the same time as little as possible to make up a useful Linux-compatibility-experience in FreeBSD. I know, this is not a descriptive explanation. And it is not on purpose. There are no fixed rules what has to be inside or what not. It “matured

HOWTO create a new linux_base port

FreeBSD is in need of a new linux_base port. It is on my TODO list since a long time, but I do not get the time to create one. I still do not have the time to work on a new one, but when you read this, I managed to get the time to create a HOWTO which describes what needs to be done to create a new linux_base port.

I will not describe how to create a new linux_base port from scratch, I will just describe how you can copy the last one and update it to something newer based upon the existing infrastructure for RPM packages.

Specific questions which come up during porting a new Linux release should be asked on [email protected],  there are more people which can answer questions than here in my blog. I will add useful information to this HOWTO if necessary.

In the easy case most of the work is searching the right RPMs and their dependencies to use, and to create the plist.

Why do we need a new linux_base port?

The current linux_base port is based upon Fedora 10, which is end of life since December 2009. Even Fedora 13 is already end of life. Fedora 16 is supposed to be released this year. From a support point of view, Fedora 15 or maybe even Fedora 16 would be a good target for the next linux_base port. Other alternatives would be to use an extended lifetime release of another RPM based distribution, like for example CentOS 6 (which seems to be based upon Fedora 12 with backports from Fedora 13 and 14). Using a Linux release which is told to be supported for at least 10 years, sounds nice from a FreeBSD point of view (only minor changes to the linux ports in such a case, instead of creating a complete new linux_base each N+2 releases like with Fedora), but it also means additional work if you want to create the first linux_base port for it.

The mysteries you have to conquer if you want to create a new linux_base port

What we do not know is, if Fedora 15/16, CentOS 6, or any other Linux release will work in a supported FreeBSD release. There are two ways to find this out.

The first one is to take an existing Linux system, chroot into it (either via NFS or after making a copy into a directory of a FreeBSD system), and to run a lot of programs (acroread, skype, shells, scripts, …). The LTP testsuite is not that much useful here, as it will test mostly kernel features, but we do not know which kernel features are mandatory for a given userland of a Linux release.

The second way of testing if a given Linux release works on FreeBSD is to actually create a new linux_base port for it and test it without chrooting.

The first way is faster, if you are only interested in testing if something works. The second way provides an easy to setup testbed for FreeBSD kernel developers to fix the Linuxulator so that it works with the new linux_base port. Both ways have their merits, but it is up to the person doing the work to decide which way to go.

The meat: HOWTO create a new linux_base port

First off, you need a system (or a jail) without any linux_base port installed. After that you can create a new linux_base port (= lbN), by just making a copy of the latest one (= lbO). In lbN you need to add lbO as a CONFLICT, and in all other existing linux_base ports, you need to add lbN as a conflict.

Change the PORTNAME, PORTVERSION, reset the PORTREVISION in lbN, and set LINUX_DIST_VER  to the new Linux-release version in the lbN Makefile (this is used in PORTSDIR/Mk/bsd.linux-rpm.mk and PORTSDIR/Mk/bsd.linux-apps.mk).

If you do not stay with Fedora, there is some more work to do before you can have a look at chosing RPMs for installation. You need to have a look at PORTSDIR/Mk/bsd.linux-rpm.mk and add some cases for the new LINUX_DIST you want to use. Do not forget to set LINUX_DIST in the lbN Makefile to the name of the distribution you use. You also need to augment the LINUX_DIST_VER check in PORTSDIR/Mk/bsd.linux-rpm.mk with some LINUX_DIST conditionals. If you are lucky, the directory structure for downloads is similar to the Fedora structure, and there is not a lot to do here.

When this is done, you can have a look at the BIN_DISTFILES variable in the lbN Makefile. Try to find similar RPMs for the new Linux release you want to port. Some may not be available, and it may also be the case that different ones are needed instead. I suggest to first work with the ones which are available (make makesum, test install and create plist). After that you need to find out what the replacement RPMs for non-existing ones are. You are on your own here. Search around the net, and/or have a look at the dependencies in the RPMs of lbO to determine if something was added as a dependency of something else or not (if not, forget about it ATM). When you managed to find replacement RPMs, you can now have a look at the dependencies of the RPMs in lbN. Do not add blindly all dependencies, not all are needed in FreeBSD (the linux_base ports are not supposed to create an environment which you can chroot into, they are supposed to augment the FreeBSD system to be able to run Linux programs in ports like they where FreeBSD native programs). What you need in the linux_base ports are libraries, config and data files which do not exist in FreeBSD or have a different syntax than in FreeBSD (those config or data files which are just in a different place, can be symlinked), and basic shell commands (which commands are needed or not… well… good question, in the past we made decisions what to include based upon problem reports from users). Now for the things which are not available and where not added as a dependency. Those are things which are either used during install, or where useful to have in the past. Find out by what it was replaced and have a look if this replacement can easily be used instead. If it can be used, add it. If not, well… bad luck, we (the FreeBSD community) will see how to handle this somehow.

If you think that you have all you need in BIN_DISTFILES, please update SRC_DISTFILES accordingly and generate the distfile via  make –DPACKAGE_BUILDING makesum to have the checksums of the sources (for legal reasons we need them on our mirrors).

The next step is to have a look at REMOVE_DIRS, REMOVE_FILES and ADD_DIRS if something needs to be modified. Most of them are there to fall back to the corresponding FreeBSD directories/files, or because they are not needed at all (REMOVE_*). Do not remove directories from ADD_DIRS, they are created here to fix some edge conditions (I do not remember exactly why we had to add them, and I do not take the time ATM to search in the CVS history).

If you are lucky, this is all (make sure the plist is correct). If you are not lucky and you need to make some modifications to files, have a look at the do-build target in the Makefile, this is the place where some changes are done to create a nice user experience.

If you arrive here while creating a new linux_base port, lean back and feel a bit proud. You managed to create a new linux_base port. It is not very well tested at this moment, and it is far from everything which needs to be done to have the complete Linux infrastructure for a given Linux release, but the most important part is done. Please notify [email protected] and call for testers.

What is missing?

The full Linuxulator infrastructure for the FreeBSD Ports Collection has some more ports around a linux_base port. Most of the infrastructure for this is handled in Mk/bsd.linux-apps.mk.

UPDATE: I got some time to write how to update the Linux-infrastructure ports.

Share

How I setup a Jail-Host

Everyone has his own way of setting up a machine to serve as a host of multiple jails. Here is my way, YMMV.

Initial FreeBSD install

I use several harddisks in a Software–RAID setup. It does not matter much if you set them up with one big partition or with several partitions, feel free to follow your preferences here. My way of partitioning the harddisks is described in a previous post. That post only shows the commands to split the harddisks into two partitions and use ZFS for the rootfs. The commands to initialize the ZFS data partition are not described, but you should be able to figure it out yourself (and you can decide on your own what kind of RAID level you want to use). For this FS I set atime, exec and setuid to off in the ZFS options.

On the ZFS data partition I create a new dataset for the system. For this dataset I set atime, exec and setuid to off in the ZFS options. Inside this dataset I create datasets for /home, /usr/compat, /usr/local, /usr/obj, /usr/ports/, /usr/src, /usr/sup and /var/ports. There are two ways of doing this. One way is to set the ZFS mountpoint. The way I prefer is to set relative symlinks to it, e.g. “cd /usr; ln –s ../data/system/usr_obj obj

Converting from Courier IMAP to dovecot is easy

I have used Courier IMAP at home since a long time. As I want to update a dovecot 1.2 setup to dovecot 2.x, I decided to first have a look at dovecot 2.x at home.

Switching from Courier IMAP to dovecot is really easy. I just configured the correct path to the maildir, setup a passdb/userdb, and it was working.

The important part was the correct transfer of the passwords. I used already an userdb in Courier IMAP with MD5 passwords. For each user it has imappw=XXX with XXX similar to $1$abc.

This can be converted into a dovecot passdb/userdb line very easily:

username:{MD5-CRYPT}$1$abc::UID:GID::HOMEDIR::userdb_mail=maildir:~/path/to/maildir

The corresponding passdb/userdb settings for dovecot are:

passdb {
   args = scheme=MD5-CRYPT username_format=%u /usr/local/etc/dovecot/dovecot.pws
   driver = passwd-file
}
userdb {
   args = username_format=%u /usr/local/etc/dovecot/dovecot.pws
   driver = passwd-file
}

Compared to when I had a look the last time, dovecot is also able to use OTP as an authentication mechanism now. Unfortunately I did not find any documentation how to configure/use it.

Share

Non-default linux base ports deprecated

Yesterday I deprecated the non-default Fedora based Linux base ports. This means fc6, f7, f8 and f9 will vanish soon (I decided for one month of expiry time). This is because all of them are End of Life upstream since a long time (= no security updates).

The fc4 and f10 ones are still available — even if they are End of Life too — because FreeBSD 7.x can not use something newer than the fc4 one, and we have not tested yet a more recent Linux distribution.

Probably the most easy way to update the Linux base ports to something newer is to stay with Fedora (we have a lot of ports-infrastructure for it already). Unfortunately it is not known if something newer works without problems (missing epoll/inotify support could be a roadblock here in case it is extensively used in a more recent version).

I want to get some time to have a look if a more recent Fedora version is suitable for the use as a Linux base in FreeBSD 8.x+, but I do not have an estimate when I can start and how long it may take. In case someone already tested a more recent Fedora version feel free to share your experience.

Share

HOWTO: creating your own updated linux RPM for the FreeBSD linuxulator

Background info

The FreeBSD linux compatibility environment currently uses RPMs from Fedora 10. Unfortunately Fedora 10 is end of life since a while. For one of the RPMs (the pango one) we where aware of a security vulnerability. As we do not know if it is feasible to update the linuxulator ports to something more recent, I decided to setup a VM with Fedora 10 and generate a new RPM for the linux-f10-pango port. Thanks to Luchesar V. ILIEV for explaining me how to do this.

Setup of the VM

I used VirtualBox 4.0.4 on a Solaris 10 x86 machine. I configured a fixed size disk of 16 GB and kept the default network setup (after installing the guest tools / kernel modules I switched to virtio, as I was not able to do anything useful besides a ping) and RAM size. The CD/DVD drive was configured to use the image of the full Fedora 10 DVD for i386 systems.

Setup of Fedora 10

Booting the VM from the DVD leads to the graphical Fedora 10 install software (after chosing to install a new system on the console). There I accepted all the defaults, except for the software to install. I deselected the Office and Productivity group and selected the Software Development group. When I was asked if I want to install some additional RPMs I had a look at the complete list and installed some I thought are necessary. I do not remember anymore which ones I chose, but everything which looks related to RPM building is a good candidate.

After a while the install will be finished and you can boot into the new system (eject the DVD from the drive before reboot). After reboot chose to install the Guest Additions in the menu of the VM. This should mount the ISO image in the VM. As root execute the file for Linux. This will build some kernel modules for better integration (e.g. seamless integration of the mouse between your desktop and the VM). At this point I rebooted and configured virtio as the NIC. I also had to configure the network settings by hand, as the GUI tool did not safe all the settings correctly.

Update and install of required RPMs

After the VM was up and the network configured, I updated the entire system (chose System Update in the menu). To update the pango port, I had to install the libthai-devel RPM. I had the RPM for it (and all the files I need to build a new pango RPM) already downloaded, so I did a “yum install /path/to/rpm

A new linux-f10-pango port is ready

In the last days I took (and even had) the time to install a VM with Fedora 10, updated all the packages after installation, and created a new linux-f10-pango port (v 1.28.3). I did this because the port has a security vulnerability according to our VuXML DB and there where more and more reports in the last months from users which had a problem with this.

During the update of the port I noticed that the port does not contain a FORBIDDEN entry, just portaudit complains about it because there is an entry in the VuXML. That is not nice. I was told that the ports slush will be lifted soon (I need to bump some PORTREVISIONs), this means that I can commit the update probably tomorrow, just in time when the new RPM should hit the FreeBSD distribution infrastructure (MASTER_SITE_LOCAL is updated once a day from a specific folder in our home directories).

Thanks to Luchesar V. ILIEV for the nice writeup of what to install in Fedora 10 to be able to build RPMs, and the description of how to build your own RPM.

Share

Jumstart/JET for FreeBSD (brainstorming)

There are some HOWTOs out there in the net which describe some automatic network based install via PXE-booting a machine from a server which has a specific FreeBSD release in the PXE-booting area and a non-interactive config for sysinstall to install this FreeBSD version on the machine which PXE-boots this.

The setup of this is completely manual and only allows to netboot one FreeBSD version. The server-side setup for the clients is also completely manual (and only allows to install one client at a time, it seems). This is not very user-friendly, and far away from the power of Jumpstart/JET for Solaris where you create a template (maybe from another template with automatic value (IP, name, MAC) replacement) and can specify different OS releases for different clients and then just run a command to generate a good config for this.

I thought a little bit how it could be done and decided to write down all the stuff (so far 160 lines, 830 words) to not forget some details. All in all I think this could be done (at least a sensible subset) in a week or two (fulltime) if you have the hardware, motivation, and time. As always, the problems are within the details, so I may be off with my estimation a little bit (also depends upon the knowledge-level (shell, tftp, dhcpd, install–software) of the person doing this).

Unfortunately I do not know if I have the hardware at home to do something like this. I have some unused harddisks which could be used in a machine which is used temporary as a test-install-client (normally I use this machines as my Desktop… if I do not use my little Netbook instead, as I do not do much at home currently), but I’ve never checked if this machine is PXE-booting-capable (VIA KT133 chipset with a 3Com 3c905C-TX Fast Etherlink XL). I also do not have the time to do this (with the current rate of free time I would expect to need about a year), except maybe someone would call my boss and negotiate something.

I can not remember any request to have something like this on the freebsd-current, freebsd-arch or freebsd-hackers list since I read them (and that is since about at least 3.0-RELEASE). Is this because nearly nobody is interested in something like this, or are the current possibilities enough for your needs? Do you work at a place where this would be welcome (= directly used when it would be done)? If you use a simple solution to make a net-install, what is your experience with this (pros/cons)?