Category Archives: Tips & Tricks

Alexander Leidinger » FreeBSD 2012-12-23 18:22:43

From time to time I convert videos to H264. When I do this I want to get the best quality out of a give filesize. This means I create VBR videos. The question here is, how big the target video file shall be.

After searching around a little bit in the net I found a formula which is supposed to give a hint about the target filesize. Naturally this depends on the encoder (or even the encoder-version), the encoder settings and even the video.

The formula is at least a good hint for my use, so I wrote a script which calculates several filesize values for a given video (based upon the output of mediainfo, which the scripts expects in a file with the filename as an argument to the script). It calculates a CBR and a VBR value for a given video based upon the width, height and duration. It should work on all system with a POSIX compatible shell.

Example output for a video from my HD-ready cam, original filesize 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 2
Per second: 6451200.000 bps / 6300 Kibps
Total CBR: 1148313600 bytes / 1121400 KiB / 1095 MiB
Total VBR: 861235200 bytes / 841050 KiB / 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 3
Per second: 9676800.000 bps / 9450 Kibps
Total CBR: 1722470400 bytes / 1682100 KiB / 1642 MiB
Total VBR: 1291852800 bytes / 1261575 KiB / 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 4
Per second: 12902400.000 bps / 12600 Kibps
Total CBR: 2296627200 bytes / 2242800 KiB / 2190 MiB
Total VBR: 1722470400 bytes / 1682100 KiB / 1642 MiB

There are 3 sections, the difference is the “motion� value. It is a kind of multiplicator depending on the amount of motion in the video. For the videos I made myself (family videos, and even some videos of volley ball games), the first section seems to be just fine. So I reduced the original MP4 file to about 50% (not visible here is the audio size, normally I copy the original audio unmodified).

For the curious ones, the formula is

width_in_pixels * height_in_pixels * fps * motion_value * 0.07

for the bps value. The CBR value is

bps * playtime_in_seconds / 8

and the VBR value is

3 / 4 * CBR_value.

Share

Related Posts:

Alexander Leidinger » FreeBSD 2012-06-15 20:18:07

FreeBSD is on its way to move from CVS to SVN  for the version control system for the Ports Collection. The decision was made to keep the complete history, so the complete CVS repository has to be converted to SVN.

As CVS has no way to record a copy or move of files inside the repository, we copied the CVS files inside the repository in case we wanted to copy or move a file (the so called “repocopy�). While this allows to see the full history of a file, the drawback is that you do not really know when a file was copied/moved if you are not strict at recording this info after doing a copy. Guess what, we where not.

Now with the move to SVN which has a build-in way for copies/moves, it would be nice if we could record this info. In an internal discussion someone told its not possible to detect a repocopy reliably.

Well, I thought otherwise and an hour later my mail went out how to detect one. The longest time was needed to write how to do it, not to come up with a solution. I do not know if someone picked up this algorithm and implemented something for the cvs2svn converter, but I decided to publish the algorithm here if someone needs a similar functionality somewhere else. Note, the following is tailored to the structure of the Ports Collection. This allows to speed up some things (no need to do all steps on all files). If you want to use this in a generic repository where the structure is not as regular as in our Ports Collection, you have to run this algorithm on all files.

It also detects commits where multiple files where committed at once in one commit (sweeping commits).

Preparation

  • check only category/name/Makefile
  • generate a hash of each commitlog+committer
  • if you are memory-limited use ha/sh/ed/dirs/cvs-rev and store pathname in the list cvs-rev (pathname = “category-nameâ€?) as storage
  • store the hash also in pathname/cvs-rev

If you have only one item in ha/sh/ed/dirs/cvs-rev in the end, there was no repocopy and no sweeping commit, you can delete this ha/sh/ed/dirs/cvs-rev.

If you have more than … let’s say … 10 (subject to tuning) pathnames in ha/sh/ed/dirs/cvs-rev you found a sweeping commit and you can delete the ha/sh/ed/dirs/cvs-rev.

The meat

The remaining ha/sh/ed/dirs/cvs-rev are probably repocopies. Take one ha/sh/ed/dirs/cvs-rev and for each pathname (there may be more than 2 pathnames) in there have a look at pathname/. Take the first cvs-rev of each and check if they have the same hash. Continue with the next rev-number for each until you found a cvs-rev which does not contain the same hash. If the number of cvs-revs since the beginning is >= … let’s say … 3 (subject to tuning), you have a candidate for a repocopy. If it is >=  … 10 (subject to tuning), you have a very good indicator for a repocopy. You have to proceed until you have only one pathname left.

You may detect multiple repocopies like A->B->C->D or A->B + A->D + A->C here.

Write out the repocopy candidate to a list and delete the ha/sh/ed/dirs/cvs-rev for each cvs-rev in a detected sequence.

This finds repocopy candidates for category/name/Makefile. To detect the correct repocopy-date (there are maybe cases where another file was changed after the Makefile but before the repocopy), you now have to look at all the files for a given repocopy-pair and check if there is a matching commit after the Makefile-commit-date. If you want to be 100% sure, you compare the complete commit-history of all files for a given repocopy-pair.

Share

Related Posts:

Alexander Leidinger » FreeBSD 2012-05-31 17:25:32

In several previous posts I wrote about my quest for the right source format to stream video to my Sony BRAVIA TV (build in 2009). The last week-end I finally found something which satisfies me.

What I found was serviio, a free UPnP-AV (DLNA) server. It is written in java and runs on Windows, Linux and FreeBSD (it is not listed on the website, but we have an not-so-up-to-date version in the ports tree). If necessary it transcodes the input to an appropriate format for the DLNA renderer (in my case the TV).

I tested it with my slow Netbook, so that I was able to see with which input format it will just remux the input container to a MPEG transport stream, and which input format would be really re-encoded to a format the TV understands.

The bottom line of the tests is, that I just need to use a supported container (like MKV or MP4 or AVI) with H.264-encoded video (e.g. encoded by x264) and AC3 audio.

The TV is able to chose between several audio streams, but I have not tested if serviio is able to serve files with multiple audio streams (my wife has a different mother language than me, so it is interesting for us to have multiple audio streams for a movie), and I do not know if DLNA supports something like this.

Now I just have to replace minidlna (which only works good with my TV for MP3s and Pictures) with serviio on my FreeBSD file server and we can forget about the disk-juggling.

Share

Related Posts:

What you should know about SSH

Michael W. Lucas published his new book “SSH Mastery� (no link to an online store, get it from your preferred online or offline one in your part of the world).

Do you think you know a lot about SSH? I thought I did when Michael searched technical proof-readers for this book. I offered to have a look at his work in progress and he gently accepted (while I do not get money for this, I am one of the persons he thanks for  the technical review in the beginning, so I am involved somehow and as such you should take the following with a grain of salt).

I already had user restrictions in place before the review, but now I narrowed down some restrictions based upon some conditionals. I already used SSH tunnels for various things before (where legally applicable), but I learned some additional VPN techniques with SSH. I already used multiple ssh-keys for various things, but Michael provides some interesting ways of handling a large-volume of ssh-keys over multiple machines. … I really hope that my review was as valuable for Michael, as it was for me to do the review.

He ends the book with “You now know more about SSH, OpenSSH and Putty than the vast majority of IT professionals! Congratulations�, and this is true, and all that in his writing style where you can come with a problem, read about it, and leave with a solution (normally with a little bit of entertainment in between).

I know a lot of people which work daily with SSH, and they know only a small part of what is presented in this book. In my opinion this book is a must-have for every System/Database/Application/Whatever Administrator in charge of something on an UNIX-like system, and even “normal users� of SSH (no matter if they use PuTTY, or a ssh command line program on an UNIX-like system (most probably it will be OpenSSH or a clone of it)) will get some helpful information from this book.

I can only recommend it.

Related Posts:

Share

Tuning guide in the wiki

In the light of the recent benchmark discussion, a volunteer imported the tuning man-page into the wiki. Some comments at some places for possible improvements are already made. Please go over there, have a look, and participate please (testing/verification/discussion/improvements/…).

As always, feel free to register with FirstnameLastname and tell a FreeBSD committer to add you to the contributors group for write access (you also get the benefit to be able to register for an email notification for specific pages).

Share

Speed traps with chmod

I have the habit to chmod with the relative notation (e.g. g+w or a+r or go-w or similar) instead of the absolute one (e.g. 0640 or u=rw,g=r,o=). Recently I had to chmod a lot of files. As usual I was using the relative notation. With a lot of files, this took a lot of time. Time was not really an issue, so I did not stop it to restart with a better performing command (e.g. find /path –type f –print0 | xargs –0 chmod 0644; find /path –type d –print0 | xargs –0 chmod 0755), but I thought a little tips&tricks posting may be in order, as not everyone knows the difference.

The relative notation

When you specify g+w, it means to remove the write access for the group, but keep everything else like it is. Naturally this means that chmod first has to lookup the current access rights. So for each async write request, there has to be a read-request first.

The absolute notation

The absolute notation is what most people are used to (at least the numeric one). It does not need to read the access rights before changing them, so there is less I/O to be done to get what you want. The drawback is that it is not so nice for recursive changes. You do not want to have the x-bit for data files, but you need it for directories. If you only have a tree with data files where you want to have an uniform access, the example above via find is probably faster (for sure if the directory meta-data is still in RAM).

If you have a mix of binaries and data, it is a little bit more tricky to come up with a way which is faster. If the data has a name-pattern, you could use it in the find.

And if you have a non-uniform access for the group bits and want to make sure the owner has write access to everything, it may be faster to use the relative notation than to find a replacement command-sequence with the absolute notation.

Share

HOWTO add linux-infrastructure ports for a new linux_base port

In my last blog-post I described how to create a new linux_base port. This blog-post is about the other Linux–ports which make up the Linux–infrastructure in the FreeBSD Ports Collection for a given Linux-release.

What are linux-infrastructure ports?

A linux_base port contains as much as possible and at the same time as little as possible to make up a useful Linux-compatibility-experience in FreeBSD. I know, this is not a descriptive explanation. And it is not on purpose. There are no fixed rules what has to be inside or what not. It “matured� into the current shape. A practical example is, that there is no GUI-stuff in the linux_base. While you need the GUI parts like GTK or QT for software like Skype and acroread, you do not need them for headless game servers. While you may need various libraries for game servers, you may not need those for Skype or acroread. As such some standard parts are in separate ports which are named linux–LINUX_DIST_SUFFIX-NAME. For GTK and the Fedora 10 release this results in linux-f10-gtk2. Such generic ports which depend upon a specific Linux-release make up the Linux-infrastructure in the FreeBSD Ports Collection. Those ports are referenced in port-Makefiles via the USE_LINUX_APPS variable, e.g. USE_LINUX_APPS=gtk2.

If you created a new linux_base port, you need most standard infrastructure ports in a version for the Linux-release used in the linux_base port, to have the Linux-application ports in the FreeBSD Ports Collection working (if you are unlucky, some ports do not play well with the Linux-release you have chosen, but this is out of the scope of this HOWTO).

Updating Mk/bsd.linux-apps.mk

 First we need to set the LINUX_DIST_SUFFIX variable to a value suitable to the new Linux-release. This is done in the conditional which checks the OVERRIDE_LINUX_NONBASE_PORTS variable for valid values. Add an appropriate conditional, and do not forget to add the new valid value to the IGNORE line in the last else branch of the conditional.

The next step is to check the _LINUX_APPS_ALL and _LINUX_26_APPS variables. If there are some infrastructure ports which are not available for the new Linux-release, the conditional which checks the availability of a given infrastructure port for a given Linux-release needs to be modified. If at a later step you notice that there are some additional infrastructure ports necessary for the new Linux-release, _LINUX_APPS_ALL and the check-logic needs to be modified too (e.g. add a new variable for your Linux-release, add the content of the variable to _LINUX_APPS_ALL, and change the check to do the right thing).

After that two tedious parts need to be done.

For each infrastructure port there is a set of variables. The name_PORT variable contains the location of the port in the Ports Collection. Typically you do not have to change it (if you really want to change it, do not do it, fix the naming of the infrastructure port instead), because we use a naming convention here which includes the LINUX_DIST_SUFFIX. The name_DETECT variable is an internal variable, do not change it (if you create a new infrastructure port, copy it from somewhere else and make sure the name in value of the variable matches the port name in the name of the variable). Then there are several name_suffix_FILE variables. Leave the existing ones alone, and add a new one with the correct suffix for your new Linux-release. The value of the variable needs to be an important file which is installed by the infrastructure port in question. FYI: The content of the name_suffix_FILE variables are used to set the name_DETECT variables, depending on the Linux-relase the name_DETECT variables are used to check if the port is already installed. Ideally the name_suffix_FILE variable points to a library in the port. The name_DEPENDS variable lists dependencies of this infrastructure port. If the dependencies changed in your Linux-release, you need to add a conditional to change the dependency if LINUX_DIST_SUFFIX is set to your Linux-release.

Normally this is all what needs to be done in PORTSDIR/Mk/bsd.linux-apps.mk, the rest of the file is code to check dependencies and some correctness checks.

The second tedious part is to actually create all those infrastructure ports. Normally you can copy an existing infrastructure port, rename it, adjust the PORTNAME, PORTVERSION, PORTREVISION, MASTER_SITES, PKGNAMEPREFIX, DISTFILES, CONFLICTS (also in all other Linux-release versions of this infrastructure port), LINUX_DIST_VER, RPMVERSION (if set/neccesary) and SRC_DISTFILE variables, generate the distfile checksums (make makesum), and fix the plist. I suggest to script parts of this work (as of this writing Freshports counts 68 ports where the portname starts with linux-f10-).

Adding new infrastructure ports, or removing infrastructure ports for a given Linux-release

If your Linux-release does not come with a package for an existing infrastructure port, just do not create a corresponding name_suffix_FILE line. You still need to do the right thing regarding dependencies of ports which depend upon this non-existing infrastructure port (if your Linux-release comes with packages for them).

To add a new infrastructure port, copy an existing block, rename the variables, set them correctly, add a new variable for your Linux-release in the first _LINUX_APPS_ALL section, add the content of this variable to _LINUX_APPS_ALL, and change the check-logic as described above.

Final words

If you have something which installs and deinstalls correctly, feel free to provide it on [email protected] for review/testing. If you have questions during the porting, feel also free to send a mail there.

Share

HOWTO create a new linux_base port

FreeBSD is in need of a new linux_base port. It is on my TODO list since a long time, but I do not get the time to create one. I still do not have the time to work on a new one, but when you read this, I managed to get the time to create a HOWTO which describes what needs to be done to create a new linux_base port.

I will not describe how to create a new linux_base port from scratch, I will just describe how you can copy the last one and update it to something newer based upon the existing infrastructure for RPM packages.

Specific questions which come up during porting a new Linux release should be asked on [email protected],  there are more people which can answer questions than here in my blog. I will add useful information to this HOWTO if necessary.

In the easy case most of the work is searching the right RPMs and their dependencies to use, and to create the plist.

Why do we need a new linux_base port?

The current linux_base port is based upon Fedora 10, which is end of life since December 2009. Even Fedora 13 is already end of life. Fedora 16 is supposed to be released this year. From a support point of view, Fedora 15 or maybe even Fedora 16 would be a good target for the next linux_base port. Other alternatives would be to use an extended lifetime release of another RPM based distribution, like for example CentOS 6 (which seems to be based upon Fedora 12 with backports from Fedora 13 and 14). Using a Linux release which is told to be supported for at least 10 years, sounds nice from a FreeBSD point of view (only minor changes to the linux ports in such a case, instead of creating a complete new linux_base each N+2 releases like with Fedora), but it also means additional work if you want to create the first linux_base port for it.

The mysteries you have to conquer if you want to create a new linux_base port

What we do not know is, if Fedora 15/16, CentOS 6, or any other Linux release will work in a supported FreeBSD release. There are two ways to find this out.

The first one is to take an existing Linux system, chroot into it (either via NFS or after making a copy into a directory of a FreeBSD system), and to run a lot of programs (acroread, skype, shells, scripts, …). The LTP testsuite is not that much useful here, as it will test mostly kernel features, but we do not know which kernel features are mandatory for a given userland of a Linux release.

The second way of testing if a given Linux release works on FreeBSD is to actually create a new linux_base port for it and test it without chrooting.

The first way is faster, if you are only interested in testing if something works. The second way provides an easy to setup testbed for FreeBSD kernel developers to fix the Linuxulator so that it works with the new linux_base port. Both ways have their merits, but it is up to the person doing the work to decide which way to go.

The meat: HOWTO create a new linux_base port

First off, you need a system (or a jail) without any linux_base port installed. After that you can create a new linux_base port (= lbN), by just making a copy of the latest one (= lbO). In lbN you need to add lbO as a CONFLICT, and in all other existing linux_base ports, you need to add lbN as a conflict.

Change the PORTNAME, PORTVERSION, reset the PORTREVISION in lbN, and set LINUX_DIST_VER  to the new Linux-release version in the lbN Makefile (this is used in PORTSDIR/Mk/bsd.linux-rpm.mk and PORTSDIR/Mk/bsd.linux-apps.mk).

If you do not stay with Fedora, there is some more work to do before you can have a look at chosing RPMs for installation. You need to have a look at PORTSDIR/Mk/bsd.linux-rpm.mk and add some cases for the new LINUX_DIST you want to use. Do not forget to set LINUX_DIST in the lbN Makefile to the name of the distribution you use. You also need to augment the LINUX_DIST_VER check in PORTSDIR/Mk/bsd.linux-rpm.mk with some LINUX_DIST conditionals. If you are lucky, the directory structure for downloads is similar to the Fedora structure, and there is not a lot to do here.

When this is done, you can have a look at the BIN_DISTFILES variable in the lbN Makefile. Try to find similar RPMs for the new Linux release you want to port. Some may not be available, and it may also be the case that different ones are needed instead. I suggest to first work with the ones which are available (make makesum, test install and create plist). After that you need to find out what the replacement RPMs for non-existing ones are. You are on your own here. Search around the net, and/or have a look at the dependencies in the RPMs of lbO to determine if something was added as a dependency of something else or not (if not, forget about it ATM). When you managed to find replacement RPMs, you can now have a look at the dependencies of the RPMs in lbN. Do not add blindly all dependencies, not all are needed in FreeBSD (the linux_base ports are not supposed to create an environment which you can chroot into, they are supposed to augment the FreeBSD system to be able to run Linux programs in ports like they where FreeBSD native programs). What you need in the linux_base ports are libraries, config and data files which do not exist in FreeBSD or have a different syntax than in FreeBSD (those config or data files which are just in a different place, can be symlinked), and basic shell commands (which commands are needed or not… well… good question, in the past we made decisions what to include based upon problem reports from users). Now for the things which are not available and where not added as a dependency. Those are things which are either used during install, or where useful to have in the past. Find out by what it was replaced and have a look if this replacement can easily be used instead. If it can be used, add it. If not, well… bad luck, we (the FreeBSD community) will see how to handle this somehow.

If you think that you have all you need in BIN_DISTFILES, please update SRC_DISTFILES accordingly and generate the distfile via  make –DPACKAGE_BUILDING makesum to have the checksums of the sources (for legal reasons we need them on our mirrors).

The next step is to have a look at REMOVE_DIRS, REMOVE_FILES and ADD_DIRS if something needs to be modified. Most of them are there to fall back to the corresponding FreeBSD directories/files, or because they are not needed at all (REMOVE_*). Do not remove directories from ADD_DIRS, they are created here to fix some edge conditions (I do not remember exactly why we had to add them, and I do not take the time ATM to search in the CVS history).

If you are lucky, this is all (make sure the plist is correct). If you are not lucky and you need to make some modifications to files, have a look at the do-build target in the Makefile, this is the place where some changes are done to create a nice user experience.

If you arrive here while creating a new linux_base port, lean back and feel a bit proud. You managed to create a new linux_base port. It is not very well tested at this moment, and it is far from everything which needs to be done to have the complete Linux infrastructure for a given Linux release, but the most important part is done. Please notify [email protected] and call for testers.

What is missing?

The full Linuxulator infrastructure for the FreeBSD Ports Collection has some more ports around a linux_base port. Most of the infrastructure for this is handled in Mk/bsd.linux-apps.mk.

UPDATE: I got some time to write how to update the Linux-infrastructure ports.

Share

How I setup a Jail-Host

Everyone has his own way of setting up a machine to serve as a host of multiple jails. Here is my way, YMMV.

Initial FreeBSD install

I use several harddisks in a Software–RAID setup. It does not matter much if you set them up with one big partition or with several partitions, feel free to follow your preferences here. My way of partitioning the harddisks is described in a previous post. That post only shows the commands to split the harddisks into two partitions and use ZFS for the rootfs. The commands to initialize the ZFS data partition are not described, but you should be able to figure it out yourself (and you can decide on your own what kind of RAID level you want to use). For this FS I set atime, exec and setuid to off in the ZFS options.

On the ZFS data partition I create a new dataset for the system. For this dataset I set atime, exec and setuid to off in the ZFS options. Inside this dataset I create datasets for /home, /usr/compat, /usr/local, /usr/obj, /usr/ports/, /usr/src, /usr/sup and /var/ports. There are two ways of doing this. One way is to set the ZFS mountpoint. The way I prefer is to set relative symlinks to it, e.g. “cd /usr; ln –s ../data/system/usr_obj obj�. I do this because this way I can temporary import the pool on another machine (e.g. my desktop, if the need arises) without fear to interfere with the system. The ZFS options are set as follows:

ZFS options for data/system/*

Dataset

Option

Value
data/system/homeexecon
data/system/usr_compatexecon
data/system/usr_compatsetuidon
data/system/usr_localexecon
data/system/usr_localsetuidon
data/system/usr_objexecon
data/system/usr_portsexecon
data/system/usr_portssetuidon
data/system/usr_srcexecon
data/system/usr_supsecondarycachenone
data/system/var_portsexecon

The exec option for home is not necessary if you keep separate datasets for each user. Normally I keep separate datasets for home directories, but Jail-Hosts should not have users (except the admins, but they should not keep data in their homes), so I just create a single home dataset. The setuid option for the usr_ports should not be necessary if you redirect the build directory of the ports to a different place (WRKDIRPREFIX in /etc/make.conf).

Installing ports

The ports I install by default are net/rsync, ports-mgmt/portaudit, ports-mgmt/portmaster, shells/zsh, sysutils/bsdstats, sysutils/ezjail, sysutils/smartmontools and sysutils/tmux.

Basic setup

In the crontab of root I setup a job to do a portsnap update once a day (I pick a random number between 0 and 59 for the minute, but keep a fixed hour). I also have http_proxy specified in /etc/profile, so that all machines in this network do not download everything from far away again and again, but can get the data from the local caching proxy. As a little watchdog I have a little @reboot rule in the crontab, which notifies me when a machine reboots:

@reboot grep "kernel boot file is" /var/log/messages | mail -s "`hostname` rebooted" root >/dev/null 2>&1

This does not replace a real monitoring solution, but in cases where real monitoring is overkill it provides a nice HEADS-UP (and shows you directly which kernel is loaded in case a non-default one is used).

Some default aliases I use everywhere are:

alias portmlist="portmaster -L | egrep -B1 '(ew|ort) version|Aborting|installed|dependencies|IGNORE|marked|Reason:|MOVED|deleted|exist|update' | grep -v '^--'"
alias portmclean="portmaster -t --clean-distfiles --clean-packages"
alias portmcheck="portmaster -y --check-depends"

Additional devfs rules for Jails

I have the need to give access to some specific devices in some jails. For this I need to setup a custom /etc/devfs.rules file. The files contains some ID numbers which need to be unique in the system. On a 9-current system the numbers one to four are already used (see /etc/defaults/devfs.rules). The next available number is obviously five then. First I present my devfs.rules entries, then I explain them:

[devfsrules_unhide_audio=5]
add path 'audio*' unhide
add path 'dsp*' unhide
add path midistat unhide
add path 'mixer*' unhide
add path 'music*' unhide
add path 'sequencer*' unhide
add path sndstat unhide
add path speaker unhide
[devfsrules_unhide_printers=6]
add path 'lpt*' unhide
add path 'ulpt*' unhide user 193 group 193
add path 'unlpt*' unhide user 193 group 193
[devfsrules_unhide_zfs=7]
add path zfs unhide
[devfsrules_jail_printserver=8]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_printers
add include $devfsrules_unhide_zfs
[devfsrules_jail_withzfs=9]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_zfs

The devfs_rules_unhide_XXX ones give access to specific devices, e.g. all the sound related devices or to local printers. The devfsrules_jail_XXX ones combine all the unhide rules for specific jail setups. Unfortunately the include directive is not recursive, so that we can not include the default devfsrules_jail profile and need to replicate its contents. The first three includes of each devfsrules_jail_XXX accomplish this. The unhide_zfs rule gives access to /dev/zfs, which is needed if you attach one or more ZFS datasets to a jail. I will explain how to use those profiles with ezjail in a follow-up post.

Jails setup

I use ezjail to manage jails, it is more comfortable than doing it by hand while at the same time allows me to do something by hand. My jails normally reside inside ZFS datasets, for this reason I have setup a special area (ZFS dataset data/jails) which is handled by ezjail.The corresponding ezjail.conf settings are:

ezjail_jaildir=/data/jails
ezjail_use_zfs="YES"
ezjail_jailzfs="data/jails"

I also disabled procfs and fdescfs in jails (but they can be enabled later for specific jails if necessary).

Unfortunately ezjail (as of v3.1) sets the mountpoint of a newly created dataset even if it is not necessary. For this reason I always issue a “zfs inherit mountpoint � after creating a jail. This simplifies the case where you want to move/rename a dataset and want to have the mountpoint automcatically follow the change.

The access flags of  /data/jails directory are 700, this prevents local users (there should be none, but better safe than sorry) to get access to files from users in jails with the same UID.

After the first create/update of the ezjail basejail the ZFS options of basejail (data/jails/basejail) and newjail (data/jails/newjail) need to be changed. For both exec and setuid should be changed to “on� The same needs to be done after creating a new jail for the new jail (before starting it).

The default ezjail flavour

In my default ezjail flavour I create some default user(s) with a basesystem-shell (via /data/jails/flavours/mydef/ezjail.flavour) before the package install, and change the shell to my preferred zsh afterwards (this is only valid if the jails are used only by in-house people, if you want to offer lightweight virtual machines to (unknown) customers, the default user(s) and shell(s) are obviously up to discussion). At the end I also run a “/usr/local/sbin/portmaster –y –check-depends� to make sure everything is in a sane state.

For the packages (/data/jails/flavours/mydef/pkg/) I add symlinks to the unversioned packages I want to install. I have the packages in a common (think about setting PACKAGES in make.conf and using PACKAGES/Latest/XYZ.tbz) directory (if they can be shared over various flavours), and they are unversioned so that I do not have to update the version number each time there is an update. The packages I install by default are bsdstats, portaudit, portmaster, zsh, tmux and all their dependencies.

In case you use jails to virtualize services and consolidate servers (e.g. DNS, HTTP, MySQL each in a separate jail) instead of providing lightweight virtual machines to (unknown) customers, there is also a benefit of sharing the distfiles and packages between jails on the same machine. To do this I create /data/jails/flavours/mydef/shared/ports/{distfiles,packages} which are then mounted via nullfs or NFS into all the jails from a common directory. This requires the following variables in /data/jails/flavours/mydef/etc/make.conf (I also keep the packages for different CPU types and compilers in the same subtree, if you do not care, just remove the “/${CC}/${CPUTYPE}� from the PACAKGES line):

DISTDIR=  /shared/ports/distfiles
PACKAGES= /shared/ports/packages/${CC}/${CPUTYPE}

New jails

A future post will cover how I setup new jails in such a setup and how I customize the start order of jails or use some non-default settings for the jail-startup.

Share

Another root-on-zfs HOWTO (optimized for 4k-sector drives)

After 9 years with my current home-server (one jail for each service like MySQL, Squid, IMAP, Webmail, …) I decided that it is time to get something more recent (specially as I want to install some more jails but can not add more memory to this i386 system).

With my old system I had an UFS2-root on a 3-way-gmirror, swap on a 2-way-gmirror and my data in a 3-partition raidz (all in different slices of the same 3 harddisks, the 3rd slice which would correspond to the swap was used as a crashdump area).

For the new system I wanted to go all-ZFS, but I like to have my boot area separated from my data area (two pools instead of one big pool). As the machine has 12 GB RAM I also do not configure swap areas (at least by default, if I really need some swap I can add some later, see below). The system has five 1 TB harddisks and a 60 GB SSD. The harddisks do not have 4k-sectors, but I expect that there will be more and more 4k-sector drives in the future. As I prefer to plan ahead I installed the ZFS pools in a way that they are “4k-ready�. For those which have 4k-sector drives which do not tell the truth but announce they have 512 byte sectors (I will call them pseudo-4k-sector drives here) I include a description how to properly align the (GPT-)partitions.

A major requirement to boot 4k-sector-size ZFS pools is ZFS v28 (to be correct here, just the boot-code needs to support this, so if you take the pmbr and gptzfsboot from a ZFS v28 system, this should work… but I have not tested this). As I am running 9-current, this is not an issue for me.

A quick description of the task is to align the partition/slices properly for pseudo-4k-sector drives, and then use gnop temporary during pool creation time to have ZFS use 4k-sectors during the lifetime of the pool. The long description follows.

The layout of the drives

The five equal drives are partitioned with a GUID partition table (GPT). Each drive is divided into three partitions, one for the boot code, one for the root pool, and one for the data pool. The root pool is a 3-way mirror and the data pool is a raidz2 pool over all 5 disks. The remaining space on the two harddisks which do not take part in the mirroring of the root pool get swap partitions of the same size as the root partitions. One of them is used as a dumpdevice (this is –current, after all), and the other one stays unused as a cold-standby. The 60 GB SSD will be used as a ZFS cache device, but as of this writing I have not decided yet if I will use it for both pools or only for the data pool.

Calculating the offsets

The first sector after the GPT (created with standard settings) which can be used as the first sector for a partition is sector 34 on a 512 bytes-per-sector drive. On a pseudo-4k-sector drive this would be somewhere in the sector 4 of a real 4k-sector, so this is not a good starting point. The next 4k-aligned sector on a pseudo-4k-sector drive is sector 40 (sector 5 on a real 4k-sector drive).

The first partition is the partition for the FreeBSD boot code. It needs to have enough space for gptzfsboot. Only allocating the space needed for gptzfsboot looks a little bit dangerous regarding future updates, so my harddisks are configured to allocate half a megabyte for it. Additionally I leave some unused sectors as a safety margin after this first partition.

The second partition is the root pool (respectively the swap partitions). I let it start at sector 2048, which would be sector 256 on a real 4k-sector drive (if you do not want to waste less than half a megabyte just calculate a lower start sector which is divisible by 8 (-> start % 8 = 0)). It is a 4 GB partition, this is enough for the basesystem with some debug kernels. Everything else (/usr/{src,ports,obj,local}) will be in the data partition.

The last partition is directly after the second and uses the rest of the harddisk rounded down to a full GB (if the disk needs to be replaced with a similar sized disk there is some safety margin left, as the number of sectors in harddisks fluctuates a little bit even in the same models from the same manufacturing charge). For my harddisks this means a little bit more than half a gigabyte of wasted storage space.

The commands to partition the disks

In the following I use ada0 as the device of the disk, but it also works with daX or adX or similar. I installed one disk from an existing 9-current system instead of using some kind of installation media (beware, the pool is linked to the system which creates it, I booted a life-USB image to import it on the new system and copied the zpool.cache to /boot/zfs/ after importing on the new system).

Create the GPT:

gpart create -s gpt ada0

Create the boot partition:

gpart add -b 40 -s 1024 -t freebsd-boot ada0

Create the root/swap partitions and name them with a GPT label:

gpart add -b 2048 -s 4G -t freebsd-zfs -l rpool0 ada0

or for the swap

gpart add -b 2048 -s 4G -t freebsd-swap -l swap0 ada0

Create the data partition and name them with a GPT label:

gpart add -s 927G -t freebsd-zfs -l data0 ada0

Install the boot code in partition 1:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

The result looks like this:

# gpart show ada0
=>        34  1953525101  ada0  GPT  (931G)
          34           6        - free -  (3.0k)
          40        1024     1  freebsd-boot  (512k)
        1064         984        - free -  (492k)
        2048     8388608     2  freebsd-zfs  (4.0G)
     8390656  1944059904     3  freebsd-zfs  (927G)
  1952450560     1074575        - free -  (524M)

Create the pools with 4k-ready internal structures

Creating a ZFS pool on one of the ZFS partitions without preparation will not create a 4k-ready pool on a pseudo-4k-drive. I used gnop (the settings do not survive a reboot) to make the partition temporary a 4k-sector partition (only the command for the root pool is shown, for the data partition gnop has to be used in the same way):

gnop create -S 4096 ada0p2
zpool create -O utf8only=on -o failmode=panic rpool ada0p2.nop
zpool export rpool
gnop destroy ada0p2.nop
zpool import rpool

After the pool is created, it will keep the 4k-sectors setting, even when accessed without gnop. You can ignore the options I used to create the pool, they are just my preferences (and the utf8only setting can only be done at pool creation time). If you prepare this on a system which already has a zpool on its own, you can maybe specify “-o cachefile=/boot/zfs/zpool2.cache� and copy it to the new pool as zpool.cache to make it bootable without the need of a life-image for the new system (I did not test this).

Verifying if a pool is pseudo-4k-ready

To verify that the pool will use 4k-sectors, you can have a look at the ashift values of the pool (the ashift is per vdev, so if you e.g. concattenate several mirrors, the ashift needs to be verified for each mirror, and if you concattenate just a bunch of disks, the ashift needs to be verified for all disks). It needs to be 12. To get the ashift value you can use zdb:

zdb rpool | grep ashift

Setting up the root pool

One of the benefits of root-on-zfs is that I can have multiple FreeBSD boot environments (BE). This means that I not only can have several different kernels, but also several different userland versions. To handle them comfortably, I use manageBE from Philipp Wuensche. This requires a specific setup of the root pool:

zfs create rpool/ROOT
zfs create rpool/ROOT/r220832M
zpool set bootfs=rpool/ROOT/r220832M rpool
zfs set freebsd:boot-environment=1 rpool/ROOT/r220832M   # manageBE setting

The r220832M is my initial BE. I use the SVN revision of the source tree which was used during install of this BE as the name of the BE here. You also need to add the following line to /boot/loader.conf:

vfs.root.mountfrom="zfs:rpool/ROOT/r220832M"

As I want to have a shared /var and /tmp for all my BEs, I create them separately:

zfs create -o exec=off -o setuid=off -o mountpoint=/rpool/ROOT/r220832M/var rpool/var
zfs create -o setuid=off -o mountpoint=/rpool/ROOT/r220832M/tmp rpool/tmp

As I did this on the old system, I did not set the mountpoints to /var and /tmp, but this has to be done later.

Now the userland can be installed (e.g. buildworld/installworld/buildkernel/buildkernel/mergemaster with DESTDIR=/rpool/ROOT/r220832M/, do not forget to put a good master.passwd/passwd/group in the root pool).

When the root pool is ready make sure an empty /etc/fstab is inside, and configure the root as follows (only showing what is necessary for root-on-zfs):

loader.conf:
---snip---
vfs.root.mountfrom="zfs:rpool/ROOT/r220832M"
zfs_load="yes"
opensolaris_load="yes"
---snip---
rc.conf
---snip---
zfs_enable="YES"
---snip---

At this point of the setup I unmounted all zfs on rpool, set the mountpoint of rpool/var to /var and of rpool/tmp to /tmp, exported the pool and installed the harddisk in the new system. After booting a life-USB-image, importing the pool, putting the resulting zpool.cache into the pool (rpool/ROOT/r220832M/boot/zfs/), I rebooted into the rpool and attached the other harddisks to the pool (“zpool attach rpool ada0p2 ada1p2�, “zpool attach rpool ada0p2 ada2p2�):

After updating to a more recent version of 9-current, the BE looks like this now:

# ./bin/manageBE list
Poolname: rpool
BE                Active Active Mountpoint           Space
Name              Now    Reboot -                    Used
----              ------ ------ ----------           -----
r221295M          yes    yes    /                    2.66G
cannot open '-': dataset does not exist
r221295M@r221295M no     no     -
r220832M          no     no     /rpool/ROOT/r220832M  561M
Used by BE snapshots: 561M

The little bug above (the error message which is probably caused by the snapshot which shows up here probably because I use listsnapshots=on) is already reported to the author of manageBE.

Share

Converting from Courier IMAP to dovecot is easy

I have used Courier IMAP at home since a long time. As I want to update a dovecot 1.2 setup to dovecot 2.x, I decided to first have a look at dovecot 2.x at home.

Switching from Courier IMAP to dovecot is really easy. I just configured the correct path to the maildir, setup a passdb/userdb, and it was working.

The important part was the correct transfer of the passwords. I used already an userdb in Courier IMAP with MD5 passwords. For each user it has imappw=XXX with XXX similar to $1$abc.

This can be converted into a dovecot passdb/userdb line very easily:

username:{MD5-CRYPT}$1$abc::UID:GID::HOMEDIR::userdb_mail=maildir:~/path/to/maildir

The corresponding passdb/userdb settings for dovecot are:

passdb {
   args = scheme=MD5-CRYPT username_format=%u /usr/local/etc/dovecot/dovecot.pws
   driver = passwd-file
}
userdb {
   args = username_format=%u /usr/local/etc/dovecot/dovecot.pws
   driver = passwd-file
}

Compared to when I had a look the last time, dovecot is also able to use OTP as an authentication mechanism now. Unfortunately I did not find any documentation how to configure/use it.

Share

HOWTO: creating your own updated linux RPM for the FreeBSD linuxulator

Background info

The FreeBSD linux compatibility environment currently uses RPMs from Fedora 10. Unfortunately Fedora 10 is end of life since a while. For one of the RPMs (the pango one) we where aware of a security vulnerability. As we do not know if it is feasible to update the linuxulator ports to something more recent, I decided to setup a VM with Fedora 10 and generate a new RPM for the linux-f10-pango port. Thanks to Luchesar V. ILIEV for explaining me how to do this.

Setup of the VM

I used VirtualBox 4.0.4 on a Solaris 10 x86 machine. I configured a fixed size disk of 16 GB and kept the default network setup (after installing the guest tools / kernel modules I switched to virtio, as I was not able to do anything useful besides a ping) and RAM size. The CD/DVD drive was configured to use the image of the full Fedora 10 DVD for i386 systems.

Setup of Fedora 10

Booting the VM from the DVD leads to the graphical Fedora 10 install software (after chosing to install a new system on the console). There I accepted all the defaults, except for the software to install. I deselected the Office and Productivity group and selected the Software Development group. When I was asked if I want to install some additional RPMs I had a look at the complete list and installed some I thought are necessary. I do not remember anymore which ones I chose, but everything which looks related to RPM building is a good candidate.

After a while the install will be finished and you can boot into the new system (eject the DVD from the drive before reboot). After reboot chose to install the Guest Additions in the menu of the VM. This should mount the ISO image in the VM. As root execute the file for Linux. This will build some kernel modules for better integration (e.g. seamless integration of the mouse between your desktop and the VM). At this point I rebooted and configured virtio as the NIC. I also had to configure the network settings by hand, as the GUI tool did not safe all the settings correctly.

Update and install of required RPMs

After the VM was up and the network configured, I updated the entire system (chose System Update in the menu). To update the pango port, I had to install the libthai-devel RPM. I had the RPM for it (and all the files I need to build a new pango RPM) already downloaded, so I did a “yum install /path/to/rpm�. At this point I was ready to create the RPM build environment.

The RPM build environment

As a normal user I executed the command rpmdev-setuptree which creates the directory rpmbuild and populates it with some directories. Now you just need to find a suitable .spec file and put it into rpmbuild/SPECS, put the sources (and maybe patches referenced in the .spec file) into rpmbuild/SOURCES, and you are ready to go (I patched pango.spec for a more recent pango version, basically just changing the version numbers). If you want to have a custom packager and vendor attribute in the RPM, you can add a line for each to ~/.rpmmacros, e.g. %packager yournamehere and %vendor whateverisappropriate. I used my @FreeBSD.org EMail address as the packager, and FreeBSD as the vendor.

Building a RPM

I used rpmbuild –ba –target i386-redhat-linux-gnu –clean rpmbuild/SPECS/pango.spec to build the new pango RPM. If everything is OK, the resulting RPMs (a source RPM, a devel RPM, a debuginfo RPM and the RPM for the binaries) are in rpmbuild/RPMS and rpmbuild/SRPMS. For a FreeBSD port we just need the source RPM (to comply to the (L)GPL) and the RPM for the binaries.

Additional info

The i386-redhat-linux-gnu string which is used for the –target option of the rpmbuild command is what seems to be used to build the Fedora 10 RPMs. After building pango, the RPM has i686-pc-linux-gnu in some filenames instead (the default value for this setup). The binaries seem to be compiled for i386, so there should be no problem even for old systems.

Share

Cheap process monitoring (no additional software required)

I have an old system (only the hardware, it runs –current) which reboots itself from time to time (mostly during the daily periodic(8) run, but also during a lot of compiling (portupgrade)). There is no obvious reason (no panic) why it is doing this. It could be that there is some hardware defect, or something else. It is not important enough to get a high enough priority that I try hard to analyze the problem with this machine. The annoying part is, that sometimes after a restart apache does not start. So if this happens, the solution is to login and start the webserver. If the webserver would start each time, nearly nobody would detect the reboot (root gets an EMail on each reboot via an @reboot crontab entry).

My pragmatic solution (for services started via a good rc.d script which has a working status command) is a crontab entry which checks periodically if it is running and which restarts the service if not. As an example for apache and an interval of 10 minutes:

*/10 * * * *    /usr/local/etc/rc.d/apache22 status >/dev/null 2>&1 || /usr/local/etc/rc.d/apache22 restart

For the use case of this service/machine, this is enough. In case of a problem with the service, a mail with the restart output would arrive each time it runs, else only after a reboot for which the service did not restart.

Share/Bookmark

Making ZFS faster…

Currently I play a little bit around with my ZFS setup. I want to make it faster, but I do not want to spend a lot of money.

The disks are connected to an ICH 5 controller, so an obvious improvement would be to either buy a controller for the PCI slot which is able to do NCQ with the SATA disks (a siis(4) based one is not cheap), or to buy a new system which comes with a chipset which knows how to do NCQ (this would mean new RAM, new CPU, new MB and maybe even a new PSU). A new controller is a little bit expensive for the old system which I want to tune. A new system would be nice, and reading about the specs of new systems lets me want to get a Core i5 system. The problem is that I think the current offers of mainboards for this are far from good. The system should be a little bit future proof, as I would like to use it for about 5 years or more (the current system is somewhere between 5–6 years old). This means it should have SATA-3 and USB 3, but when I look at what is offered currently it looks like there are only beta-versions of hardware with SATA-3 and USB 3 support available on the marked (according to tests there is a lot of variance of the max speed the controllers are able to achieve, bugs in the BIOS, or the  controllers are attached to a slow bus which prevents to use the full bandwidth). So it will not be a new system soon.

As I had a 1GB USB-stick around, I decided to attach it to the one of the EHCI USB ports and use it as a cache device for ZFS. If someone wants to try this too, be careful with the USB ports. My mainboard has only 2 USB ports connected to an EHCI, the rest are UHCI ones. This means that only 2 USB ports are fast (sort of… 40 MBit/s), the rest is only usable for slow things like a mouse, keyboard or a serial line.

Be warned, this will not give you a lot of bandwidth (if you have a fast USB stick, the 40MBit/s of the EHCI are the limit which prevent a big streaming bandwidth), but the latency of the cache device is great when doing small random IO. When I do a gstat and have a look how long a read operation takes for each involved device, I see something between 3 msec and 20 msec for the harddisks (depending if they are reading something at the current head position, or if the harddisk needs to seek around a lot). For the cache device (the USB stick) I see something between around 1 mssec and 5 msec. That is 1/3th to 1/4th of the latency of the harddisks.

With a “zfs send� I see about 300 IOops per harddisk (3 disks in a RAIDZ). Obviously this is an optimum streaming case where the disks do not need to seek around a lot. You see this in the low latency, it is about 2 msec in this case. In the random-read case, like for example when you run a find, the disks can not keep this amount of IOops, as they need to seek around. And here the USB-stick shines. I’ve seen upto 1600 IOops on it during running a find (if the corresponding data is in the cache, off course). This was with something between 0.5 and 0.8 msec of latency.

This is the machine at home which is taking care about my mails (incoming and outgoing SMTP, IMAP and Webmail), has a squid proxy and acts as a file server. There are not many users (just me and my wife) and there is no regular usage pattern for all those services. Because of this I did not do any benchmark to see how much time I can gain with various workloads (and I am not interested in some artificial performance numbers of my webmail session, as the browsing experience is highly subjective in this case). For this system a 1 GB USB stick (which was just collecting dust before) seems to be a cheap way to improve the response time for often used small data. When I use the webmail interface now, my subjective impression is, that it is faster. I am talking about listing emails (subject, date, sender, size) and displaying the content of some emails. FYI, my maildir storage has 849 MB with 35000 files in 91 folders.

Bottom line is: do not expect a lot of bandwidth increase with this, but if you have a workload which generates random read requests and you want to decrease the read latency, it could be a cheap solution to add a (big) USB stick as a cache device.

Improving the order of directories to backup in tarsnap

I experimented a little bit with the order of directories to backup in tarsnap.

Currently I use the following sorting algorithm:

  1. least frequently changed directory-trees first
    Every change — even in meta-data — will affect the following data, as tarsnap is doing the de-duplication in fixed-width blocks (AFAIR 64k).
  2. for those directory-trees which change with the same frequency: list the bigger ones first
    Implicitly I assume that the smaller ones are much smaller than the bigger ones so that the smaller part which will be backed up will not be noticed because of the bigger change. For my use cases of tarsnap this is true.
  3. if changes in a directory-tree are much much bigger than anything else, but the directory-tree has a medium change-frequency, put it even before less-frequently changing stuff
    I do not want that a small change triggers a big backup, but a big backup can contain the remaining small part.
  4. if you backup home directories (even root’s one) and they do not contain much data, put them before directory-trees which change a lot daily
    I do not want that a login triggers the transfer of data in other directory-trees which have not changed.

HOWTO: “Blind” remote install of FreeBSD via tiny disk image

In this post I write how to install FreeBSD on a remote linux system by creating a root image prepared for mirroring on a local system. There is no (free) access to the remote console, only access via ssh.

Background story

While I was at the university, I worked remotely for an ISP (administration of several FreeBSD systems). During this time I started to use my own domain, and I was allowed to host it directly at the ISP for free. I tried to not use too much space on the harddisk (at the end about 400 MB) and to not provide something which attracted too much people to keep the bandwith on a sane level. After the university I was still available to answer questions, and I was still allowed to host my website there for free. As the number of questions can be counted with IIRC one hand since then, I decided at some point (recently, to be exact — better late than never) to move my website to a different place (I am still available if they have some questions — for free — but I do not expect to get much questions from them).

At the same time my brother decided to move to a new server hoster, as his old one decided to increase the amount of money customers have to pay for the service. So we searched together a new ISP where he either could host his server, or get a rootserver for a good price (the idea was to have my domain in a jail on a server of the company of my brother, and we share the costs for it). We found Manitu (the owner has even a blog about his business), which is even not far away from my place.

Unfortunately they do not provide FreeBSD preinstalled on their rootservers, but they offer a remote rescue system (read: booting a linux image… I assume via PXE or similar) and we knew someone who has some servers there, so I was able to get a rough idea what kind of hardware is used there (the hard facts like PCI IDs and such). The idea was to build a very small disk image, put it on the harddisk over the network via the remote rescue system, and then to configure the remainder of the harddisk(s) to use it. And here is how I did it (my brother thought “who is better suited to install a FreeBSD system remotely without access to the console of the machine (the ISP offers to hook up a KVM switch, but only during business hours and you have to pay for it) than one of the developers of FreeBSD…”).

HOWTO

In the title of this post I wrote “via a tiny disk image”. This is true for a suitable definition of tiny.

What we have in the rootserver are two 160 GB harddisks. They shall be used in a software mirror (via gmirror). The root-FS shall have about 5 GB. This is more than needed, but as this is only 3% of one harddisk, I prefer to keep it a little bit bigger than too small after an remote update or two. The machine has 2 GB of RAM. We do not expect much kernel panics (= crash dumps) there, so we do not really need >2 GB of swap (forget the rule of having twice as much swap than RAM, with the current amount of RAM in a machine you are in “trouble” when you need even the same amount of swap than RAM). I decided to go with 1 GB of swap (mirrored too, to prevent a harddisk failure to take down the machine), this is more than enough. The rest of the harddisk will be used for jails, the distfiles/packages for/of the ports, and as WRKDIRPREFIX when building ports.

Now, pushing/pulling a 160 GB image over the network to install a system is not really something I want to do. I would prefer to transfer less than 500 MB (that is 0.3% of the entire disk) to get this job done, and this is feasible. Due to an error or two I had to transfer the image several times until everything was working, so it was more in the area of maybe 2 GB (~1% of the entire disk).

First let us define some variables in the shell, this way you just need to change the values in one place and the rest is copy&paste:

ROOTFS_SIZE=5G
ROOTFS_NAME=root0
FILENAME=rootfs

Then change your current directory to a place where you have enough space for the image. There we will create a container for the image, and make it ready for partitioning:

truncate -s ${ROOTFS_SIZE} ${FILENAME}
mdconfig -a -t vnode -f ${FILENAME}

Create the rootfs:

# create one active FreeBSD slice with $ROOTFS_SIZE GB
fdisk -e /dev/mdX
gmirror label ${ROOTFS_NAME} /dev/mdXsY
bsdlabel -w /dev/mirror/${ROOTFS_NAME}
# create an "a"-partition for everything
bsdlabel -e /dev/mirror/${ROOTFS_NAME}
newfs -U /dev/mirror/${ROOTFS_NAME}a

Mount the new rootfs to /mnt and install FreeBSD:

mount /dev/mirror/${ROOTFS_NAME}a /mnt
cd /usr/src
make buildworld >&! buildworld.log
make buildkernel >&! build_generic.log
make installworld DESTDIR=/mnt
make distribution DESTDIR=/mnt
make installkernel DESTDIR=/mnt

Now you need to create /mnt/etc/rc.conf (set the defaultrouter, the IP address via ifconfig_IF (and do not forget to use the right IF for it), the hostname, set sshd_enable to yes, add an user (I used “vipw –d /mnt/etc”) which is in the wheel group so you can login remotely, and maybe other things you are interested in), /mnt/etc/resolv.conf, /mnt/etc/hosts. Finally, do not forget to load the gmirror module, it will safe a lot of head-scratching (yes, an echo would be shorter, but WP converts the single-quotes to double-quotes), and add the rootfs to the fstab:

cat > /mnt/boot/loader.conf <<EOT
geom_mirror_load="YES"
EOT
echo "/dev/mirror/root0a / ufs rw,noatime 1 1" >/mnt/etc/fstab

Now we are ready to install, the image can be unmounted:

umount /mnt

The final steps are to login into the rescue console (theoretically you should be able to overwrite even a running system, but then you need to make sure there is a way to power-cycle the system remotely to force a reboot) of the new system and to install via a compressed ssh connection (my remote rescue system is a linux system, so linux-syntax has to be used). The login to the rescue console is not shown here, but the install from the remote system is simple (this assumes the image resides on a system which is accessible from the new system):

ssh -C -o CompressionLevel=9 user@myhost cat /path/to/${FILENAME} | dd of=/dev/hda bs=1m

Alternatively you can compress it with bzip2 on your system and add an bunzip2 into the pipe above. This way you could even use an HTTP server to fetch the file from, but then the command to fetch the file needs to have a way to output the file it downloads on stdout. Both ways of transferring the image depend upon the stability of the transfer. If the connection is cut, the system can not boot anymore. So if you do not have a rescue console which is independent from the content of the harddisk, you better have a plan B.

If I did not make an error here (I did this some months ago, I hope I did not forget to write down some important step and also corrected all steps which where not written down correctly), you did everything correctly too, and the remote system does not need other kernel modules loaded, you can now reboot the new system, cross your fingers for some moments, and then login to the new system.

Post install TODO

Now you should have a basic system up and running, and you can start to configure it.

I added two more FreeBSD partitions, 1 GB for swap, and the rest of the harddisk (I took care to not have the last partition cover also the last sector of the harddisk, else gmirror will think it has to mirror the entire harddisk and not only the last partition) as one big partition. For the second harddisk I made the same partitioning as for the first harddisk.

Then I created two more gmirrors, one mirror for the swap, and one for the rest of the space. The mirror for the swap I created with the option “-F”, to not synchronize the partition after a power failure (not necessary for swap). All two I also created with the “-n” option, to not sync the contents (there is nothing yet, so it does not matter what is written there).

Now just a quick “bsdlabel” on the two mirrors to create a “b”-partition for the swap and a “d”-partition for the large partition. To add the swap it is just an “echo /dev/mirror/swap0b none swap sw 0 0 >/etc/fstab”. Do not forget to add the big partition to the fstab (after doing a “newfs” off course).

To be sure it will survive a reboot, do a quick test of manually mounting the big partition (use the easy manual way via “mount /place/where/it/is/mounted” after adding it to the fstab, not the complete manual way, this is meant to test the fstab entry too), and a “swapon –a” to test the swap. I also made a reboot test, just to be sure everything is ok.

The final step was to add the partition on the second harddisk to the rootfs (“gmirror insert ${ROOTFS_NAME} /dev/…”).

Share/Bookmark

Easy library dependencies detection for ports

In the last days I committed some scripts to $PORTSDIR/Tools/scripts which help in detecting the explicit library dependencies of installed ports. You just have to run $PORTSDIR/Tools/scripts/explicit_lib_depends.sh with the package-name of the installed port (alternatively you can give the path to the registered port, e.g. /var/db/pkg/gnome-terminal-2.18.1). One of the scripts which are called needs portupgrade installed. As an example here’s the complete output of a script run with the gnome-terminal port:

# /usr/ports/Tools/scripts/explicit_lib_depends.sh gnome-terminal-2.18.1
USE_FREETYPE=yes
USE_GETTEXT=yes
USE_GNOME+=atk
USE_GNOME+=esound
USE_GNOME+=gconf2
USE_GNOME+=glib20
USE_GNOME+=gnomevfs2
USE_GNOME+=gtk20
USE_GNOME+=libartlgpl2
USE_GNOME+=libbonobo
USE_GNOME+=libbonoboui
USE_GNOME+=libglade2
USE_GNOME+=libgnome
USE_GNOME+=libgnomecanvas
USE_GNOME+=libgnomeui
USE_GNOME+=libxml2
USE_GNOME+=orbit2
USE_GNOME+=pango
USE_GNOME+=vte
USE_ICONV=yes
USE_XORG+=ice
USE_XORG+=sm
USE_XORG+=x11
USE_XORG+=xau
USE_XORG+=xcursor
USE_XORG+=xdmcp
USE_XORG+=xext
USE_XORG+=xfixes
USE_XORG+=xft
USE_XORG+=xi
USE_XORG+=xinerama
USE_XORG+=xrandr
USE_XORG+=xrender
audiofile:${PORTSDIR}/audio/libaudiofile
avahi-client:${PORTSDIR}/net/avahi
avahi-common:${PORTSDIR}/net/avahi
avahi-glib:${PORTSDIR}/net/avahi
cairo:${PORTSDIR}/graphics/cairo
dbus-1:${PORTSDIR}/devel/dbus
dbus-glib-1:${PORTSDIR}/devel/dbus-glib
expat:${PORTSDIR}/textproc/expat2
fontconfig:${PORTSDIR}/x11-fonts/fontconfig
gnome-keyring:${PORTSDIR}/security/gnome-keyring
jpeg:${PORTSDIR}/graphics/jpeg
png:${PORTSDIR}/graphics/png
popt:${PORTSDIR}/devel/popt
startup-notification-1:${PORTSDIR}/x11/startup-notification

All those libraries are directly referenced (dynamically linked in) in the binaries or libs of the gnome-terminal port, indirect (dependencies of dependencies) ones are not listed (and not needed). So the script allows to quickly produce a list of libs/ports which

  • should be referenced in the port Makefile (new port development, port updates)
  • allows to check if a port lists all referenced LIB_DEPENDS (port maintenance)

To do this with all your ports you can run

for port in /var/db/pkg/*; do
echo $port:
/usr/ports/Tools/scripts/explicit_lib_depends.sh $port
done > explicit_depends.txt

Not all USE_* switches of the Ports Collection are handled yet, if you stumble upon such a case, feel free to send me a mail.

Share/Bookmark