Nore’s FreeBSD Blog

Hi, it’s nearing the end of GSoC 2008 and I’ve updated my wiki ( http://wiki.freebsd.org/AndersNore/pkg_improved ) about the pkg_improved project and there are testing instructions for those brave enough. Please test and please report bugs. There are several new features to the pkg-tools and there are some speed improvements made. Lately I’ve been cleaning some of the code mostly fixing bugs and writing documentation in the form of man-pages.

Now the new date feature is doing something useful, I’ve added an option (-M) to pkg_info which compares installtimes of packages similar to the comparison of versions. So now you can do “pkg_info -M ‘*>2008 07 18′ ” and it will display all pacakges installed after the date YYYY MM DD, you can also use the -x or -X option with the -M option. I’m also planning to make use of this in pkg_delete so you can delete packages within date-ranges. Also pkg_convert will have an option so you can make the installtime of packages (who do not have a @comment DATE field) equal e.g., to the changetime of +DESC in /var/db/pkg/pkgname/ (which is correct for most packages). I’ve also decided to cache the installtime which makes this operation a lot faster than calling read_plist for every package to find the installtime.

I’ve also done some improvements to the ‘pkg_add -r’ progress indication output, which now displays the speed of the download as well as percentage progress.

Hello, it’s been a while since I’ve updated my blog and it was about time. I’ve come back from my vacation this week and I have added two new features to pkg_improved which I think are nice features to have.

The first feature is that pkg_add registers the installation time (seconds-since-epoch) in +CONTENTS as a “@comment DATE: 123124124″, pkg_info can then see the installation dates of installed ports with the new -n option (for viewing the date in human readable form) or with the -N option (prints out seconds-since-epoch). The reason why I added this is because I’ve seen people ask how they can get the installation time of packages, and in my opinion I don’t think that the filesystem should be the only holder of such information (e.g. fetching modification time of files contained in /var/db/pkg/*).

The second feature is dependency fixing (I’ve seen that portmaster does this), and what it does is to reconstruct the +REQUIRED_BY file for packages being installed with pkg_add or when pkg_create -O option is called from ports. In detail it checks every installed package for a @pkgdep that matches the installing package it add’s it to the installing package +REQUIRED_BY file. This slows down pkg_add somewhat but I think this should be the correct behaviour for pkg_add.

I’ve also added a percentage progress status to pkg_add -r (remote fetching packages). You can see instructions for testing the improved tools on my blog: http://wiki.freebsd.org/AndersNore/pkg_improved

Today I’m going on a vacation to Greece for one week. I’ll be back next Thursday (10th). I’ve polished my project pkg_improved so I think that people could test it out to see if it works. There’s not a whole lot of change, just some speed gains here and there, you can check the CHANGES file posted for more details. You could also make a diff with /usr/src/usr.sbin/pkg_install/ but you will get a whole lot of syntax changes =(

If you want to test it, then just compile and install (NB! this will replace your normal pkg_* tools). You could always get back your original pkg_* tools by compiling and installing from /usr/src/usr.sbin/pkg_install . If you don’t want to replace your pkg_* tools you should just compile it and rename the executables (NB! You will not get proper behaviour from ports unless you change some PKG_* variables in /usr/ports/Mk/bsd.port.mk (see PKG_CMD). The first thing you should do after install is to run pkg_convert to cache the existing flat database, after that you don’t have to run it again unless you use the original pkg_* programs.

There’s some ideas that I’ve been looking into like adding date’s to installed packages (perhaps a @comment in +CONTENTS), recovering +REQUIRED_BY in pkg_add (after e.g. pkg_delete -f zip; pkg_add zip). These should probably be posted to a mailing list, but I’ll wait until I get home from Greece to post them.

pkg_improved: http://home.no.net/andenore/pkg_improved.tar.gz

(please e-mail me if you find bug’s and the like)

This has been somewhat of a amputated week, last weekend I was away for a bicycle race and this weekend I’m going to a wedding, so it hasn’t been a lot of time for programming. Anyways I’ve added human-readable output for pkg_info’s -s option (the old is also available with the -b option). The environment variable PKG_DBCACHE_FILE is used to specify where the cache-file is located. And I’ve added features for the pkg_convert program that checks if key’s exists within the database and the printing of the data indexed in the database. The pkg_create -O option has been modified to cache information as well and installing ports from /usr/ports/ will then work if you specify the PKG_CMD in /usr/ports/Mk/bsd.port.mk to point to this pkg_create.

I’m hoping to release a test-version next week so that I can receive comments and bugs which I’m sure there is. On Thursday next week I’m going on a one week vacation to Greece so that you will have a good time to test the tools and fill my mailbox

Changes

These are the main things I’ve done so far some abstract and some more specific, if you want to check out the code you can get it from the perforce depot: //depot/projects/soc2008/andenore_pkginstall/

pkg_convert:

• Added a convert program (pkg_convert) that indexes information to PKG_DBCACHE_FILE (defaults to $PKG_DBDIR/pkgcache.db)

pkg_info:

• Improved performance for -W option using the dbcache
• A quick profiling of pkg_info revealed that the default behavior for pkg_info was to call read_plist for each package even though not needed so I’ve added a check for this and it runs significantly faster.

pkg_add:

• Indexes information to dbcache according to the add

pkg_delete:

• “Deindexes” information according to the delete

lib:

• Changed methods in match.c to use cache if available
• Added global variable ‘database’ of type DB* (the database object)
• Added database helper functions

I’ve done some quick profiling of pkg_version and it spends much of it’s time in read_plist too (just to fetch the origin). So I plan to index the origin of each package and speed this up in pkg_version.

Now I’m finally done with my exams for the semester and travelled home where I will spend the summer.

This week I started to work on my project which is adding .db support to pkg_install. I have written some database helper functions and a simple converter program that indexes portnames in /var/db/pkg to the database file which is going to act as a cache rather than a replacement for the whole flat-database. I have rewritten some functions in pkg_info that uses the cache and it works well and seemingly faster although I have not taken time measurements (writing a time comparison script is on my todo-list).

Next week I will continue to improve the converter to index more information that is needed and gradually rewrite the pkg-tools to make use of the database.