BSDCan 2014 – Ports and Packages WG

July 18, 2014 by · Leave a Comment 

Baptiste Daroussin started the session with a status update on package building. All packages are now built with poudriere. The FreeBSD Foundation sponsored some large machines on which it takes around 16 hours to build a full tree. Each Wednesday at 01:00UTC the tree is snapshot and an incremental build is started for all supported released, the 2 stable branches (9 and 10) and quarterly branches for 9.x-RELEASE and 10.x-RELEASE. The catalogue is signed on a dedicated signing machine before upload. Packages can be downloaded from 4 mirrors (us-west, us-east, UK, and Russia) and feedback so far has been very positive.

He went on to note that ports people need better coordination with src people on ABI breakage. We currently only support i386 and amd64, with future plans for ARM and a MIPS variant. Distfiles are not currently mirrored (since fixed), and while it has seen no progress, it’s still a good idea to build a pkg of the ports tree itself.

pkg 1.3 will include a new solver, which will help 'pkg upgrade' understand that an old packages needs to be replaced with a newer one, with no more need for 'pkg set' and other chicanery. Cross building ports has been added to the ports tree, but is waiting for pkg-1.3. All the dangerous operations in pkg have now been sandboxed as well.

EOL for pkg_tools has been set for September 1st. An errata notice has gone out that adds a default pkg.conf and keys to all supported branches, and nagging delays have been added to ports.

Quarterly branches based on 3 month support cycle has been started on an evaluation basis. We’re still unsure about the manpower needed to maintain those. Every quarter a snapshot of the tree is created and only security fixed, build and runtime fixed, and upgrades to pkg are allowed to be committed to it. Using the MFH tag in a commit message will automatically send an approval request to portmgr and an mfh script on Tools/ makes it easy to do the merge.

Experience so far has been good, some minor issues to the insufficient testing. MFHs should only contain the above mentioned fixes; cleanups and other improvements should be done in separate commits only to HEAD. A policy needs to be written and announced about this. Do we want to automatically merge VuXML commits, or just remove VuXML from the branch and only use the one in HEAD?

A large number of new infrastructure changes have been introduces over the past few months, some of which require a huge migration of all ports. To speed these changes up, a new policy was set to allow some specific fixes to be committed without maintainer approval. Experience so far has been good, things actually are being fixed faster than before and not many maintainers have complained. There was agreement that the list of fixes allowed to be committed without explicit approval should be a specific whitelist published by portmgr, and not made too broad in scope.

Erwin Lansing quickly measured the temperature of the room on changing the default protocol for fetching distils from MASTER_SITE_BACKUP from ftp to http. Agreement all around and erwin committed the change.

Ben Kaduk gave an introduction and update on MIT’s Athena Environment with some food for thought. While currently not FreeBSD based, he would like to see it become so. Based on debian/ubuntu and rolled out on hundreds of machines, it now has it’s software split into about 150 different packages and metapackages.

Dag-Erling Smørgrav discussed changes to how dependencies are handled, especially splitting dependencies that are needed at install time (or staging time) and those needed at run time. This may break several things, but pkg-1.3 will come with better dependency tracking solving part of the problem.

Ed Maste presented the idea of “package transparency”, loosely based on Google’s Certificate Transparency. By logging certificate issuance to a log server, which can be publicly checked, domain owners can search for certificates issued for their domains, and notice when a certificate is issued without their authority. Can this model be extended to packages? Mostly useful for individually signed packages, while we currently only sign the catalogue. Can we do this with the current infrastructure?

Stacy Son gave an update on Qemu user mode, which is now working with Qemu 2.0.0. Both static and dynamic binaries are supported, though only a handful of system call are supported.

Baptiste introduced the idea of having pre-/post-install scripts be a library of services, like Casper, for common actions. This reduces the ability of maintainers to perform arbitrary actions and can be sandboxed easily. This would be a huge security improvement and could also enhance performance.

Cross building is coming along quite well and most of the tree should be able to be build by a simple 'make package'. Major blockers include perl and python.

Bryan Drewery talked about a design for a PortsCI system. The idea is that committer easily can schedule a build, be it an exp-run, reference, QAT, or other, either via a web interface or something similar to a pull request, which can fire off a build.

Steve Wills talked about using Jenkins for ports. The current system polls SVN for commits and batches several changes together for a build. It uses 8 bhyve VMs instances, but is slow. Sean Bruno commented that there are several package building clusters right now, can they be unified? Also how much hardware would be needed to speed up Jenkins? We could duse Jenkins as a fronted for the system Bryan just talked about. Also, it should be able to integrate with phabricator.

Erwin opened up the floor to talk about freebsd-version(1) once more. It was introduced as a mechanism to find out the version of user land currently running as uname -r only represents the kernel version, and would thus miss updates of the base system that do no touch the kernel. Unfortunately, freebsd-version(1) cannot really be used like this in all cases, it may work for freebsd-update, but not in general. No real solution was found this time either.

The session ended with a discussion about packaging the base system. It’s a target for FreeBSD 11, but lots of questions are still to be answered. What granularity to use? What should be packages into how many packages? How to handle options? Where do we put the metadata for this? How do upgrades work? How to replace shared libraries in multiuser mode? This part also included the quote of the day: “Our buildsystem is not a paragon of configurability, but a bunch of hacks that annoyed people the most.”

Thanks to all who participated in the working group, and thanks again to DK Hostmaster for sponsoring my trip to BSDCan this year, and see you at the Ports and Packages WG meet up at EuroBSDCon in Sofia in September.

Monthly dashboard

July 13, 2014 by · Leave a Comment 

Thursday morning, 5 AM, I couldn’t sleep. I thought I could use the time before
work to do something useful, so I started handling a few PRs for FreeBSD. After
a couple of commits, a warm shower, and just before heading to work, I quickly
browsed through my irc backlog and suddently got very sad: someone was angrily
asking why bug reports were being ignored for such a long time, pushing for his
own PR to finally be given some consideration.

Thinking about it in the bus to work I realized that this guy was right to
complain: when a bug is reported it should ideally be fixed right away. Still I
was feeling sad because being on the other side of the fence I know how much
dedication volunteers put into FreeBSD, but I was not sure everybody was aware
of this. I had to find something to express this dedication.

That’s how the idea of the monthly dashboard came: simple figures that can tell
a whole story. See for yourself with this dashboard that can be found in portmgr
monthly report for June 2014:


Monthly dashboard

Number of messages to portmgr@: 564 (+53%)
Number of commits on ports: 3,717 (+17%)
Number of ports PRs closed: 873 (+25%)
Active ports committers: 147 (+10%)


 

Isn’t it amazing? Nearly 4,000 updates on the ports tree and nearly 900 problem
reports closed in a single month!

That’s a tremendous amount of work done by our committers. Take Linux for
example: with more than twice as much contributors during the same period four
times less commits were applied to the Linux kernel than to the FreeBSD ports
tree [1].

Those figures pay tribute to our committers, and I am pleased to see that the
activity keeps growing. I personnally believe there has never been a better time
to start contributing to the FreeBSD ports tree with all those new features
currently being introduced. So come and join the party!


[1] Statistics taken from http://www.ohloh.net/p/linux:
962 commits done by 344 contributors (activity recorded from Jun 9 2014 to Jul 9 2014)

Getting to know your portmgr-lurker: William Grzybowski

July 5, 2014 by · Leave a Comment 

From July to the end of October two new lurkers will have the opportunity to get insights into FreeBSD portmgr internals, namely William Grzybowski and Nicola Vitale. William was the first to answer our interview so let’s get to know him a bit better.

 

Name

William Grzybowski

Committer name

wg

Inspiration for your IRC nick

my name initials

 TLD of origin

.br

 Current TLD (if different from above)

Occupation

Software engineer

 Blog

None

Inspiration for using FreeBSD

Stability and simplicity

Who was your first contact in FreeBSD

I don’t recall, too long ago!

Who was your mentor(s)

culot, jpaetzel

What was your most embarrassing moment in FreeBSD

Breaking INDEX, but hey, who ever didn’t? ;)

vi(m) /  emacs / other

vim

What keeps you motivated in FreeBSD

The passion of everyone in the zoo about it. You can very easily see a
guy angry by someone’s else commit like stealing candy from his son :)

Favorite musician/band

AC/DC

What book do you have on your bedside table

book? what is that? :)

coffee / tea / other

coffee, beer (but real beer, no corn!)

How would you describe yourself

Calm and mind-centered

sendmail / postfix / other

postfix

Do you have a hobby outside of FreeBSD

Play soccer, gym

What is your favorite TV show

Fringe

Claim to Fame

I was mentored by culot and I survived! Just kidding, I have none, I am a joke!

What did you have for breakfast today

An orange and a slice of yogurt cake

What sports team do you support

Gremio Foot-Ball Porto Alegrense, brazillian soccer team

What else do you do in the world of FreeBSD

I do work mostly in python ports and help closing PRs.

2014Q3 Branched

July 1, 2014 by · Leave a Comment 

The 2014Q3 branch has just been branched and the package builder has been
updated to use that branch. This means that the next update on the quarterly
packages will be on the 2014Q3 branch.

What happened during the last 3 months:
- 177 different committers have participated
- 9918 commits happened
- diffstat says: 23646 files changed, 554070 insertions(+), 577210 deletions(-)

What does that means for users:
- default Java is now 1.7
- massive conversion to stagedir (93% of the ports are now properly staged)
- massive improvement of the usage of libtool (which reduces a lot overlinking)
- new USES: mono, objc, drupal, gecko, cpe, gssapi, makeinfo
- new Keywords for plist: @sample, @shell
- LibreOffice has been updated to 4.2.5
- Firefox has been updated to 30.0
- Firefox-esr has been updated to 24.6
- Default postgresql has moved from 9.0 to 9.2
- nginx has been updated to 1.6.0
- Default lua is 5.2
- subversion has been split into multiple ports for each features
- On FreeBSD 9-STABLE and 10-STABLE the default xorg 1.12.4 (for default binary
packages it is still 1.7.7)
- Improved QA checking in the infrastructure
- Info files are handle correctly even if base has been built WITHOUT_INFO
- Ancient emacs version has been cleaned out