pkg(8) is now the only package management tool

The ports tree has been modified to only support pkg(8) as package management system for all supported version of FreeBSD.

if you were still using pkg_install (pkg_* tools) you will have to upgrade your system.

The simplest way is

cd /usr/ports/ports-mgmt/pkg
make install

then run

pkg2ng

You will have lots of warning, don’t be scared, they are expected, pkg_*  databases used to get easily mangled. pkg2ng is most of the time able to deal
with it.

If however you encounter a problem then please report to [email protected]

A tag has been applied to the ports tree if you need to get the latest ports tree before the EOL of pkg_install:

https://svn.FreeBSD.org/ports/tags/PKG_INSTALL_EOL

A branch has been created if some committers want to provides updates on the for pkg_install users:

https://svn.FreeBSD.org/ports/branches/pkg_install

Please note that this branch is not officially maintained and that we strongly recommend that you do migrate to pkg(8)

The ports tree is now stage only

The ports tree is now fully staged (only 2% has been left unstaged, marked as broken and will be removed from the ports tree if no PR to stage them are pending in bugzilla).

I would like to thank every committer and maintainers for their work on staging!
It allowed us to convert more than 23k packages to support stage in only 11 months!

Staging is a very important state, it allows us to right now be able to run quality testing scripts on the packages (which already allowed to fix tons of hidden problems) and it allows use to be able to build packages as a regular user!

It also opens the gates to new features that users have been requesting for many years:

  • flavors
  • multiple packages

Expect those features to happen in the near future.

helping out with VC4

I've had a couple of questions about whether there's a way for others to contribute to the VC4 driver project.  There is!  I haven't posted about it before because things aren't as ready as I'd like for others to do development (it has a tendency to lock up, and the X implementation isn't really ready yet so you don't get to see your results), but that shouldn't actually stop anyone.

To get your environment set up, build the kernel (https://github.com/anholt/linux.git vc4 branch), Mesa (git://anongit.freedesktop.org/mesa/mesa) with --with-gallium-drivers=vc4, and piglit (git://anongit.freedesktop.org/git/piglit).  For working on the Pi, I highly recommend having a serial cable and doing NFS root so that you don't have to write things to slow, unreliable SD cards.

You can run an existing piglit test that should work, to check your environment: env PIGLIT_PLATFORM=gbm VC4_DEBUG=qir ./bin/shader_runner tests/shaders/glsl-algebraic-add-add-1.shader_test -auto -fbo -- you should see a dump of the IR for this shader, and a pass report.  The kernel will make some noise about how it's rendered a frame.

Now the actual work:  I've left some of the TGSI opcodes unfinished (SCS, DST, DPH, and XPD, for example), so the driver just aborts when a shader tries to use them.  How they work is described in src/gallium/docs/source/tgsi.rst. The TGSI-to_QIR code is in vc4_program.c (where you'll find all the opcodes that are implemented currently), and vc4_qir.h has all the opcodes that are available to you and helpers for generating them.  Once it's in QIR (which I think should have all the opcodes you need for this work), vc4_qpu_emit.c will turn the QIR into actual QPU code like you find described in the chip specs.

You can dump the shaders being generated by the driver using VC4_DEBUG=tgsi,qir,qpu in the environment (that gets you 3/4 stages of code dumped -- at times you might want some subset of that just to quiet things down).

Since we've still got a lot of GPU hangs, and I don't have reset wokring, you can't even complete a piglit run to find all the problems or to test your changes to see if your changes are good.  What I can offer currently is that you could run PIGLIT_PLATFORM=gbm VC4_DEBUG=norast ./piglit-run.py tests/quick.py results/vc4-norast; piglit-summary-html.py --overwrite summary/mysum results/vc4-norast will get you a list of all the tests (which mostly failed, since we didn't render anything), some of which will have assertion failed.  Now that you have which tests were assertion failing from the opcode you worked on, you can run them manually, like PIGLIT_PLATFORM=gbm /home/anholt/src/piglit/bin/shader_runner /home/anholt/src/piglit/generated_tests/spec/glsl-1.10/execution/built-in-functions/vs-asin-vec4.shader_test -auto (copy-and-pasted from the results) or PIGLIT_PLATFORM=gbm PIGLIT_TEST="XPD test 2 (same src and dst arg)" ./bin/glean -o -v -v -v -t +vertProg1 --quick (also copy and pasted from the results, but note that you need the other env var for glean to pick out the subtest to run).

Other things you might want eventually: I do my development using cross-builds instead of on the Pi, install to a prefix in my homedir, then rsync that into my NFS root and use LD_LIBRARY_PATH/LIBGL_DRIVERS_PATH on the Pi to point my tests at the driver in the homedir prefix.  Cross-builds were a *huge* pain to set up (debian's multiarch doesn't ship the .so symlink with the libary, and the -dev packages that do install them don't install simultaneously for multiple arches), but it's worth it in the end.  If you look into cross-build, what I'm using is rpi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc and you'll want --enable-malloc0returnsnull if you cross-build a bunch of X-related packages.

FreeBSD Foundation announces IPsec Enhancement Project

The Internet Protocol Security (IPsec) suite is used to implement virtual private networks on FreeBSD and other operating systems. As the networking world continues its transition from 1 to 10, to 40 gigabit per second speeds, and faster, improvements in IPsec’s cryptographic building blocks are necessary to keep pace. The FreeBSD Foundation is pleased to announce that long-time FreeBSD developer John-Mark Gurney is adding modern AES modes to FreeBSD’s cryptographic framework and IPsec. This project is co-sponsored by the FreeBSD Foundation and Netgate, a leading vendor of BSD-based firewalls and networking gear.

The project adds new encryption modes while also importing infrastructure updates from OpenBSD giving FreeBSD users unprecedented support for high performance, encrypted communications.  New modes include AES-CTR and AES-GCM with hardware acceleration using Intel’s AES-NI instructions. According to John-Mark, “on a modern 64-bit x86 CPU one core can process about 1 gigabyte per second of data” using the new AES-GCM mode.

Concurrent with this project, FreeBSD committer and pfSense employee Ermal Luçi will update the FreeBSD IPsec stack to take advantage of the new cryptographic modes.

Jim Thompson, a co-owner of both Netgate and ESF (the company behind pfSense), said “We are pleased to contribute to this project.  Our interest in high-performance IPsec is obvious, however we also recognize the importance of contributing this capability to the FreeBSD project. Not only because our own software is based on FreeBSD, but for the benefit it brings to the entire community.  We plan to have AES-GCM support for IPsec with AES-NI acceleration available in the 2.2 release of pfSense software.”

The project is currently in progress, with a planned completion at the end of September 2014.

PC-BSD 10.0.3 Preview: Lumina Desktop

As we are getting ready for PC-BSD 10.0.3, I wanted to share a little preview of what to expect with the Lumina desktop environment as you move from version 0.4.0 to 0.6.2.

To give you a quick summary, pretty much everything has been updated/refined, with several new utilities written specifically for Lumina. The major new utility is the “Insight” file manager: with ZFS snapshot integration, multimedia player, and image slideshow viewer capabilities built right in by default. It also has a new snapshot utility and the desktop configuration utility has been completely rewritten. I am going to be listing more details about all the updates between the versions below, but for those of you who are not interested in the details, you can just take a look at some screenshots instead.…  :-)

Lumina10-0-3--1

 

Lumina10-0-3--2

Lumina10-0-3--3

 

Lumina10-0-3--4

Lumina10-0-3--5

 

==== FULL UPDATE DETAILS ====

(Moving from 0.4.0 to 0.6.2)
Desktop

- A desktop plugin system has been implemented, with two plugins available at the moment (a calandar plugin, and an application launcher plugin).
– The panel plugin system has been refined quite a bit, with transparency support for the panel itself and automatic plugin resizing for example.
– A new panel plugin has been added: the system dashboard. This plugin allows control over the audio volume, screen brightness, and current workspace, while also displaying the current battery status (if applicable) and containing a button to let the user log out (or shutdown/restart the system).
– The user button panel plugin has been re-implemented as well, and incorporating the functionality of the desktopbar plugin. Now the user has quick access to files/application in the ~/Desktop folder, as well as the ability to add/remove shortcuts to system applications in the desktop folder with one click.
– New backgrounds wallpapers and project logo (courtesy of iXsystems).

NOTE: Users of the older versions of the Lumina desktop will have their configuration files returned to the defaults after logging in to the new version for the first time.


Utilities
The new file manager (lumina-fm, also called “Insight”):
Features:
– Browse the system and allow the bookmarking of favorite directories
– Simple multimedia player to allow playing/previewing multimedia files
– Image slideshow viewer for previewing image files
– Full ZFS file/directory restore functionality if ZFS snapshots are available
– Menu shortcuts to quickly browse attached/mounted devices
– Tabbing support for browsing multiple directories at once
– Standard file/directory management (copy/paste/delete/create)
– Supported multimedia and image formats are auto-detected on start, so if a particular file is not recognized, you just need to install the appropriate library or plugin on your system to provide support (none required by default).

The new screenshot utility (lumina-screenshot):
Features:
– Simple utility to create/save screenshots on the system.
– Can capture the entire system, or individual windows.
– Can delay the image capture for a few seconds as necessary
– Automatically assigned to the “Print Screen” keyboard shortcut by default, but also listed in the application registry under utilities.

The configuration utility (lumina-config):
Features:
– Competely new implementation
– Configure desktop appearance (background image, add desktop plugins)
– Configure panels (location, color/transparency, size, manage plugins, up to 2 panels supported per screen)
– Configure right-click menu plugins
– Manage/set global keyboard shortcuts (including shortcuts for adjusting audio volume or screen brightness)
– Manage/set default applications for the system by categories or individually
– Manage session options (enable numlock on log in, play audio chimes)
– Manage/set applications/files to be launched on log in
– Manage window system options (appearance, mouse focus policy, window placement policy, number of workspaces)

The application/file opener utility (lumina-open):
– Update the overall appearance of the application selector window.
– Fully support registered mime-types on the system now, and recommend those applications as appropriate.

ZFS support in libvirt

An upcoming release of libvirt, 1.2.8 that should be released early September, will include an initial support of managing ZFS volumes.

That means that it's possible to boot VMs and use ZFS volumes as disks. Additionally, it allows to control volumes using the libvirt API. Currently, supported operations are:

  • list volumes in a pool
  • create and delete volumes
  • upload and download volumes

It's not possible to create and delete pools yet, hope to implement that in the next release.

Defining a pool

Assume we have some pools and want to use one of them in libvirt:

# zpool list
NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT
filepool 1,98G 56,5K 1,98G 0% - 0% 1.00x ONLINE -
test 186G 7,81G 178G 0% - 4% 1.00x ONLINE -

Let's take filepool and define it with libvirt. This could be done using this virsh command:

virsh # pool-define-as --name zfsfilepool --source-name filepool --type zfs
Pool zfsfilepool defined

virsh # pool-start zfsfilepool
Pool zfsfilepool started

virsh # pool-info zfsfilepool
Name: zfsfilepool
UUID: 5d1a33a9-d8b5-43d8-bebe-c585e9450176
State: running
Persistent: yes
Autostart: no
Capacity: 1,98 GiB
Allocation: 56,50 KiB
Available: 1,98 GiB

virsh #

As you can see, we specify a type of the pool, its source name, such as seen in zpool list output and a name for it in libvirt. We also need to start it using the pool-start command.

Managing volumes

Let's create a couple of volumes in our new pool.


virsh # vol-create-as --pool zfsfilepool --name vol1 --capacity 1G
Vol vol1 created

virsh # vol-create-as --pool zfsfilepool --name vol2 --capacity 700M
Vol vol2 created

virsh # vol-list zfsfilepool
Name Path
------------------------------------------------------------------------------
vol1 /dev/zvol/filepool/vol1
vol2 /dev/zvol/filepool/vol2

virsh #

Dropping a volume is also easy:

virsh # vol-delete --pool zfsfilepool vol2
Vol vol2 deleted

Uploading and downloading data

Let's upload an image to our new volume:

virsh # vol-upload --pool zfsfilepool --vol vol1 --file /home/novel/FreeBSD-10.0-RELEASE-amd64-memstick.img 

... and download

virsh # vol-download --pool zfsfilepool --vol vol1 --file /home/novel/zfsfilepool_vol1.img

Note: if you would check e.g. md5 sum of the downloaded files, the result would be different as downloaded file will be of the same size as a volume. However, if you trim zeros, it'll be the same.

$ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img 
MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c
MD5 (zfsfilepool_vol1.img) = a77c3b434b01a57ec091826f81ebbb97
$ truncate -r FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img
$ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img
MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c
MD5 (zfsfilepool_vol1.img) = e8e7cbd41b80457957bd7981452ecf5c
$

Booting a VM from volume

Finally got to the most important part. In use a volume as disk device for VM 'devices' section of the domain XML should be updated with something like this:


<disk type='volume' device='disk'>
<source pool='zfsfilepool' volume='vol1'/>
<target dev='vdb' bus='virtio'/>
</disk>

Few notes

Note #1: this code is just a few weeks old, so quite likely there are some rough edges. Feel free to report problems to novel%freebsd.org if you spot any problems.

Note #2: this code is FreeBSD-only for now. However, it should not be hard to make it work on Linux with zfsonlinux.org. Its developers were kind enough to add some useful missing flags in some of the CLI tools. However, these changes are not available in any released version so far. There are some more minor differences between zfs on Linux and FreeBSD, but that should not be hard to address. I was planning to get to it as soon as a new version of zfs on linux with the necessary flags is available. However, if you are interested in that and ready to help with testing -- feel free to poke me so it could be done sooner.


MeetBSD California 2014

MeetBSD California 2014 (https://www.meetbsd.com/), Western Digital Campus, San Jose, United States 1 - 2 November, 2014. MeetBSD 2014 uses a mixed unConference format featuring both scheduled talks and community-driven events such as birds-of-a-feather meetings, lightning talks, and speed geeking sessions.

X with glamor on vc4

Today I finally got X up on my vc4 driver using glamor.  As you can see, there are a bunch of visual issues, and what you can't see is that after a few frames of those gears the hardware locked up and didn't come back.  It's still major progress.
2014-08-21 16.16.37
The code can be found in my vc4 branch of mesa and linux-2.6, and the glamor branch of my xf86-video-modesetting.  I think the driver's at the point now that someone else could potentially participate.  I've intentionally left a bunch of easy problems -- things like supporting the SCS, DST, DPH, and XPD opcodes, for which we have piglit tests (in glean) and are just a matter of translating the math from TGSI's vec4 instruction set (documented in tgsi.rst) to the scalar QIR opcodes.

Happy 20th birthday FreeBSD ports tree!

It all started with this commit from Jordan Hubbard on August 21, 1994:

Commit my new ports make macros
Still not 100% complete yet by any means but fairly usable at this stage.

Twenty years later the ports tree is still there and actively
maintained. A video was prepared to celebrate the event and to thank
all of you who give some of their spare time and energy to the project!

Farewell beloved Canadian

Last month, our beloved Canadian Thomas Abthorpe decided to step
down from his portmgr-secretary position. While I suspect this is secretly
related to his pool of Canadian jokes having dried up, the official reason is
that Thomas wants to focus more on his private and professional lives for the
moment. Needless to say, the whole ports community is in mourning.

BSDCan Trip Report: Baptiste Daroussin

The next trip report is from Baptiste Daroussin:

Thanks to the FreeBSD Foundation I was able to attend BSDCan 2014.

I arrived in Ottawa on Tuesday evening and went directly to the Royal Oak where I met other FreeBSD developers.

On Wednesday, the DevSummit started with the FreeBSD future plans where I was mainly interested in pushing subjects like packaging base, dma(8) integration, improvements in kqueue, and status of the toolchain.

The afternoon was mainly spent meeting with many other developers to talk face to face on subjects which usually take a while to resolve via mail.

Thursday started with the ports and package session where I talked about the status of the package distribution: from building packages to distributing packages on the FreeBSD cluster. I gave a brief status about pkg(8). We talked about the pkg_tools decomission. We had a long and interesting discussion about the future of the ports tree. The other subjects we talked about were packaging-base, continuous integration of the ports tree, cross building packages, and the license framework.

Like the previous day, I spent the afternoon discussing pkg(8) with other developers, as well as phabricator, and discussing with clusteradm about different possibilities for distributed "extra" packages repositories.

On Friday and Saturday the main conference took place. There were plenty of different interesting talks I went to.

The main interesting one for me was " The architecture of the new solver in pkg" by Vsevolod Stakhov as it gave me more details about his wonderful work on pkg during GSoC 2013!

This conference has been really succesful for me. It was the first time we were able to get 4 pkg developers together: Vsevolod Stakhov (vsevolod@), Bryan Drewery (bdrewery@), Matthew Seaman (matthew@), and myself. I found it really productive to exchange ideas, share problems, and simply have discussion.

This conference also allows me to talk with clusteradm people, in particular Glen Barber (gjb), Peter Wemm (peter@), and Sean Bruno (sbruno@)

There was also the opportunity for 4 portmgrs, a future portmgr, and a former portmgr to have an informal meeting which was really great!

July/August Issue of The FreeBSD Journal Now Available

The fourth issue of the online FreeBSD Journal is now available! The issue is all about FreeBSD and Virtualization and includes topics such as FreeBSD on Amazon's EC2, and FreeBSD's own native virtualization system, bhyve. Plus, you'll find pieces on Xen, the USE Method, and more. The FreeBSD Journal is available at the Apple, Google, and Kindle stores at $19.99/year for six (6) issues or $6.99 for a single issue. Not a subscriber? Find out more and subscribe today!


BSDCan Trip Report: Mark Linimon

The next trip report is from Mark Linimon:

The first day, Tuesday, was an unoffficial day, spent socializing.

The Developer's Summit began Wednesday.  My main interest was to attend the "FreeBSD future plans" session.   Of particular interest was the discussion about how Release Engineering should look in the future. The ports team has done a great deal of work to decouple ports releases from src releases.  This required both changes in the way packages were built, as well as a substantial amount of new hardware to be able to build multiple package sets simultaneously.  (Much of this hardware was purchased by the Foundation).  This was the first change that many of the src and docs people had been brought up to speed on these developments.

Thursday, of course, my main interest was the Ports and Packages session.

In the evening, I was invited to an informal meeting with the various Ports Management Team (portmgrs) who were in attendance.  (I had previously served for several years on this team.)  Somehow, I was "volunteered" to rejoin the Ports Management team with an "advisor" status.  Clearly, peer pressure works.

Friday the conference itself started.  I spent some of the day trying to catch up on rest from the hectic first two days, and then socialized in the evening.

On Saturday, the most interesting session was the FreeNAS development talk.  While it was informative, there was also an opportunity to heckle John Hixson.

Perhaps the most important task that I accomplished during the conference was to sit down with Bryan Drewery and discuss future software improvements to the Ports Monitoring System (portsmon), which I wrote.

portsmon has survived many changes in FreeBSD.  The first was from CVS to SVN.  More recently, the ports build farm has been switched over from the old portbuild codebase to a completely rewritten system.  Our discussion dealt with the changes that I needed to make to port over to the new system; what the future changes to the new system would be; and changes that I requested that would make portsmon's job easier.  These changes have now been incorporated.  The next task is to catch up with the change from GNATS to Bugzilla; by that point, all of the inputs to portsmon will have been switched over from their initial codebase.