PC-BSD 10.0.3 Preview: Lumina Desktop

As we are getting ready for PC-BSD 10.0.3, I wanted to share a little preview of what to expect with the Lumina desktop environment as you move from version 0.4.0 to 0.6.2.

To give you a quick summary, pretty much everything has been updated/refined, with several new utilities written specifically for Lumina. The major new utility is the “Insight” file manager: with ZFS snapshot integration, multimedia player, and image slideshow viewer capabilities built right in by default. It also has a new snapshot utility and the desktop configuration utility has been completely rewritten. I am going to be listing more details about all the updates between the versions below, but for those of you who are not interested in the details, you can just take a look at some screenshots instead.…  :-)

Lumina10-0-3--1

 

Lumina10-0-3--2

Lumina10-0-3--3

 

Lumina10-0-3--4

Lumina10-0-3--5

 

==== FULL UPDATE DETAILS ====

(Moving from 0.4.0 to 0.6.2)
Desktop

- A desktop plugin system has been implemented, with two plugins available at the moment (a calandar plugin, and an application launcher plugin).
– The panel plugin system has been refined quite a bit, with transparency support for the panel itself and automatic plugin resizing for example.
– A new panel plugin has been added: the system dashboard. This plugin allows control over the audio volume, screen brightness, and current workspace, while also displaying the current battery status (if applicable) and containing a button to let the user log out (or shutdown/restart the system).
– The user button panel plugin has been re-implemented as well, and incorporating the functionality of the desktopbar plugin. Now the user has quick access to files/application in the ~/Desktop folder, as well as the ability to add/remove shortcuts to system applications in the desktop folder with one click.
– New backgrounds wallpapers and project logo (courtesy of iXsystems).

NOTE: Users of the older versions of the Lumina desktop will have their configuration files returned to the defaults after logging in to the new version for the first time.


Utilities
The new file manager (lumina-fm, also called “Insight”):
Features:
– Browse the system and allow the bookmarking of favorite directories
– Simple multimedia player to allow playing/previewing multimedia files
– Image slideshow viewer for previewing image files
– Full ZFS file/directory restore functionality if ZFS snapshots are available
– Menu shortcuts to quickly browse attached/mounted devices
– Tabbing support for browsing multiple directories at once
– Standard file/directory management (copy/paste/delete/create)
– Supported multimedia and image formats are auto-detected on start, so if a particular file is not recognized, you just need to install the appropriate library or plugin on your system to provide support (none required by default).

The new screenshot utility (lumina-screenshot):
Features:
– Simple utility to create/save screenshots on the system.
– Can capture the entire system, or individual windows.
– Can delay the image capture for a few seconds as necessary
– Automatically assigned to the “Print Screen” keyboard shortcut by default, but also listed in the application registry under utilities.

The configuration utility (lumina-config):
Features:
– Competely new implementation
– Configure desktop appearance (background image, add desktop plugins)
– Configure panels (location, color/transparency, size, manage plugins, up to 2 panels supported per screen)
– Configure right-click menu plugins
– Manage/set global keyboard shortcuts (including shortcuts for adjusting audio volume or screen brightness)
– Manage/set default applications for the system by categories or individually
– Manage session options (enable numlock on log in, play audio chimes)
– Manage/set applications/files to be launched on log in
– Manage window system options (appearance, mouse focus policy, window placement policy, number of workspaces)

The application/file opener utility (lumina-open):
– Update the overall appearance of the application selector window.
– Fully support registered mime-types on the system now, and recommend those applications as appropriate.

ZFS support in libvirt

An upcoming release of libvirt, 1.2.8 that should be released early September, will include an initial support of managing ZFS volumes.

That means that it's possible to boot VMs and use ZFS volumes as disks. Additionally, it allows to control volumes using the libvirt API. Currently, supported operations are:

  • list volumes in a pool
  • create and delete volumes
  • upload and download volumes

It's not possible to create and delete pools yet, hope to implement that in the next release.

Defining a pool

Assume we have some pools and want to use one of them in libvirt:

# zpool list
NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT
filepool 1,98G 56,5K 1,98G 0% - 0% 1.00x ONLINE -
test 186G 7,81G 178G 0% - 4% 1.00x ONLINE -

Let's take filepool and define it with libvirt. This could be done using this virsh command:

virsh # pool-define-as --name zfsfilepool --source-name filepool --type zfs
Pool zfsfilepool defined

virsh # pool-start zfsfilepool
Pool zfsfilepool started

virsh # pool-info zfsfilepool
Name: zfsfilepool
UUID: 5d1a33a9-d8b5-43d8-bebe-c585e9450176
State: running
Persistent: yes
Autostart: no
Capacity: 1,98 GiB
Allocation: 56,50 KiB
Available: 1,98 GiB

virsh #

As you can see, we specify a type of the pool, its source name, such as seen in zpool list output and a name for it in libvirt. We also need to start it using the pool-start command.

Managing volumes

Let's create a couple of volumes in our new pool.


virsh # vol-create-as --pool zfsfilepool --name vol1 --capacity 1G
Vol vol1 created

virsh # vol-create-as --pool zfsfilepool --name vol2 --capacity 700M
Vol vol2 created

virsh # vol-list zfsfilepool
Name Path
------------------------------------------------------------------------------
vol1 /dev/zvol/filepool/vol1
vol2 /dev/zvol/filepool/vol2

virsh #

Dropping a volume is also easy:

virsh # vol-delete --pool zfsfilepool vol2
Vol vol2 deleted

Uploading and downloading data

Let's upload an image to our new volume:

virsh # vol-upload --pool zfsfilepool --vol vol1 --file /home/novel/FreeBSD-10.0-RELEASE-amd64-memstick.img 

... and download

virsh # vol-download --pool zfsfilepool --vol vol1 --file /home/novel/zfsfilepool_vol1.img

Note: if you would check e.g. md5 sum of the downloaded files, the result would be different as downloaded file will be of the same size as a volume. However, if you trim zeros, it'll be the same.

$ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img 
MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c
MD5 (zfsfilepool_vol1.img) = a77c3b434b01a57ec091826f81ebbb97
$ truncate -r FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img
$ md5 FreeBSD-10.0-RELEASE-amd64-memstick.img zfsfilepool_vol1.img
MD5 (FreeBSD-10.0-RELEASE-amd64-memstick.img) = e8e7cbd41b80457957bd7981452ecf5c
MD5 (zfsfilepool_vol1.img) = e8e7cbd41b80457957bd7981452ecf5c
$

Booting a VM from volume

Finally got to the most important part. In use a volume as disk device for VM 'devices' section of the domain XML should be updated with something like this:


<disk type='volume' device='disk'>
<source pool='zfsfilepool' volume='vol1'/>
<target dev='vdb' bus='virtio'/>
</disk>

Few notes

Note #1: this code is just a few weeks old, so quite likely there are some rough edges. Feel free to report problems to novel%freebsd.org if you spot any problems.

Note #2: this code is FreeBSD-only for now. However, it should not be hard to make it work on Linux with zfsonlinux.org. Its developers were kind enough to add some useful missing flags in some of the CLI tools. However, these changes are not available in any released version so far. There are some more minor differences between zfs on Linux and FreeBSD, but that should not be hard to address. I was planning to get to it as soon as a new version of zfs on linux with the necessary flags is available. However, if you are interested in that and ready to help with testing -- feel free to poke me so it could be done sooner.


MeetBSD California 2014

MeetBSD California 2014 (https://www.meetbsd.com/), Western Digital Campus, San Jose, United States 1 - 2 November, 2014. MeetBSD 2014 uses a mixed unConference format featuring both scheduled talks and community-driven events such as birds-of-a-feather meetings, lightning talks, and speed geeking sessions.

X with glamor on vc4

Today I finally got X up on my vc4 driver using glamor.  As you can see, there are a bunch of visual issues, and what you can't see is that after a few frames of those gears the hardware locked up and didn't come back.  It's still major progress.
2014-08-21 16.16.37
The code can be found in my vc4 branch of mesa and linux-2.6, and the glamor branch of my xf86-video-modesetting.  I think the driver's at the point now that someone else could potentially participate.  I've intentionally left a bunch of easy problems -- things like supporting the SCS, DST, DPH, and XPD opcodes, for which we have piglit tests (in glean) and are just a matter of translating the math from TGSI's vec4 instruction set (documented in tgsi.rst) to the scalar QIR opcodes.

Happy 20th birthday FreeBSD ports tree!

It all started with this commit from Jordan Hubbard on August 21, 1994:

Commit my new ports make macros
Still not 100% complete yet by any means but fairly usable at this stage.

Twenty years later the ports tree is still there and actively
maintained. A video was prepared to celebrate the event and to thank
all of you who give some of their spare time and energy to the project!

Farewell beloved Canadian

Last month, our beloved Canadian Thomas Abthorpe decided to step
down from his portmgr-secretary position. While I suspect this is secretly
related to his pool of Canadian jokes having dried up, the official reason is
that Thomas wants to focus more on his private and professional lives for the
moment. Needless to say, the whole ports community is in mourning.

BSDCan Trip Report: Baptiste Daroussin

The next trip report is from Baptiste Daroussin:

Thanks to the FreeBSD Foundation I was able to attend BSDCan 2014.

I arrived in Ottawa on Tuesday evening and went directly to the Royal Oak where I met other FreeBSD developers.

On Wednesday, the DevSummit started with the FreeBSD future plans where I was mainly interested in pushing subjects like packaging base, dma(8) integration, improvements in kqueue, and status of the toolchain.

The afternoon was mainly spent meeting with many other developers to talk face to face on subjects which usually take a while to resolve via mail.

Thursday started with the ports and package session where I talked about the status of the package distribution: from building packages to distributing packages on the FreeBSD cluster. I gave a brief status about pkg(8). We talked about the pkg_tools decomission. We had a long and interesting discussion about the future of the ports tree. The other subjects we talked about were packaging-base, continuous integration of the ports tree, cross building packages, and the license framework.

Like the previous day, I spent the afternoon discussing pkg(8) with other developers, as well as phabricator, and discussing with clusteradm about different possibilities for distributed "extra" packages repositories.

On Friday and Saturday the main conference took place. There were plenty of different interesting talks I went to.

The main interesting one for me was " The architecture of the new solver in pkg" by Vsevolod Stakhov as it gave me more details about his wonderful work on pkg during GSoC 2013!

This conference has been really succesful for me. It was the first time we were able to get 4 pkg developers together: Vsevolod Stakhov (vsevolod@), Bryan Drewery (bdrewery@), Matthew Seaman (matthew@), and myself. I found it really productive to exchange ideas, share problems, and simply have discussion.

This conference also allows me to talk with clusteradm people, in particular Glen Barber (gjb), Peter Wemm (peter@), and Sean Bruno (sbruno@)

There was also the opportunity for 4 portmgrs, a future portmgr, and a former portmgr to have an informal meeting which was really great!

July/August Issue of The FreeBSD Journal Now Available

The fourth issue of the online FreeBSD Journal is now available! The issue is all about FreeBSD and Virtualization and includes topics such as FreeBSD on Amazon's EC2, and FreeBSD's own native virtualization system, bhyve. Plus, you'll find pieces on Xen, the USE Method, and more. The FreeBSD Journal is available at the Apple, Google, and Kindle stores at $19.99/year for six (6) issues or $6.99 for a single issue. Not a subscriber? Find out more and subscribe today!


BSDCan Trip Report: Mark Linimon

The next trip report is from Mark Linimon:

The first day, Tuesday, was an unoffficial day, spent socializing.

The Developer's Summit began Wednesday.  My main interest was to attend the "FreeBSD future plans" session.   Of particular interest was the discussion about how Release Engineering should look in the future. The ports team has done a great deal of work to decouple ports releases from src releases.  This required both changes in the way packages were built, as well as a substantial amount of new hardware to be able to build multiple package sets simultaneously.  (Much of this hardware was purchased by the Foundation).  This was the first change that many of the src and docs people had been brought up to speed on these developments.

Thursday, of course, my main interest was the Ports and Packages session.

In the evening, I was invited to an informal meeting with the various Ports Management Team (portmgrs) who were in attendance.  (I had previously served for several years on this team.)  Somehow, I was "volunteered" to rejoin the Ports Management team with an "advisor" status.  Clearly, peer pressure works.

Friday the conference itself started.  I spent some of the day trying to catch up on rest from the hectic first two days, and then socialized in the evening.

On Saturday, the most interesting session was the FreeNAS development talk.  While it was informative, there was also an opportunity to heckle John Hixson.

Perhaps the most important task that I accomplished during the conference was to sit down with Bryan Drewery and discuss future software improvements to the Ports Monitoring System (portsmon), which I wrote.

portsmon has survived many changes in FreeBSD.  The first was from CVS to SVN.  More recently, the ports build farm has been switched over from the old portbuild codebase to a completely rewritten system.  Our discussion dealt with the changes that I needed to make to port over to the new system; what the future changes to the new system would be; and changes that I requested that would make portsmon's job easier.  These changes have now been incorporated.  The next task is to catch up with the change from GNATS to Bugzilla; by that point, all of the inputs to portsmon will have been switched over from their initial codebase.

Receive Side Scaling: figuring out how to handle IP fragments

The TL:DR; of this is - IP fragments are annoying.

If everything was awesome and there were never IP fragments, all TCP and UDP frames would always have the TCP/UDP header stamped on them, and the NIC could hash the TCP/UDP header in hardware to calculate the destination queue to receive traffic on.

However, everything isn't awesome and there will be cases where IP frames are fragmented. When this happens, the first frame in the fragment has the IPv4 header and the TCP/UDP header - but the subsequent fragments only have the IPv4 header. That means there's not enough information in the rest of the fragments to hash them to the same hash value and thus hardware queue as the first fragment - only the first has the full IPv4+TCP/UDP information.

The Intel and Chelsio NICs will hash on all packets that are fragmented by only hashing on the IPv4 details. So, if it's a fragmented TCP or UDP frame, it will hash the first fragment the same as the others - it'll ignore the TCP/UDP details and only hash on the IPv4 frame. This means that all the fragments in a given IP datagram will hash to the same value and thus the same queue.

But if there are a mix of fragmented and non-fragmented packets in a given flow - for example, small versus larger UDP frames - then some may be hashed via the IPv4+TCP or IPv4+UDP details and some will just be hashed via the IPv4 details. This means that packets in the same flow will end up being received in different receive queues and thus highly likely be processed out of order.

The Linux intel driver code flipped off IPv4+UDP hashing a while ago - they hash UDP frames by their IPv4 details only and then do whatever other load balancing in the kernel they choose. I found this and updated the FreeBSD drivers to do the same. This should result in less out of order UDP frames for UDP heavy workloads. I'm not sure about the Chelsio driver yet - when I convert it to the RSS framework it'll disable IPv4+UDP hashing if that isn't enabled at boot time. This is a good stop-gap, but it's not the whole story.

TCP is where it gets annoying. People don't want to flip off IPv4+TCP hashing as they're convinced that the TCP MSS negotiation and path-MTU discovery stuff will prevent there from being any IP fragmented TCP frames. But, well, that's not really viable in the real world. There are too many misconfigured networks out there and IP fragmentation does occur. So this is also a problem for TCP. This means that the IPv4 fragmented TCP frames in those sessions will come into another receive queue and CPU and this will show up as out of order data.

So, what's this all have to do with receive side scaling?

With RSS, there's a well defined hash for packets and a configuration for what the operating system and NICs are supposed to be doing. It's entirely possible that we'll configure IPv4+TCP to be hashed and also entirely possible we'll see IP fragments showing up on other CPUs. So in order to have the TCP stack run on the right CPU, the IP fragments need to be assembled on whichever CPU they're received upon and then re-injected into the correct destination queue to run on the correct CPU.

Fortunately the FreeBSD netisr scheme makes this easy.

So what I'm doing in my branch (and what will soon show up in -HEAD) is thus:


  • UDP is still hashed as IPv4-only frames for now. I'll change that later to hash on IPv4+UDP and have things reinjected on the correct destination RSS bucket / netisr queue / CPU.
  • I create one netisr thread, pinned to a CPU, for each RSS CPU that's defined.
    • Ideally I'd create one netisr thread for each RSS bucket and pin that, but that'll come later.
  • IP fragments will be hashed to whatever the IPv4 hash calculates, so fragment reassembly will occur on some CPU;
    • .. and it's the same CPU for all frames in a fragmented datagram.
  • Then when the fragment is reassembled, a software hash is calculated for the newly reassembled frame.
    • If RSS is configured to hash for IPv4 only, then it'll see that the hash on the reassembled datagram matches the configured hash for that packet type and reuse it.
    • So, if it's UDP right now, it'll see that UDP is only hashing on IPv4 details and reuse it.
    • .. but if IPv4+UDP hashing is configured, it'll software hash the packet and assign the new flow type and RSS hash.
  • Then, it'll reinject the frame into netisr to be requeued and reprocessed.
  • .. this uses the nh_m2cpuid function to calculate the destination CPU for the given RSS hash.
    • If it's handled on the same destination CPU then it'll be handled.
    • If it's handled on a different destination CPU then it'll be queued to that netisr and dispatched appropriately.
This works. It's not great, and I'd rather the IP fragment reassembly code was much more efficient, but it's correct. I'm going for correctness here to begin with.

Now, before you ask - yes, IPv6 has fragments and yes, I have to do the same thing for IPv6 flows. Most of the code is written.

Finally - the same thing applies to things like IPv4 tunnels, IPv6-in-IPv4 tunnels, IPSEC tunnels and the like. The NIC hashes the packets on the IPv4 header details but once the packet is de-encapsulated, it needs to be reinjected back into the correct CPU for further processing.

Using the xdev target with qemu-user-static on #FreeBSD

I’ve been playing with building ports for ARM on an AMD64 machine via a bunch of tools.  The duct tape and bailing wire is a bit thick with this method, but if you keep at it, this should work.

1. build armv6 chroot:
make buildworld TARGET=arm TARGET_ARCH=armv6
make installworld TARGET=arm TARGET_ARCH=armv6 DESTDIR=/armv6
make distribution TARGET=arm TARGET_ARCH=armv6 DESTDIR=/armv6

2. build xdev
make xdev TARGET=arm TARGET_ARCH=armv6 NOSHARED=y

3. move xdev into chroot
mv /usr/armv6-freebsd /armv6/usr/

4. add toolchain to make.conf:
CFLAGS+=-integrated-as
CC=/usr/armv6-freebsd/usr/bin/cc
CPP=/usr/armv6-freebsd/usr/bin/cpp
CXX=/usr/armv6-freebsd/usr/bin/c++
AS=/usr/armv6-freebsd/usr/bin/as
NM=/usr/armv6-freebsd/usr/bin/nm
LD=/usr/armv6-freebsd/usr/bin/ld
OBJCOPY=/usr/armv6-freebsd/usr/bin/objcopy
SIZE=/usr/armv6-freebsd/usr/bin/size
STRIPBIN=/usr/armv6-freebsd/usr/bin/strip
5. Install qemu-static-user from ports and copy into jail:
pkg instlal qemu-static-user
mkdir -p /armv6/usr/local/bin
cp /usr/local/bin/qemu-arm /armv6/usr/local/bin/

6. setup binmiscctl to handle armv6 translations:
binmiscctl add armv6 –interpreter “/usr/local/bin/qemu-arm” –magic “x7fx45x4cx46x01x01x01x00x00x00x00x00x00x00x00x00x02x00x28x00″ –mask “xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxfexffxffxff” –size 20 –set-enabled

7. mount devfs and ports if needed
mount -t devfs devfs /armv6/dev
mount -t nullfs /usr/ports /armv6/usr/ports

8. chroot
chroot /armv6

Foundation is Accepting Travel Grant Applications for EuroBSDCon 2014

Calling all FreeBSD developers needing assistance with travel expenses to EuroBSDCon 2014.

The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please fill out and submit  the Travel Grant Request Application at http://www.freebsdfoundation.org/documents/TravelRequestForm.pdf by August 15th, 2014 to apply for this grant.

This program is open to FreeBSD developers of all sorts (kernel hackers,  documentation authors, bugbusters, system administrators, etc).  In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.

If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your direct request to us.

There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.

bsdtalk243 – mandoc with Ingo Schwarze

Interview about mandoc with Ingo Schwarze.  The project webpage describes mandoc as "a suite of tools compiling mdoc, the roff macro language of choice for BSD manual pages, and man, the predominant historical language for UNIX manuals."

Recorded at BSDCan 2014.

File Info: 16Min, 8MB.

Ogg Link: http://cis01.uma.edu/~wbackman/bsdtalk/bsdtalk243.ogg