PC-BSD 10.1.2: an Interview with Kris Moore

PC-BSD 10.1.2 has been released, so we thought we’d talk to project lead, Kris Moore, to see what’s in store!

KrisMug2015

Q: What new features and improvements are available in PC-BSD 10.1.2?

This quarterly update got a LOT of new features, partly so we would have time to really play with them before the 10.2 release later this summer.  Most of them are security and privacy focused, but there are some other neat things as well.  In no particular order, these are some of the best ones:

* PersonaCrypt – Our new privacy utility that offers a few cool new features.

In its default mode, it can setup your entire user $HOME directory on a geli-backed external device, such as a fast SSD Flash Stick (Using the 256GB Corsair Flash Voyager GTX here). At the login screen, you will then enter your normal user password, along with a decryption password to mount your home-directory. The GELI key is split in two, so you can “pair” the memory stick with your system, so even if the stick is stolen, and somebody gets the password, it is still worthless without the “paired” system. Another benefit of this technology is that you can bring your work with you when you travel between desktops, or jump from a desktop to laptop. I’m using it now for conferences and being on the road, since all my development work and important data is on my $HOME, meaning I can just unplug and keep working on my laptop without  having to keep files in sync.

Another side of PersonaCrypt is something we call “Stealth” mode, which allows you to do a desktop login with a one-time GELI-key encrypted $HOME directory with no personal data. Think of it as privacy mode  in a web-browser, but for your entire desktop session. This also plays nicely with the following new feature.

* Tor transparent proxy support

10.1.2 includes an easy way to switch between your normal internet connectivity, and “Tor” mode by clicking a single button on the system tray. In Tor mode, the firewall acts as a transparent proxy, forcing all internet traffic to be routed through the Tor network, including DNS requests. All other traffic which may expose your system on the internet is blocked. This goes a step beyond just running Tor as a browser proxy, since you can’t always trust plugins (cough *flash/java*) and other apps to behave properly.

* 4K Monitor support

While most things already worked with 4K monitors, we did go through our toolchain and fixed a number of scaling bugs, mouse cursors and fonts which needed tweaking to look better “out of box”. I’m currently using an nvidia card with DisplayPort, and running in 4K at 60Hz, and yes my eyes thank me every day.

* Encrypted ZFS replication

Our Life-Preserver utility got an update to support doing backups to an encrypted zpool, using GELI and iSCSI. Both the zpool and GELI are initialized on the client side, so that when doing backups, nothing ever goes “over the wire” that isn’t already encrypted. LP also gives the ability to export the connection settings and GELI key to a password-protected GELI file, which means I don’t have to copy the plaintext key to another system for importing the pool / recovery. It also can be used by the install media to do a “bare-metal” restore from a backup, which is handy if your system dies badly. I’m currently using this feature to backup my PersonaCrypt home-directory.

LP was also updated to now do per-dataset replication, allowing us to do recursive replication with excludes, and preparing us for resumable ZFS send/recv in the near future.

* Media Center Support

Our installer now provides support to install Kodi or PlexHomeTheater.  Along with the automatic update feature, it makes PC-BSD great for the 10ft experience. My evil plan going forward is to add Steam Big Picture mode to this list, possibly for 11.

* LibreSSL / OpenNTPD

We switched all our ports to now using LibreSSL instead of OpenSSL, along with moving from NTP -> OpenNTPD. This is done in response to there being so many PR’s against this two particular pieces of code, hopefully reducing the number of PR’s we are vulnerable to.

* AppCafe web-interface

The new AppCafe web-interface is now live, allowing remote management of PBI/pkg, and basic jail support. This means you can run a  PC-BSD/TrueOS box headless, and manage your Applications, Jails, and Updates all via web-browser or mobile device. This is our first release of the web-based system, and we already have plans on adding more features to it, in particular to jail management and support for 3rd party repositories in
jails.

* IPFW Firewall

We’ve also switched our default firewall from PF over to IPFW, with the plan of enabling VIMAGE for 10.2 later this year. Once we ship with VIMAGE enabled, we will then be doing some planned updates to the Warden networking support.

Q: What feature(s) do you think will have the biggest impact for users?

Personally I think PersonaCrypt is just the coolest. Since I travel a bit, it is really nice to be able to bring my entire home development environment and workstation with me. I can actually shut the machine off at home now since all my files and work comes with me.  I can also see how that would be helpful for those who alternate between home and office, not needing to lug around a laptop to work from home.

From a marketing perspective though, we are hearing from a lot of dis-enfranchised Linux users. Over things like ShellShock, Heartbleed, SystemD and the like. While Tor mode isn’t for everybody, that coupled with LibreSSL, PersonaCrypt and the like helps demonstrate that PC-BSD is very security conscious, making some of the best security and privacy technology more user-friendly.

Q:  Did any of these features pose a particular challenge?

Switching over to LibreSSL was quite a challenge, I worked with  Bernard Spil (Whom should be getting a port-commit bit soon), to find and fix lots of packages that still used really old and vulnerable OpenSSL functionality. It was really eye-opening, and I think proves the point why we should somewhat frequently “cull” the old and obsolete code, because you never know what is still lurking around that uses it.

Q: What features do you plan on tackling next?

For 10.2 we will be working on getting VIMAGE enabled and better supported by Warden. Along with that will be updates to the AppCafe for more jail functionality, and beefing up the number of PBIs in our tree which have configuration UIs. I also plan on doing some infrastructure work, so that we can do more frequent builds of packages for -CURRENT and EDGE users, along with other architectures down the road.

Q: What does the upgrade path to 10.1.2 look like?

Upgrading from 10.1.1 is pretty easy. Our new updater went into the previous release, and it allows doing a “background” update of all your packages to the 10.1.2 set. Basically a new boot-environment is created, and fresh 10.1.2 packages are installed into it via a chroot. This means you can keep working throughout the update, and when its done you only have to reboot to get into the new BE and be on 10.1.2 with the new features.

Another Data Center Site Visit – NYI

No Systems Administrators Were Harmed While Writing This Blog Entry

Mmm...  Freshly-unboxed servers.  There really is nothing better to wake up to in the morning.

Well, okay, coffee.  But new servers - definitely second.

In late April, the FreeBSD Foundation generously purchased more machines to keep the FreeBSD.org infrastructure operating smoothly.  While the new servers are not yet in production (a task the Cluster Administrators will undertake while at BSDCan in June), we have planned far in advance what we intend to do with the new hardware.

In mid-May, I spent several days at our East-Coast US colocation facility, racking, cabling, installing, and configuring the new servers.

As They Say in Real-Estate: Colocation, Colocation, Colocation

The new hardware is located at New York Internet in Bridgewater, New Jersey, who generously provides colocation services to the FreeBSD Project.  They have an amazing staff, and whether we are on-site or working with them through their ticket system, are always friendly, knowledgeable, and of course, helpful.

New Hardware Specs

In total, 14 new SuperMicro-based machines were purchased from iXsystems, for this site. They are all 1U servers, each with four 1TB drives, redundant power supplies, and gigabit ethernet.

Ten of the machines are single-socket Intel(R) Xeon(R) E3-1230 v3 CPUs, running at 3.30GHz (4-core with hyper-threading, providing 8 logical CPU threads), with 32 GB RAM.

Four of the machines are dual-socket Intel(R) Xeon(R) E5-2630 v3 CPUs, running at 2.40GHz (8-core with hyper-threading, providing 16 logical CPU threads per socket, 32 threads total), with 64 GB RAM.

For the Inner Geek in All of Us

No blog post about new hardware would be complete without pictures, right?

This is the front view of the ten single-socket machines:


And this is the back view of the same machines:


(Yes, I really do label each end of the network and serial console cables.)

Then, this is the front view of the four dual-socket machines (sorry for the blurry picture, though):


And the back view of the dual-socket machines:


Last but not Least...

On behalf of the FreeBSD Foundation and the FreeBSD Cluster Administration Team, I would like to thank New York Internet for generously providing the colocation space for our east-coast site, the NYI network operations team for all of their assistance during my visit, and especially all of the community investors that have generously donated to the FreeBSD Foundation.  All of your support is greatly appreciated.

Thank you for helping make FreeBSD better!

Lumina Desktop Status Update/FAQ

With the amount of changes to the Lumina desktop environment and the increasing number of questions/comments we are receiving, I thought it would be a good time to post a status report about it and answer many of the commonly asked questions.

As always, please post any bug reports or feature requests on the PC-BSD Bug tracker.

If you wish to get involved in the development of the Lumina Desktop Environment, you can find the source code in our repository on GitHub.

Lumina Desktop FAQ/Status Update:

What is the Lumina Desktop Environment?

  • The Lumina Desktop Environment (sometimes referred to as Lumina-DE or just Lumina) is a BSD-licensed, FreeDesktop-compliant graphical interface for a desktop operating system.
  • It has been written from scratch in C++/Qt5 (it is not based on any existing DE code-base). It uses Fluxbox for the window management in the background and xscreensaver for screensaver/screenlocking functionality. The only other runtime dependencies are a couple of small X11 utilities (xnumlock, xbrightness, xterm, xrandr).
  • This results in a very lightweight, very smooth desktop experience with minimal system overhead.

What is the current development status?

  • While the current version (0.8.4) is still considered to be “Beta” quality, that is primarily because there are still a couple external utilities which have not had in-house replacements written yet (Fluxbox and xscreensaver primarily).
  • In its current state it is very stable and completely usable on a daily basis for both developers and non-developers alike.
  • The only caveat for its “beta” status is that since there are some backend systems which are still in major development, we reserve the right to break a user’s customized settings during the transition to these new systems (the window manager and screensaver settings are where I anticipate this happening once the replacement systems are in place). However, we already have a good track record of backwards compatibility with older settings formats, and will continue to work on ensuring a smooth transition between different versions of the desktop (even during heavy development/backend changes).

What are some of the features of Lumina?

  • Restore files from ZFS snapshots directly through the Insight file manager (lumina-fm).
  • View picture slideshows or play multimedia files through the Insight file manager.
  • Completely configurable – the interface is designed around a plugin-based mentality where everything you see/use is a compiled-in plugin (ensuring that every plugin is guaranteed to work). This means that you can create an interface custom-tailored to your specific needs.
  • Easy-to-use configuration utility (lumina-config). This single utility controls all the different configuration options for the desktop in one simple location.
  • Total system search capabilities through the lumina-search utility (without having a daemon eating memory in the background).
  • Screenshot functionality through the lumina-screenshot utility (also tied to the “print screen” key by default).
  • Multi-monitor support through xrandr (every monitor is treated as a distinct screen, with it’s own background, interface, etc..). Lumina also provides a graphical utility for adding/removing monitors from within the Lumina session (lumina-xconfig).
  • Simple system controls through the “System Dashboard” for things like audio volume, screen brightness, battery status/notifications, and workspace switching.
  • Easily select which application to use when opening a file and set that as the default for the next time. If you happen to remove that default application, it will automatically re-open the application selection dialog the next time you open that type of file again. This utility (lumina-open) is also tied into the commonly used xdg-open utility so that 3rd party applications will automatically use this utility for opening files outside of the application scope.

How is it different from other desktop environments?

  • Designed on PC-BSD, specifically for the BSD community at large (although it is easily ported to any OS, including Linux distros).
  • Does not require any of the commonly-used desktop implementation frameworks (DBUS, policykit, consolekit, systemd, HALD, etc..).
  • Does not come bundled with any “end-user” applications (web browsers, email clients, multimedia software, office suites, etc..). The only utilities that Lumina brings to the table by default are the ones written specifically for the project and are generally for background/utilitarian functionality (the largest utility is the file manager).
  • Simple text-based configuration file for setting system-wide defaults for new users. This allows distributors of the desktop to easily pre-set the system defaults/interface so it just works for the end user.
  • Plugin-based interface design. This allows the user to make the desktop as light/heavy as desired (within reason) simply by choosing which plugins to have running on their desktop/panels.

Are there any PC-BSD specific features?

  • Hardware-level screen brightness control (sysctl control through pc-sysconfig on PC-BSD 10.1.2+).
  • Links to the PC-BSD Control Panel and AppCafe are embedded directly into the Lumina interface for system control/modifications.
  • Detects when system updates are in progress and prevents the user from shutting down the system until that is complete (you can log out of Lumina and then shutdown the computer from the PCDM login manager if absolutely necessary) .
  • Designed to work hand-in-hand with the PC-BSD utilities (mounttray, AppCafe, Life Preserver, etc.) preventing conflicts between DE and OS utilities.

Is it available for any other operating systems?

  • There are currently source files in the project for building/using Lumina on: FreeBSD/PC-BSD, OpenBSD, DragonFlyBSD, kFreeBSD, Debian, and generic “Linux” distributions.
  • Please contact the package management/distribution team for your particular OS to determine the availability of pre-compiled packages.

What does it look like?

  • Due to the flexibility of the interface design and the ease with which the themes may be customized, the Lumina desktop can take many various forms. Here are a just a few of the variations that I have assembled on my system:
Click to view slideshow.
  • As we continue to create more plugins with every new version of Lumina, the variability of the interface just continues to grow. If you have a particular plugin/feature that you would like, please post a feature request on the PC-BSD bug tracker!

What plugins are currently available (as of version 0.8.4)?

  • Panel Plugins: AppLauncher, AppMenu, BatteryMonitor, Clock, DesktopBar, DesktopSwitcher, HomeButton, Spacer, SystemDashboard, SystemTray, TaskManager(with or without grouping), UserButton.
  • Desktop Plugins: AppLauncher, AudioPlayer, Calendar, DesktopView, Notepad
  • Menu Plugins: AppLauncher, AppMenu, LineBreak, OpenTerminal, OpenFileManager, SettingsMenu, WindowList
  • More plugins are being created all the time!

What is being worked on right now?

  • Right now, we are mainly focused on expanding the scope of the desktop through creation of new plugins and enhancement of the existing plugins.
  • The next “big” change planned is the move to a new window manager which is in the process of being written right now.  This change will correspond with version 0.9.0.
  • We are also working on tweaking the default themes and colors right now, and are looking into adding new themes & color schemes “out of box” for people to be able to simple select and use.

A new window manager? Is something wrong with Fluxbox?

  • Fluxbox is a great WM, and the Lumina project will be still be using it for quite some time yet while the replacement is being written. The main reason a new WM is planned is for better integration with the Lumina desktop in the following areas:
    • Interface/font scaling
    • Keyboard shortcuts (particularly with using the default application registrations in Lumina)
    • Addition of WM “modes” for use on various types of devices.
    • Allow compositing between windows (for better transparency effects)
    • Uniform Theme/Appearance
  • In addition, having a custom WM specifically for Lumina allows us to be able to combine the functionality of a few common background daemons into a single utility (since at their core they all revolve around responding to particular base system events).
    • Window Manager (ICCCM/EWMH events)
    • Screensaver/Screen Locker (All Input Events – transparent watcher)
    • Keyboard shortcuts (Keyboard Input Events)
    • Application message logger/viewer (some EWMH events – transparent watcher)
    • Power management (system events through devd/HALD/other)
  • Having all these systems within the same utility allows us to be able to create a much tighter level of inter-functionality between them. For a few quick examples:
    • Temporarily disable the screensaver while a window is full-screen (such as when watching a video)
    • While the screen is locked, prevent the window manager from showing any new windows on the screen and stop all keyboard shortcut handling (preventing possible security issues), while still allowing some notifications and background procedures.
    • Enable different levels of power management while the screen is locked or the screensaver is active.
  • This is a fairly massive undertaking, so I want to re-iterate that I expect it to be quite some time until this new window manager is ready for use and we will continue to use Fluxbox in the meantime. The Lumina desktop will remain within the 0.8.x series of version numbers while this new utility is in development.

How can I get involved?

  • If you are a C++/Qt developer, there are tons of areas where you could contribute! Some of them are: finding/fixing bugs, writing new interface plugins, adding functionality to existing plugins, and more!
  • If you are a CSS developer, then you might be interested in making new themes for Lumina (using QSS: it is very similar to CSS but specifically for Qt applications/widgets). These themes can be created/modified directly within the Lumina configuration utility, and also provides an easy way to directly implement the user’s current color scheme. Once you have a theme put together that you like, you can just send us that file (located in ~/.lumina/themes) and we can get that included with any future versions of the desktop.
  • If you want to create new color schemes, you can do so via the Lumina configuration utility. Once you have one that you like, just send in that file (located in ~/.lumina/colors) and we can get that included with any future versions of the desktop.
  • If you found a bug or have an idea for a cool new feature, you can post bug reports or feature requests on the PC-BSD bug tracker!

How can I contact the developer(s) with questions/comments/snide remarks?

 

HotFix release to 10.1.2 – Now available

A minor hotfix update to the 10.1.2 ISO’s has been released today. This includes fixes to advanced installation using raidz, cache and log devices, as well as a fix to the text-installer when booted in UEFI mode. Users who have already installed 10.1.2 will not need to download, and can instead online-update to install any fixes.

Download Now

PC-BSD 10.1.2 Released

The PC-BSD team is pleased to announce the availability of the next PC-BSD / TrueOS quarterly release, 10.1.2.

 

PC-BSD 10.1.2 Notable Changes:

  •  New PersonaCrypt Utility
    • Allows moving all of users $HOME directory to an encrypted USB Drive. This drive can be connected at login, and used across different systems
    • Stealth Mode — Allows login to a blank $HOME directory, which is encrypted with a one-time GELI key. This $HOME directory is then discarded at logout, or rendered unreadable after a reboot
  • Tor mode — Switch firewall to running transparent proxy, blocking all traffic except what is routed through Tor.
  • Migrated to IPFW firewall for enabling VIMAGE in 10.2
  • Added sound configuration via the first boot utility
  • Support for encrypted iSCSI backups via Life-Preserver, including support for bare-metal restores via installer media
  • New HTML handbook, updated via normal package updates
  • Media Center support allowing direct login to Kodi and PlexHomeTheater for the 10ft user experience
  • Switch to new AppCafe interface, with remote support via web-browser
  • Improvements to Online Updater, along with GRUB nested menus for Boot-Environments
  • Migrate all ports to using LibreSSL instead of OpenSSL
  • Switch from NTPD to OpenNTPD
  • Lumina desktop 0.8.4
  • Chromium 42.0.2311.135
  • Firefox 38.0
  • NVIDIA Driver 346.47
  • Pkg 1.5.2

 

Updating

Users currently running the 10.1.1 release can now apply the updates via their Update Manager utility. Users running TrueOS or CLI can start the update with the following command:

# pc-updatemanager pkgupdate

 

Getting media

10.1.2 DVD/USB media can be downloaded from the following URL via HTTP or Torrent. http://www.pcbsd.org/en/download.html

Reporting Bugs

Found a bug in 10.1.2? Please report it (with as much detail as possible) to our bugs database. https://bugs.pcbsd.org

Contacting Us

freebsd-wifi-build, or "wait, you can run freebsd on atheros MIPS access points? where do I get that?"

I've been running FreeBSD at home as my primary internet/wifi access for a few years now. It's cheap, it's easy to do, and I've tried very hard to wrap up the whole process into a mostly-simple build system that spits out a useful image to use.

It's pretty simple in concept - I take FreeBSD-HEAD, build it with some cut-down options, create a custom filesystem image with some custom boot scripts and a custom configuration file, and provide an image that you can TFTP (using a serial console and ethernet cable) or upload directly to the AP if it supports it.

The supported hardware list is here:

https://github.com/freebsd/freebsd-wifi-build/wiki/Supported-Boards

Now, it's not a huge list like OpenWRT, but that's mostly because I don't have an infinite supply of Atheros MIPS based routers. I think I'll get some of the TP-Link Archer series stuff next.

Building it is pretty simple:

https://github.com/freebsd/freebsd-wifi-build/wiki

You checkout the build repo, check out FreeBSD-HEAD, install a couple of packages, and run the build for your board. Once it's done, the images for your board appear in ../tftpboot/. There's a wiki page for each of the supported boards with a walkthrough with how to get FreeBSD going on it.

It comes up on 192.168.1.20/24 with 'user' and 'root' users, with no password. So, the first thing you should do after installation is telnet in, configure /etc/cfg/rc.conf with your actual LAN IPs, set the user/root passwords, and then 'cfg_save' to save things. Then, reboot and voila!

The configuration file format looks like FreeBSD but it isn't. I'm keeping it somewhat hierarchical-looking in naming but flat in implementation so I can migrate it to something like a sqlite or luci backend in the future.

https://github.com/freebsd/freebsd-wifi-build/wiki/Config-Overview

It's good enough for me to be able to set up an AP to be a bridge with a management IP address and configure the ethernet switch. Others have added ipfw support to do NAT and firewalling - I'm going to add configuration rules for NAT, IPFW and routing soon so it's all integrated.

It's FreeBSD, all the way through:


$ uname -a
FreeBSD tl-wdr3600 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r282406M: Wed May 6 22:27:16 PDT 2015 adrian@lucy-11i386:/usr/home/adrian/work/freebsd/head-embedded/obj/mips/mips.mips/usr/home/adrian/work/freebsd/head-embedded/src/sys/TL-WDR4300 mips
$ ifconfig wlan0 list sta
ADDR AID CHAN RATE RSSI IDLE TXSEQ RXSEQ CAPS FLAG
18:ee:69:15:f4:12 2 1 26M 37.0 45 2703 51888 EPS AQEHTRM RSN HTCAP WME
04:e5:36:0d:1b:0d 1 1 19M 23.0 15 1524 47072 EPS AQEPHTR RSN HTCAP WME
cc:3a:61:0e:33:a0 3 1 19M 32.0 30 2585 43072 EPS AQEPHTR RSN HTCAP WME
40:0e:85:1a:f1:69 4 1 19M 25.0 30 1138 54800 EPS AQEPHTR RSN HTCAP WME
00:0f:13:97:14:54 5 1 54M 30.0 45 1808 57424 EPS AE RSN
00:22:fa:c2:d1:20 6 1 26M 24.5 0 574 57776 EPS AQEHTRS RSN HTCAP WME

So if you'd like a FreeBSD based device to act as your home gateway, this is where you can start. It's not pfsense, but it's designed to run on things much smaller than pfsense supports and it's a good introduction into the world of FreeBSD embedded.

bsdtalk253 – George Neville-Neil

An interview with George Neville-Neil about the recently published 2nd edition of The Design and Implementation of the FreeBSD Operating System.

File Info: 30Min, 15MB

Ogg Link: https://archive.org/download/bsdtalk253/bsdtalk253.ogg

PC-BSD 11.0-CURRENTMAY2015 images now available

The PC-BSD project is pleased to announce the availability of our 11.0-CURRENTMAY2015 images.

WARNING: These images are considered “bleeding-edge” and should be treated as such.

The DVD/USB ISO files can now be downloaded from this URL.

We hope to continue rolling these –CURRENT images as a way for testers and developers to tryout both FreeBSD and PC-BSD bleeding edge features, often months before a planned release. These images include a full PKG repository compiled for that months image. Users of this system will also be able to “upgrade” when the next monthly image is published.

This is a great way to test features and report bugs well before the release cycle begins for the next major .0 release.

For bugs in PC-BSD, please report to https://bugs.pcbsd.org.

For FreeBSD / Port / Kernel / World bugs, please report to https://bugs.freebsd.org/bugzilla/enter_bug.cgi.

Updating from 10.1-RELEASE

There was a bug we found in freebsd-update when upgrading from 10.1-RELEASE to 11.0.

This has been fixed in 10.1-RELEASE-p21 (Check the output of freebsd-version), please update to it before attempting to change branch to 11.0.

Once you have the –p21 fix, run the following to “upgrade” your system to 11.0-CURRENTMAY2015svn282515

# pc-updatemanager chbranch 11.0-CURRENTMAY2015svn282515

This process will take a while, downloading new packages / world / kernel for the system. When done you can reboot, and the updater will finish up the update process.

PC-BSD 10.1.2-RC1 Now Available

The PC-BSD team is pleased to announce the availability of RC1 images for the upcoming quarterly 10.1.2 release.

Please test these images out and report any issues found on our bug tracker.

PC-BSD 10.1.2 Notable Changes

  • New PersonaCrypt Utility allows moving all of users $HOME directory to an encrypted USB Drive. This drive can be connected at login, and used across different systems
  • Stealth Mode allows login to a blank $HOME directory, which is encrypted with a one-time GELI key. This $HOME directory is then discarded at logout, or rendered unreadable after a reboot
  • Tor mode switches the firewall to running transparent proxy, blocking all traffic except what is routed through Tor
  • Migrated to IPFW firewall for enabling VIMAGE in 10.2
  • Added sound configuration via the first boot utility
  • Support for encrypted iSCSI backups via Life-Preserver, including support for bare-metal restores via installer media
  • New HTML handbook, updated via normal package updates
  • Media Center support allowing direct login to Kodi and PlexHomeTheater for the 10ft user experience
  • Switch to new AppCafe interface, with remote support via web-browser
  • Improvements to Online Updater, along with GRUB nested menus for Boot-Environments
  • Migrate all ports to using LibreSSL instead of OpenSSL
  • Switch from NTP to OpenNTPD
  • Lumina desktop 0.8.4
  • Chromium 42.0.2311.90
  • Firefox 37.0.2
  • NVIDIA Driver 346.47
  • Pkg 1.5.1

Updating

Users currently running the EDGE package repo can now update their packages via the updater GUI or “pc-updatemanager” utility to be brought up to date with RC1.

Updates for users on the 10.1.1 / PRODUCTION repo will be available once 10.1.2-RELEASE is announced.

Getting media

10.1.2-RC1 DVD/USB media can be downloaded from here via HTTP or Torrent.

Reporting Bugs
———————————————
Found a bug in 10.1.2? Please report it (with as much detail as possible) to our bugs database.

PC-BSD 10.1.2-RC1, Lumina Desktop 0.8.4 Released!

The PC-BSD team is pleased to announce the availability of RC1 images for the upcoming quarterly 10.1.2 release. Please test these images out and report any issues found on our bug tracker.

What else is new in PC-BSD 10.1.2? How about a new version of the Lumina Desktop Environment! PC-BSD users who stick to the “Production” branch of packages will find that the Lumina desktop has evolved/improved an incredible amount since the last quarterly update for PC-BSD (10.1.1), so I highly recommend that you try it out! The release notes for this new version are also listed at the bottom of this announcement for those of you who have been tracking along with its development, so please try it out and let us know what you think!

 

PC-BSD 10.1.2 notable Changes

  • New PersonaCrypt Utility
    • Allows moving all of users $HOME directory to an encrypted USB Drive. This drive can be connected at login, and used across different systems
    • Stealth Mode — Allows login to a blank $HOME directory, which is encrypted with a one-time GELI key. This $HOME directory is then discarded at logout, or rendered unreadable after a reboot
  • Tor mode — Switch firewall to running transparent proxy, blocking all traffic except what is routed through Tor.
  • Migrated to IPFW firewall for enabling VIMAGE in 10.2
  • Added sound configuration via the first boot utility
  • Support for encrypted iSCSI backups via Life-Preserver, including support for bare-metal restores via installer media
  • New HTML handbook, updated via normal package updates
  • Media Center support allowing direct login to Kodi and PlexHomeTheater for the 10ft user experience
  • Switch to new AppCafe interface, with remote support via web-browser
  • Improvements to Online Updater, along with GRUB nested menus for Boot-Environments
  • Migrate all ports to using LibreSSL instead of OpenSSL
  • Switch from NTP to OpenNTPD
  • Lumina desktop 0.8.4
  • Chromium 42.0.2311.90
  • Firefox 37.0.2
  • NVIDIA Driver 346.47
  • Pkg 1.5.1 Updating

Package Availability:

  • Users currently running the EDGE package repo can now update their packages via the updater GUI or “pc-updatemanager” utility to be brought up to date with RC1. Updates for users on the 10.1.1 / PRODUCTION repo will be available once 10.1.2-RELEASE is announced.

Getting media

Reporting Bugs

  • Found a bug in PC-BSD 10.1.2 or Lumina 0.8.4? Please report it (with as much detail as possible) to our bugs database. https://bugs.pcbsd.org

 

Lumina Desktop changes since version 0.8.3:

Panel Improvements:

  • Add mouse tracking support
  • Add support for variable-length panels (a percentage of the screen edge length).
  • Add support for pinning the panel to a particular location on the screen edge (either corner, or centered)
  • Automatically re-scale the panel size if the monitor used in the previous session was a different screen resolution.
  • For hidden panels, have 1% of the panel size be visible on the screen while it is “hidden” (rather than a hard-coded pixel size). This is better for high-resolution screens.
  • Remove the restriction that panels be on opposite screen edges.

New options/usage for lumina-search:

  • Easily change file/dir search preferences on a temporary basis
  • New command-line flags for starting searches instantly
  • Search functionality integrated into the Insight file manager. The Ctrl-F keyboard shortcut or the “Search” menu option will start a search for a file/directory with the current directory as the starting point.
  • A “Search” button has been added to the  home directory browser in the user menu. This will allow the user to easily start searching for a file/dir within the selected directory.

New “Favorites” system backend:

  • This new backend is much faster and more reliable than the old system of sym-links.
  • Your favorites should be automatically converted to the new format when you log into the new version of Lumina.

New Utility: lumina-fileinfo

  • This utility allows the user to view basic file information, such as timestamps, owner/group info, file size, and read/write permissions.
  • If the file is a XDG desktop shortcut (that the user has permission to modify), this utility also provides the ability to make changes to that shortcut.
  • This can easily be used by right-clicking on files in the desktop view plugin or within the Insight file manager and selecting the “Properties” option.
  • A big thank you to contributor William (william-os4y on GitHub) for writing this utility!!

Other Random Improvements:

  • Better application recommendations for files/URL’s (especially for web browsers or email clients).
  • Major cleanup of XCB library usage.
  • Hardware-brightness controls now used for PC-BSD by default (if supported by the system hardware).
  • Putting the system into the suspend state is now supported for PC-BSD/Debian.
  • New clock display formats.
  • A large number of session cleanup improvements
  • A large number of session initialization improvements (including resetting the user’s previous screen brightness and audio volume settings).
  • New default keyboard shortcuts for tiling the open windows on the screen (new user configurations only)
  • Better support for the URL input format when required by an application.
  • Make the user’s “log out” window appear much faster when activated.

Errata:

  • There is a known bug in Lumina 0.8.4 regarding “unlocked” desktop plugins. The close/maximize buttons for the plugin are unresponsive when using Qt 5.4.1, preventing the user from easily removing/maximizing a desktop plugin. We are still looking into this, but at the moment it appears to be a bug in Qt itself. As a temporary workaround, you can simply right-click on the titlebar for the unlocked plugin and select close/maximize from the menu instead.

From the Trenches, Tips & Tricks Edition: Hacking "/ on ZFS" and GELI Encrypted Drives, the Old-School Way

Glen Barber is back to kick off our latest From The Trenches series: The Tips and Tricks Edition. 

All my personal machines run FreeBSD.

In fact, all my personal machines run FreeBSD-CURRENT. I do this primarily to keep track of changes that get committed to the head branch, so I can personally test changes (for the things I use, at least) before they get merged to the stable branches.

As one of the Release Engineers, I find it essential that, whenever possible, I find issues so they can be corrected before they are part of a release.

My primary work machine is a laptop, currently a Lenovo Thinkpad T540p. I picked this laptop, and all the other laptops before it, because it met my minimum requirements for a primary workstation: it is capable of supporting a large amount of RAM (16GB for my Thinkpad, 8GB for all previous laptops), an Intel Core i7 CPU, and I could replace the DVD drive with a second hard drive.

In addition to these hardware requirements, I also have a few personal requirements of any workstation - the drives must be encrypted, and the underlying filesystem must be ZFS.

For me, it is not so much about the data I have *on* the laptop that I need to protect, but the kinds of things within the FreeBSD Project I am permitted access. Without encrypted drives, a lost or stolen laptop would absolutely be my worst possible nightmare, because I only have my login passphrase protecting my data (GPG key, SSH keys, and so on).

Recent FreeBSD releases allow "/ on ZFS" installation with the option to enable GELI-based encryption. This predates my original installation, however, since each laptop I have purchased for the past several years used the hard drives from the previous laptop. According to zpool history, the installation was at least two and a half years ago, but I know it is much longer than that, because of zfs recv being one of the first things zpool history reports.

So, I needed to do things the old-fashioned way, and manually create the GELI-backed providers and perform the "/ on ZFS" installation myself.

While bsdinstall(8) may now cover the majority of use cases for such installations, there may be cases where someone specifically needs to do something a certain way that the installer does not provide.

Because I only had one hard drive in the system when the system was initially installed (a long time ago), I will only refer to one hard drive when describing the steps I used to perform the installation, for now.

I installed the system using the 9.0-RELEASE or 9.1-RELEASE memory stick installer (memstick.img), I cannot remember which, but that detail is not as important, since I did not use the installer anyway.

When I booted from the memory stick, the two drives recognized on the system were the internal hard drive, /dev/ada0, and the external USB flash drive for the installation, /dev/da0. The first menu screen has three options available: "Install", "Shell", "Live CD".

I selected "Live CD", and logged in as root (no password is necessary for the "Live CD" functionality). The hard drive did not have an operating system. Because I purchased the hard drive, in addition to the laptop, with the intention of replacing the laptop's drive, I did not need to remove any partitions from an existing installation. If I did need to remove partitions, I would have done so with:
# gpart destroy -F ada0
Here is where some technical details become important:
  • While you can install "/ on ZFS" on a drive partitioned with MBR (Master Boot Record), using GPT is far easier. In fact, I have forgotten much about how MBR partitioning is actually done.
  • When doing full disk encryption, you must keep /boot contents separate, otherwise loader(8) and the kernel will not be available when the BIOS hands over control to the operating system. As such, /boot should be given its own partition on the disk left unencrypted, and the rest of the system on its own encrypted partition.
I created four partitions on the drive. The first partition is for the boot blocks (not to be confused with the /boot contents), the second partition is for /boot, the third is for the encrypted system, and the fourth is for swap.
# gpart create -s gpt ada0
# gpart add -t freebsd-boot -s 512k -i 1 -l gptboot ada0
# gpart add -t freebsd-zfs -s 10G -i 2 -l bootfs ada0
# gpart add -t freebsd-swap -s 10G -i 3 -l swapfs ada0
# gpart add -t freebsd-zfs -s 180G -i 4 -l rootfs ada0
I decided to put the swap partition between the /boot partition and the rest of the system, in case I needed to increase or decrease the size of the /boot partition, it would be far easier (and safer) to do.

Then, I loaded the necessary kernel modules for ZFS and GELI:
# kldload /boot/kernel/opensolaris.ko
# kldload /boot/kernel/zfs.ko
# kldload /boot/kernel/geom_eli.ko
Now that GELI functionality is available, I created the backend provider for the ZFS dataset:
# geli init -b -a HMAC/SHA256 -e AES-CBC -l 256 
-s 4096 /dev/ada0p4
Then I attached the GELI provider, and wrote data from /dev/randomto the new device /dev/ada0p4.eli:
# geli attach ada0p4
# dd if=/dev/random of=/dev/ada0p4.eli bs=4096

This took a while on the system this hard drive was originally installed, so I probably got coffee at this point. :-)

When the dd(1) command finished, I continued the installation.

I created temporary directories to use to import the pools after they were created:
# mkdir /tmp/zroot
# mkdir /tmp/zboot
Keep in mind, I am installing from a memory stick image, which by default, is read-only. The /tmp directory is writable, however, because it is a md(4)-backed memory disk filesystem.
# zpool create -O checksum=fletcher4 -O atime=off 
-m /tmp/zboot zboot /dev/ada0p2
# zpool create -O checksum=fletcher4 -O atime=off
-m /tmp/zroot zroot /dev/ada0p4.eli
Then I made a few ZFS datasets for various paths:
# for i in var var/log var/tmp var/db usr usr/home 
usr/compat usr/ports
usr/local tmp; do
zfs create zroot/${i}
done
I also made a separate ZFS dataset for the "bootfs" contents, and set the mountpoint to the /boot directory in the temporary working directory:
# zfs create zboot/boot
# zfs set mountpoint=/tmp/zroot/boot zboot/boot
On the memory stick installation media, the distribution sets are located in /usr/freebsd-dist. I extracted their contents into the newly-created filesystem:
# cd /tmp/zroot
# for i in base kernel lib32; do
tar -xf /usr/freebsd-dist/${i}.txz -C .
done
Then I wrote the bootcode to the first partition of the drive:
# gpart bootcode -b /tmp/zroot/boot/pmbr 
-p /tmp/zroot/boot/gptzfsboot -i 1 ada0
Because the "bootfs" (/boot) and "rootfs" (everything else) are both ZFS, I needed to use the gptzfsboot bootcode for the "freebsd-boot" partition.

Now the system is installed, but I needed to make a few modifications before I was ready to reboot. In particular, set a root password, edit /etc/fstab to enable swap, edit /etc/rc.conf to enable the zfs rc(8) startup script, and edit /boot/loader.conf to load the geom_eli.ko, opensolaris.ko, and zfs.ko kernel modules at boot.
# chroot /tmp/zroot
# passwd root
[enter password]
# echo '/dev/gpt/swapfs none swap sw 0 0'
>> /etc/fstab
# echo 'zfs_enable="YES"' >> /etc/rc.conf
# echo 'geom_eli_load="YES"' >> /boot/loader.conf
# echo 'zfs_load="YES"' >> /boot/loader.conf
# exit
Before rebooting, I needed to make a few adjustments to where /boot from the zboot/boot dataset would be mounted at boot.
# zfs umount zboot/boot
# zfs set mountpoint=/realboot zboot/boot
This now makes the /boot directory mount as /realboot, so I then needed to point /boot in the zroot dataset to the correct place. This was easily solved with a symbolic link:
# cd /tmp/zroot
# ln -s boot /realboot
Now when the system boots, the filesystem will look something like this:
/bin
/sbin
/boot -> /realboot
/realboot
[...]
Finally, I needed to unmount the zroot dataset, and fix its mountpoints. I only needed to change the zroot mountpoint itself, since all children datasets adjusted their paths automatically.
# zfs umount -a
# zfs set mountpoint=/ zroot
At this point, the installation was complete. I rebooted the laptop, entered the GELI passphrase for /dev/ada0p4.eli when prompted, and was greeted by the "login: " prompt we have all grown to love.

PC-BSD and 4K — Oh my!

I had recently seen some chatter on IRC about 4K monitor support. One of the people discussing stated that PC-BSD didn’t support 4K monitors, which was curious to me, since I hadn’t tried it yet. This week I did something about that, and ended up getting two 4K monitors and a new video card (With triple DisplayPort) to do some testing.

The Hardware

2x — ASUS PB287Q 28-Inch Screen LED-Lit 4K Monitor (Amazon)

Zotac GeForce GTX 960 2GB GDDR5 PCI Express 3.0 HDMI DVI DisplayPort (Amazon)

The Setup

The setup process was thus:

1. Remove the old monitors and video card

2. Connect the new monitors and video card

3. Go into monitor menus — Enable DisplayPort 1.2 mode

4. Boot up, selecting the “Run Display Wizard” option at the boot-loader.

5. Click “Accept”

That’s it! The display wizard properly detected both 4k monitors, and set them up in a proper dual-head configuration. Logging into KDE resulted in it detecting the new monitors, and automatically re-configuring the display to take advantage of the new resolution. At this point the system was fully usable, font-scaling even worked properly for Firefox and such.

Having 4K setup I did find a few minor issues, which are already being fixed in time for PC-BSD 10.1.2 next week, such as changing the mouse cursor to something which supports larger sizes /  resolutions better. The only real hiccup was figuring out that the monitor ships with DisplayPort set to 1.1 mode, only giving 30hz mode out of box. Changing that in the monitors menus to DisplayPort 1.2 mode, enabled 60hz, giving a silky-smooth desktop experience at 4K resolutions. Not sure if that is something typical of all 4K monitors, or just this model in particular, but if you can’t go beyond 30hz, you may want to check that in your monitor documentation.

Anyway, I hope this information is of use to anybody else who is thinking of taking the 4K plunge. Of course I would love additional feedback from other users who are running 4K, to see if there are other tweaks we can make for optimal support out of box. Let us know on the bug tracker, mailing lists or new forums!

Intel DDIO, LLC cache, buffer alignment, prefetching, shared locks and packet rates.

I've been digging into the low level behaviour of high throughput packet classification and pushing for my job. The initial suggestions from everyone was "use netmap!" Which was cool, but it only seems to to fast packet work if you're only ever really flipping packets between receive and transmit rings. Once you start actually looking into the payload, you start having to take memory misses and things can slow down quite a bit. An L3 miss (ie, RAM access) on Sandybridge is ~50ns. (There's also costs involved in walking the TLB, but I won't cover that here.)

For background: http://7-cpu.com/cpu/SandyBridge.html .

But! Intel has this magical thing called DDIO. In theory (and there's a lot of theory here), DMA is done via a small (~10%) fraction of LLC (L3) cache, which is shared between all cores. If the data is already in cache when the CPU accesses it, it will be quick. Also, if you then wish to DMA out data from something in cache, it doesn't have to get flushed to memory first - it's just DMAed straight out of cache.

However! When I was doing packet bridge testing (using netmap + bridge, 64 byte payloads), I noticed that I was doing a significant amount of memory bandwidth. It wasn't quite at the rate of 10G worth of bridged data, but DDIO should be doing almost all of that work for me at 64 byte payloads.

So, to reproduce: run netmap bridge (eg 'bridge -i netmap:ix0 -i netmap:ix1') and run pkt-gen between two nodes.

This is the output of 'pcm-memory.x 1' from the intel-pcm toolkit (which is available as a binary package on FreeBSD.)

---------------------------------------||---------------------------------------
--                   System Read Throughput(MB/s):    300.68                  --
--                  System Write Throughput(MB/s):    970.81                  --
--                 System Memory Throughput(MB/s):   1271.48                  --
---------------------------------------||---------------------------------------

The first theory - the bridging isn't occuring fast enough to service what's in LLC before it gets flushed out by other packets. So, assume:

  1. It's 1/10th of the LLC - which is 1/10th of an 8 core * 2.5MB per core setup, is ~ 2MB.
  2. 64 byte payloads are being cached.
  3. Perfect (!) LLC use.
That's 32,768 packets at a time. Now, netmap is doing ~ 1000 packets a batch and it's keeping up line rate bridging on one core (~14 million packets per second), so it's not likely that.

Ok, so what if it's not perfect LLC usage?

Then I thought back to cache line aliasing and other issues that I've previously written about. What if the buffers are perfectly aligned (say, 2048 byte aligned) - the cache line aliasing effects should also manifest themselves as low LLC utilisation.

Luckily netmap has a twiddle - 'dev.netmap.buf_size' / 'dev.netmap.priv_buf_size'. They're both .. 2048. So yes, the default buffer sizes are aligned, and there's likely some very poor LLC utilisation going on.

So, I tried 1920 - that's 2048 - (2 * 64) - ie, two cache lines less than 2048.


---------------------------------------||---------------------------------------
--                   System Read Throughput(MB/s):    104.92                  --
--                  System Write Throughput(MB/s):    382.32                  --
--                 System Memory Throughput(MB/s):    487.24                  --
---------------------------------------||---------------------------------------

It's now using significantly less memory bandwidth to do the same thing. I'm guessing this is because I'm now using the LLC much more efficiently.

Ok, so that's nice - but what about when it comes time to actually look at the packet contents to make decisions?

I've modified a copy of bridge to do a few things, mostly inspired by netmap-ipfw:
  • It does batch receive from netmap;
  • but it then looks at the ethernet header do decap that;
  • then it gets the IPv4 src/dst addresses;
  • .. and looks them up in a (very large) traditional hash table.
I also have a modified copy of pkt-gen that will use completely random source and destination IPv4 addresses and ports, so as to elicit some very terrible behaviour.

With an empty hash set, but still dereferencing the ethernet header and IPv4 source/destination, handling a packet at a time, no batching, no prefetching and only using one core/thread to run:

buf_size=2048:
  • Bridges about 6.5 million pps;
  • .. maxes out the CPU core;
  • Memory access: 1000MB/sec read; 423MB/sec write (~1400MB/sec in total).
buf_size=1920:
  • Bridges around 10 million pps;
  • 98% of a CPU core;
  • Memory access: 125MB/sec read, 32MB/sec write, ~ 153MB/sec in total.
So, it's a significant drop in memory throughput and a massive increase in pps for a single core.

Ok, so most of the CPU time is now spent looking at the ethernet header in the demux routine and in the hash table lookup. It's a blank hash table, so it's just the memory access needed to see if the bucket has anything in it. I'm guessing it's because the CPU is loading in the ethernet and IP header into a cache line, so it's not already there from DDIO.

I next added in prefetching the ethernet header. I don't have the code to do that, so I can't report numbers at the moment. But what I did there was I looped over everything in the netmap RX ring, dereferenced the ethernet header, and then did per-packet processing. This was interesting, but I wanted to try batching out next. So, after some significant refactoring, I arranged the code to look like this:
  1. Pull in up to 1024 entries from the netmap receive ring;
  2. Loop through, up to 16 at a time, and place them in a batch
  3. For each packet in a batch do:
    1. For each packet in the batch: optional prefetch on the ethernet header
    2. For each packet in the batch: decapsulate ethernet/IP header;
    3. For each packet in the batch: optional prefetch on the hash table bucket head;
    4. For each packet in the batch: do hash table lookup, decide whether to forward/block
    5. For each packet in the batch: forward (ie, ignore the forward/block for now.)
I had things be optional so I could turn on/off prefetching and control the batch size.

So, with an empty hash table, no prefetching and only changing the batch size, at buf_size=1920:
  • Batch size of 1: 10 million pps;
  • Batch size of 2: 11.1 million pps;
  • Batch size of 4: 11.7 million pps.
Hm, that's cute. What about with prefetching of ethernet header? At buf_size=1920:
  • Batch size of 1: 10 million pps;
  • Batch size of 2: 10.8 million pps;
  • Batch size of 4: 11.5 million pps.
Ok, so that's not that useful. Prefetching on the bucket header here isn't worthwhile, because the buckets are all empty (and thus NULL pointers.)

But, I want to also be doing hash table lookups. I loaded in a reasonably large hash table set (~ 6 million entries), and I absolutely accept that a traditional hash table is not exactly memory or cache footprint happy. I was specifically after what the performance was like for a traditional hash table. Said hash table has 524,288 buckets, and each points to an array of IPv4 addresses to search. So yes, not very optimal by any measure, but it's the kind of thing you'd expect to find in an existing project.

With no prefetching, and a 6 million entry hash table:

At 2048 byte buffers:
  • Batch size of 1: 3.7 million pps;
  • Batch size of 2: 4.5 million pps;
  • Batch size of 4: 4.8 million pps.
At 1920 byte buffers:
  • Batch size of 1: 5 million pps;
  • Batch size of 2: 5.6 million pps;
  • Batch size of 4: 5.6 million pps.
That's a very inefficient hash table - each bucket is going to have around 11 IPv4 entries in it, and that's checking almost a cache line worth of IPv4 addresses in it. Not very nice. But, it's within a cache line worth of data, so in theory it's not too terrible.

What about with prefetching? All at 1920 byte buffers:
  • Batch size of 4, ethernet prefetching: 5.5 million pps
  • Batch size of 4, hash bucket prefetching: 7.7 million pps
  • Batch size of 4, ethernet + hash bucket prefetching: 7.5 million pps
So in this instance, there's no real benefit from doing prefetching on both.

For one last test, let's bump the bucket count from 524,288 to 2,097,152. These again are all at buf_size=1920:
  • Batch size of 1, no prefetching: 6.1 million pps;
  • Batch size of 2, no prefetching: 7.1 million pps;
  • Batch size of 4, no prefetching: 7.1 million pps;
  • Batch size of 4, hash bucket prefetching: 8.9 million pps.
Now, I didn't quite predict this. I figured that since I was reading in the full cache line anyway, having up to 11 entries in it to linearly check would be cheap. It turns out that no, that's not exactly true.

The difference between the naive way (no prefetching, no batching) to 4-packet batching, hash bucket prefetching is not trivial - it's ~ 50% faster. Going all the way to a larger hash bucket was ~75% faster. Now, this hash implementation is not exactly cache footprint friendly - it's bigger than the LLC, so with random flows and thus no real useful cache behaviour it's going to degrade to quite a few memory accesses.

This has been quite a fun trip down the optimisation peephole. I'm going to spend a bunch of time writing down the hardware performance counters involved in analysing this stuff and I'll look to write a follow-up post with details about that.

One final things: threads and locking. I wanted to clearly demonstrate the cost of shared read locks on a setup like this. There's been lots of discussions about the right kind of locking and concurrency strategies, so I figured I'd just do a simple test in this setup and explain how terrible it can get.

So, no read-locks between threads on the hash table, batch size of 4, hash bucket prefetching, buf_size=1920:
  • 1 thread: 8.9 million pps;
  • 4 threads: 12 million pps.
But with a read lock on the hash table lookups:
  • 1 thread: 7 million pps;
  • 4 threads: 4.7 million pps.
I'm guessing that as I add more threads, the performance will drop.

Even taking a rwlock as a reader lock in pthreads is expensive - it's purely just an atomic increment/decrement in FreeBSD, but it's still not free. I'm getting the lock once for two hash table lookups - ie, the source and destination IP hash table lookups are done under one lock. I'm sure if I took the lock for the whole batch hash table lookup it'd work out a little better on a small number of CPU cores, but I think this demonstrates my point - read locks aren't going to cut it when you have a frequently accessed thing to protect.

The best bit about this post? The prefetching, terrible (large) hash table performance and general cache abuse is not new. Doing batching on superscalar Intel CPUs is not new. Documenting DDIO effectiveness using non-power-of-two-aligned buffer sizes is new, but it's just a rehash of the existing cache aliasing effect. But, I now have a little test bed to experiment with these things without having to try and involve the rest of a kernel.

Yes, I'll publish code soon.

ICC profiles and “startx” on FreeBSD

Beyond photography, I find that a calibrated screen is more comfortable to use, especially if you have two or more monitors with uneven panels and colors rendering.

You can buy a professional colorimeter for a not-so-small amount of money, but fortunately, there’s an open-source alternative. Richard Hughes developed ColorHug, a free open-source colorimeter that you can build yourself or order from him. I bought one myself shortly after it was released. It gives very good results, comparable to professional devices.

To calibrate the screen, FreeBSD lacks the appropriate applications in the Ports tree. The ColorHug is provided with a modified Fedora live CD with all the required programs installed and configured. Just boot your computer using this CD, plug the device and the wizard starts automatically, offerring you to calibrate all connected monitors. The process is quite straightforward and takes from a few minutes to 20-25 minutes, depending on the accuracy you want. When it’s finished, you have your shiny new profile(s) in ~/.local/share/icc. Profiles are named after monitor’s name and the date of the calibration. You need to copy those files to the same location on your FreeBSD desktop.

Now, you need to configure your computer to use the generated profile(s). GNOME, KDE and any other modern desktop environment come with display configuration tools which are easy to use. For those preferring a more lightweight environment and startx, things are a bit more complicated. Here is what I had to do.

  1. ICC profiles are managed by colord (graphics/colord). This daemon is started by dbus automatically. However, it seems that, after a fresh install, the service fails to start as reported in bug 197946 by several users of darktable. Running the service manually once “fixed” the problem for me:

    /usr/local/libexec/colord

  2. colord finds ICC profiles located in ~/.local/share/icc without further configuration. However, monitors are probed and managed by the X.Org server so another tool is required to declare all monitors in colord during session startup. I used xiccd, compiled from source (I need to make a port for it).

  3. Now, you need to start xiccd but PolKit and ConsoleKit get in the way as soon as xiccd tries to communicate with colord. Debugging this is where I spent most of the time… So here is the solution:

    • ConsoleKit is started by a login manager such as XDM, GDM, KDM and so on. If you don’t use one, you need to enable ConsoleKit in /etc/pam.d/login:

      # ConsoleKit.
      session  optional  /usr/local/lib/pam_ck_connector.so  nox11

    • To test if ConsoleKit works, log out and log in again on the console and run ck-list-sessions. You should get something like:

      Session1:
      	unix-user = '...'
      	realname = '...'
      	seat = 'Seat1'
      	session-type = ''
      	active = TRUE
      	x11-display = ''
      	x11-display-device = ''
      	display-device = '/dev/ttyv0'
      	remote-host-name = ''
      	is-local = TRUE
      	on-since = '2015-04-06T08:12:57.167178Z'
      	login-session-id = ''
      	idle-since-hint = '2015-04-06T08:13:27.293605Z'

      The “active = TRUE” line is the important one.

    • You need to start your X session with ConsoleKit. Modify your ~/.xinitrc to start your session with ck-launch-session:

      exec ck-launch-session dbus-launch --exit-with-session ~/.xstart

      dbus-launch starts dbus under the ConsoleKit session and ~/.xstart (of course, choose the name you want) is a script running all applications requiring a working dbus and ConsoleKit environment.

  4. My .xstart launches xiccd, loads ICC profiles and starts a window manager:

    #!/bin/sh 
    
    set -e
    
    if [ -x $HOME/Projects/graphics/xiccd/install/bin/xiccd ]; then
            # xiccd, compiled from source, is present, let's start it.
            $HOME/Projects/graphics/xiccd/install/bin/xiccd &
    
            # It takes some time to declare all screens in colord. Wait here
            # until colord knows about the laptop's monitor.
            #
            # To know the name of the monitors, use "colormgr get-devices".
            while ! colormgr get-devices | grep -q xrandr_LVDS_(...); do 
                    sleep 1 
            done 
    
            # Load the ICC profile for the laptop's monitor.
            #
            # To list the ICC profile IDs, use "colormgr get-profiles".
            colormgr device-add-profile 
                /org/freedesktop/ColorManager/devices/xrandr_LVDS_(...) 
                /org/freedesktop/ColorManager/profiles/icc_(...)
    
            # Do this again for each possible external screens.
            if colormgr get-devices | grep -q xrandr_EA244UHD_(...); then 
                    # NEC EA244UHD. 
                    xrandr --output LVDS --off 
                    xrandr --output HDMI-0 --mode 1920x1080 --rate 60 
    
                    colormgr device-add-profile 
                        /org/freedesktop/ColorManager/devices/xrandr_EA244UHD_(...) 
                        /org/freedesktop/ColorManager/profiles/icc_(...)
            fi 
    
            if colormgr get-devices | grep -q xrandr_SONY_TV_(...); then 
                    # Sony Bravia KDL-40WE5. 
                    xrandr --output LVDS --off
    
                    colormgr device-add-profile 
                        /org/freedesktop/ColorManager/devices/xrandr_SONY_TV_(...) 
                        /org/freedesktop/ColorManager/profiles/icc_(...)
            fi
    fi
    
    # Start window manager.
    exec awesome

Now, after startx(1), all monitors display the same colors!

With a single monitor to manage, oyranos (graphics/oyranos) was working for me. There is no dependency to colord, ConsoleKit or PolKit and is way simpler to use. However, it didn’t want to manage my external monitor (or two monitors at a time, I don’t know). This made an opportunity to play with colord. I just didn’t expect ConsoleKit or PolKit to join the party :)