new job!

Yesterday was my first day working at Broadcom. I've taken on a new role as an open source developer there. I'm going to be working on building an MIT-licensed Mesa and kernel DRM driver for the 2708 (aka the 2835), the chip that's in the Raspberry Pi.

It's going to be a long process. What I have to work with to start is basically sample code. Talking to the engineers who wrote the code drops we've seen released from Broadcom so far, they're happy to tell me about the clever things they did (their IR is pretty cool for the target subset of their architecture they chose, and it makes instruction scheduling and register allocation *really* easy), but I've had universal encouragement so far to throw it all away and start over.

So far, I'm just beginning. I'm still working on getting a useful development environment set up and building my first bits of stub DRM code. There are a lot of open questions still as to how we'll manage the transition from having most of the graphics hardware communication managed by the VPU to having it run on the ARM (since the VPU code is a firmware blob currently, we have to be careful to figure out when it will stomp on various bits of hardware as I incrementally take over things that used to be its job).

I'll have repos up as soon as I have some code that does anything.

PC-BSD at SouthEast LinuxFest

SouthEast LinuxFest will take place June 20–22 at the Sheraton Charlotte Airport in Charlotte, NC. Registration for this event  is free.

There will be a BSD booth in the expo area on both Friday and Saturday from 9:00–17:00. As usual, we’ll be giving out a bunch of cool swag, PC-BSD DVDs, and FreeNAS CDs, as well as accepting donations for the FreeBSD Foundation. Ken Moore will present “PBI v10: Application Management Made Easy” at 13:30 on Friday and Dru Lavigne will present “ZFS 101″ at 13:30 on Saturday. The BSDA certification exam will also be held at 11:00 On Sunday.

10.0.2-RC2 Available for Testing

PC-BSD 10.0.2-RC2 images are now online for testing from our download site.

This will (hopefully) be our last RC before releasing 10.0.2 officially sometime on or around the 23rd. We have addressed or fixed most tickets related to the 10.0.2 release, so if you are still running into any issues, please report them using our Trac database.

Users running EDGE or earlier 10.0.2 images can upgrade their packages to the RC2 versions via AppCafe or Package Manager.

Thanks for all your help testing, and the issues reported so far!

FreeBSD 9.3-BETA3 Now Available

FreeBSD 9.3-BETA3 Now Available


The third BETA build of the 9.3-RELEASE release cycle is now available on the FTP servers for the amd64, i386, ia64, powerpc, powerpc64 and sparc64 architectures.

The image checksums can be found in the PGP-signed announcement email.

ISO images and, for architectures that support it, the memory stick images are available here:

    http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.3/

(or any of the FreeBSD mirror sites).

If you notice problems you can report them through the normal GNATS PR system or on the -stable mailing list.

Please note, as the FreeBSD bug tracking system is undergoing maintenance, the PR system may be unavailable.  Problem reports submitted this maintenance period are being queued for later processing.

If you would like to use SVN to do a source based update of an existing system, use the "stable/9" branch.

A list of changes since 9.2-RELEASE are available on the stable/9 release notes page here:


Changes between 9.3-BETA2 and 9.3-BETA3 include:
  • A new ttys(5) flag, onifconsole, has been added, which activates ttyu0 if the device is an active kernel console.
  • The NFSv4 server now allows creating a hard link to a symbolic link, as was allowed in NFSv3.
  • OpenSSL has been updated to 0.9.8za.
  • A deadlock caused by incorrect reference counts has been fixed in the usb(4) driver.
  • The arc4random(3) library has been updated to match that in FreeBSD-CURRENT.
  • The amount of data collected by hwpmc(4) has been increased to work with modern processors and available RAM.
  • A new pmcstat(8) flag, '-l', has been added, which ends event collection after the specified number of seconds.
The freebsd-update(8) utility supports binary upgrades of amd64 and i386 systems running earlier FreeBSD releases.  Systems running earlier FreeBSD releases can upgrade as follows:

    # freebsd-update upgrade -r 9.3-BETA3

During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly.

    # freebsd-update install

The system must be rebooted with the newly installed kernel before continuing.

    # shutdown -r now

After rebooting, freebsd-update needs to be run again to install the new userland components:

    # freebsd-update install

It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 8.x.  Alternatively, the misc/compat8x port can be installed to provide other compatibility libraries, afterwards the system must be rebooted into the new userland:

    # shutdown -r now

Finally, after rebooting, freebsd-update needs to be run again to remove stale files:

    # freebsd-update install

Love FreeBSD?  Support this and future releases with a donation to the FreeBSD Foundation!

FreeBSD 9.3-BETA3 Available

The third BETA build for the FreeBSD-9.3 release cycle is now available. ISO images for the amd64, i386, ia64, powerpc, powerpc64 and sparc64 architectures are available on most of our FreeBSD mirror sites.

BSDCan Trip Report: Li-Wen Hsu

The next trip report is from Li-Wen Hsu:

I am very excited for having the chance to join the most important and largest annual BSD conference. Thanks to the FreeBSD Foundation, it's my first year to attend BSDCan. The main motivation for attending is that I'm in part of the project started by Craig Rodrigues, Jenkins CI for FreeBSD, and and I am honored to be invited join that group.
 

I arrived in Ottawa on May 13th. After checking into Residence and taking a short nap to ease jet-lag, I went to the Royal Oak Pub to join the pre-party of the developers. Sean Bruno quickly recognized me and introduced me to other developers. I talked with Steve Wills, Mark Linimon, Gavin Atkinson, and met Peter Wemm, my roommate.
 

The first day of the Developer Summit started with presentations about changes to the support plan and brainstorming about FreeBSD 11. During the break, I spoke to Mark Johnston, who completed the last piece of axge(4), our first USB 3.0 to gigabit ethernet adapter driver. It is written mainly by Kevin Lo and I provided some fixes in rx/tx routines. During our chat, we discussed the performance issues of axge(4) where he discovered there might be a limitation of calling rx/tx routines numbers per second in the USB stack. This is done by just a few lines of DTrace code. I was totally shocked by that and decided that I should learn more about it.
 

In the afternoon, I joined The Java working group where Greg Lewis introduced the history, current status, and we discussed the future plans of the Java port. We talked about how to improve user experience, support for important Java software, and the known problems of Java on FreeBSD. There was also a discussion on how to to get more developers who want to develop Java applications on FreeBSD. We think that DTrace support might be attractive for people who run Java on FreeBSD.

We had Thai food in the Hacker Lounge for dinner. George V. Neville-Neil and I talked about how to make more people develop or support their software on FreeBSD. I showed him Travis CI which is used by many open source projects developed on GitHub for their continuous integration needs. However, it cannot support FreeBSD in the near future. Being a ports committer for several years, I feel that many projects are willing to support FreeBSD, but lack the environment and experience. I think we should work more with external communities to address this. We discussed the possibility of establishing such a service for FreeBSD, following the successful model of RedPorts. There are many tricky parts and security issues to consider. Furthermore, the most important part is manpower. If any reader is interested in helping, please contact me.
 

On the second day of the Developer Summit, I went to the Continuous Integration and Testing with Jenkins for FreeBSD working group in the morning. In the first part of the meeting, Craig introduced Jenkins and how it is utilized in the FreeBSD.org cluster. He also covered the internal architecture of jenkins.FreeBSD.org. In the second part of the meeting, we discussed the next steps to work on this year. Craig and I helped Julio Merino set up a Jenkins instance on his laptop and Julio quickly hacked Kyua to let it generate JUnit output which can be parsed by Jenkins. This is very exciting to us because it means that we can have a trackable and easy-to-read continuous integration report. We believe this can help developers and contributors to produce higher quality code and to find items they can start to work on.
 

In the afternoon, I joined the Documentation Translation System Session where Benedict Reuschling introduced a process to translate documentation just like using gettext for software i18n and l10n. This process is done by translating docbook XML files to .po files with po4a, then translators can use their favorite po file editor to concentrate on the content instead of struggling with non-human readable XML files. It is also possible to establish a "translation memory" to remember the phrases and sentences that have already been translated for sharing between documents, which reduces duplicated work from translators. We also talked about a wish: one web-based system where casual translators can fix a translation by clicking a mouse while the backend takes care of the rest. The doc committers or another contributor can commit the change back to the doc repository.
 

Finally, I asked about continuous integration for the doc tree. Warren Block suggested that we can run igor for checking the errors, however there are some false positives that would bother people. During BSDCan, I joined two of the Doc Sprints. One night I asked Warren about "safe parameters" for igor and I quickly hacked igor to generate the checkstyle format XML output and pass it to the Jenkins checkstyle plugin. I presented the proof of concept on the second night. It is really great that people thought it is useful and encouraged me to setup it as a job on jenkins.FreeBSD.org. Warren will help me with this. In the future, this could also integrate with Phabricator as a "lint" tool for being as a filter.
 

I always wanted to revive the Traditional Chinese Document Translation Project. Fortunately, about two weeks after BSDCan, a volunteer sent a mail to the freebsd-doc mailing list stating that he wants to contribute to the Traditional Chinese translation of the FreeBSD Handbook. After discussing with him and with the help from people on EFNet/#bsddocs and doceng, I converted zh_TW from Big5 to UTF-8 in the doc tree for making future translation easier. This is really a good restart and I hope more people can join and we can have a complete Traditional Chinese handbook and other documents soon.
 

The following two days were the BSDCan sessions. The starting keynote speaker was Karl Lehenbauer from FlightAware. The most rememberable might be the slide "A billion dollars + Linus <= good people with a rigorous engineering process doing BSD." Also, there are some slides about FreeBSD tuning at FlightAware.
 

In Luigi Rizzo's talk In-kernel OpenvSwitch on FreeBSD, I learned a lot about how to port the Linux kernel subsystem to FreeBSD. In his work, he provides netlink sockets for the FreeBSD kernel. This is very good news because I always heard that people who are familiar with Linux want this feature.
 

Patrick Kelsey gave the libuinet talk. This is used to port the FreeBSD TCP/IP stack to userland. It means that the resource needed for a connection is in userspace memory and the kernel only needs to provide a packet interface, such as netmap. This is useful for an application that handles lots of concurrent connections. Patrick became a committer recently and I hope more work from him can bring libuinet closer to HEAD soon.
 

Pawel Jakub Dawidek and Mariusz Zaborski talked about their work on Capsicum and Casper. In this talk, they presented the lack of traditional security mechanism (setuid(2), chroot(2), P_SUGID, setrlimit(2)) provided by the system, and how an easier protection is provided by Capsicum. The Casper daemon provides services to sandboxed processess which do not have the necessary rights.
 

The FreeBSD Developer Summit and BSDCan overlap for one day. May 16 is the public track of the dev summit and I attended two sessions. Michal Dubiel from Semihalf gave the status update of OpenStack and OpenContrail on a FreeBSD host. I am glad that there are companies which invest in cloud technologies for FreeBSD. I hope this can be in production soon and maybe the FreeBSD cluster can have some setup to enrich developer resources. Another session I attended is lightweight reference counting, by Gleb Smirnoff. Using counter(9) in FreeBSD 10 as a reference count is really a brilliant idea. I am looking forward to seeing performance improve with this solution.
 

Arun Thomas gave a good tutorial of ARM and how BSD supports it. For a person like myself with no experience in embedded systems, it was a good start. Now I can have more fun with my Raspberry Pi.
 

Julio Merino talked about The FreeBSD Test Suite, which is really important to me and the FreeBSD continuous integration group. He also announced the plan to combine Kyua and Jenkins in the session. He hopes that the tests can be more complete and the CI pipeline can be more mature. There are still lots of things to be done!
 

Matt Ahrens presented the goals of the OpenZFS project and its current status. With this project, platforms like Illumos, FreeBSD, Linux, and OS X can directly interact with a shared, platform-independent ZFS code base. This will greatly reduce the effort to port changes between platforms and the tests can also be shared. The future of the OpenZFS features are amazing, including resumable zfs send/recv and device removal, which can make a system administrator's€™ life much easier.
 

The ASLR talk by Shawn Webb was awesome and this is definitely a feature that a paranoid person like myself  hope will be merged to the main trunk soon. It seems to still have problems on ARM so he also asked for help from people with ARM experience.
 

On return home, I was surprised that Peter and I were on the same flight and sat next to each other. We talked about Linux containers and the projects that use it like Docker, which is the part that FreeBSD is not doing well. Currently, the resource limitation of the lightweight containers is not really complete. He said that the way we using servers, or the "computing nodes", are changing in lightspeed and we should not be left behind. We both agree that a modern operating system should put more effort in cloud and mobile solutions.
 

I would like to thank the FreeBSD Foundation again for sponsoring me to attend this great event. I made new friends and met people who only know each other on the Internet before. We shared many good ideas and it is really awesome to know there are so many people working on FreeBSD. I hope I can participate again next year!

PC-BSD at Texas LinuxFest

There will be a BSD booth at Texas LinuxFest to be held June 13 and 14 at the Austin Convention Center in downtown Austin, Texas. Several members of the PC-BSD team will be at the booth and will be giving out cool swag, PC-BSD DVDs, FreeNAS CDs, and accepting donations for the FreeBSD Foundation. Registration for this event is $30 or $60 and the expo area is open on Friday from 12:00–18:00 and on Saturday from 10:00 to 18:00.

On Saturday, Josh Smith will present  Introduction to PC-BSD 10.0 and Dru Lavigne will present Graphical ZFS Management Tools.

The BSDA certification exam will also be available on Friday at 10:00. The cost of the exam is $75.

10.0.2-PRERELEASE ISO Available for Testing

The next 10.0.2-PRERELEASE ISO is now available for testing and can be downloaded from
http://download.pcbsd.org/iso/10.0-RELEASE/testing/amd64/.

If you have a spare system or virtual machine, consider testing this image. If you find any bugs, report them at https://trac.pcbsd.org so we can take a look at fixing them before 10.0.2 is released later this month.

NOTE: if you plan to use AppCafe in this image, go to Configure -> Repository Settings and change it to “Edge”. Do this before attempting to upgrade within AppCafe; otherwise, if you reboot or logout, you will not be able to successfully log back in again.

Hacking on Receive Side Scaling (RSS) on FreeBSD

RSS is a Microsoft invention that tries to keep a given TCP or UDP flow (and I think IP, but I haven't yet tried that) on a given CPU core. The idea is to try and keep both flow-local data and flow-local locking on a single CPU core, increasing the chances that data is hot in the CPU core cache and reducing the chance of lock overhead.

You can find the RSS overview and programming details here:

http://msdn.microsoft.com/en-us/library/windows/hardware/ff567236(v=vs.85).aspx

RSS and supporting technology has been making its way into FreeBSD for quite some time but it's not in any real shape that application developers can take advantage of.

Firstly, there's "PCBGROUPS", which looks to group PCB (protocol control block) data for a connection local to a CPU. Instead of there being one global PCB table for the system (well, VIMAGE for FreeBSD - each virtual image instance has its own PCB table) with one lock protecting it, there's now multiple PCB tables, one per "thing". Here, the thing is whatever the kernel developer thinks is worth grouping them by.

http://www.ece.rice.edu/~willmann/pubs/paranet_usenix.pdf

Now, until the RSS work went in, this code was in FreeBSD but sat unused. A kernel developer could provide the hooks needed to map TCP (and maybe UDP later) flows to a "thing" and have that map to a PCB group table - but it required some glue to stamp incoming connections and outgoing packets with some identifier (which we call a "flowid" in FreeBSD) with something that can map to said "thing". Then whenever a PCB lookup was needed, it would first try the lookup in the table mapped to by the mapping between the "flowid" and "thing" - if it was successful, it wouldn't have to use the global PCB table to do the lookup.

This is only good for established connections - creating and destroying a connection still requires manipulating that global PCB table and the single PCB table lock. I'm going to ignore fixing that for now, as that is a bigger issue.

Then Robert Watson added the RSS work done under contract to Juniper Networks, Inc. RSS provides one kind of mapping between the flowid from the NIC and which CPU to run work on. So that part worked great - but there wasn't any way for the application user to take advantage of it. Additionally, there's no driver awareness of it yet - I'll discuss this shortly.

So I grabbed a bunch of this work whilst at Netflix and tried to make sense of it. It turns out that if you can keep the work local to a CPU, a lot of the lock contention in the networking stack melts away. Here's what's going on:

  • The receive thread(s) in the NIC driver processing packets are typically doing direct dispatch to the network stack - so they're running the receive side of the TCP stack;
  • .. and the receive side of the network stack includes ACKs, which triggers the transmit side of the network stack;
  • There's typically some deferred thread(s) in the NIC driver transmitting packets to each NIC queue;
  • There's also application threads trying to queue data to the TCP socket, which also can dig into the socket and TCP stack state, which involves grabbing locks;
  • And there's also timers firing to update state, and doing this involves grabbing locks.
Without RSS and without lining everything up on CPU cores, all the above can run on different cores. Whenever any of them try running at the same time, lock contention can occur and that particular task can stop. If the lock contention blocks the transmit or receive NIC threads, then not only is that connection affected - the whole NIC processing is affected.

There's still lock contention in the network stack - especially if you're doing a lot of new, short connections. The good folk at Verisign are working on that particular corner of the problem so I'm happy to defer to them.

So, I ended up doing a bunch of little pieces to get this lined up right:
  • The per-CPU timer callwheels can now be optionally pinned to their CPU cores, so timer events running on CPU X actually do run on CPU X (yes, that was amusing to find..);
  • There's support in the TCP stack for per-CPU timers, but it's not enabled by default;
  • ... and it also didn't query RSS, netisr or anything to figure out how to map a flowid to a given CPU to run a timer on;
  • Then to make matters worse, incoming TCP sessions didn't have a flowid assigned to the PCB until after the first data packet was read - which meant that the initial timer work would all assume CPU 0 and any queries on that particular PCB would return flowid=0 - so it would not find it in the right PCBGROUP.
So those are fixed in FreeBSD-HEAD. The per-CPU TCP timer and pinned-CPU timers aren't enabled by default - I'll only flip that on when I'm confident that the RSS stuff is working.

So that lets all the RSS stuff correctly work. But there wasn't a nice way to query the per-connection flowid or RSS information. So I then extended netstat to have 'R' as a flag - it returns the flowid and the flowid type. I'll add RSS information once I have a nice way to extract it out in bulk. It's still a good diagnostic tool to ensure that the IPv4/IPv6 hashing is working correctly.

Then I had to teach a driver about RSS so I could actually test it all out. I have some igb(4) hardware at home, so I did the minimal work required to teach it about the RSS key and assigning things to the correct CPUs. It's still incomplete but it's good enough to get off the ground. I'll go into more details about the driver requirements in a follow-up blogpost.

Finally, how are application developers supposed to use it? I'll cover that particular bit in another follow-up blog post as there's quite a lot to cover there.

FreeBSD 9.3-BETA2 Available

The second BETA build for the FreeBSD-9.3 release cycle is now available. ISO images for the amd64, i386, ia64, powerpc, powerpc64 and sparc64 architectures are available on most of our FreeBSD mirror sites.

Playing nice with others. git(1) and patches on #FreeBSD

I’ve been spending a lot of time massaging a branch of patches and other assorted bits and pieces for QEMU user mode on github

This led me down the path of being a good git user and contributor, so I’ll leave these notes for myself and others in the event you come into a situation where you need FreeBSD to play nice with people who are very git(1) centric.

After an update by [email protected] to the devel/git port, you can now install git(1) and have it work out of the box.  The most frustrating thing, after using git for like 5 minutes, is to figure out how to extract a patch out of it and send it all pretty-like to the mailing list(s) that would be consuming the patch.

In its simplest incarnation, you can simply reference a commit hash and us it to generate a patch via git format-patch, but this will give you the entire commit diff between the referenced version and HEAD.  This, in my case generated approximately 3000 patch files.

e.g. git format-patch –output-directory ~/patches –to=”[email protected]” c60a1f1b2823a4937535ecb97ddf21d06cfd3d3b

What I want, is a diff of one revision, which requires a start and ending hash:

format-patch –output-directory ~/patches –to=”[email protected]” c60a1f1b2823a4937535ecb97ddf21d06cfd3d3b…c6ad44bb288c1fe85d4695b6a48d89823823552b

Now I send this to the mailing lists via my client.  Here is where I kind of head-desked a bit.  If you are like me and run a mail server yourself and you use SSL with self-signed certs, then this little bit if for you.  I lost about an hour trying to figure this little bit out.

The way to dump patches from your patch director (~/patches) is to use:

git send-email patches/*

This will use the following variables in your git environment:

sendemail.smtpserver=mail.ignoranthack.me
sendemail.smtpencryption=ssl
sendemail.smtpuser=[email protected]
sendemail.smtpserverport=465
sendemail.smtpsslcertpath=
sendemail.annotate=yes

Notice the empty “sendemail.smtpcertpath” variable.  Without that set to EMPTY, git would repeatedly fail on the self-signed cert that I use.  So, I’m pretty sure something still isn’t setup correctly.  However, it must be set to EMPTY and not undefined.  Else, you will repeatedly fail with certificate validation errors.

Getting to know your portmgr@ – Steve Wills

It is my pleasure to introduce Steve Wills, the newest member of the portmgr team. Steve has done a tremendous work on the ports tree, especially in the field of testing and quality. Here is a short interview to get to know him better.

Name
Steve Wills

Committer name
swills

Inspiration for your IRC nick
Boring, it’s my userid.

TLD of origin
.us

Current TLD (if different from above)
Same.

Occupation
Sysadmin.

When did you join portmgr@
2014

Blog
Used to have one, use twitter more now (@swills)

Inspiration for using FreeBSD
Simplicty and learning.

Who was your first contact in FreeBSD
Can’t recall, it was ages ago.

Who was your mentor(s)
pgollucci

What was your most embarrassing moment in FreeBSD
Trying to migrate Ruby default version from 1.8 to 1.9 and having to roll back.

Boxers / Briefs / other
Heh, question assume survey taker is male, which I am, but I think we need to
work on diversity (but not in that “hey, let’s work on diversity and get some
women” way, but more in that we make something everyone wants to use)

What is your role in your circle of friends
The FreeBSD user. ;)

vi(m) / emacs / other
vi(m)

What keeps you motivated in FreeBSD
New users, new committers.

Favourite musician/band
I listen to a decent variety of stuff, but I suppose the thing I come back to
most is NIN.

What book do you have on your bedside table
I have an iPad by my bed, which I bought to read, but mostly I browse news on
it.

coffee / tea / other
Don’t drink caffeine, so don’t drink coffee much. I do drink good beer tho.

Do you have a guilty pleasure
Good dark chocolate. :)

How would you describe yourself
Mostly standard in many ways, husband, father, FreeBSD hacker, sysadmin, in
that order.

sendmail / postfix / other
Sendmail, tho dma is nice too.

Do you have a hobby outside of FreeBSD
Used to play guitar, still have one, don’t find time to pick it up much any
more.

What is your favourite TV show
Futurama

Claim to Fame
Ported Acidwarp from DOS to svgalib.

What did you have for breakfast today
Everything bagel with plain cream chese.

What sports team do you support
The only sport I watch is University of North Carolina Basketball.

What else do you do in the world of FreeBSD
ruby ports, perl ports sometimes

What can you tell us about yourself that most people don’t know
I was an employee at Red Hat way way back

Any parting words you want to share
Not really.

What is your .sig at the moment
Null.

Steve

FreeBSD 9.3-BETA1 Now Available

FreeBSD 9.3-BETA1 Now Available

The first BETA build of the 9.3-RELEASE release cycle is now available on the FTP servers for the amd64, i386, ia64, powerpc, powerpc64 and sparc64 architectures.

The image checksums can be found in the PGP-signed announcement email.

ISO images and, for architectures that support it, the memory stick images are available here:

    http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/9.3/

(or any of the FreeBSD mirror sites).

If you notice problems you can report them through the normal GNATS PR system or on the -stable mailing list.

Please note, as the FreeBSD bug tracking system is undergoing maintenance, the PR system may be unavailable.  Problem reports submitted this maintenance period are being queued for later processing.

If you would like to use SVN to do a source based update of an existing system, use the "stable/9" branch.

A list of changes since 9.2-RELEASE are available on the stable/9 release notes page here:


The freebsd-update(8) utility supports binary upgrades of amd64 and i386 systems running earlier FreeBSD releases.  Systems running earlier FreeBSD releases can upgrade as follows:

    # freebsd-update upgrade -r 9.3-BETA1

During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly.

    # freebsd-update install

The system must be rebooted with the newly installed kernel before continuing.

    # shutdown -r now

After rebooting, freebsd-update needs to be run again to install the new userland components:

    # freebsd-update install

It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 8.x.  Alternatively, the misc/compat8x port can be installed to
provide other compatibility libraries, afterwards the system must be rebooted into the new userland:

    # shutdown -r now

Finally, after rebooting, freebsd-update needs to be run again to remove stale files:

    # freebsd-update install

Love FreeBSD?  Support this and future releases with a donation to the FreeBSD Foundation!  https://www.freebsdfoundation.org/donate/