It all started with this commit from Jordan Hubbard on August 21, 1994:
Commit my new ports make macros
Still not 100% complete yet by any means but fairly usable at this stage.
Twenty years later the ports tree is still there and actively
maintained. A video was prepared to celebrate the event and to thank
all of you who give some of their spare time and energy to the project!
Last month, our beloved Canadian Thomas Abthorpe decided to step
down from his portmgr-secretary position. While I suspect this is secretly
related to his pool of Canadian jokes having dried up, the official reason is
that Thomas wants to focus more on his private and professional lives for the
moment. Needless to say, the whole ports community is in mourning.
Thanks to the FreeBSD Foundation I was able to attend BSDCan 2014.
I arrived in Ottawa on Tuesday evening and went directly to the Royal Oak where I met other FreeBSD developers.
On Wednesday, the DevSummit started with the FreeBSD future plans where I was mainly interested in pushing subjects like packaging base, dma(8) integration, improvements in kqueue, and status of the toolchain.
The afternoon was mainly spent meeting with many other developers to talk face to face on subjects which usually take a while to resolve via mail.
Thursday started with the ports and package session where I talked about the status of the package distribution: from building packages to distributing packages on the FreeBSD cluster. I gave a brief status about pkg(8). We talked about the pkg_tools decomission. We had a long and interesting discussion about the future of the ports tree. The other subjects we talked about were packaging-base, continuous integration of the ports tree, cross building packages, and the license framework.
Like the previous day, I spent the afternoon discussing pkg(8) with other developers, as well as phabricator, and discussing with clusteradm about different possibilities for distributed "extra" packages repositories.
On Friday and Saturday the main conference took place. There were plenty of different interesting talks I went to.
The main interesting one for me was " The architecture of the new solver in pkg" by Vsevolod Stakhov as it gave me more details about his wonderful work on pkg during GSoC 2013!
This conference has been really succesful for me. It was the first time we were able to get 4 pkg developers together: Vsevolod Stakhov (vsevolod@), Bryan Drewery (bdrewery@), Matthew Seaman (matthew@), and myself. I found it really productive to exchange ideas, share problems, and simply have discussion.
This conference also allows me to talk with clusteradm people, in particular Glen Barber (gjb), Peter Wemm (peter@), and Sean Bruno (sbruno@)
There was also the opportunity for 4 portmgrs, a future portmgr, and a former portmgr to have an informal meeting which was really great!
The first day, Tuesday, was an unoffficial day, spent socializing.
The Developer's Summit began Wednesday. My main interest was to attend the "FreeBSD future plans" session. Of particular interest was the discussion about how Release Engineering should look in the future. The ports team has done a great deal of work to decouple ports releases from src releases. This required both changes in the way packages were built, as well as a substantial amount of new hardware to be able to build multiple package sets simultaneously. (Much of this hardware was purchased by the Foundation). This was the first change that many of the src and docs people had been brought up to speed on these developments.
Thursday, of course, my main interest was the Ports and Packages session.
In the evening, I was invited to an informal meeting with the various Ports Management Team (portmgrs) who were in attendance. (I had previously served for several years on this team.) Somehow, I was "volunteered" to rejoin the Ports Management team with an "advisor" status. Clearly, peer pressure works.
Friday the conference itself started. I spent some of the day trying to catch up on rest from the hectic first two days, and then socialized in the evening.
On Saturday, the most interesting session was the FreeNAS development talk. While it was informative, there was also an opportunity to heckle John Hixson.
Perhaps the most important task that I accomplished during the conference was to sit down with Bryan Drewery and discuss future software improvements to the Ports Monitoring System (portsmon), which I wrote.
portsmon has survived many changes in FreeBSD. The first was from CVS to SVN. More recently, the ports build farm has been switched over from the old portbuild codebase to a completely rewritten system. Our discussion dealt with the changes that I needed to make to port over to the new system; what the future changes to the new system would be; and changes that I requested that would make portsmon's job easier. These changes have now been incorporated. The next task is to catch up with the change from GNATS to Bugzilla; by that point, all of the inputs to portsmon will have been switched over from their initial codebase.
If everything was awesome and there were never IP fragments, all TCP and UDP frames would always have the TCP/UDP header stamped on them, and the NIC could hash the TCP/UDP header in hardware to calculate the destination queue to receive traffic on.
However, everything isn't awesome and there will be cases where IP frames are fragmented. When this happens, the first frame in the fragment has the IPv4 header and the TCP/UDP header - but the subsequent fragments only have the IPv4 header. That means there's not enough information in the rest of the fragments to hash them to the same hash value and thus hardware queue as the first fragment - only the first has the full IPv4+TCP/UDP information.
The Intel and Chelsio NICs will hash on all packets that are fragmented by only hashing on the IPv4 details. So, if it's a fragmented TCP or UDP frame, it will hash the first fragment the same as the others - it'll ignore the TCP/UDP details and only hash on the IPv4 frame. This means that all the fragments in a given IP datagram will hash to the same value and thus the same queue.
But if there are a mix of fragmented and non-fragmented packets in a given flow - for example, small versus larger UDP frames - then some may be hashed via the IPv4+TCP or IPv4+UDP details and some will just be hashed via the IPv4 details. This means that packets in the same flow will end up being received in different receive queues and thus highly likely be processed out of order.
The Linux intel driver code flipped off IPv4+UDP hashing a while ago - they hash UDP frames by their IPv4 details only and then do whatever other load balancing in the kernel they choose. I found this and updated the FreeBSD drivers to do the same. This should result in less out of order UDP frames for UDP heavy workloads. I'm not sure about the Chelsio driver yet - when I convert it to the RSS framework it'll disable IPv4+UDP hashing if that isn't enabled at boot time. This is a good stop-gap, but it's not the whole story.
TCP is where it gets annoying. People don't want to flip off IPv4+TCP hashing as they're convinced that the TCP MSS negotiation and path-MTU discovery stuff will prevent there from being any IP fragmented TCP frames. But, well, that's not really viable in the real world. There are too many misconfigured networks out there and IP fragmentation does occur. So this is also a problem for TCP. This means that the IPv4 fragmented TCP frames in those sessions will come into another receive queue and CPU and this will show up as out of order data.
So, what's this all have to do with receive side scaling?
With RSS, there's a well defined hash for packets and a configuration for what the operating system and NICs are supposed to be doing. It's entirely possible that we'll configure IPv4+TCP to be hashed and also entirely possible we'll see IP fragments showing up on other CPUs. So in order to have the TCP stack run on the right CPU, the IP fragments need to be assembled on whichever CPU they're received upon and then re-injected into the correct destination queue to run on the correct CPU.
Fortunately the FreeBSD netisr scheme makes this easy.
So what I'm doing in my branch (and what will soon show up in -HEAD) is thus:
- UDP is still hashed as IPv4-only frames for now. I'll change that later to hash on IPv4+UDP and have things reinjected on the correct destination RSS bucket / netisr queue / CPU.
- I create one netisr thread, pinned to a CPU, for each RSS CPU that's defined.
- Ideally I'd create one netisr thread for each RSS bucket and pin that, but that'll come later.
- IP fragments will be hashed to whatever the IPv4 hash calculates, so fragment reassembly will occur on some CPU;
- .. and it's the same CPU for all frames in a fragmented datagram.
- Then when the fragment is reassembled, a software hash is calculated for the newly reassembled frame.
- If RSS is configured to hash for IPv4 only, then it'll see that the hash on the reassembled datagram matches the configured hash for that packet type and reuse it.
- So, if it's UDP right now, it'll see that UDP is only hashing on IPv4 details and reuse it.
- .. but if IPv4+UDP hashing is configured, it'll software hash the packet and assign the new flow type and RSS hash.
- Then, it'll reinject the frame into netisr to be requeued and reprocessed.
- .. this uses the nh_m2cpuid function to calculate the destination CPU for the given RSS hash.
- If it's handled on the same destination CPU then it'll be handled.
- If it's handled on a different destination CPU then it'll be queued to that netisr and dispatched appropriately.
I’ve been playing with building ports for ARM on an AMD64 machine via a bunch of tools. The duct tape and bailing wire is a bit thick with this method, but if you keep at it, this should work.
1. build armv6 chroot:
make buildworld TARGET=arm TARGET_ARCH=armv6
make installworld TARGET=arm TARGET_ARCH=armv6 DESTDIR=/armv6
make distribution TARGET=arm TARGET_ARCH=armv6 DESTDIR=/armv6
2. build xdev
make xdev TARGET=arm TARGET_ARCH=armv6 NOSHARED=y
3. move xdev into chroot
mv /usr/armv6-freebsd /armv6/usr/
4. add toolchain to make.conf:
5. Install qemu-static-user from ports and copy into jail:
pkg instlal qemu-static-user
mkdir -p /armv6/usr/local/bin
cp /usr/local/bin/qemu-arm /armv6/usr/local/bin/
6. setup binmiscctl to handle armv6 translations:
binmiscctl add armv6 –interpreter “/usr/local/bin/qemu-arm” –magic “x7fx45x4cx46x01x01x01x00x00x00x00x00x00x00x00x00x02x00x28x00″ –mask “xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxfexffxffxff” –size 20 –set-enabled
7. mount devfs and ports if needed
mount -t devfs devfs /armv6/dev
mount -t nullfs /usr/ports /armv6/usr/ports
The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please fill out and submit the Travel Grant Request Application at http://www.freebsdfoundation.org/documents/TravelRequestForm.pdf by August 15th, 2014 to apply for this grant.
This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc). In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.
If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your direct request to us.
There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.
Recorded at BSDCan 2014.
File Info: 16Min, 8MB.
Ogg Link: http://cis01.uma.edu/~wbackman/bsdtalk/bsdtalk243.ogg
I’m very please to announce the release of pkg 1.3.0
This version is the result of almost 9 month of hard work
Here are the statistics for the version:
- 373 files changed, 66973 insertions(+), 38512 deletions(-)
- 29 different contributors
Please not that for the first time I’m not the main contributor, and I would
like to particularly thanks Vsevold Stakhov for all the hard work he has done to
allow us to get this release out. I would like also to give a special thanks to
Andrej Zverev for the tons of hours spending on testing and cleaning the bug
So much has happened that it is hard to summarize so I’ll try to highlight the
- New solver, now pkg has a real SAT solver able to automatically handle
conflicts and dynamically discover them. (yes pkg set -o is deprecated now)
- pkg install now able to install local files as well and resolve their
dependencies from the remote repositories
- Lots of parts of the code has been sandboxed
- Lots of rework to improve portability
- Package installation process has been reworked to be safer and handle properly
the schg flags
- Important modification of the locking system for finer grain locks
- Massive usage of libucl
- Simplification of the API
- Lots of improvements on the UI to provide a better user experience.
- Lots of improvements in multi repository mode
- pkg audit code has been moved into the library
- pkg -o A=B that will overwrite configuration file from cli
- The ui now support long options
- The unicity of a package is not anymore origin
- Tons of bug fixes
- Tons of behaviours fixes
- Way more!
Thank you to all contributors:
Alberto Villa, Alexandre Perrin, Andrej Zverev, Antoine Brodin, Brad Davis,
Bryan Drewery, Dag-Erling Smørgrav, Dmitry Marakasov, Elvira Khabirova, Jamie
Landeg Jones, Jilles Tjoelker, John Marino, Julien Laffaye, Mathieu Arnold,
Matthew Seaman, Maximilian Gaß, Michael Gehring, Michael Gmelin, Nicolas Szalay,
Rodrigo Osorio, Roman Naumann, Rui Paulo, Sean Channel, Stanislav E. Putrya,
Vsevolod Stakhov, Xin Li, coctic
Bapt on behalf of the pkg@