I have just shipped the new FreeBSD Forums box. It will be much nicer to have a more powerful box and lots more RAM to play with. Maybe even let us implement things like Varnish finally!
In a few days, I’ll be heading to the FOSDEM conference in Brussels again this year. On Saturday, you’ll most likely find me around the FreeBSD booth representing the FreeBSD Foundation, so if you’re there drop by to say hi, discuss the Foundation’s work, pick up a Foundation flyer, check out the swag, or make a donation. On Sunday, I’ll be in the BSD DevRoom where there will be some interesting presentations and discussions. Remember, FOSDEM is free to attend. Hope to see you there!
The most recent newsletter, which talks about what the foundation did the last half of last year.
Lastly, but probably most importantly, people can donate here:
File Info: 24Min, 12MB.
The most exciting news to report is that we raised $426,000 through our fundraising efforts. We were overwhelmed by the generosity of the FreeBSD community. We would like to thank everyone who made a contribution to FreeBSD by either making a financial donation to the foundation or volunteering on the Project.
We published our semi-annual newsletter in December. If you have not already done so, please take a moment to read this publication to find out how we supported the FreeBSD Project and community during the second half of 2011. There are also two great testimonials in the newsletter from TaxiMagic and the Apache Software Foundation.
The Foundation sponsored EuroBSDCon 2011 which was held in The Netherlands, October 6-9. And, we sponsored six developers to attend the conference. We sponsored the Bay Area Vendor Summit in November. We were represented at LISA '11, Dec 7-8 in Boston MA.
We are a proud sponsor of AsiaBSDCon 2012, which will be held in Tokyo, Japan, March 22-25.
The Foundation funded the completed Feed-Forward Clock Synchronization Algorithms Project by the University of Melbourne. We approved two new projects at the beginning of 2012: analyzing the performance of FreeBSD's IPv6 stack by Bjoern Zeeb, and implementing the auditdistd daemon by Pawel Jakub Dawidek
We purchased more servers and other hardware for the FreeBSD co-location centers at Sentex, NYI, and ISC.
The work above, as well as many other tasks which we do for the FreeBSD Project, could not be done without donations. Please help us by making a donation or asking your company to make a donation. We would be happy to send marketing literature to you or your company. Find out how to make a donation at our donate page.
Find out more up-to-date Foundation news by reading our blog and Facebook page.
Release of FreeBSD 9.0 Delivers More Power to Serve
Today, the FreeBSD Foundation announced the recent release of FreeBSD 9.0. FreeBSD 9.0-RELEASE raises the bar for open source operating systems in terms of file system reliability, IPv6-readiness, networking capabilities, compiler and toolchain technologies, and security. Many of its new features directly benefit system administrators, application developers, and companies that use or base their products on FreeBSD.
"FreeBSD 9.0 represents the culmination of over two years of ground-breaking work in operating system performance, reliability, and security," said Ken Smith, Release Engineer for the FreeBSD Project. "We are proud to dedicate this release to the memory of Dennis M. Ritchie, one of the founding fathers of the UNIXÂ® operating system, whose vision and work laid the foundations for FreeBSD."
Filesystem changes in this release provide great benefits to both UFS and ZFS users. When installing with UFS, softupdates journaling (UFS+SUJ) is automatically enabled. UFS+SUJ uses an intent log which safely eliminates the need for a long filesystem check and recovery process, even after an unclean shutdown.
ZFS has been updated to version 28 which supports data deduplication, triple parity RAIDZ3, snapshot holds, log device removal, zfs diff, zpool split, zpool import -F, and read-only zpool import.
FreeBSD 9.0 also introduces the Highly Available STorage (HAST) framework which provides transparent storage of the same data across several systems connected by a TCP/IP network. In combination with other high availability features of FreeBSD like the CARP fail-over protocol, HAST makes it possible to build a highly available storage cluster that is resistant to hardware failures.
Continuing its heritage of innovating in the area of security research, FreeBSD 9.0 introduces Capsicum. Capsicum is a lightweight framework which extends a POSIX UNIX kernel to support new security capabilities and adds a userland sandbox API. Originally developed as a collaboration between the University of Cambridge Computer Laboratory and Google and sponsored by a grant from Google, FreeBSD was the prototype platform and Chromium was the prototype application. FreeBSD 9.0 provides kernel support as an experimental feature for researchers and early adopters. Application support will follow in a later FreeBSD release and there are plans to provide some initial Capsicum-protected applications in FreeBSD 9.1.
"Google is excited to see the award-winning Capsicum work incorporated in FreeBSD 9.0, bringing native capability security to mainstream UNIX for the first time," said Ulfar Erlingsson, Manager, Security Research at Google.
FreeBSD has been been an early adopter and active participant in the IPv6 community since FreeBSD 4.0 was released in 2000 with the KAME reference implementation of IPv4/IPv6 networking support. In addition, the FreeBSD Project has been serving releases from IPv6-enabled servers for more than 8 years and FreeBSDâ€™s website, mailing lists, and developer infrastructure have been IPv6-enabled since 2007. FreeBSD 9.0 introduces IPv6-only snapshots which completely remove IPv4 from the operating system.
2012 has been called the 'year of IPv6' and "the FreeBSD project is well positioned to be one of the leaders in IPv6-Only validation work," stated Bjoern Zeeb, member of the FreeBSD Release Engineering Team and recipient of the 2010 Itojun Service Award for his significant improvements in open source implementations of IPv6. "The growing usage of FreeBSD's IPv6 networking stack by appliance builders, integration of a more flexible interface configuration, and the implementation of new standards such as Secure Neighbor Discovery, DNS Options for Router Advertisements, and CPE Requirements, makes FreeBSD 9.0 the perfect open source operating system to build your IPv6 deployments and products on."
Other new features include:
- userland DTrace has been added to supplement kernel-level DTrace
- the FreeBSD world and kernel can now be compiled using the BSD-licensed LLVM toolchain
- resource limit actions can be applied to processes, users, login classes, and jails
- the addition of a pluggable congestion framework and five new TCP congestion control algorithms
- HPN-SSH is enabled by default and increases transfer speeds on long, high bandwidth network links
- NFSv4 support added
- flattened device trees (FDT) allows for hardware resource enumeration and simplifies configuration on embedded platforms
About the FreeBSD Foundation
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting the FreeBSD Project and community. The Foundation gratefully accepts donations from individuals and businesses, using them to fund and manage projects, sponsor FreeBSD events, Developer Summits and provide travel grants to FreeBSD developers. In addition, the Foundation represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. The FreeBSD Foundation is entirely supported by donations. More information about The FreeBSD Foundation is available on the web.
About The FreeBSD Project
The FreeBSD Project provides an up-to-date and scalable modern operating system that offers high-performance, security, and advanced networking for personal workstations, Internet servers, routers, and firewalls. The FreeBSD packages collection includes popular software like the Apache web server, GNOME, KDE, X.org, Python, Firefox, and over 23,000 software suites. FreeBSD can be found on the Internet.
Yesterday I committed some more configs to generate doxygen documentation of FreeBSD-kernel drivers. I mechanically generated missing configs for subdirectories of src/sys/dev/. This means there is no dependency information included in the configs, and as such you will not get links e.g. to the PCI documentation, if a driver calls functions in the PCI driver (feel free to tell me about such dependencies).
IfÂ you want to generate the HTML or PDF version of some subsystem, just go to src/tools/kerneldoc/subsys/ an run â€œmakeâ€? to get a list of targets to build. As an example, â€œmake dev_soundâ€? will generate the HTML version for the sound system, â€œmake pdf-dev_soundâ€? generates the PDF version. The sound system is probably the most â€œniceâ€? example, as it includes a page with TODO items, and has even some real API docs instead of just the call-graphs and such automatically generated information.
There is more documentation than only for those drivers, I just listed those as there are at least parts of doxygen documentation inside.
Thanks to everyone that helped. Especially Andrew, Tim, and Daniel.
The commit is here: https://github.com/puppetlabs/puppet/pull/338
There is a huge discussion going on on hackers@ about how FreeBSD is not suitable for large installations (anymore?). As of this writing, the discussion seems to get some discussion-clusters. We have some sub-topics which could lead to some good improvements.
One subtopic is the release engineering. Some changes like a more guided approach of what should be merged to which branch, the frequency of releases and maybe some kind of long-term-branch(es). There is some discussion to get maybe some joined-funding in some way from interested parties to pay someone to take care about long-term-branch(es).
Another subtopic is the way bugs are handled in our old bugtracking software and how patches go unnoticed there.
And both of them are connected (parts more, parts less) by what can be done in a volunteer project.
To me it looks like the proposals â€œjustâ€? need some refinements and some â€œvolunteersâ€? to put value (this means man powerÂ and/or money) to what they said.
What I want to discuss here is, how tools could help with making PRs/patches more visible to developers (there is already the possibility to get emails from the small bugbuster-team about patches in PR database, but you have to ask them to get them) and how to make it more easy to get patches into FreeBSD.
Making bugs more visible to developers
The obvious first: We need a different bugtracking system. We already know about it. There is (or wasâ€¦) even someone working IIRC on an evaluation of what could be done and how easy/hard it would be. I am not aware of any outcome, despite the fact that it is months (or even a year) since this was announced. I do not blame anyone here, I would like to get time to finish some FreeBSD volunteer work myself.
In my opinion this needs to be handled in a commercial way. Someone needs to be officially paid (with a deadline) to produce a result. Unfortunately there is the problem that the requirements are in a way, that people do not have to change their workflows/procedures.
IIRC people ask that they should be able to send a mail to the bugtracker without the need for authentication. Personally I think the bugtracking issue is in a state where we need to change our workflows/procedures. It is convenient to get mails from the bugtracker and only have to reply to the mail to add something. On the other hand, if I report bugs somewhere, and if I really care about the problem resolution, I am willing login to whatever interface to get this damn problem solved.
Sending a problem report from the system where I have the issue in an easy way is a very useful feature. Currently we have send-pr for this and it uses emails. This means it requires a working email setup. As an user I do not care if the tool uses email or HTTP or HTTPS, I just want to have an easy way to submit the problem. I would not mind if I first have to do a â€œsend-problem register me@tldâ€? (once), â€œsend-problem login me@tldâ€? (once per system+user I want to send from) and then maybe a â€œsend-problem template write_template_here.txtâ€? (to get some template text to fill out), edit the template file and then run â€œsend-problem send my_report.txt file1 file2 â€¦â€?. That would be a different workflow, but still easy.
Email notifications are surely needed, but if I really care about a problem, I can be bothered to register first. So in my opinion, we need a different bugtracker desperately enough that we need to drop our requirements regarding our current workflow/procedures (even if it means we can not get a command line way of submitting bugs at all). The primary goal of the software needs to be to make it easy to track and resolve bugs. The submission of bugs shall be not hard too.Â If I look at the state of the world as it is ATM, I would say a webinterface with authentication is not a big burden to take if I really want to get my problem fixed. Some command line tool would be nice to have, but regarding the current state of our bugtracker it needs to be optional instead of a hard requirement.
Apart from making it easy to track and resolve problems, the software also needs to be able to make us aware of the biggest problems. Nowâ€¦ you may ask what is a big problem. Wellâ€¦ IMO it does not matter to you what I think is big or small here. The person with a problem needs to decide what is a big problem to him. And people with the same problem need to be able to tell that it is also a big problem for them. So aÂ feature which allows to â€œvoteâ€? or â€œ+1â€³ or â€œAOLâ€? (or however you want to call it) would allow to let users with problems voice their opinion upon the relevance of the problem to our userbase. This also means there needs to be a way to see the highest voted problems. An automatic mail would be best, but as above this is optional. If I as a developer really care about this, I can be bothered to login to a webinterface (or maybe someone volunteers to make a copy & paste and send a mailâ€¦ we need to be willing to rethink our procedures).
Getting patches more easy into a FreeBSD branch
It looks to me that this topic is requires a little bit more involvement from multiple tools. In my opinion we need to switch to a distributed version control system. One which allows to easily create my own branch of FreeBSD on my own hardware, and which allows to let other users use my branch easily (if I want to allow other to branch from my branch). It also needs to be able to let me push my changes towards FreeBSD. Obviously not directly into the official sources, but into some kind of staging area. Other people should be able to have a look at this staging area and be able to review what I did. They need to be able to make some comments for others to see, or give some kind of (multi-dimensional?-)rating for the patch (code quality / works for me / does not work / â€¦). Based upon the review/rating and maybe some automated evaluation (compile test / regression test / benchmark run) a committer could push the patch into the official FreeBSD tree (ideal would be some automated notification system, a push button solution for integration and so on, but as above we should not be afraid if we do not get all the bells and whistles).
If we would have something like this in place, creating some kind of long-term-release branch could be used more easily in a colaborative manner. Companies which use the same long-term-release branch could submit their backports of fixes/features this way. They also could see if similar branches (there could be related but different branches, like 9.4-security-fixes-only <= 9.4-official-errata-only <= 9.4-bugfixes <= 9.4-bugfixes-and-driverupdates <= â€¦) could be merged to their in-house branch (and maybe consequently push-back to the official branch they branched from if the patch comes from a different branch).
It does not matter here if we would create a fixed set of branches for each release, or if we only create some special-purpose branches based upon the phase of the moon (ideally we would create a lot of branches for every release, companies/users can cherry pick/submit what they want, and the status of a long-term-branch is solely based upon the inflow of patches and not by what the security team or release manager or a random developer thinks it should beâ€¦ but the reality will probably be somewhere in the middle).
I do not know if tools exists to make all this happen, or which tools could be put together to make it happen. I also did not mention on purpose toolsÂ I am aware of whichÂ already provide (small) parts of this. These are just some ideas to think about. Interested parties are invited to join the discussion on hackers@ (which is far away from discussing specific tools or features), but you are also free to add some comments here.
Just to make it officially official, now that FreeBSD 9.0 has been shipped, the ports slush state is now been lifted.
Ports committers are now entitled to perform sweeping commits. Keep in mind that -exp runs are always a good idea if you think there is a significant change to the ports tree.
And just remember, PLEASE TRY TO NOT BREAK THE INDEX!
on behalf of portmgr@
CTL is a disk and processor device emulation subsystem originally written for Copan Systems under Linux starting in 2003. It has been shipping in Copan (now SGI) products since 2005. It was ported to FreeBSD in 2008, and thanks to an agreement between SGI (who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is available under a BSD-style license. The intent behind the agreement was that Spectra would work to get CTL into the FreeBSD tree.
We spoke to Ken about the benefits of CTL and this is what he had to say:
CTL offers a number of benefits, but here are a couple of ways we can use it:
- It provides the missing piece needed to turn a FreeBSD box into an external RAID array. All you need is a Fibre Channel card (Qlogic 4Gb and 8Gb cards work now), and some disks, and with ZFS managing the disks and CTL providing the target interface, you've got an external RAID array. End users can do it, or companies can use it as the foundation for a FreeBSD-based storage appliance.
- CTL provides a test framework for CAM. When we implement new features and command support in CAM, we can immediately test out the new features in CTL. For instance, when I implemented SCSI descriptor sense support last year, I actually first implemented descriptor sense in CTL. That way, when I implemented it in CAM, I had a way to test everything out. I was able to test, through sense data injection, scenarios that would be impossible to trigger using an ordinary hard disk. You can't make a hard disk return every possible error and combination of errors, but with CTL, you can do that. So I was able to more fully test everything out and gain confidence that the descriptor sense code worked properly.
For people who want to test it, you don't need a Fibre Channel card, you can actually create LUNs and use CTL without any new hardware. With the CTL CAM SIM, the LUNs are visible on the internal 'ctl2cam0' bus. Here's what the output of camcontrol devlist -v looks like:
scbus6 on ctl2cam0 bus 0:
<> at scbus6 target -1 lun -1 ()
The da(4) driver is attached to the CTL LUNs, and you can do normal I/O to them just as you would any other disk.
One thing to watch out for is that if you use a block device (as opposed to a file or a ramdisk) as your backing store for CTL, you will need to disable synchronize cache support with ctladm realsync off. The reason is that g_dev_strategy() does not support the BIO_FLUSH command, and will panic with a KASSERT. That is something that needs to be fixed.
Files and ramdisks work fine without disabling flush support.
There is a brief description of how to create LUNs and enable ports in the CTL README.ctl.txt file in sys/cam/ctl, and the ctladm(8) man page also describes all of the options and provides some examples.
CTL is pretty functional, and should work well in most cases, but I am certainly interested in any feedback on it. The README has a to-do list, and I'm also planning on doing some performance optimizations as well.
One of the next things we need is more hardware support for various boards that support target mode. (e.g. other Fibre Channel, iSCSI and FCoE boards) It would also be nice to get CTL working with the Firewire target driver.
Registration for the expo is $10, or $70 if you would like to also attend the SCALE presentations.
The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please fill out and submit the Travel Grant Request Application by February 20, 2012 to apply for this grant.
This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc). In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.
Your request should be based on a realistic and economical estimate of travel costs (economy airfare, trainfare, ...), accommodations (conference hotel and sharing a room), and registration or tutorial fees. If there are other sponsors willing to cover costs, such as your employer or the conference, we prefer that you talk to them first, as our budget is limited. We are happy to split costs with you or another sponsor, such as just covering airfare or board.
If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your request.
If your application is approved, we will authorize you to seek reimbursement up to a limit. We consider several factors, including our overall and per-event budgets, and the benefit to the community by funding your travel. We reimburse costs based on receipts, and by check or bank transfer. And, we do not cover your costs if you end up having to cancel your trip. We require you to submit a report on your trip, which we may show to current or potential sponsors, and may include in our semi-annual newsletter or this blog.
There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.
The FreeBSD audit facility provides fine-grained, configurable logging of security-relevant events. One of the key purposes of logging security events is postmortem analysis in case of system compromise. Currently the kernel can push audit records directly into a file or make them available through the /dev/auditpipe device. Because audit logs are stored locally by the kernel, an attacker has access to them once the system is compromised, which enables him to remove trails of his activity.
The goal of the auditdistd project is to securely and reliably distribute audit records over the TCP/IP network from a local auditdistd daemon to a remote auditdistd daemon. In case of source system compromise, the attacker's activity can be analysed using data collected by the remote system, as only the remote system's audit logs can still be trusted.
The project will conclude in February 2012.
Last year, Bjoern improved FreeBSD IPv6 support, allowing the possibility to build a FreeBSD system without IPv4 support. This project will continue on this work and concentrate on the kernel, looking at the performance of FreeBSD's IPv6 stack. Various parties have seen lower performance when comparing IPv4 to IPv6 on FreeBSD. While the numbers seem to differ between releases the causes are mostly unknown.
The project will carry out a detailed performance analysis starting with benchmarking IPv6 to IPv4 to get up-to-date numbers to better understand where we are. It will then continue to identify the origins of differences in performance, and where possible, directly address them or identify areas of future work. Having initial benchmark numbers will allow changes to be evaluated by re-running the measurements and quantifying the improvements.
"As the world starts to roll out IPv6 and traffic patterns shift from IPv4 to IPv6, not only correctness and stability, but also feature parity and performance matter," said developer Bjoern Zeeb. "Getting the performance numbers aligning with IPv4 will ensure that our users will not need more resources when using IPv6."
"ISC uses FreeBSD extensively across our server infrastructure and have provided IPv6 services to the community since 2002," commented Peter Losher, ISC Sr. Operations Engineer. "We are excited to support The FreeBSD Foundation and Bjoern's efforts to improve IPv6 performance in FreeBSD."
Bjoern Zeeb is a consultant based in Germany and has been an active FreeBSD committer since 2004. He is currently also a member of the FreeBSD Security and Release Engineering teams.