Monthly Archives: March 2012

Generating all combinations of n elements taken t at a time

I don’t usually share the code I write for my research projects, unless someone asks for it. Anyway, today I had fun writing some C code, and I think it may turn out to be useful for other people. It’s written almost well enough to be acceptable, so here you are.

I implemented Chase’s sequence algorithm to generate the bit-strings corresponding to each combination of n elements taken t at a time. The algorithm is described in Donald E. Knuth’s “The Art Of Computer Programming”, Vol. 4, Fasc. 3, page 13. I implemented it in C and added very few comments, so refer to TAOCP if you have any doubts.

The code is distributed under the BSD 2-clauses License.

Download chaseseqcombgen.c

Posted from Providence, Rhode Island, United States.

Accepting Travel Grant Applications for BSDCan 2012

Calling all FreeBSD developers needing assistance with travel expenses for BSDCan 2012.

The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please fill out and submit the Travel Grant Request Application by April 6, 2012 to apply for this grant.

This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc). In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.

(1) You request funding based on a realistic and economical estimate of travel costs (economy airfare, trainfare, ...), accommodations (conference hotel and sharing a room), and registration or tutorial fees. If there are other sponsors willing to cover costs, such as your employer or the conference, we prefer you talk to them first, as our budget is limited. We are happy to split costs with you or another sponsor, such as just covering airfare or board.

If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your direct request to us.

(2) We review your application and if approved, authorize you to seek reimbursement up to a limit. We consider several factors, including our overall and per-event budgets, and (quite importantly) the benefit to the community by funding your travel.

Most rejected applications are rejected because of an over-all limit on travel budget for the event or year, due to unrealistic or uneconomical costing, or because there is an unclear or unconvincing argument that funding the applicant will directly benefit the FreeBSD Project. Please take these points into consideration when writing your application.

(3) We reimburse costs based on actuals (receipts), and by check or bank transfer. And, we do not cover your costs if you end up having to cancel your trip. We require you to submit a report on your trip, which we may show to current or potential sponsors, and may include in our semi-annual newsletter.

There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.

Call for Testers: Upcoming LibreOffice Port with Localizations

Many PC-BSD users have been looking for localized versions of the LibreOffice PBI (which, like the underlying FreeBSD port, is currently only available in English). The FreeBSD LibreOffice porting team is working on the next version of the port which will be localized.

Baptiste Daroussin from the porting team describes the current state of the upcoming port in this mailing list post.

If you are comfortable compiling FreeBSD ports, have been looking for an excuse to try compiling with clang, and would like to assist with the testing of the upcoming port, read that mailing list post first as it contains technical details that you will need to get started. Note that the provided Makefiles are already configured to use clang, meaning that you can type make as usual.

FreeBSD Activity++

Lately I have become more active then the last year in total for FreeBSD. I committed several enhancements that were in my queue already for a long time, but finally came into the FreeBSD tree. Some too late for 8.3 which is upcoming and some not.

I also understand merging much better then before, taking the time for a commit and making a few mistakes really helps, and ofcourse the community is not too shy to mention my faults :-)

One of the things that got committed is the force setting of carp, whether it’s a master or backup node (make sure pre-empt is disabled if you do this, else it will just rollback within seconds :-)), which eases maintenance for example. One of the other things is that I am using an extensive network of local jails now that service my needs, most seperated items. I also started writing an Howto (or bsdmag article if there is interest) to demonstrate and tell how we setup most items. One of the things that is still causing me headaches is that we have nullfs rw mounts of /home to the jails because mailservers need to write, ftp servers need to write etc. But if we were sharing these information sources via NFS we would have had the same challenges :-)

I feel good in taking the time for FreeBSD again, and I would like to hear recommendations on what targets I can persue in FreeBSD (low hanging fruit is good enough for the time being) and additional things, also please comment on the nullfs mounts (rw,nosuid) to enhance my security level and which makes my article even better :)

Foundation at NELF

NorthEast Linux Fest will be held in Worcester, MA on March 17. Registration is free for this event and there will be a FreeBSD booth. If you're in the area, drop by the booth to say hi, check out the Foundation swag, discuss funded projects, or make a donation to the Foundation.

2012 Budget

The Foundation's 2012 budget has been published and is available as a PDF download. The total budget is $595,350, more than double the 2011 budget. Highlights include:

  • $125,850 allocated for FreeBSD infrastructure equipment
  • $60,000 allocated for conference sponsorships and travel grants
  • $200,000 allocated for funded projects
Much of the work that was sponsored in 2011 was integrated into FreeBSD 9.0, greatly enhancing its features. We look forward to the work that will occur in 2012 and welcome project proposals. If you are interested in submitting a proposal, see the Project Proposal Procedures and Project Proposal Help on the documents page of the website.


Testimonial: Experts Exchange

Andrew Alsup, Site Director of Experts Exchange recently provided the following testimonial:

Experts Exchange is a technology help site. We've developed a patented Q&A system that makes finding solutions easier. People ask questions and experts from around the world provide reliable answers.

Since 2009, Experts Exchange has used FreeBSD as its operating system of choice for all external facing web servers and internal tools hosting.

About 3 years ago, Experts Exchange did extensive performance testing of our website on several flavors of Linux and UNIX, including FreeBSD 7. FreeBSD outperformed all other tested operating systems in both page load time metrics and Lucene search index performance testing.  Since that time, we’ve migrated our developer, test and production servers to FreeBSD.  We’ve also used FreeBSD jails to host our developer test servers and have come to rely on FreeBSD throughout our local office datacenter and our co-location datacenter stack.

Because of our investment in FreeBSD, we continue to have several system administrators working on FreeBSD enhancements to improve our production environment and their changes hopefully help other companies and users as well.

A big thank you to the FreeBSD community for your open, friendly attitude and how much you’ve contributed to the growth of Experts Exchange!

New Funded Project: Grow Mounted Filesystems

The FreeBSD Foundation is pleased to announce that Edward Tomasz Napierala has been awarded a grant to implement the ability to grow filesystems while they are mounted.

"Users of FreeBSD in a virtualized environment will be pleased with the increased ease of deployment afforded by the ability to grow mounted filesystems," said Ed Maste, Director, The FreeBSD Foundation.

This project will add GEOM and filesystem changes that are necessary to increase the size of both UFS and ZFS filesystems while a filesystem is mounted read-write. This project will provide the additional benefit of online provisioning of virtual instances.

The Foundation is pleased to be working with Edward again. He was previously awarded a grant to implement resource containers and a simple per-jail resource limits mechanism. This work was
included in FreeBSD 9.0 RELEASE.

This project is expected to be completed by October 2012.

FreeBSD

The first RC build for the FreeBSD-8.3 release cycle is now available. ISO images for the amd64, i386, and pc98 architectures are available on most of our FreeBSD mirror sites.

George Neville-Neil Joins Board of Directors

The FreeBSD Foundation is pleased to announce the addition of George Neville-Neil to its board of directors.

George has been dabbling in the BSD world since his undergraduate days in the mid-1980s. He was granted his commit bit in 2004, and has served two terms on the FreeBSD Core team between 2006 and 2010.

In 2011, he started organizing the semi-annual FreeBSD Vendor Summits that gather commercial customers of FreeBSD along with project members in order to facilitate the movement of technologies between the Project and its customers.

George co-authored "The Design and Implementation of the FreeBSD Operating System." His technical contributions are mostly within the FreeBSD network stack with occasional forays into other parts of the system.

"The Foundation is at a pivotal point in our growth as a company," said Deb Goodkin, Secretary/Treasurer, The FreeBSD Foundation. "We believe with George's previous and current involvement in FreeBSD, that he will be a significant contributor in helping us achieve our ambitious goals this year."

Trip Report: FOSDEM 2012

The Foundation recently sponsored travel expenses for Brooks Davis to attend FOSDEM 2012. Brooks provided the following trip report:

I arrived in Brussels the afternoon of Thursday the 2nd.  After checking in to my hotel I headed into the city center and met up with a group of FreeBSD and Étoilé developers for dinner.  On Friday I toured the city and worked on my presentation for Sunday before joining the crowd at Delrium Cafe for the FOSDEM Beer Event and made some new friends.

Saturday began with the traditional opening talk followed by an interesting talk by Robert Dewar or AdaCore on how they are a profitable company selling a rather expensive GPL licensed product to a very conservative client base (mostly defense and safety critical applications). Other than avoiding costs associated with source escrow, it wasn't clear that being open source gave them much advantage, but he had a compelling argument that it didn't hurt either.

In the afternoon I attended a number of talks in the Vitalization and Cloud track. The Aeolus Project and Open Cloud Interoperability with CompatibleOne covered some related efforts to provide interoperability and portability between cloud providers. Both were definitely works in progress, but showed promise. A more interested talk in that track was "Infrastructure as an open-source project" by Ryan Lane of Wikimedia Labs. He talked about the way the had placed most of their infrastructure definition for system setup, configuration management, and monitoring into a git repository to allow project contributors to view and submit patches to the infrastructure. This appears to be growing the number of volunteer contributors to the admin of Wikimedia infrastructure, but has shifted the bottleneck for support staff so now they spend a significant portion of their time review patches to the infrastructure. After that talk a group of FreeBSD Developers adjourned to a local restaurant for dinner and drinks.

Sunday I ran the BSD License Operating System Devroom. We had a room in the new K building at ULB which was very nice. All the equipment was built in and just worked. We had a total of ten talks with topics including MINIX 3 and BSD, an Introduction to pkgng (FreeBSD's new package system), four talks on pkgsrc and related topics, two talks on FreeBSD toolchain work, one on tablet support in NetBSD, and one on Lua in the NetBSD kernel. The latter was the best attended talk of the day with the room being just about full.  After the talks we attempted a brief brainstorming session on ways we could collaborate.  Unfortunately we were pressed for time and while we got a discussion going, we didn't have the necessary time to develop concrete ideas. While most of our talks were on BSD OSes, we did talk about expanding the focus of the room a bit further to include other non-Linux OSes next year and received generally positive vibes about that idea.  All in all we had a good set of presentations and I learned quite a bit about other projects so it was personally successful.  Further information on the devroom talks can be found here.

After cleaning up the devroom I met up with a FreeBSD developer for dinner and then joined a group of Étoilé developers at Delirum Cafe. Monday morning I returned to London and proceeded to Cambridge for two weeks of FreeBSD hacking, mostly focused on the toolchain.

Why FreeBSD – My Personal view.

Hi,

Just a review of myself since i’ve need it for my talk :-)

I’ve started playing with Linux in 2000/1, and since then I have been switching between Linux and Windows from time to time. By the end of 2001, I switched completely to Linux because I had enough of windows blue screens problem. After some hardcore testings of other distributions like debian, slax, crux, fedora, suse and gentoo, I finally stayed with Gentoo for 1 ½ years. But after portage got more and more buggy, I started to look for a new challenge. I’ve met some BSD Guys in IRC, and after a while I setup FreeBSD in dual boot and started playing with it. I was really surprised on how fast and stable it runs, and also the community impressed me with their helpfulness. If you ask anything over the mailing list, you will always get helpful answers from the developers and users. My experience with Linux community is totally the opposite. It was common for me to get answers such as “RTFM” or “Try google”, which is not what I hoped for. I still remember today how many times I had to reinstall Linux just to fix up some mess. After a while with Linux I have the impression that Linux is going more and more towards mainstream, which is of course nice for the end users, but the fact that anyone can install Linux without knowing what happened in the background makes it even worse. Not to mention about the kernel size, which I have always seen some new stuff in, but the old codes were left unmaintained. This leads to a bigger kernel with more security holes, which leads to system instability. The kernel is now 63MB, which is so much bigger compared to FreeBSD kernel.

I believe that Linux now has a big user base where 70% of the users do not understand what is occurring in the background or if they even know what ‘terminal’ is Thank you Ubuntu, Suse, Fedora. Well alright I think I stop here because you maybe start thinking that I’m bashing Linux, which I am not :-) . I also have a lot of good experience with Linux :) and of course I won’t forget which user base I was from :) .

In November 2005, I moved completely to FreeBSD. After a while using FreeBSD, I was thinking on how to give something back to the community. I started submitting patches to the FreeBSD project, and in Jun 2006 I got my FreeBSD ports commit bit. Since then I am one of the most active member in the Project. At least, this is something that I could do in return of being able to enjoy a good and a stable system with a big choice of applications that is easy to handle using the package management system.

There are a lot of things that I love about FreeBSD, and I bet you know most of them already. FreeBSD has a clean structure that makes it so easy to understand. It is a rock solid system, very stable and easy to maintained. For applications, you have a choice between package system, if you want it fast, or the famous ports system, if you wish to compile applications yourselves with specific options. Another great thing about ports system is its easy framework, which provides flexibility to FreeBSD users to build their own stuff.

FreeBSD’s user base is probably not as big as Linux’s, but I can safely say that FreeBSD’s community support is one of the best in the open source world. As I have mentioned previously, I have not been in a more helpful community than FreeBSD’s. If you’re in the mailing list or a community member, you’ll know what I mean :) .

FreeBSD has a lot to offer to new users. The deep learning curve is probably tough for a first timer, but all is worth it. And FreeBSD has the best gift to new users, the handbook. FreeBSD handbook is one of the best documented operating system, and this is another reason why I like FreeBSD so much.

Well there another 5000 reason…

So long miwi :-)

Home made pkgng repositories

By popular demand I have been requested a tutorial about how to maintain your own pkgng repositories.

Currently the best solution for that is to use poudriere (ports-mgmt/poudriere).

Let say you want to maintain 2 repositories: 8.2-RELEASE i386 and 9.0-RELEASE amd64.

For that you will need an amd64 (to support both amd64 and i386 box with 9.0 binary support and 8.2 binary support, a default 9.0-RELEASE amd64 should be enough :)

poudriere only depends on zfs and sh, so you won't need much, just a zfs pool available.

First: install poudriere:

$ make -C /usr/ports/ports-mgmt/poudriere install clean

To be able to build the repository the host would also need to have pkgng installed (no need to convert it to pkgng anyway.)

$ make -C /usr/ports/ports-mgmt/pkg install clean

Now configure your poudriere, defining some configuration to /usr/local/etc/poudriere.conf:

BASEFS=/poudriere
ZPOOL=myzpool
FTPHOST=ftp.freebsd.org
POUDRIERE_DATA=/poudriere_data
RESOLV_CONF=/etc/resolv.conf
DISTFILES_CACHE=/usr/ports/distfiles

This should be enough (see poudriere.conf.sample for more informations)

Create the default ports tree:

$ poudriere ports -c

Create the two jails (in fact it will be chroot :))

$ poudriere jails -c -j 82i386 -v 8.2-RELEASE -a i386
$ poudriere jails -c -j 90amd64 -v 9.0-RELEASE -a amd64

Make them pkgng aware

$ mkdir /usr/local/etc/poudriere.d
$ echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/82i386-make.conf
$ echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/90amd64-make.conf

Add the list of packages you want to build:

$ cat ~/mylist1
editors/vim
www/nginx
$ cat ~/mylist2
www/firefox
editors/libreoffice

If you want special options just add them to you different make.conf in poudriere.d

to share options between all the jails managed by poudriere, add them to /usr/local/etc/poudriere.d/make.conf

You can now create your packages:

$ poudriere bulk -f ~/mylist1 -j 82i386
$ poudriere bulk -f ~/mylist2 -j 90amd64

When finished you will get 2 pkgng repositories in /poudrieredata/packages/82i386-default and /poudrieredata/packages/90amd64-default

You can now provide them through your webserver.

On you client boxes just:

$ echo "packagesite: http://yoururl/82i386-default" >> /usr/local/etc/pkg.conf

or

$ echo "packagesite: http://yoururl/90amd64-default" >> /usr/local/etc/pkg.conf

If your client box already has packages from an old installation, first convert it to pkgng

$ fetch http://yoururl/90amd64-default/Latest/pkg.txz
$ tar xf ./pkg.txz -s ",/.*/,,g" "*/pkg-static"
$ ./pkg-static add ./pkg.txz
$ pkg2ng

Now you can forget about pkg_install and pkg2ng

Just use pkgng (for example):

$ pkg update
$ pkg upgrade
$ pkg install firefox

To update your repository:

$ poudriere ports -u # this update your default ports tree
$ poudriere bulk -f ~/mylist1 -j 82i386 -k
$ poudriere bulk -f ~/mylist2 -j 90amd64 -k

bonus poudriere will rebuild only what has changed and what is impacted by this change.

Once it is done simply upgrade your client box:

$ pkg update
$ pkg upgrade

More informations on poudriere here

PC-BSD at NELF

NorthEast LinuxFest will be held in Worcester, MA on March 17. We will be given away PC-BSD DVDs and cool swag at the FreeBSD booth in the expo area. Registration is free for this event and there will be an after-party at a local pub. Not a bad way to spend St. Patrick’s Day if you’re in the NorthEast.

New Project: NAND Flash Support

The FreeBSD Foundation is pleased to announce that Semihalf, an embedded solutions company, has been awarded a grant to bring their comprehensive NAND Flash file system and storage stack to FreeBSD. This technology enables FreeBSD to natively manage NAND Flash devices, satisfying a crucial requirement for many applications needing access to fast, reliable, non-volatile storage.

FreeBSD is widely used as the OS foundation of embedded appliances both small and large. Semihalf's NAND Flash stack opens new opportunities for FreeBSD in this space, where size, cost, or performance, mandate the use of direct attached NAND Flash.

Made possible by matching funds from Juniper Networks, this FreeBSD Foundation grant covers the costs of transferring technology developed for Juniper Networks by Semihalf to the FreeBSD project. This will ensure that the NAND framework meets community standards and can be easily maintained and enhanced.

Highlighting the return on investment offered by this kind of technology transfer, FreeBSD Foundation president Justin T. Gibbs, noted: "Open sourcing enhancements that do not expose 'business critical intellectual property' reduces the cost of managing a FreeBSD distribution that has been customized for a product. The NAND subsystem is a perfect example of how technology transfer benefits both the FreeBSD community and its commercial users. We'd like to thank Semihalf and Juniper for partnering with us to make the code available under a BSD license"

The NAND Flash subsystem consists of a driver framework for NAND controllers and memory chips, a NAND device simulator and a fault tolerant, log-structured file system, tailored to meet the unique challenges of NAND flash storage. The package includes all the tools, utilities and documentation needed to deploy this technology in custom applications.

"A reliable file system that supports NAND Flash is critical for Juniper's ongoing success," said Marcel Moolenaar, Distinguished Engineer, Juniper Networks. "But since storage isn't Juniper's core business, we were eager to find a solution that would put the implementation and support of the file system in the most capable hands. We reached out to Semihalf and ultimately the Foundation to help us achieve our goals. Juniper cannot be more pleased to have the NAND Flash file system and NAND Flash framework present in the next major FreeBSD version as a standard feature and under the care of the community."

"We are very glad to have the NAND framework made available for the general FreeBSD audience, reaffirming the system as a versatile platform for appliances and other embedded and industrial designs," said Rafal Jaworowski of Semihalf.

The Foundation is pleased to be working with Semihalf again. They were previously awarded a grant to bring "Flattened Device Tree" support to FreeBSD. This new feature in FreeBSD 9.0 has been well received by the FreeBSD community.