Monthly Archive for December, 2009

Alexander Leidinger: Some fixes for ZFS on 7-stable (more testers wanted)

Published on December 30, 2009 in FreeBSD, Open Source, Software, Source and kernel. Closed

Due to the problems with a 7-stable machine, I had a look at some unmerged fixes for ZFS (58 changes not merged).

I backported some of those changes from 8-stable to 7-stable, I have this running on one 7-stable machine. I would like to get some more feedback for it (even an “it works for me” would be great). The main part of this change is that the FreeBSD taskqueue is used now instead of the opensolaris one (and some other changes which may improve the ZFS experience).

It would also be nice if someone could have a look at the FIRST_THREAD_IN_PROC part. Can there be more than one thread at this place (I do not think so) and I should use FOREACH_THREAD_IN_PROC_instead?

How to apply:

  • cd /usr/src/
  • fetch http://www.Leidinger.net/FreeBSD/test/releng7_zfs_merge3.diff
  • fetch http://www.Leidinger.net/FreeBSD/test/opensolaris_taskq.c
  • fetch http://www.Leidinger.net/FreeBSD/test/taskq.h
  • mv taskq.h sys/cddl/contrib/opensolaris/uts/common/sys/taskq.h
  • mv opensolaris_taskq.c sys/cddl/compat/opensolaris/kern/opensolaris_taskq.c
  • patch –p 0 –quiet <releng7_zfs_merge3.diff
  • ignore the 2 .rej files
  • rm –f sys/cddl/compat/opensolaris/sys/taskq_impl.h*
  • rm –f sys/cddl/compat/opensolaris/sys/taskq.h*
  • rm –f sys/cddl/contrib/opensolaris/uts/common/os/taskq.c*
  • rebuild kernel

I do not list all of those 16 of 58 outstanding patches which are covered here, a detailed list can be found on the stable and fs mailinglists.

Share/Bookmark

Dru Lavigne: Letter from the President

Published on December 29, 2009 in newsletter. Closed
From the December Foundation Newsletter:

In 2009, the FreeBSD project had the misfortune of losing two long time contributors: John Birrell and Jean-Marc Zucconi. I chatted with John recently, during this year's BSDCAN, so his death was all the more shocking. It forced me to recognize my own mortality and to consider what contributions from our lives remain after we pass away. Reviewing the heritage of FreeBSD it becomes clear that our work on this project takes on a life of its own. John and Jean-Marc's efforts live on in FreeBSD.

The value of the FreeBSD legacy has become even more apparent to me with my return, after an almost 10 year break, to working on FreeBSD during my day job. The kernel has the familiarity of an old friend, even though it has been improved in countless ways by dozens of new contributors. Although the principles and best practices engendered by the FreeBSD project have changed little in the last decade, they are no less relevant today. The faces and challenges may have changed, but the qualities of FreeBSD - solid design, high performance, stability - have not.

Trying to communicate this real value of FreeBSD can be difficult. Valuations typically take current features into account, but neglect to consider the community that created and will support those features. And what value can you place on future, community driven, improvements to FreeBSD? There is no guarantee that every feature you need will be added to the system in the time frame you need it, but the collaborative environment created by the FreeBSD project makes it very likely. The greatest asset of FreeBSD is our community. If we continue to invest in the FreeBSD community, I have every confidence that the FreeBSD legacy will endure.

The FreeBSD Foundation is doing its part to continue FreeBSD's legacy. Through conservative management and careful planning we have improved our financial position even when faced with the most severe recession in 50 years. Our growing strength and capabilities, all made possible by your generous donations, are proof that the FreeBSD Foundation will be a faithful supporter of FreeBSD for many years to come. Together, building on the legacy left by John, Jean-Marc, and the countless contributors before them, we will ensure the future of FreeBSD.

Ivan Voras: What’s cooking for FreeBSD 9?

Published on December 29, 2009 in 9 and FreeBSD. Closed

I've already started a new page :)

Read more...

Dru Lavigne: FreeBSD Project Receives Bad Code Offset

Published on December 28, 2009 in financial support and good code. Closed
The Bad Code Offsets project of The Alliance for Code Excellence is "a way to undo the bad code other people have written without actually replacing the bad code. Much like carbon offsets, money used to buy Bad Code Offsets goes towards open-source projects which not only produce good code, but produce software that helps developers build good software".

The FreeBSD Project was recently added as a supported project and the FreeBSD Foundation will be receiving a check for $500. Thanks to those who suggested the FreeBSD project be added and to the Alliance for supporting good code!

Dru Lavigne: Google Summer of Code Mentor Summit 2009

Published on December 28, 2009 in gsoc and trip report. Closed
Brooks Davis recently reported on his trip to the Google Summer of Code Mentor Summit 2009. The Foundation assisted in some of his travel costs to this event. Brook writes:

The Google Summer of Code Mentor Summit was held at the Google campus in Mountain View October 24th and 25th. I represented the FreeBSD project at the event along with Tim Kientzle who was one of our other program admins.

Google runs the Mentor Summit as an un-conference which means that attendees pick topics they want to discuss, others indicate interest in them, and then rooms are allocated based on demand. Sessions were about things including general open source process issues, education, technical collaborations, and individual project meetings.

One of the highlights of Saturday was a session on multi-core and other acceleration technologies like GPUs. The session didn't come to a strong consensus other than incorporating these technologies is difficult. The most concrete thing that came up was the idea of putting the technologies behind widely used APIs so they automatically provide benefit. Another topic of discussion was debugging tools and
techniques. I think that is an area where FreeBSD is sometimes ahead with technologies like witness and now DTrace.

The other session of interest was an impromptu session of non-Linux OSes including DragonFlyBSD, FreeBSD, Haiku, and NetBSD. It was mostly people talking about current status. One thing other projects seemed to want was a way to take advantage of FreeBSD network drivers by providing more common interfaces. In concept this was interesting, but probably isn't some thing that would make a whole lot of sense for FreeBSD given that we're rethinking and redesigning the interfaces we have to meet modern performance requirements.

The most useful session on Sunday was when Tim and I grabbed a small conference room and started reviewing our GSoC admin materials for next year. Based on that information, we plan to start engaging with FreeBSD developers in January in anticipation of a 2010 program.

Over all the conference was interesting and fun. It was good to talk to
people from other projects that don't often attend the BSD conference.
I'm not sure anything concrete came of those interactions, but it was
probably useful nonetheless.

Martin Wilke: Call for tester: VirtualBox 3.1.2 for FreeBSD

Published on December 27, 2009 in FreeBSD, FreeBSD_EN, Ports, Vbox and VirtualBox. Closed
Hi All, Changelog from VirtualBox is available here: http://www.virtualbox.org/wiki/Changelog Changes in the port: - VirtualBox and the guest additions have been updated to 3.1.2. - Port has been renamed to virtualbox-ose to reflect that we are using the OSE version. Requested by: mm@ - A seperate port for the kernel modules has been created: virtualbox-ose-kmod - A seperate port for guest additions for [...]

Will Backman: bsdtalk183 – Randal L. Schwartz

Published on December 23, 2009 in Uncategorized. Closed
Four years of BSDTalk.

Interview with Randal Schwartz. We talk about his early experiences with BSD, permissive licenses, OpenBSD, OpenSolaris, perl, the BSDFund credit card, and the Floss Weekly podcast.

File Info: 24Min, 12MB.

Ogg Link:
http://cisx1.uma.maine.edu/~wbackman/bsdtalk/bsdtalk183.ogg

Martin Wilke: KOffice2 for FreeBSD avalible

Published on December 21, 2009 in FreeBSD, FreeBSD_EN, KDE, Koffice and Ports. Closed
The KDE FreeBSD team is proud to announce the release of KOffice2 suite for FreeBSD. The official KOffice 2.1.0 notes can be found here. We’d like to say thanks to all helpers, testers and submitters.

Remko Lodder: FreeBSD Foundation end of year Fundraise

Published on December 21, 2009 in FreeBSD. Closed

Hello, it’s the time again. The FreeBSD Foundation is looking for a few spare bucks here and there to reach the goal of $300.000. So I was wondering, if we all make a contribution of $1 or more, we will reach that easily. I donated $150 so you should consider donating a few spare bucks as well. The original announcement is found below.

Since the start of our ‘Be Counted!’ campaign in August of this year, over 350 new and returning donors have contributed to the FreeBSD Foundation. With your help, we are now 50% of the way to meeting our 2009 fund raising goal. Thank you donors, for your support! Now, in these last few weeks of 2009, the FreeBSD Foundation needs the support of those who have yet to donate to take us the rest of the way.

The recession has hit everyone hard. For many, every possible expense has been cut, and what spending they do is out of strict necessity. Unfortunately the challenges facing FreeBSD are undiminished by recessions and the technological landscape continues to change at a rampant pace. That is why the FreeBSD Foundation nearly doubled its 2008 budget for 2009 and needs your support so we can avoid cutting our investments in 2010.

If you benefit from FreeBSD, please donate so:

development projects are funded to support emerging technologies such as solid state disks, USB 3.0, machine and network virtualization, highly parallel processors, clustering, and data replication.
BSD conferences continue around the globe.
students and contributors have the opportunity to attend conferences and developer summits.
the infrastructure of computers and equipment supporting our community can be maintained.
the FreeBSD community is grown through marketing and outreach to users and businesses.
FreeBSD trademarks are protected and the project has access to legal counsel.
FreeBSD continues to serve as the foundation for research and enterprise.
Every donation, no matter its size, makes this work possible. As a non-profit with very low overhead, your donation is the best way to invest in FreeBSD. Please make that investment today so we can meet our dual goals for 2009 of 1000 donors and $300,000.

You can make a donation (including recurring subscriptions) by going to: http://www.freebsdfoundation.org/donate/.

FreeBSD News Flash: New committer: Ryusuke SUZUKI (doc/ja_JP, www/ja)

Published on December 20, 2009 in Uncategorized. Closed

Dru Lavigne: Thank You!

Published on December 17, 2009 in donate. Closed
We would like to thank everyone who has donated to the FreeBSD Foundation this year. We have raised $182,778 towards our 2009 goal of $300,000! We are almost 2/3 of the way to reaching our goal! Oh, and BTW, we have had 663 donors this year. This is compared to just over 300 this time last year. This is important not only to help us keep our Public Charity Status, but it shows there are many users who are passionate about FreeBSD and want to show their support.

With the weakened economy we have been very conservative with our spending this year. But, like each previous year we have increased the amount we have spent on the FreeBSD Project and community. We were blown away with the number of project proposals we received this year. We were able to fund 7 projects this year. Unfortunately we didn't have the budget to fund all the proposals we received.

This coming year we want to double the amount we spend on project development. In order to accomplish this, we need to meet our fund-raising goal.

Why do we need donations?

The goal of the FreeBSD Project is to provide software that may be used for any purpose -- and without strings attached. Our mission is to support the FreeBSD Project and community. Our funding comes from people like you – those who are determined to keep FreeBSD free!

How have we spent the money this year?

• Sponsored FreeBSD related conferences like BSDCan, EuroBSDCon, AsiaBSDCon, KyivBSD, and DCBSDCon.

• Provided 15 travel grants and funding to individuals to attend these and other conferences this year.

• Provided grants for projects that improve FreeBSD, like wireless mesh support, FreeBSD TCP stack improvements, new console driver, safe removal of disk devices, flattened device tree, and high available storage projects.

• Provided equipment for developers working to improve FreeBSD and projects like the NetPerf cluster. We purchased servers, USB analyzers, power controllers, and parts for computer repairs for the Project. We also paid for shipping of equipment to various projects.

• Provided legal support for the project on issues like understanding the GPLv3 impact on FreeBSD, providing a privacy policy, trademark ownership and permission, and other legal issues that come up.

How can you help?

Your financial support is critical for the FreeBSD Project. Please help us keep FreeBSD free. Any amount you can donate will help. And thank you for your continued support of the FreeBSD Foundation.

Alexander Leidinger: Stabilizing 7-stable…

Published on December 16, 2009 in FreeBSD, Jails, Open Source, Problem(s), Software, Source, Userland and kernel. Closed

The 7-stable system on which I have stability problems after an update from 7.1 to 7.2/7-stable is now semi-stable.

The watchdog reboots after one minute of no reaction (currently it is able to run 3–4 hours), and the jails come up without problems now.

The problem with the jails was, that e.g. the mysql-server startup went into the STOP state because TTY-input was “requested”. I solved the problem by using /dev/null as input on jail-startup. On –current I do not see this behavior (I have a 9-current system with a lot of jails which reboots every X days, and there mysql does not go into the STOP state).

I also start the jails in the background, so that one blocking jail does not block everything (done like in –current).

To say this with code:

--- /usr/src/etc/rc.d/jail      2009-02-07 15:04:35.000000000 +0100
+++ /etc/rc.d/jail      2009-12-16 17:03:12.000000000 +0100
@@ -556,7 +556,8 @@
 fi
 _tmp_jail=${_tmp_dir}/jail.$$
 eval ${_setfib} jail ${_flags} -i ${_rootdir} ${_hostname} \
-                       \\"${_addrl}\\" ${_exec_start} > ${_tmp_jail} 2>&1
+                       \\"${_addrl}\\" ${_exec_start} > ${_tmp_jail} 2>&1 \\
+                       </dev/null

 if [ "$?" -eq 0 ] ; then
 _jail_id=$(head -1 ${_tmp_jail})
@@ -623,4 +624,4 @@
 if [ -n "$*" ]; then
 jail_list="$*"
 fi
-run_rc_command "${cmd}"
+run_rc_command "${cmd}" &

I also identified 57 patches for ZFS which are in 8-stable, but not in 7-stable (I do not think they could solve the deadlock, but I do not really know, and now that there is one FS on ZFS, I would like to get as much fixed as possible). Some of them should be merged, some would be nice to merge, and some I do not care much about (but if they are easy to merge, why not…). I already have all revisions and the corresponding commit logs available in an email-draft.

Now I just need to write a little bit of text and find some people willing to help (some of the changes need a review if they are applicable to 7-stable, and everything should be tested on a scratch-box).

Share/Bookmark

Will Backman: bsdtalk 182 – FreeNAS with Josh Paetzel from iXsystems

Published on December 16, 2009 in Uncategorized. Closed
A quick update on FreeNAS with Josh Paetzel from iXsystems.

File Info: 12Min, 6MB.

Ogg link:
http://cisx1.uma.maine.edu/~wbackman/bsdtalk/bsdtalk182.ogg

Alexander Leidinger: Stability problems with 7-stable

Published on December 15, 2009 in FreeBSD, Open Source, Problem(s) and Software. Closed

On the machine where I host this blog, I have/had some stability problems.

Last week I updated the machine from FreeBSD 7.1-pX to 7.2-p5 (GENERIC kernel in both cases). 5–10 Minutes after the reboot into the new version the machine had a deadlock. After some roadblocks (ordering a KVM-switch from the hoster, the KVM-switch not working with a proxy (during lunchtime at work), a broken video-capture of the KVM-switch and a replacement on Monday morning to not pay the WE-fees), I spend a big part of the night to get it stable. I tried disabling SMP, enabling INVARIANTS and WITNESS, changing the scheduler, cutting the software mirror (to rule out a mismatch between the content of the disks after all the hard reboots) and updating to 7-stable.

Unfortunately nothing helped. :(

Googling a little bit around (it is a AMD Dual-Core system with NVidia MCP61 chipset) was leading me to a post on the mailinglists from 2008 which talks about an issue with the buffer cache. I do not know if this is still an issue (I have send a email to kib@ to ask about it), and my scenario is not the same as the one which is described in the mail, but because of this I decided to switch one of the two UFS mirrors to ZFS.

The first boot into the ZFS caused again a reboot after some minutes (I do not know if it was because of a memory exhausted panic, or because of a deadlock), but as I did not tune the kernel for ZFS I am tempted to believe that I should not count that. Now, after tuning the kernel (increasing the kmem_size to 700M, no prefetching, limiting the ARC to 40M) it is up since nearly 2h (as of this writting… crossing fingers). Before it was not able to survive more than some minutes with just the jail for the mails up. Now I not only have the mail-jail up, but also the jail for the blog (one jail still disabled, but I will take care about that after this post).

I do not know if only increasing the kmem_size would have helped with the problem, but as I was testing a GENERIC kernel + gmirror module in the beginning, I expected that the auto-tuning of this value should have been enough for such a simple setup (2GB RAM, 2 disks with 3 partitions each, one partition pair for root, one for swap, one for the jails).

I hope that I stabilized the system now. It may be the case that I will test some patches in case someone comes up with something, so do not be surprised if the blog and email to me is a little bit flaky.

Share/Bookmark

Dru Lavigne: Update on HAST Project

Published on December 14, 2009 in Uncategorized. Closed
Pawel Jakub Dawidek has completed the first milestone for the High Available Storage Project. He writes:

I want to report that first milestone of the HAST project is complete.

Summary of the work that have been done:

  • Implementation of hastd daemon.
  • Implementation of hastctl utility to manage hastd daemons.
  • GEOM_GATE class was extended so that the caller can specify the name of GEOM provider. Before only /dev/ggateX names were supported. HAST will use /dev/hast/<resource_name> names.
  • Implementation of communication protocol. There is abstraction layer on top and below there are three protocols implemented currently:
  1. proto_tcp4 - It is used for communication between primary and secondary nodes.
  2. proto_uds - (UDS - UNIX Domain Socket) It is used for communication between hastctl and hastd.
  3. proto_socketpair - It is used for communication between main hastd daemon and worker processes forked from it.
  • Implementation of nv (name-value) API, which allows to easy create packets containing name-value pairs. It is used for entire communication through the protocols above. It is also responsible for managing correct byte-order.
  • Implementation of ebuf (extendable buffer) API, which provides a way to extend given buffer by adding data at the back, but also at the front without reallocating it and copying the data very often (or never).
  • Implementation of logging API (pjdlog). The API decides if messages should be logged on standard output/error (before going into background) or to syslog (when we daemonize). It also provides some shortcuts for logging a message and exiting, etc. It supports notion of debug level and can skip messages intended for higher debug level than requested.
  • Implementation of configuration file parser in lex/yacc. Configuration file is designed in a way that it can be kept identical on both nodes.
  • Checksumming and compression for the data is not one of the project's goal, but the stubs are there, so this can be added easly.
  • A lot of care was taken to be able to handle more nodes in the future. This is not implemented and in not project goal, but I wanted to make it ready for future improvements.

HAST can be used by starting hastd daemons on both nodes:

# hastd -h
hastd: [-dh] [-c config]

Then on the secondary node we mark all resources as secondary:

# hastctl -h
hastctl: [-d] [-c config]
hastctl: [-d] [-c config] status [all | name ...]

# hastctl secondary all

On the primary node we mark all resources as primary:

# hastctl primary all

The hastd daemon running on primary node will connect to the secondary node and fork a child to handle communication. There is a socketpair between parent and child so that they can communicate. Primary node creates two connections: one for incoming data and one for outgoing data. There are seven threads in total for each working resource:

1. ggate_recv

Thread receives ggate I/O requests from the kernel and passes them to appropriate threads:
WRITE - always goes to both local_send and remote_send threads
READ (when the block is up-to-date on local component) -only local_send thread
READ (when the block isn't up-to-date on local component) -only remote_send thread
DELETE - always goes to both local_send and remote_send threads
FLUSH - always goes to both local_send and remote_send threads

2. local_send

Thread reads from or writes to local component.
If local read fails, it redirects it to remote_send thread.

3. remote_send

Thread sends request to secondary node.

4. remote_recv

Thread receives answer from secondary node and passes it to ggate_send thread.

5. ggate_send

Thread sends answer to the kernel.

6. ctrl

Thread handles control requests from the parent.

7. guard

Thread guards remote connections and reconnects when needed, handles signals, etc.

On the secondary node when both connections are successfully established it forks a worker process, which operates using four threads:

1. recv

Thread receives requests from the primary node.

2. disk

Thread reads from or writes to local component and also handles DELETE and FLUSH requests.

3. send

Thread sends requests back to primary node.

4. ctrl

Thread handles control requests from the parent.

At this point primary and secondary nodes can communicate and requests are properly replicated. IO errors on local read failure are handled by redirecting read request to remote node. Replicated storage can be accessed through /dev/hast/<resource_name> GEOM provider.

I'm confident that the first milestone is complete. If you have any questions, I'll be happy to answer them. If you have any suggestions or comments, I'll also be happy to hear them.

Ulf Lilleengen: Locale fix

Published on December 12, 2009 in FreeBSD. Closed

After looking for a long time as to why my default locale in gnome changed after a recent upgrade, I finally found out where to change the locale setting. The problem was that gnome did not seem to pick up my system locale settings, and the norwegian characters in my terminal came up as question marks.

As the gnome login manager (gdm) got rewritten, there is now no way to change this locale at the login screen unless it was picked up by gdm. But, as always, reading the documentation helps. After reading

http://library.gnome.org/admin/gdm/2.28/configuration.html.en

I discovered that I could just edit

~/.dmrc

and write this:

[Desktop]
Language=en_US.UTF-8
Layout=no

to set the correct locale!

Dru Lavigne: Colin Percival on Supporting the Foundation

Published on December 10, 2009 in developers, donating and tarsnap. Closed
Developers not only benefit from the work done by the Foundation, they also help to financially support it. Colin Percival writes about donating to the Foundation on his blog. His blog post begins:

As a FreeBSD user and developer, I obviously care about the success of FreeBSD. I make a small contribution towards this success via my role as Security Officer; but the time I spend working on my Tarsnap online backup service prevents me from making as much of a direct contribution as I would like. Fortunately the FreeBSD Foundation does an excellent job of supporting FreeBSD development; but like most such organizations, they are funded entirely by donations and are always in need of more. In light of this, I am pleased to announce that I will be donating all of the profits made by Tarsnap for the month of December to the FreeBSD Foundation.

Much of the work done by the FreeBSD Foundation is done behind the scenes, but the importance of the work they do is undeniable. They sponsor a range of important FreeBSD development which would likely not get done otherwise; in the past year this has included...

Ivan Voras: Nicer things in life – VMWare 4, ZFS, virtual hot-plugs

Published on December 10, 2009 in FreeBSD, VMWare and ZFS. Closed

Sometimes, when things go right, they are really beautiful to watch. Like adding a virtual (SCSI) drive to a running virtual machine freshly copied from ESXi 3.5 to vSphere 4, seeing that drive simply appear in the guest system in a hot-plug fashion, creating a ZFS pool on it and watching iostat as data is migrated from the old drive which has gotten to be too small (and uses UFS), to the new one on which the magic of ZFS will allow the file system to grow indefinitely (as the damn application is greedy in a stupid way), and seeing that "pausing" problems ZFS caused on ESXi 3.5 really are solved in 4.

Read more...

Ivan Voras: SoftUpdates – the next generation (or SU+J)

Published on December 9, 2009 in FreeBSD, Journalling, SoftUpdates, UFS and file systems. Closed

Jeff Roberson is working on an addition to UFS softupdates which includes a tiny journal to keep track of things like free space and inode orphaning that were left to background fsck to deal with in the original implementation of softupdates. Basically, it's still the same old softupdates but without the need for (bg)fsck to be run for recovery!

Read more...

Gleb Kurtsou: pefs and l2filter moved to github

Published on December 9, 2009 in FreeBSD, l2filter and pefs. Closed

I’ve just moved pefs and l2filter development to github. Hope it helps people follow development.

pefs repository (github.com/glk/pefs) can be used to to compile and run pefs without applying any patches.

pefs changelog:
* support running on msdosfs
* enable dircache only on file systems that are known to support it
* add man page
* add pefs getkey command
* intial implementation of pefs PAM module

l2filter repository (github.com/glk/l2filter) contains only patches. There is fresh patch against 8-STABLE with some minor improvements comparing to 7-STABLE version. 9-CURRENT patch is a bit outdated at the moment, as I’m waiting for Luigi Rizzo to finish ipfw refactoring work first.

Besides I’ve moved my blog to glebkurtsou.blogspot.com/
Please update your bookmarks. I do not intend to update blog freebsdish any more.