Archive for the 'FreeBSD' Category

Alexander Leidinger: Linuxulator D-Trace probes committed to current

A while ago I committed the linuxulator D-Trace probes I talked about earlier. I waited a little bit for this announcement to make sure I have not broken anything. Nobody complained so far, so I assume nothing obviously bad crept in.

The >500 probes I committed do not cover the entire linuxulator, but are a good start. Adding new ones is straight forward, if someone is interested in a junior–kernel–hacker task, this would be one. Just ask me (or ask on emulation@), and I can guide you through it.

Share

Alexander Leidinger: linux_base-c6

Seems I forgot to announce that the linux_base-c6 is in the Ports Collection now. Well, it is not a replacement for the current default linux base, the linuxulator infrastructure ports are missing and we need to check if the kernel supports enough of 2.6.18 that nothing breaks.

TODO:

To my knowledge, nobody is working on anything of this. Anyone is welcome to have a look and provide patches.

Share

Ivan Voras: BSDCan 2012 – Day 2

The second, and unfortunately the last day of BSDCan was filled with interesting talks, again with much overlap. There are simply so many interesting things going on in FreeBSD that all of them simply don't fit in just two days of conferencing! From all of those, I'd recommend (even though I wasn't able to attend some of them) the talks on netmap, ZFS, AWS, pkgng and IPv6 security - don't miss them when the videos go online!

Read more...

Ivan Voras: DevSummit 2012 day 2 and BSDCan 2012 day 1

The second day of the DevSummit continued with interesting technical discussions in the Virtualization track, which was paralleled with the Teaching OS Courses track and the Administration and Toolchain tracks. The BSDCan day began with an epic bagpipe performance followed by full four tracks of highly interesting topics - unfortunate as there is much overlap. I gave my talk on Bullet Cache which describes some of the more interesting technical aspects and presents new performance measurements.

Read more...

Ivan Voras: BSDCan 2012 – DevSummit

Another year - another BSDCan! It's very nice and even comforting to see such a large number of familiar faces again, and even more as the ranks are filled in by fresh new developers. The conference and the Developers' Summit before it promise a great program and a great time for the BSDers.

Read more...

Brad Davis: Scripted Install of FreeBSD 9

Continuing in the theme of automation, here is a distilled guide on how I do an install on FreeBSD 9.

http://freebsd.so14k.com/freebsd9_scripted_install.shtml

It is pretty basic, here are some of the highlights:

  1. GPT disk layout
  2. ZFS Only Install (which could be easily converted to UFS)
  3. The Script is nice and short!

Baptiste Daroussin: Fresh blood needed for LibreOffice

For some time now, I have been maintaining pretty much alone the LibreOffice port for FreeBSD, trying to provide all the features LibrOffice has to our users.

I managed to port the 3.3.x series of LibreOffice to FreeBSD some time ago and tried to keep updating it as fast as possible as soon as new releases were out.

At the beginning I was thinking about maintaining 2 concurrent of LibreOffice at the same time: legacy and "normal" to reflect the upstream support. But this take too much time, I'll kill libreoffice-legacy in the next weeks.

The problem is that it is really time consuming, not that it is really complex, the LibreOffice upstream is really nice and really responsive, other maintainers from the BSD community, has also been really helpful.

Some time ago, I decided to create office@ to help maintaining every office related ports maintained on FreeBSD, it worked quite well, sunpoet@ taking good care for examples of the different hunspell/hyphen/mythes dictionnaries/thesaurus, pfg and maho taking care of Apache OpenOffice and all of us working on the third party libraries/fonts/etc needed by all those big office project.

But I'm still missing help on LibreOffice itself, I just committed 3.5.2 some days ago, and 3.5.3 is almost there.

We also still have known issues on 3.5.2:

  • Doesn't build on recent -HEAD (problem with clang 3.1): all the fixes are in the upstream git, but need to be tracked and backported.
  • Doesn't work propertly with base lpd.
  • Doesn't build using the WITH_DEBUG option
  • Doesn't build with gcc from ports (gcc from base is too old for libreoffice)

All the libreoffice work is done on redports svn

if you have an account and are willing to help tell me so that you can have access to the office svn on redports.

Alexander Leidinger: DTrace in GENERIC (-current)

In case you have not noticed yet, KDTRACE_HOOKS is now in the GENERIC kernel in FreeBSD-current. This means you just need to load the DTrace modules and can use DTrace with the GENERIC kernel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you invest reading it.

Share

Remko Lodder: DSPAM

Since recent (with the very great help of Ion-Mihai Tetu, a fellow FreeBSD committer and developer for dspam) we (JR-Hosting) are running our anti-spam infrastructure on DSPAM. We stopped using SpamAssassin after some testing and resolving problems. The interesting fact is that we share most directories through nullfs so that both the webjail and the mailjail share data and our users are able to modify settings, see their stats etc. Very great and after overcoming our issues (local delivery was not OK in the beginning and the webjail was not able to properly use the MySQL database backend at first, which was odd because the main system WAS looking into it and the webjail wasn’t), it works just fine. Ofcourse it is still learning but it seems that it finds spam efficiently and quick, and it’s footprint is much much lower then SpamAssassin was. I might want to figure out how to run the daemonized version as per advise of Ion-Mihai, till then it works as a deliveryagent.

I am writing a ‘hosting environment howto’ (or something that will largely look like that) in which I will write about the setup as well.

Martin Wilke: [XORG-DEV] trunk is unstable

Dear All,
As I mentioned already in twitter/fb, our xorg-dev/trunk repo is currently
completely unstable because we are trying now to get xorg 7.7 RC ready. Our
wiki page will be updated soon. Testers and feedback are welcomed, but
please make sure you know what you are doing. If you like to discuss with
us directly, please join us on irc efnet/#freebsd-xorg.

- Martin

Martin Wilke: Please welcome Xorg 7.5.2

The Xorg Team is pleased to announce the next round of Xorg updates.
The team created a new flag called WITH_NEW_XORG that users can include
in /etc/make.conf. This was created for the intel KMS work being done
althouthough It probably works for other chips. Unfortunately, the intel
KMS driver will only work on FreeBSD 9(RELENG|STABLE) or 10/HEAD users.
Older version of FreeBSD will not be supported. Intel users will need
to patch their source manually with Konstantin’s KMS kernel patch to get
the newer chips to work. Please carefully read UPDATING entry.

Changes:

– libdrm 2.4.31 (including KMS support)
– mesa 7.11.2
– xorg-server 1.10.6
– a lot of new Graphic Drivers.

I would like to thank:

Koop Mast
Eitan Adler
Niclas Zeising
and all helpers and testers from x11@.

Alexander Leidinger: Linuxulator progress

This weekend I made some progress in the linuxulator:

  • I MFCed the reporting of some linux-syscalls to 9-stable and 8-stable.
  • I updated my linuxulator-dtrace patch to a recent -current. I already compiled it on i386 and arundel@ has it compiled on amd64. I counted more than 500 new DTrace probes. Now that DTrace rescans for SDT probes when a kernel module is loaded, there is no kernel panic anymore when the linux module is loaded after the DTrace modules and you want to use DTrace. I try to commit this at a morning of a day where I can fix things during the day in case some problems show up which I did not notice during my testing.
  • I created a PR for portmgr@ to repocopy a new linux_base port.
  • I set the expiration date of linux_base-fc4 (only used by 7.x and upstream way past its EoL) and all dependent ports. It is set to the EoL of the last 7.x release, which can not use a later linux_base port. I also added a comment which explains that the date is the EoL of the last 7.x release.

Share

Adrian Chadd: And the winner of the most committing committer to src/sys over the last 12 months is ..

.. well, it was me. Up until last week. marius@ snuck in to take my place. I guess all of those commits I did about this time last year to start bringing FreeBSD's ath(4) support up to scratch are finally expiring out of the 12 month window.

(Source: http://people.freebsd.org/~peter/commits.html)

.. but I wouldn't call myself the most important committer. Or the most active. What I'd call myself is the "most active fixing a sorely needed corner of the codebase."

What I _could_ have done is simply do all my work in a branch and then merge it back into -HEAD when I was done. And, for about 6 months, this is what I did. The "if_ath_tx" branch is where I did most of the initial TX aggregation work.

But as time goes on, your branch diverges more and more from the master branch (-HEAD in FreeBSD) and you are faced with some uncomfortable decisions.

If you stay on the same branch point and never merge in anything from your master branch, you _may_ have a stable snapshot of code, but who knows how stable (or relevant) your work will be when you merge it back into master.

You have no idea if your work will break anything in master and you have no idea if changes in master have broken your work.

As time goes on, the delta between your branch point and the master branch increases, making it even more difficult to do that final merge back. It also has the side effect of making it increasingly likely that problems will occur with the merge (your code breaking master, master breaking your code, etc.)

So as uncomfortable as it was - and as much as I wanted things to stay stable - I did press through with relatively frequent merging. This means:
  • I would pick specific development targets to work towards, at which point I'd stop developing and go into a code review/tidyup/testing phase;
  • I'd do frequent merges from master back into my branch during active development - I wouldn't leave this until I was ready to merge my work back into master;
  • Once I reached my development target and had done sufficient testing - including integrating changes from master back into my branch - I'd kick off a semi-formal review (read: email freebsd-wireless@) and call for testers/review;
  • Only _then_ would I merge what was suitable back into master.

I wouldn't merge everything from my branch into master. In my instance, there were some debugging extensions that were easy to maintain (read: lots of device_printf() calls) but weren't suitable for FreeBSD-HEAD. But I merged the majority of my work each time.

But that doesn't always work. I managed to merge a bunch of ath(4), ath_hal(4) and net80211 fixes back into -HEAD as appropriate. But the TX aggregation code was .. well, rather large. So I attempted to break up my commit into as many small, self-contained functional changes as possible. Yes, there was a big "here's software TX queue and aggregation" as a big commit at the end but I managed to peel off more than 30% of that in the lead-up commits.

Why bother doing that?

Two words - version bisection. Once I started having users report issues, they would report something like "FreeBSD-HEAD revision X worked, revision Y didn't." (If I were lucky, of course.) Or, they'd note that a certain snapshot from a certain day worked, but the next day had a regression. If I had committed everything as one enormous commit after having spent 6 + months on the branch, I'd be in for a whole lot of annoying line-by-line debugging of issues. Instead, I was able to narrow down most of the regressions by trying all the different commits.

Now that 802.11n ath(4) TX aggregation and general 802.11n support is in the tree, I only use branches for larger scale changes that take a couple of weeks. For example, when fixing up the reset path to not drop any TX/RX frames. I do most of the bugfixing in FreeBSD-HEAD. I could do it in a branch and then do monthly merges, but I then have the same problems I've listed above.

In summary: don't underestimate how helpful it is to break down your commits into little, piecemeal, self-contained functional changes. It has the side effect of making you look really good in the committer statistics.

Adrian Chadd: The initial introduction into "it’s the NIC, stupid!"

I've finally hit that dreaded condition which hardware guys hate - where a NIC is behaving badly.

In my case, an IBM/Lenovo Thinkpad T60 has been modified (not by me) to take an Atheros AR9280 NIC. Unfortunately, the NIC was proving to be very unstable when doing 802.11n throughput. The investigations did show I was doing something slightly incorrect with TX descriptors (and I've since fixed that) but the stability issues remained.

The Atheros NICs can expose some host interface error conditions via the AR_INTR_SYNC_CAUSE register. These include PCI(e) transaction timeouts, illegal chip access (eg whilst the MAC is asleep), parity errors, and other rather nice things. FreeBSD's HAL and Linux ath9k does have the register definition for what the bits do - but unfortunately we don't keep statistics.

In my particular case:
  • I'd see AR_INTR_SYNC_LOCAL_TIMEOUT occur. This is because a PCI(e) transaction didn't complete in time. I can tune these timeouts via a local register but that's not the point - I was seeing these errors when receiving only beacons from the access point. That's a bit silly.
  • I'd also see AR_INTR_SYNC_RADM_CPL_DLLP_ABORT, which is an indication that the PCIe layer isn't behaving well.

I swapped it out with another AR9280 based NIC and suddenly all the instabilities have gone away. No TX hangs, no missed TX interrupts. Everything looks great.

So as an open source developer, I want to try and put some tools into the hands of the community to be able to debug what's going on - or, if that's not possible, at least get some indication that things are going wrong. Right now the only thing people see is "I see TX timeouts, it must be the driver/chip fault." There's too much going on to be able to conclude that.

My game plan is this:

  • Implement statistics keeping for each of the SYNC interrupts and expose those via a diagnostic interface. Ben Grear has done something similar for Linux ath9k after a private email discussion. He's also seeing MAC sleep accesses, so it's quite likely we'll start finding/squishing these.
  • Take the offending laptop/NIC to the office and attach it to a very expensive and fancy looking PCIe analyser. I'm hoping we'll find something really silly occuring - like lots of sleep state transitions, or a high number of parity errors.
  • Try documenting this a lot better so users are able to understand what's going on when their NIC is misbehaving.

Ivan Voras: Google Summer of Code

Time flies! The last year's barely passed and here we are again - another Google Summer of Code. Even though the deadline for student's submissions is less than 24 hours ahead, I'd still take the opportunity to call anyone interested, anywhere in the world, to submit a proposal - it's fun to participate!

Read more...

Dag-Erling Smorgrav: On testing

Last fall, I wrote a completely new configuration parser for OpenPAM Lycopsida. Although the new parser was far more robust than the one it replaced, it was large, unwieldy, and suffered from a number of issues relating to whitespace handling, which stemmed from reusing some old code which unfortunately was thoroughly documented and therefore could not be easily modified. So I decided to rewrite it again, from scratch this time.

Then I did what I should have done last fall but didn't: I wrote some unit tests. And of the first dozen or so tests I came up with, three failed, revealing two different bugs—one of them fairly serious.

There's a lesson in here somewhere...

Brad Davis: PXE Booting FreeBSD 9

I have thrown together a quick guide to get FreeBSD 9 to PXE Boot:

http://freebsd.so14k.com/freebsd9_pxe.shtml

In FreeBSD 9, a few things have changed. If you have an old PXE environment from FreeBSD 8, you will want to make note of the following:

  • No more mfsroot.
  • Which means, no more changes to /boot/loader.conf, it should be empty infact.
  • You need the new pxeboot binary from 9, do not try using an old one.

Brad Davis: anoncvs1/cvsup14 Update

Just a quick note. I have cvsup14 running again, and has been for awhile.

Anoncvs I have not gotten around to setting up yet, hopefully soon.

Remko Lodder: FIXED: FreeBSD Jails PHP dirname WordPress

Dear Reader,

I had fixed the issue. Instead of using nullfs to get access to the /usr/home directories, I am using unionfs, which basically does the same for my goals (unless someone corrects me in misunderstanding things) and this does not seem to generate the same issues. Various sites are now running happily behind the WWW Jail. Time to finish my document on how I did setup the entire beast.

Thanks all for listening, helping, and giving tips (Alexander and Miroslav!)

Remko Lodder: HELP: FreeBSD Jails PHP dirname WordPress

So, I am still building up my jail structure and the last few evenings I was testing the FreeBSD jail wrt. PHP, Apache22-mpm-itk and wordpress.

Things started to break when I redirected external traffic to the jail. It seemed that require_once(dirname(dirname(__FILE__))) . ‘/wp-load.php’; does not work from within the jail.

I decided to do a little test and testing reveals that in a stand alone configuration the dirnames behave exactly the same, in both the host and the jail. Printing the directive within WordPress (when loading the admin pages f.ex.) reveals a ‘.’ instead of the ‘/path’ . It is resolvable by adding a ‘.’ to the directive so that wp-admin/admin.php loads the ../wp-load.php file instead of ‘/path/to/wordpress/wp-load.php’. Though this sounds very sily todo.

Did someone else encounter this? I Do not want to change enforcement of the statfs to some other value since the defaults should be good enough (given the testsript).

Relevant details: the /usr/home where the public_html files live, are nullfs rw mounted from the host and are available in the jail. The jail does username/group lookups through Ldap, and can see the various users. Apache had been build with the ITK patches so that every host runs under his/her own user. I do not see obvious differences between the regular host and the jail, the only real difference is the internal/external addresses used in the vhost configuration, but that is kinda obvious to me.

Let me know :-)