Monthly Archives: March 2012

Interview with Kris Moore

As part of their Developer’s Corner series, iXsystems has posted an interview with lead PC-BSD developer, Kris Moore. The interview hints at some of the exciting features being developed for 9.1.

Speaking of exciting features, we expect the next testing snapshot to be ready in a week or so. This snapshot will include the changes to Warden which will make it easier than ever to manage jails on a PC-BSD system.

brd’s notes

I have thrown together a quick guide to get FreeBSD 9 to PXE Boot:

http://freebsd.so14k.com/freebsd9_pxe.shtml

In FreeBSD 9, a few things have changed. If you have an old PXE environment from FreeBSD 8, you will want to make note of the following:

  • No more mfsroot.
  • Which means, no more changes to /boot/loader.conf, it should be empty infact.
  • You need the new pxeboot binary from 9, do not try using an old one.

Evilcoder

Dear Reader,

I had fixed the issue. Instead of using nullfs to get access to the /usr/home directories, I am using unionfs, which basically does the same for my goals (unless someone corrects me in misunderstanding things) and this does not seem to generate the same issues. Various sites are now running happily behind the WWW Jail. Time to finish my document on how I did setup the entire beast.

Thanks all for listening, helping, and giving tips (Alexander and Miroslav!)

Remko Lodder » FreeBSD 2012-03-20 13:00:26

So, I am still building up my jail structure and the last few evenings I was testing the FreeBSD jail wrt. PHP, Apache22-mpm-itk and wordpress.

Things started to break when I redirected external traffic to the jail. It seemed that require_once(dirname(dirname(__FILE__))) . ‘/wp-load.php’; does not work from within the jail.

I decided to do a little test and testing reveals that in a stand alone configuration the dirnames behave exactly the same, in both the host and the jail. Printing the directive within WordPress (when loading the admin pages f.ex.) reveals a ‘.’ instead of the ‘/path’ . It is resolvable by adding a ‘.’ to the directive so that wp-admin/admin.php loads the ../wp-load.php file instead of ‘/path/to/wordpress/wp-load.php’. Though this sounds very sily todo.

Did someone else encounter this? I Do not want to change enforcement of the statfs to some other value since the defaults should be good enough (given the testsript).

Relevant details: the /usr/home where the public_html files live, are nullfs rw mounted from the host and are available in the jail. The jail does username/group lookups through Ldap, and can see the various users. Apache had been build with the ITK patches so that every host runs under his/her own user. I do not see obvious differences between the regular host and the jail, the only real difference is the internal/external addresses used in the vhost configuration, but that is kinda obvious to me.

Let me know :-)

Concurrency in the TX path and when it all falls down..

I'm still (yes, still) hacking on the FreeBSD 802.11n ath(4) TX path. I'm trying to find and fix issues that creep up before I can flip on the 11n code by default.

I've become increasingly aware of the lack of locking in net80211, including some of the A-MPDU TX session management code. I know that I'll eventually have to plan out and implement some locking, but for now I'm just trying to squish whatever issues are showstoppers.

A user approached the freebsd-wireless list about two weeks ago and noticed that his 802.11n session was hanging after a bit of use. If he tried 802.11g, things worked fine. He tried SMP - and things worked fine. The symptoms? A number of frames seemed to be "stuck" and sitting in a software TX queue, not being transmitted. But other frames would be TXed fine, so it wasn't as simple as a totally confused TX BAW (Block-ack Window.)

After I managed to reproduce the issue locally, I discovered what was going on.

It was concurrency.

Specifically, that there's multiple places where ath_start() is being called, thus multiple concurrent TX is occuring.

Now, in the non-aggregate method, net80211 is doing all the sequence number assignment. I'm not so sure that in the normal case, the sequence numbers are being allocated in a consistent, sequential way - if the net80211 TX code is able to be called concurrently from multiple threads, sequence numbers can and will be occasionally "raced" and allocated in the reverse order that they're submitted to the driver. But I'm not here to fix that (however I'll eventually have to.)

In the aggregate method, the driver is doing the aggregation and sequence number assignment. For now, the driver is also doing the TX BAW tracking and frame queuing. So imagine this sequence of events:

* a frame is submitted via ath_start();
* since aggregation is enabled, it's allocated a sequence number;
* it's then thrown into the software queue;
* then at some later stage, the software queue is checked, the frame is popped out of the list and if it's inside the BAW, it's added to the BAW and TX;ed
* adding the frame to the TX BAW slides the left hand edge of the BAW to be at that sequence number. Any frames TXed with a seqno _less_ than this will be treated as outside of the BAW and put back onto the software queue for now.

Now, the locking only occurs at:

* the time the frame has the sequence number allocated;
* then when the queue is checked and the frame is popped off.

If two or more threads are allocating sequence numbers and doing work, it's quite possible that thread A will allocate (say) seqno 5, thread B will allocate seqno 6 and then add it to the BAW before thread A can. Then when thread A tries to do some work, it finds the queue has a frame with seqno 5 in it - and since it's before the left hand edge of the BAW (which is at 6, as it was successfully pushed to the hardware), it won't be transmitted and will stay in the software queue until the BAW sequence numbers wrap around to 0 and catch up.

Now, linux ath9k/mac80211 doesn't have this problem. The TX pathway is totally serialised, which means that even if multiple threads are trying to TX, only one thread will be able to enter the TX code at any time. The other threads get blocked.

So how can I solve this? The easy solution would be to serialise FreeBSD's net80211 and ath driver TX code. That way all of this nonsense will go away. For the net80211 side of things this may work - the legacy TX path, where sequence numbers are allocated by the stack, could benefit from serialisation. The throughput isn't ever going to be that great, so we wouldn't really hurt from it. But the trouble is making absolutely sure that the driver also does the same - even if I push the frames into the queue in order and ensure that they have sequential sequence numbers, there's no guarantee that a driver with concurrent entry paths into XXX_start() will de-queue the frames and push them into the hardware in the same order.

So what I've chosen to do instead is to ignore the legacy part for now, and not serialise anything. Instead, I'm doing the sequence number allocation (for aggregation, remember) -at the time I'm about to add the frame to the BAW and TX it to the hardware-. Ie, until the frame actually is able to be added to the BAW, it won't _have_ a sequence number. Since this action is done behind a lock, it's guaranteed to be sequential. The trick here is to only allocate the sequence numbers once it's known for certain that the frame _will_ be going out to the hardware.

For the legacy path, it's also likely worthwhile delaying the sequence number allocation until it's just about to go to the ifnet queue. That way the frames on the ifnet queue have sequence numbers that are in order. Then I need to fix each driver (ugh) to make sure they're dequeued fine.

I've written up the aggregation change and this so far works quite well. I don't want to tackle the legacy path yet or fix other drivers, not until I've verified this works. What I also should do is write some test cases to verify that indeed sequence numbers are being presented to the driver in order, so I can identify this happening in the wild.

Downtime

I haven’t been able to read email sent to [email protected] or [email protected] for five days, due to a series of unfortunate incidents involving dodgy power supplies and the fragility of ZFS boot in FreeBSD. Work and other duties prevented me from addressing the issue in a more timely manner, but I am now regaining control. Luckily, neither my ~30 GB IMAP spool nor any other data was lost, nor did my backup MX bounce any mail. My IMAP server is now back up with a small UFS SU+J boot / root partition instead of ZFS. I am still unable to read email, but that should be fixed within 24 hours.

I also uncovered an annoying but luckily not fatal bug in the Cyrus IMAP server. When TLS is configured, the IMAP daemon stores state for each TLS session in a DB file. If that file is corrupted, the server will start, but it will refuse any incoming IMAP or LMTP connections, and will instead spit out a stream of completely unhelpful error messages. The only recourse is to delete the TLS session state database; I set up an rc script to do that at boot time, so hopefully this won’t bite me again.

Downtime

I haven't been able to read email sent to [email protected] or [email protected] for five days, due to a series of unfortunate incidents involving dodgy power supplies and the fragility of ZFS boot in FreeBSD. Work and other duties prevented me from addressing the issue in a more timely manner, but I am now regaining control. Luckily, neither my ~30 GB IMAP spool nor any other data was lost, nor did my backup MX bounce any mail. My IMAP server is now back up with a small UFS SU+J boot / root partition instead of ZFS. I am still unable to read email, but that should be fixed within 24 hours.

I also uncovered an annoying but luckily not fatal bug in the Cyrus IMAP server. When TLS is configured, the IMAP daemon stores state for each TLS session in a DB file. If that file is corrupted, the server will start, but it will refuse any incoming IMAP or LMTP connections, and will instead spit out a stream of completely unhelpful error messages. The only recourse is to delete the TLS session state database; I set up an rc script to do that at boot time, so hopefully this won't bite me again.

Alexander Leidinger

I just updated to a recent -current and tried the new nullfs. Sockets (e.g. the MySQL one) work now with nullfs. No need to have e.g. jails on the same FS and hardlink the socket to not need to use TCP in MySQL (or an IP at all for the jail).

Great work!

Share

Why Juniper Gives Back to the FreeBSD Community

Michael Bushong, Senior Director of Product Management for Junos Software at Juniper Networks, wrote today on Giving Back to the FreeBSD Community.  With his permission, his text is repeated here:

I have worked at Juniper in various capacities for the past 11 years. I started off as an engineering trainer, teaching our engineers how to develop on Junos. The classes I developed and delivered included a heavy dose of architecture, a bit of product training, and some obligatory process topics. In the hundreds of classes I delivered over the first few years of my career here, I learned pretty quickly that my students had an emotional tie to two specific areas.

Let’s face it, engineers love architecture. Our software and hardware developers love hearing the ins and outs of how we design and construct our platforms. But beyond that, they wanted to know about why we made certain decisions and what our more philosophical positions were. To that end, the stuff that really killed in these classes was the magic behind our engineering DNA and what was important to us as a company. Looking back now this might not seem surprising, but in the moment I guess I didn’t fully appreciate how deeply the need to understand not just what we do but why we do it is ingrained in an engineer’s psyche. .

So after 11 years at the company, how would I describe our engineering philosophy? In a word, maybe it all comes down to karma. We all accept help when we need it, but it is perhaps more important to give back when you can. So whenever I get a chance to do something meaningful for the people and organizations who have helped us become a networking leader, I jump at the opportunity.

Juniper’s flagship software, Junos, has a 15-year history with the FreeBSD community. At the heart of Juniper’s network operating system is a FreeBSD kernel. FreeBSD provides the underlying software architecture on top of which Juniper delivers its networking capabilities. The networking functionality is basically a set of applications that run on top of FreeBSD. The kernel then handles all the communication between these applications, along with basic OS stuff like memory allocation and scheduling.

Juniper doesn’t just use FreeBSD, we actually expose it to our customers. For instance, if you log onto a Juniper platform, you can access a FreeBSD shell, which gives you some pretty nifty tools. You get access to a file system so you can move things around (configurations, logs, etc). We actually make extensive use of this internally in our lab environment. We have a system set up so we automount directories so that our engineers can just copy configurations and binaries from their Unix directories onto devices they are testing on. We also use many of FreeBSD’s debug and tracing faculties. In that regard, FreeBSD is more than just a component in our software; it is in fact a big part of engineering life more broadly.

When you benefit so heavily from the work of so many selfless individuals, you just have to give back when you can. So when I found out that the FreeBSD assets were moving from a datacenter in Santa Clara to another datacenter literally right across the street, I jumped at the opportunity to help. The cluster administrator for the hosting datacenter reached out to one of our FreeBSD committers and asked if we could provide some switches so they could support the work. In response, we gave them three EX3200s with full support contracts – free of charge.

In the end, this is just one more way that we can help serve a broader community. It wasn’t the first and it won’t be the last time that we try to pitch in and help out. But the experience reinforced what I knew from my years as a trainer. Our engineering DNA is strong here, and when we have the chance to give back, people will go out of their way to be a part of it.

New CentOS linux_base for testing soonish

It seems my HOWTO create a new linux_base port was not too bad. There is now a PR for a CentOS 6 based linux_base port. I had a quick look at it and it seems that it is nearly usable to include into the Ports Collection (the SRPMs need to be added, but that can be done within some minutes).

When FreeBSD 8.3 is released and the Ports Collection open for sweeping commits again, I will ask portmgr to do a repo-copy for the new port and commit it. This is just the linux_base port, not the complete infrastructure which is needed to completely replace the current default linuxulator userland. This is just a start. The process of switching to a more recent linux_base port is a long process, and in this case depends upon enough support in the supported FreeBSD releases.

Attention: Anyone installing the port from the PR should be aware that using it is a highly experimental task. You need to change the linuxulator to impersonate himself as a linux 2.6.18 kernel (described in the pkg-message of the port), and the code in FreeBSD is far from supporting this. Anyone who wants to try it is welcome, but you have to run FreeBSD-current as of at least the last weekend, and watch out for kernel messages about unsupported syscalls. Reports to [email protected] please, not here on the webpage.

Share

New opportunities in the linuxulator

Last weekend I committed some dummy-syscalls to the linuxulator in FreeBSD-current. I also added some comments to syscalls.master which should give a hint which linux kernel had them for the first time (if the linux man–page I looked this up in is correct). So if someone wants to experiment with a higher compat.linux.osrelease than 2.6.16 (as it is needed for a CentOS based linux_base), he should now get some kernel messages about unimplemented syscalls instead of a silent failure.

There may be some low-hanging fruits in there, but I did not really verify this by checking what the dummy syscalls are supposed to do in linux and if we can easily map this to existing FreeBSD features. In case someone has a look, please send an email to emulation@FreeBSD.org.

Share

PC-BSD Teams With DuckDuckGo to Provide Enhanced, Secure Web Searches

From the press release:

DuckDuckGo is a general-purpose search engine that offers real privacy, less spam/clutter and instant answers. Well-known and widely acclaimed for its privacy policy, DuckDuckGo does not store or track user information, ensuring a truly private search experience.

The most recent edition of PC-BSD updates the suggestion list of the search bar with DuckDuckGo to provide users with a discreet, clutter-free search option. Additionally, DuckDuckGo offers many benefits including the ability to use shortcuts to directly search many websites and instant answers that provide topic summaries from a variety of web sources.

PC-BSD users concerned with security can be confident in the knowledge that requests submitted through DuckDuckGo will remain confidential. “We are pleased to make DuckDuckGo available to PC-BSD users, providing a reliable, yet completely anonymous search experience,

System Update for LXDE

If you have installed the LXDE window manager, a system update is now available in Update Manager that addresses several bugs reported by users. The description for the update indicates that changes to menu-cache-0.3.2_2 were implemented in order to fix an issue within the LXDE desktop which caused the menu to constantly refresh every 2–4 seconds.

Call for Testers: Network Manager

We’re looking for testers for Control Panel -> Network Manager. In particular, we need feedback from those who use 3G or PPP to connect. We’ve gotten feedback from several non-native English speakers who are new to BSD networking that the 3G/PPP tab doesn’t work “out of the box

Ports Feature Freeze for FreeBSD 8.3 is now in effect

FreeBSD 8.3 RC1 has been pulicly announced, it is now time for the the
Ports Feature Freeze.

Normal upgrade, new ports, and changes that only affect other branches
will be allowed without prior approval, but with the extra

Feature safe: yes

tag in the commit message. Any commit that is sweeping, i.e. touches a
large number of ports, infrastructural changes, commits to ports with
unusually high number of dependencies, and any other commit that requires
the rebuilding of many packages will not be allowed without prior explicit
approval from portmgr@ after that date.

Thomas
on behalf of portmgr@