Monthly Archives: January 2011

HeatMaps again…

Today I stumbled again over some HeatMaps from Brendan Gregg (of DTrace–fame). This time it was the PDF of his presentation at the LISA 2010 conference. It shows nicely how he plans to evolve it from a single-machine (like in Analytics for Oracle Storage products) to the cloud. It is a very good overview about what kind of intuitive performance visualization you can do with this.

I would love to see something for FreeBSD (or other performance data). Maybe someone could take the DTraceTazTool as a base or hint and write something generic which works for a lot of things…

There are just too much nice and interesting things out there, and not enough time for all of them.

Share

FreeBSD/mips for Cavium Octeon

I've recently improved the ABI selection in FreeBSD/mips. Now it is all handled via MACHINE_ARCH or TARGET_ARCH. mipsel and mipseb are the o32 32-bit versions. mips64eb and mips64el are for n64 64-bit versions.I've knocked together a script for building an image for the Cavium Octeon eval boards that have CF that can boot using uboot from SDK 1.9.0 and 2.0.0 (and likely earlier versions, but I've not tested them). It creates two partitions: a FAT partition for the kernel and BSD partition for the rest of FreeBSD. It also takes care of building a big endian ufs system on a little endian system. You can find the script here. I've also create an image as well you can grab here. The script contains instructions for how to create the CF image.Enjoy

FreeBSD/mips for Cavium Octeon

I've recently improved the ABI selection in FreeBSD/mips. Now it is all handled via MACHINE_ARCH or TARGET_ARCH. mipsel and mipseb are the o32 32-bit versions. mips64eb and mips64el are for n64 64-bit versions.

I've knocked together a script for building an image for the Cavium Octeon eval boards that have CF that can boot using uboot from SDK 1.9.0 and 2.0.0 (and likely earlier versions, but I've not tested them). It creates two partitions: a FAT partition for the kernel and BSD partition for the rest of FreeBSD. It also takes care of building a big endian ufs system on a little endian system. You can find the script here. I've also create an image as well you can grab here. The script contains instructions for how to create the CF image.

Enjoy

Accepting Travel Grant Applications for AsiaBSDCon 2011

Calling all FreeBSD developers needing assistance with travel expenses to AsiaBSDCon 2011.

The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please download, fill out and submit the Travel Grant Request Application by February 11, 2011 to apply for this grant.

This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc). In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.

You request funding based on a realistic and economical estimate of travel costs, accommodations, and registration or tutorial fees. If there are other sponsors willing to cover costs, such as your employer or the conference, we prefer you talk to them first, as our budget is limited. We are happy to split costs with you or another sponsor, such as just covering airfare or board.

If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your direct request to us.

We review your application and if approved, authorize you to seek reimbursement up to a limit. We consider several factors, including our overall and per-event budgets, and (quite importantly) the benefit to the community by funding your travel.

Most rejected applications are rejected because of an over-all limit on travel budget for the event or year, due to unrealistic or uneconomical costing, or because there is an unclear or unconvincing argument that funding the applicant will directly benefit the FreeBSD Project. Please take these points into consideration when writing your application.

We reimburse costs based on receipts, and by check or bank transfer. And, we do not cover your costs if you end up having to cancel your trip. We require you to submit a report on your trip, which we may show to current or potential sponsors, and may include in our semi-annual newsletter.

There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.

Accepting Travel Grant Applications for AsiaBSDCon 2011

Calling all FreeBSD developers needing assistance with travel expenses to AsiaBSDCon 2011.

The FreeBSD Foundation will be providing a limited number of travel grants to individuals requesting assistance. Please download, fill out and submit the Travel Grant Request Application by February 11, 2011 to apply for this grant.

This program is open to FreeBSD developers of all sorts (kernel hackers, documentation authors, bugbusters, system administrators, etc). In some cases we are also able to fund non-developers, such as active community members and FreeBSD advocates.

You request funding based on a realistic and economical estimate of travel costs, accommodations, and registration or tutorial fees. If there are other sponsors willing to cover costs, such as your employer or the conference, we prefer you talk to them first, as our budget is limited. We are happy to split costs with you or another sponsor, such as just covering airfare or board.

If you are a speaker at the conference, we expect the conference to cover your travel costs, and will most likely not approve your direct request to us.

We review your application and if approved, authorize you to seek reimbursement up to a limit. We consider several factors, including our overall and per-event budgets, and (quite importantly) the benefit to the community by funding your travel.

Most rejected applications are rejected because of an over-all limit on travel budget for the event or year, due to unrealistic or uneconomical costing, or because there is an unclear or unconvincing argument that funding the applicant will directly benefit the FreeBSD Project. Please take these points into consideration when writing your application.

We reimburse costs based on receipts, and by check or bank transfer. And, we do not cover your costs if you end up having to cancel your trip. We require you to submit a report on your trip, which we may show to current or potential sponsors, and may include in our semi-annual newsletter.

There's some flexibility in the mechanism, so talk to us if something about the model doesn't quite work for you or if you have any questions. The travel grant program is one of the most effective ways we can spend money to help support the FreeBSD Project, as it helps developers get together in the same place at the same time, and helps advertise and advocate FreeBSD in the larger community.

FreeBSD 8.2-RC2 Available

The second Release Candidate build for the FreeBSD-8.2 release cycle is now available. ISO images for Tier-1 architectures can be downloaded from most of the FreeBSD mirror sites. Please see the official announcement for further details about this release.

FOSDEM 2011

In a few weeks, I’ll be heading to the FOSDEM conference in Brussels again this year. I’ll spend most of my time at the FreeBSD booth for the FreeBSD Foundation, so if you’re there drop by to say hi, discuss the Foundation’s work, pick up a Foundation flyer, check out the swag, or make a donation. There will also be a BSD DevRoom where there will be some interesting presentations and discussions that I might attend. Remember, FOSDEM is free to attend. Hope to see you there!

Requesting Project Proposal Submissions

The FreeBSD Foundation is pleased to announce that we are soliciting the submission of proposals for work relating to any of the major subsystems or infrastructure within the FreeBSD operating system. Proposals will be evaluated based on desirability, technical merit and cost-effectiveness.

To find out more about the proposal process please read this PDF which is available for download from our website.

 
The deadline for proposal submissions is February 15, 2011. Notification of acceptance or denial of proposals will be by March 17, 2011.

Proposals are open to all developers, including non-FreeBSD committers, but developers without access to commit to the source tree must provide details about how the completion guidelines will achieved.

Requesting Project Proposal Submissions

The FreeBSD Foundation is pleased to announce that we are soliciting the submission of proposals for work relating to any of the major subsystems or infrastructure within the FreeBSD operating system. Proposals will be evaluated based on desirability, technical merit and cost-effectiveness.

To find out more about the proposal process please read this PDF which is available for download from our website.

 
The deadline for proposal submissions is February 15, 2011. Notification of acceptance or denial of proposals will be by March 17, 2011.

Proposals are open to all developers, including non-FreeBSD committers, but developers without access to commit to the source tree must provide details about how the completion guidelines will achieved.

FreeBSD Foundation at FOSDEM

Erwin Lansing from the FreeBSD Foundation will be available at the FreeBSD booth during FOSDEM, to be held in Brussels, Belgium, February 5-6. If you're at this event, drop by to say hi, discuss the Foundation's work, pick up a Foundation flyer, check out the swag, or make a donation. FOSDEM is free to attend.

There will be a BSD devroom during FOSDEM with network/internet connectivity and projectors. DevRooms are a place for teams to discuss, hack and publicly present latest directions, lightning talks, news and discussions.

Philip Paeps, a FOSDEM organizer and FreeBSD committer, will be proctoring the BSDA examination. More information about the examination is available here.

FreeBSD Foundation at FOSDEM

Erwin Lansing from the FreeBSD Foundation will be available at the FreeBSD booth during FOSDEM, to be held in Brussels, Belgium, February 5-6. If you're at this event, drop by to say hi, discuss the Foundation's work, pick up a Foundation flyer, check out the swag, or make a donation. FOSDEM is free to attend.

There will be a BSD devroom during FOSDEM with network/internet connectivity and projectors. DevRooms are a place for teams to discuss, hack and publicly present latest directions, lightning talks, news and discussions.

Philip Paeps, a FOSDEM organizer and FreeBSD committer, will be proctoring the BSDA examination. More information about the examination is available here.

FreeBSD laptop as a Wireless Access Point for an iPhone

Recently I was on a holiday where the provider of my iPhone had no signal, but where the provider of my 3G modem for the laptop did have a signal. At least my glass was half-full!

In the past I have tried to setup Bluetooth between my laptop and my iPhone, and that resulted in a night of hard work and no effort. This time I tried a different approach: Instead of using Bluetooth for communication, I transformed the FreeBSD laptop into a wireless access point.

The command to change the wireless card from a normal client to a wireless access point are:

[~] edwin@lappie>cat wlan-iphone
#!/bin/sh ifconfig wlan0 destroy
ifconfig wlan0 create \ wlandev ath0 \ wlanmode hostap \ bssid \ authmode open \ ssid "My iPhone WiFi" ifconfig wlan0 up
ifconfig wlan0 inet 10.0.0.1 netmask 255.255.255.0
sleep 1 sysctl -a net.inet.ip.forwarding=1 service isc-dhcpd restart

Notes:

  • Experiment with the "open" authmode before you set it to a more secure alternative. Get things working first, then secure then properly.
  • The interface "ath0" is the Atheros wireless interface of this laptop.

The 3G connection is setup via ppp(8) and to enable NAT on the outgoing packets, you need to enter the following command or add it to the right label in your ppp.conf:

ppp ON lappie> dial
Ppp ON lappie>
PPp ON lappie>
PPp ON lappie> Warning: 0.0.0.0/0: Change route failed: errno: No such process
PPP ON lappie> nat enable yes
PPP ON lappie> 

And to make sure that the connected clients get their IP address, you should run the ISC DHCP server with for example the following configuration:

option domain-name "";
option domain-name-servers 8.8.8.8; default-lease-time 150;
max-lease-time 300; ddns-update-style none; authoritative; log-facility local7; subnet 10.0.0.0 netmask 255.255.255.0 { range 10.0.0.10 10.0.0.99; option routers 10.0.0.1;
}

Notes:

  • 8.8.8.8 is the Google DNS server.

Everything is working now, your glass is full again! :-)

FreeBSD laptop as a Wireless Access Point for an iPhone

Recently I was on a holiday where the provider of my iPhone had no signal, but where the provider of my 3G modem for the laptop did have a signal. At least my glass was half-full!

In the past I have tried to setup Bluetooth between my laptop and my iPhone, and that resulted in a night of hard work and no effort. This time I tried a different approach: Instead of using Bluetooth for communication, I transformed the FreeBSD laptop into a wireless access point.

The command to change the wireless card from a normal client to a wireless access point are:

[~] edwin@lappie>cat wlan-iphone
#!/bin/sh ifconfig wlan0 destroy
ifconfig wlan0 create \ wlandev ath0 \ wlanmode hostap \ bssid \ authmode open \ ssid "My iPhone WiFi" ifconfig wlan0 up
ifconfig wlan0 inet 10.0.0.1 netmask 255.255.255.0
sleep 1 sysctl -a net.inet.ip.forwarding=1 service isc-dhcpd restart

Notes:

  • Experiment with the "open" authmode before you set it to a more secure alternative. Get things working first, then secure then properly.
  • The interface "ath0" is the Atheros wireless interface of this laptop.

The 3G connection is setup via ppp(8) and to enable NAT on the outgoing packets, you need to enter the following command or add it to the right label in your ppp.conf:

ppp ON lappie> dial
Ppp ON lappie>
PPp ON lappie>
PPp ON lappie> Warning: 0.0.0.0/0: Change route failed: errno: No such process
PPP ON lappie> nat enable yes
PPP ON lappie> 

And to make sure that the connected clients get their IP address, you should run the ISC DHCP server with for example the following configuration:

option domain-name "";
option domain-name-servers 8.8.8.8; default-lease-time 150;
max-lease-time 300; ddns-update-style none; authoritative; log-facility local7; subnet 10.0.0.0 netmask 255.255.255.0 { range 10.0.0.10 10.0.0.99; option routers 10.0.0.1;
}

Notes:

  • 8.8.8.8 is the Google DNS server.

Everything is working now, your glass is full again! :-)

4k drive update

Just to let you know what the current status is wrt. 4k drives:

It looks like the consensus in the industry (meaning everyone except Western Digital) is to announce dual sector sizes, i.e. 512-byte logical sectors on top of 4096-byte physical sectors.

Ivan Voras has taken the initiative to organize a 4k BoF at BSDCan, although judging from the (private) email exchange on the subject, it's quite possible that a decision will be made before then. Currently, it looks like we're moving towards having the low-level driver report a 512-byte sector size and 4096-byte stripe width (and, if necessary, an appropriate offset) to GEOM. This preserves backward compatibility, but announces to GEOM consumers that it is a good idea to do I/O in 4096-byte blocks and align data structures on 4096-byte boundaries. All that remains is then to make sure that those GEOM consumers we care about (particularly ZFS) take advantage of this information.

The situation for WD “Advanced Format” drives is a bit more complex, because they announce 512-byte logical sectors. The only solution I can see is to add a quirk system to the ada driver (and possibly to ata as well, if we still care about it) similar to the ones we have for SCSI and USB devices, and match the model number. I believe /WD\d+[A-Z]+RS/ should match all existing Advanced Format drives with no false positives.

4k drive update

Just to let you know what the current status is wrt. 4k drives:

It looks like the consensus in the industry (meaning everyone except Western Digital) is to announce dual sector sizes, i.e. 512-byte logical sectors on top of 4096-byte physical sectors.

Ivan Voras has taken the initiative to organize a 4k BoF at BSDCan, although judging from the (private) email exchange on the subject, it's quite possible that a decision will be made before then. Currently, it looks like we're moving towards having the low-level driver report a 512-byte sector size and 4096-byte stripe width (and, if necessary, an appropriate offset) to GEOM. This preserves backward compatibility, but announces to GEOM consumers that it is a good idea to do I/O in 4096-byte blocks and align data structures on 4096-byte boundaries. All that remains is then to make sure that those GEOM consumers we care about (particularly ZFS) take advantage of this information.

The situation for WD “Advanced Format

Ports Feature Freeze for 7.4 and 8.2

phase is under way.
Normal upgrade, new ports, and changes that only affect other branches
will be allowed without prior approval but with the extra Feature safe:
yes tag in the commit message.  Any commit that is sweeping, i.e.
touches a large number of ports, infrastructural changes, commits to
ports with unusually high number of dependencies, and any other commit
that requires the rebuilding of many packages will not be allowed
without prior explicit approval from portmgr@ after that date.
Thomas on behalf of portmgr@

AQM/ECN in FreeBSD

After reading Jim Gettys investigations about the problems current buffer sizes of network equipment provoke (which may even have implications in the net neutrality debate), I had a look at which active queue management (AQM) algorithms with or without explicit congestion notification (ECN) FreeBSD supports.

It looks like there is not much implemented (if the best solution would be implemented, it would not matter how much there is, but unfortunately there is no best solution). Other systems offer more. RED is implemented, but even the inventor/researcher of RED thinks the algorithm needs some improvements (he is in the process of preparing a paper about this, as Jim Gettys reveals). Blue/SFBlue is not implemented (a more turnkey-solution than the current RED implementation). PID controller (which may or may not be something someone wants to use in this case… no idea about its pros/cons in this regard, but it is referenced in the AQM article on Wikipedia) is also not implemented.

Regarding ECN for FreeBSD you can find more or less no real documentation in the net (at least with a simple “ECN FreeBSD

How big are the buffers in FreeBSD drivers?

Today I have read an interesting investigation and problem analysis from Jim Gettys.

It is a set of articles he wrote over several months and is not finished writing as of this writing (if you are deeply interested in it go and read them, the most interesting ones are from December and January and the comments to the articles are also contributing to the big picture). Basically he is telling that a lot of network problems users at home (with ADSL/cable or WLAN) experience  are because buffers in the network hardware or in operating systems are too big. He also proposes workarounds until this problem is attacked by OS vendors and equipment manufacturers.

Basically he is telling the network congestion algorithms can not do their work good, because the network buffers which are too big come into the way of their work (not reporting packet loss timely enough respectively try to not lose packets in situations where packet loss would be better because it would trigger action in the congestion algorithms).

He investigated the behavior of Linux, OS X and Windows (the system he had available). I wanted to have a quick look at the situation in FreeBSD regarding this, but it seems at least with my network card I am not able to see/find the corresponding size of the buffers in drivers in 30 seconds.

I think it would be very good if this issue is investigated in FreeBSD, and apart from maybe taking some action in the source also write some section for the handbook which explains the issue (one problem here is, that there are situations where you want/need to have such big buffers and as such we can not just downsize them) and how to benchmark and tune this.

Unfortunately I even have too much on my plate to even further look into this. :( I hope one of the network people in FreeBSD is picking up the ball and starts playing.

Share