Monthly Archives: November 2009

New Gecko World coming :)

Here is a small Status update about our work on gecko stuff: beat@ already removed all references to www/mozilla. Also the switch from USE_GECKO= xulrunner firefox mozilla to USE_GECKO= libxul is mostly done. Some small parts are still in the tree and we are working on them. Afterwards xulrunner will be removed. Finally the first steps for upcoming Thunderbird3 and [...]

Announcing GNOME 2.28.1. for FreeBSD!

Presenting GNOME 2.28.1 for FreeBSD. The official release notes for this release can be found at http://library.gnome.org/misc/release-notes/2.28/ . Officially, this is mostly a polishing release in preparation for GNOME 3.0 due in about a year.On the FreeBSD front, though, a lot went into this release. Major thanks goes to kwm and avl who did a lot of the porting work for this release. In particular, kwm brought in Evolution MAPI support for better Microsoft Exchange integration. Avl made sure that the new gobject introspection repository ports were nicely compartmentalized so that large dependencies aren't brought in wholesale.But, every GNOME team member (ahze, avl, bland, kwm, mezz, and myself) contributed to this release.Other major improvements include an updated HAL with better volume probing code, ufsid integration, and support for volume names containing spaces (big thanks to J.R. Oldroyd); a new WebKit; updated AbiWord; an updated Gimp; and a preview of the new GNOME Shell project (thanks to Pawel Worach).The FreeBSD GNOME Team would like to that the following additional contributors to this release whose patches and testing really helped make it a success: Andrius Morkunas Dominique Goncalves Eric L. Chen J.R. Oldroyd Joseph S. Atkinson Li Pawel Worach Romain Tarti?re Thomas Vogt Yasuda Keisuke Rui Paulo Martin Wilke (and an extra shout out to miwi and pav for the pointyhat runs) We would like to send this release out to Alexander Loginov (avl) in hopes that he feels better soon.

OpenRD console access on FreeBSD.

This morning, I received the Open-RD I bought a few days ago and started playing with my new device. One of the first thing I wanted to access was (of course) the system console. While the default setup provides a SSH daemon and even GDM and a full desktop, my goal is to have FreeBSD on this device and move the services I run at home from my personal computer to this low-consumption computer (and I don't intend to switch to GNU/Linux).

Since the console is usually available on the serial port, I looked for some cable to connect the OpenRD to my computer. Unfortunately, I didn't have the required cable. I then spent about two hour with various serial cables, cables looking like serial cables but which where not serial cables, parallel SUB-D25 connectors, a serial SUB-D25 cross-over cable and a soldering iron to build a null-modem SUB-D9 serial cable. Unfortunately, when I connected my computer to the OpenRD, I was still not able to access the console using cu(1).

After reading a bit of documentation, I realised I missed the fact that the console was available on USB and not via the serial port. However, FreeBSD did not recognised the chip the OpenRD uses to provide serial over USB support.

Using usbconfig(8), I could get the vendor ID and product ID the uftdi(4) driver was lacking. I patches the source code and a few minutes later could enjoy a set of new devices when I plug the OpenRD on my computer:

% ls /dev/cuaU*
/dev/cuaU0
/dev/cuaU0.init
/dev/cuaU0.lock
/dev/cuaU1
/dev/cuaU1.init
/dev/cuaU1.lock

I can now access the OpenRD console using cu(1):

% cu -l /dev/cuaU1 -s 115200

\o/ I can boot a FreeBSD 8.0-STABLE arm kernel!

I filled-in the usb/140951 problem report in the FreeBSD GNATS with the patch. If you own an OpenRD and want to have access to the console, the patch is just waiting for you.

FreeBSD 8.0 released into the wild

After the final preparations have been wrapped up, FreeBSD 8.0 is a fact and ready for download or upgrade. The all the new features and improvements are too numerous to mention, ranging from networking, scalability, virtualization, wireless, ZFS to USB. See the official announcement, or even better, a comprehensive article by Remko Lodder, for more information.

Related posts:

  1. FreeBSD Download statistics FreeBSD download statistics disc1 and dvd1 ISO downloads per architecture...

Related posts brought to you by Yet Another Related Posts Plugin.

Welcome to FreeBSD 8!

Welcome to FreeBSD 8!

In this article I will write about the latest release from FreeBSD, 8.0. This is a major version that offers new functionality and much improved parts of the code.

Why are major releases so special?

Major releases offer the possibility to include changes in the system that are not allowed in stable branches. This could be new applications or API/ABI interfaces, or serious changes to datastructures and things like that.

FreeBSD brings out major releases every 18 months. Ofcourse the 18 months may vary, depending on the amount of new features and problems found while preparing for the first .0 release. The schedule should guide as a reference on when to expect something new.

FreeBSD 8?

FreeBSD 8 will be the latest version of FreeBSD that is currently on the market, it was preceded by FreeBSD 6 and 7, where we found features like ZFS, Improved Jail support, the widely known Danish Axe was used to further take out the GIANT lock, and many more features that had been in the previous release.

What will FreeBSD 8 offer us?

FreeBSD 8 will offer us the following:

USB Storage hotplug removal

Previously it was not possible to take out lets say an USB storage device while the operating system was running, without crashing it. The system code is updated to make it possible to take out an USB storage device without the system crashing. This will make many people happy!

Multi-IP/IPv6 jails

With FreeBSD 8 it will be possible to assign multiple IPv4 addresses to a jail, as well as assigning IPv6 addresses. People that offer hosting services can greatly enjoy these features, a jailed webserver for example can now use many more addresses and do proper SSL virtualhosting.

NFSv4 support

NFSv4 support is added to the FreeBSD Operating System, while this is still experimental, it gives you a NFSv4 client, as well as a server that can service NFSv4 nodes. NFSv4 gives you better security access control’s, a stateful protocol, performance improvements, etc.

Enhanced ULE scheduler

The ULE scheduler is much improved for FreeBSD 8, this allows even better scheduling and responses to systems running FreeBSD 8, especially when used with SMP systems. Systems running the FreeBSD 8 operating system, have this scheduler enabled by default.

DTrace

DTrace is a suite of applications imported from Solaris. DTrace will assist you in profiling your system and applications, data which you can use to make the system perform even better under your specific configuration. Every key developer needs a tool like this.

Wireless Mesh Networking

The Wireless Mesh Networking (802.11s) is an experimental feature and early adoption framework that gives the ability to setup a Meshed Wireless network. Currently most Wireless networks are build around one central access points, with leaf nodes / bridges / repeaters attached to them. This experimental feature offers the ability to ‘perfectly’ roam accross the network, without being tied to one specific Access Point.

VIMAGE

VIMAGE, is an network virtualisation project that aims in delivering multiple networking related instances. So for example you can give your jails, their own vimage, enabling them to run IPSEC, Packet filter(s), setting their own routing tables etc. So this actually makes your jails, more or less independant servers.

Multiple routing tables

Multiple routing tables, enable you to select different routing tables for different services. For example you can setup different routes for your webserver(s) and mailserver(s). The “setfib” utility is added to provide a management layer for these FIB’s (Routing tables).

Equal Cost multipath routing

Equal-Cost Multi-Path Routing (ECMP) is a new feature under FreeBSD 8, that enables the administrator to loadbalance traffic by equally balancing traffic over various routes. In theory this could increase the bandwidth that you can use, so be sure to check this out!

FreeBSD ports Parallel building

The FreeBSD ports framework is adopted to that multiple ports can be build at once. People with multi core systems will definitly see significant improvements in this region. While this update is not limited to the 8.0 release itself, it is the first release that will have this feature onboard. (after fetching the ports tree ofcourse).

MPSafe TTY

The TTY subsystem is one of the last sections of the entire operating system that had not been rewritten or massively updated in the last decade or two. FreeBSD 8 will have a replaced version that no longer uses the GIANT lock. That reduces the impact of the lock even more and will assist with fine grained multiprocessor capabilities. It is also more modular and will make it easier to hook into the new layer. This will help in lagging console and X.org sessions.

Procstat

Procstat is a new application, eliminating the need for procfs(4), it gives the user information about processes, command line arguments, kernel thread stacks, and many other functions that could assist people in troubleshooting and debugging.

Textdumps

Textdumps had been written to make it much easier to get coredump information back from the Operating System after fatal events occured.

The application had been setup to generate a tar archive that includes several text files with valuable troubleshooting information. Users that send this tar file to one of the appropriate lists, make a much better chance in getting their problem fixed. Before this tool one needed to do this manually, often making this an impossible step for the average user. Not only will this help the user that is facing problems, but also the bugbusters that will have all required information available without needing to explain on how to get the information there.

Experimental AHCI driver

An experimental AHCI driver had been added to the tree, which gives direct and native AHCI commands through the CAM layer. This includes the NCQ feature (Native Command Queueing).

Gvinum v2

Gvinum had been updated and reworked, making it much more production ready and usable then before.

Disk utility upgrades

Several disk utility applications had been updated, or adjusted to become the default. For example the GEOM_PART utility becomes the defacto standard for disk slicing. One should pay attention when this comes in to play though, since the labels might be read differently with GEOM_PART your devices might be enumerated in a different way then before. Make sure you are actively involved with the upgrade and make sure you have serial access to the machine in case of problems!

Beyond that GPT partitions are now capable of being boot from, and the bsdlabel utility supports up to 26 partitions now.

ProPolice Stack Protector

The ProPolice SSP framework is added to the builds, protecting the kernel and userland (where configured) from being stack-smashed. This will make it harder to exploit stack based buffer overflows.

ZFS

ZFS is updated to version 13, and sees several improvements. The experimental warning is removed, making it a production ready filesystem which you can use with very large datapools. Make sure you have enough memory when playing with it though!

Wireless Virtual AP

The Wireless code is updated to support Virtual AP’s. Within the /etc/rc.conf file you are now able to clone the ‘direct hardware interface’ like ath0 into a wlan device. If you clone this more often you will be able to support more SSID’s and things like that. For small companies or freaky home users, you can setup your own protected SSID, and a guest SID which visitors can use and is protected differently.

VirtualBox Host support

The FreeBSD ports tree saw VirtualBox added to the tree, making FreeBSD 8 one of the possible candidates for Virtual Machines delivered through VirtualBox. This makes it a good alternative for VMWare.

NULL-pointer dereferences made harder

The FreeBSD Security Team included a little rewrite of the NULL pointer handling within FreeBSD. Programs are no longer able to make use of the NULL pointer by default and will be stacked on a random address location. This feature makes FreeBSD a bit safer against NULL pointer dereference vulnerabilities.

lukemftpd build disabled by default

Personally I axed the build of lukemftpd. This is one of the two ftp daemons that we have in the base system, and was opted for removal a long time ago. That had happened and the daemon is no longer available by default on the latest version of FreeBSD. Further removal of the lukemftpd software will be investigated to see what consequences will popup because of that.

How to upgrade?

Now that we spoiled you with all the new features in FreeBSD 8, there is probably interest in how one can upgrade to the FreeBSD 8 release.

Before explaining the process of upgrading the system, I expect that you already have a FreeBSD system running on 7.2 or 7-STABLE, if you are unsure on how to get that going, please refer to my previous article in BSDMag which explains this in great detail (and it includes pictures too!).

Ofcourse it is also be possible to install FreeBSD 8 from CD/DVD, please browse to ftp://ftp.langcode.FreeBSD.org/pub/FreeBSD/ISO-IMAGES-yourarchitecturehere/8.0
and fetch the proper media. All you have to do is follow the instructions that are printed on screen. If you need more details about this, there is an excellent handbook that will guide you through it, or again my previous article will assist you. The FreeBSD handbook can be found here:

http://www.FreeBSD.org/doc/en/books/handbook/

In some cases there are localised versions, like the Dutch Handbook:

http://www.FreeBSD.org/doc/nl/books/handbook

replace ‘/nl’ by ‘/yourlangcode’ to see whether your version also exists.

As mentioned we will be upgrading a running 7.2 / 7-STABLE system to FreeBSD 8.0-RELEASE.

We use “csup” for this, which will fetch our source code. This will lead up to a new system and new kernel.

For the instructions that follow, you will need “root” access, if you do not have this access, please consult your system administrator and ask him for help on how to upgrade FreeBSD as “root”.

Copy over the example csup file to /root and begin editing:

# cp /usr/share/examples/cvsup/stable-supfile /root/8-release
# vi /root/8-release

Navigate to the line which mentions ‘default host’, and change it to a server near you, so for example in the Netherlands you would use:

*default host=cvsup.nl.FreeBSD.org

Now find the part that has ‘default release’ mentioned. We need to update that as well. Change the contents of the tag parameter to RELENG_8_0. Thus the line would look like something as:

*default release=cvs tag=RELENG_8_0

Note that there is another tag that will give you the actual release bits (RELENG_8_0_0_RELEASE), but this will not get updated for Errata Notice’s and Security Advisories, making it an unwanted tag to use.

After making sure the above updates were done, we will need to update our tree. If you are a survivor of my previous article you will have an old tree and old objects tree which you might want to save for future reference.

# mv /usr/src /usr/src.releng7
# mv /usr/obj /usr/obj.releng7

Now we can download / update the new tree.

# csup /root/8-release

Wait for it to finish, and after that we can build a new kernel and world. We will not do modifications to the kernel and world this will be very straightforward.

# cd /usr/src
# make buildworld && make buildkernel

This will generate both a new world, and a new kernel, which we will use later on.

When this had been done, we will update the configuration files by issueing:

# mergemaster -p

This will update the configuration files that are needed before the new release can be started at all. If no output is given then the current configuration is sufficient to start the 8.0 version later on.

Configuration had been updated, so we can start by installing the kernel, after which we need to reboot the system:

# make installkernel
# reboot

The system will restart, when the bootloader starts, press a key and/or start the system in single user mode. In case you do not have the menu; you should issue “boot -s” on the command line so that the system starts in single user mode.

Assuming that nothing went wrong and the system starts normally into the single user mode, we can continue by installing the new world.

# cd /usr/src
# make installworld

Next we need to run a new mergemaster, but this time without any additional flags, so that “optional” configuration files will get updated.

# mergemaster

Walk through all differences and make updates where you think they are appropriate. Be carefull though, if you update passwd and group, you might no longer be able to login remotely. I have been there, done it and got the t-shirt. Again I’d assume you made all the changes you wanted to, without messing up with the system itself. We need to reboot one more time before we can be a happy 8.0 user.

# reboot

And the system will reboot and restart all services under the new world order.

Updating the jails:

In my previous article we also installed jails, that will do our hosting needs, without compromising the host machine. They are not updated through the way described above. Though a fairly simple way of doing that is by installing the new world over the old jail-world. For this you need to stop the jails, because otherwise the system might not be able to update essential files because they are in use.

Once again navigate to the usr/src tree so that we can start by updating the jails. I’d assume you have stopped the jails already so we are safe to go.

# cd /usr/src

Next, update the jails master infrastructure:

# make installworld DESTDIR=/home/j/mroot

However, please do note that it might be possible that certain ports need to be rebuild. Carefully inventorize what kind of applications you currently use, and check whether they might be using libraries that are no longer provided. After you did this you are free to cleanup old libraries:

# cd /usr/src
# make delete-old-libs
# make delete-old

Again this is at your own risk. If you fail to update bash for example to use the new libraries, you are no longer able to restart it if you relogin. Again I have been there, done it and, got a nice second t-shirt!

Conclusion:

So now you are running a fresh and modern 8 system, which you can use to do various kind of (new) things. Not only did we update your base system, we also updated your jail infrastructure and you also updated several ports so survive this cleanup. You also got an impression about what new features you can use under FreeBSD, so I’d suggest you start playing with them now. In case you find problematic things, please consider sending an email to the various mailing lists of FreeBSD, located at http://lists.FreeBSD.org

In case of a more serious problem, which cannot be resolved by the various mailing lists, please consider sending in a bug report so that the developer crew can act accordingly and if there is enough information available resolve the problem you are facing.

Do realise though: Most people working on FreeBSD is a volunteer, who all work for bosses or are being hired for different projects. They also have families, kids, and a social life that demand attention. Please take care when sending in a problem report. Make the report as detailed as possible, send in the textdump if that is available, and give people time to respond. Most often a gentle prod works better then bashing around.

References:

I used Ivan Voras’ list of “Whats cooking for FreeBSD 8″ as my main source for new and improved updates. You can find the list at the following URL:

http://ivoras.sharanet.org/freebsd/freebsd8.html

About the Author:

Remko Lodder is a 26 year old FreeBSD enthusiast, in his spare time he likes being with his son and girlfriend, playing with FreeBSD systems, and wearing various FreeBSD hats to help the community. In his professional life Remko is an Unix Engineer for Snow B.V. in the Netherlands mostly focussing on Firewalls and Security (Checkpoint/Juniper etc.). You can contact him by sending an email to: [email protected], or if you want to hire him send an email to [email protected] or check out their website on: http://snow.nl

video hackfest

I got back from the video hackfest a day ago. It was a very productive meeting, even if most of it was getting across the features and requirements of our various projects to the other projects involved. I finally understood why using GL in gstreamer is hard, why gstreamer wants to be that way, why we're changing the locking model in cairo, and other details of projects I'm interested but not actively involved in. And I got a few chances to go over how GL works with threading and the ridiculous context model that it has.

Out of that, I ended up with the project of making GLX allow binding a single GL context into multiple threads. This should largely fix the disaster that is GL multithreading. The basic idea is: I have a collection of threads that want to work on a single context because they're sharing all the same objects and want to have some sort of serialization into a GL command stream. If we pass the context around between the threads, unbinding and rebinding, you get this forced command stream flushing that will kill performance. If we make multiple contexts, then at the transition of changed objects between one thread and another you have to flush in the producer and re-bind the object in the consumer. Whether or not that could perform well, we determined that in the gstreamer model we couldn't know in time whether the producer is going to be in a different thread than the consumer: you'd have to flush every time just like passing a single context around.

So here comes a simple hack: Just rip out the piece of the spec that says you can't bind one context into multiple threads. Tell the user that if they do this, locking is up to them. It's not an uncommon position for projects to take, and it will let us do exactly what we want in gstreamer: everyone works in the same context[1], and when you want access to the GL context, just grab the lock for the context and go.

Development trees are at:
http://cgit.freedesktop.org/~anholt/piglit/log/?h=mesa-multithread-makecurrent
http://cgit.freedesktop.org/~anholt/mesa/log/?h=mesa-multithread-makecurrent

The testcase gets bad rendering at the moment. So I made the testcase for the non-extended version, and it still didn't render, with either i965 or swrast. Next step is to test my testcase against someone else's GL.

Incidentally, Apple's GL spec allows binding a single context to multiple threads like this. Windows GL doesn't.

[1] OK, so that's not exactly true. I'm assuming that elements negotiate a single context through some handwavy caps magic -- people have said that this is possible. You can still end up with two contexts, though, like with the following pipeline: videotestsrc ! glupload ! glfilterblur ! gldownload ! gamma ! glupload ! glimagesink. The second group of gl elements doesn't know about the first, or have any way to communicate with them. But if each element calls glXMakeCurrent at the start, it'll be approximately free for the one-context case, and just work for the multiple-contexts case.

video hackfest

I got back from the video hackfest a day ago. It was a very productive meeting, even if most of it was getting across the features and requirements of our various projects to the other projects involved. I finally understood why using GL in gstreamer is hard, why gstreamer wants to be that way, why we're changing the locking model in cairo, and other details of projects I'm interested but not actively involved in. And I got a few chances to go over how GL works with threading and the ridiculous context model that it has.

Out of that, I ended up with the project of making GLX allow binding a single GL context into multiple threads. This should largely fix the disaster that is GL multithreading. The basic idea is: I have a collection of threads that want to work on a single context because they're sharing all the same objects and want to have some sort of serialization into a GL command stream. If we pass the context around between the threads, unbinding and rebinding, you get this forced command stream flushing that will kill performance. If we make multiple contexts, then at the transition of changed objects between one thread and another you have to flush in the producer and re-bind the object in the consumer. Whether or not that could perform well, we determined that in the gstreamer model we couldn't know in time whether the producer is going to be in a different thread than the consumer: you'd have to flush every time just like passing a single context around.

So here comes a simple hack: Just rip out the piece of the spec that says you can't bind one context into multiple threads. Tell the user that if they do this, locking is up to them. It's not an uncommon position for projects to take, and it will let us do exactly what we want in gstreamer: everyone works in the same context[1], and when you want access to the GL context, just grab the lock for the context and go.

Development trees are at:
http://cgit.freedesktop.org/~anholt/piglit/log/?h=mesa-multithread-makecurrent
http://cgit.freedesktop.org/~anholt/mesa/log/?h=mesa-multithread-makecurrent

The testcase gets bad rendering at the moment. So I made the testcase for the non-extended version, and it still didn't render, with either i965 or swrast. Next step is to test my testcase against someone else's GL.

Incidentally, Apple's GL spec allows binding a single context to multiple threads like this. Windows GL doesn't.

[1] OK, so that's not exactly true. I'm assuming that elements negotiate a single context through some handwavy caps magic -- people have said that this is possible. You can still end up with two contexts, though, like with the following pipeline: videotestsrc ! glupload ! glfilterblur ! gldownload ! gamma ! glupload ! glimagesink. The second group of gl elements doesn't know about the first, or have any way to communicate with them. But if each element calls glXMakeCurrent at the start, it'll be approximately free for the one-context case, and just work for the multiple-contexts case.

OS X: Flyswatter, JTAG & RS232

>Since I lent my RS232/USB adapter to a friend it was nice opportunity to experiment with Flyswatter on-board RS232 port. OS X does not support FTDI devices out of the box, but it's not a problem for OpenOCD, which uses generic interface to communicate JTAG board. Things get a little bit more complicated when you're trying to get on-board RS232 port working.

I started with installing stock drivers from FTDI site. Current version supports Snow Leopard. With these drivers installed system started to detect two COM ports but JTAG part stopped to work as OpenOCD failed to claim USB device.

Long story short: you'll need to edit kext content to prevent first usb device from being recognized as a virtual COM port. Nothing fancy, simple text editor would do the trick for all you need to edit is plain XML file. Find line <key>FT2232C_A</key> and delete it along with following <dict> element. Reload kext using kextunload/kextload and voila - JTAG works, "cu -s 115200 -l cu.usbserial-FS000000B" works.

End-of-Year Fund Raising Drive

The FreeBSD Foundation is kicking off their End-of-Year Fund Raising Drive! Their goal this year is to raise $300,000. So far, they have raised just over $159,000 so they are little more than half way towards their goal. Please consider making a donation and help spread the word by tweeting and blogging about the drive.

Click here to find out how to make a donation.

End-of-Year Fund Raising Drive

We're kicking off our End-of-Year Fund Raising Drive! Our goal this year is to raise $300,000. So far, we have raised just over $159,000 so we are little more than half way towards our goal. Please consider making a donation and help us to spread the word by tweeting and blogging about the drive.

Click here to find out how to make a donation.

Testing tarsnap

I am impressed. Yes, really. It seems tarsnap DTRT.

I made a test with tarsnap. I made a backup of some data (a full backup of everything is kept on a ZFS volume on an external disk which is only attached to make a full backup once in a while) of one of my systems. This data is 1.1 GB in size (most of it is /usr/src checked out via subversion and extended with some patches