simon’s blog

A month or so before EuroBSDCon 2007 conference the FreeBSD.org systems at Yahoo! had gotten IPv6 connectivity with the main web server and mail servers being accessible via IPv6. The FreeBSD wiki was still IPv4 only as was (and still is) is running in a jail.

At the conference I talked to Bjoern A. Zeeb (AKA bz@) about the issue with IPv4 only jails and he was interested in making a patch so FreeBSD jails could support IPv6 and the FreeBSD wiki could be accessible via IPv6.

I should poke Bjoern regularly about making the patch, which I failed miserably at, but he got work done on the patch anyway. A few weeks ago he sent me the IPv6 jail patch for me to try out. Since life should be interesting I didn’t try it on a test server, but on the production web server sky which hosts the FreeBSD wiki and more. Just in case there were any problems I made sure I was around to recover things in case the system blew up, but none of that happened. In fact, since I installed the patch on sky a week ago there haven’t been any problems (that I know of at least). Granted there aren’t much IPv6 traffic, but the IPv4 part have been under its normal load.

So far the main FreeBSD.org DNS record for the wiki has not been updated to include the AAAA records, so people will use IPv6 if they have it, but that expected to come soon. For now people can try out the wiki using IPv6 by accessing http://v6.wiki.nitro.dk/. It has a slight (100%) likeness with the IPv4 wiki, but… IPv6!

For people interested in the patch the work is being done in the FreeBSD Perforce repository at //depot/user/bz/jail/.... I am sure Bjoern will post appropriate public patch when he think it is ready. Credit should also go to Pawel Jakub Dawidek (AKA pjd@) who made the multi IP(v4) jail patch which Bjoern based his patch on. Thanks to Bjoern and Pawel for the work making this possible!

Now I just need to actually get around to setting up IPv6 at home, so I can actually try out the IPv6 wiki myself in anything other than lynx from other hosts… any year now.

When I started the “sky” project (the new jail based www.FreeBSD.org) I never expected how much magic was involved, or how long it would take to set up a new www.FreeBSD.org from scratch, so that’s why the project has been going slow for a while.

Over the last month or so the current www.FreeBSD.org has had severe hardware problems which caused it to crash often. That is of course rather annoying but the positive thing is that it has given me motivation for finishing up the setup on sky. Now that EuroBSDCon 2007 is over I will also have more time to do other FreeBSD stuff again.

I am currently at the FreeBSD Developers summit and I’m mainly working on sky. It’s been a while since I messed with it so just upgrading ports etc. in the jails has taken some time, but I’m not done yet – so stay tuned for more updates.

Oh, and if www.FreeBSD.org is down, try wwwfe.FreeBSD.org which is the main FreeBSD website running on sky.FreeBSD.org.

I finally got tired at looking at the hostname “wikitest”, so I decided to move the FreeBSD wiki to sky.FreeBSD.org. This also means that the wiki can now be fully “official” and has been renamed to wiki.FreeBSD.org. I took the opportunity to familiarize myself some more with how a moinmoin installation works so I did spend a good part of a weekend doing the migration but now there are fewer direct hacks in the wiki and I actually somewhat knows where the files are. The small downside to moving the wiki, and the main reason I haven’t done this before, is that I have a bit less freedom configuring the jails on sky since I now have to be a bit careful not to accidentally break the wiki. The move actually happened over a week ago, I just didn’t get around to writing about it before.

The current FreeBSD.org “monitoring system” consists of running “ruptime | grep down” from cron every hour. This is actually very effective compared to the simplicity, but it doesn’t catch e.g. when squid on www.FreeBSD.org die due to the disk being being full. To better detect this kind of errors I have I have been working on setting up Nagios for FreeBSD.org to be able to find out quickly when stuff crash. The configuration of the Nagios installation still isn’t complete, but at least it does warn me about major outages now. Thanks to the Nagios install by Erwin Lansing I also get mails if the FreeBSD Nagios crash so that part is also covered.

In unrelated news FreeBSD 4.X is no longer supported by the FreeBSD Security Team, so that is very nice that we finally could drop the support since FreeBSD 4.x has diverted quite a lot from FreeBSD 5/6/7 by now (or rather the other way around). It was getting increasingly difficult backporting fixes etc. for Security Advisories. RIP FreeBSD 4.

So, it has been a while since I written about the progress of sky, the next primary FreeBSD.org web server, and the reason is rather simple… there haven’t really been any progress.
Getting sky up and running turned out to be a bigger task than I initially imagined, and things have a tendency of getting in the way.
The move of all the main FreeBSD.org systems took some time, even though I only helped with preparations and fixing things which was broken after the move. Peter Wemm handled all the on-site stuff for the move itself.
When sky will be ready is still not certain, since I don’t have any illusions about fewer things taking up time, but who knows…

I haven’t written an update for the status on sky.FreeBSD.org (the next www.FreeBSD.org) for a while and that’s unfortunately since there isn’t much news. “Things” have a tendency to get in the way… That said, some progress has been made, e.g. I think all CGI scripts except man.cgi have been updated to work with perl 5.8, so that’s one less issue which has to be dealt with.

If anybody is wondering which things I’m talking about that is taking my time it’s e.g. FreeBSD-SA-06:19.openssl, FreeBSD-SA-06:20.bind, and FreeBSD-SA-06:21.gzip which took some of my time in the last couple of weeks (not that I was the only one working on them – far from – but I handle a part of them).

FreeBSD-SA-06:20.bind was a particularly cooperative advisory. It was mainly written by philip@, remko@, and myself with language fixes by cperciva@ and brueffer@. And of cause the thanks to both philip@ and remko@ for that is prodding them to write more. The thanks to brueffer@ was another advisory for him to proofread.

And when you, the reader, find all the typo’s and grammar errors in this blog post you know why I never write an advisory entirely by myself – somebody always need to check my Danglish :-).

My vacation has ended (a few weeks ago now), so progress on sky has slowed down due to less “FreeBSD time” in general and even less time for sky setup since various other things has used up most “FreeBSD time”.

I’m currently trying to setup some of the backend magic required by the CGI scripts. For some scripts that is quite a lot of things that need to be setup behind the scenes, so this takes some time.

It also turned out that nobody has tried to run many of the CGI scripts on Perl 5.8, so it also takes some time to get the minor nits fixed for things changed since Perl 5.0.

So, overall things are progressing with sky, but it will take some time before it’s all done.

On Tuesday I’m giving a presentation at AAUUG in Aarhus about “The FreeBSD Security Officer function” and on Saturday I’m giving the same presentation at BSD-DK in Copenhagen. Since I haven’t made a presentation of this type before I’m a bit excited about how that’s going to turn out… I hope people will find it interesting… time will tell :-).

After some more talk on #bsddocs (@EFnet), with many ideas for names for the new web-server, I decided on sky based on bluesky which erwin@ suggsted. bluesky just seemed so long to type all the time… :-). DNS has been updated so sky.FreeBSD.org is the new official name.

The jail containing the static web pages is now mostly working, though it still need more magic in the apache configuration for all the reverse proxy, aliases etc. being done on the current setup. People have suggested using various other smaller web servers, but due to all the magic configuration we have that’s just not possible.

The cgi jail is also configured and most CGI script now works. That said I’m sure some tweaking is still needed to get all to run. In the process I also found one script which should have been removed long ago, so that has now been removed from CVS.

To take some of the load of the server for CGI scripts we use squid as a reverse proxy / HTTP accelerator. The reverse proxy jail, and the squid proxy within it, is also set up and working. In the latest stable squid version they changed how to set up a HTTP accelerator but they haven’t yet updated most of the documentation, so it took a bit of time find out out how to configure it (and I’m still not entirely sure I did it right, even if it works…).

peter@ got the FreeBSD.org firewall updated to allow traffic to the static and cgi jails, so basically all external requirements is done, and I just need to finish it all.

The work on nnwww is continuing. I’m trying to find a better name of the box, but so far no genius names have been found, though the members of #bsddocs had some… interesting suggestions. iwantapony.freebsd.org and drososucks.freebsd.org were both discarded…

The rest of the FreeBSD 4 -> 6 migration was rather painless and there were basically no problems related to the upgrade itself. That said, I did spent some time doing silly things, which took some hours to track down as just me doing silly things.

I have now started with the jail setup. This will be done with “lightweight” jails where most parts are shared via read-only nullfs mounts to (hopefully) make it less painful to maintain. Basically each “major” service will be put in its own jail.

So far the “cvsup” and “build” jails are partially ready. The cvsup jail continuously keeps the local CVSup mirror updated and build jail simply contains the build of the www/ repository. Other builds (like portaudit-db) will probably be added to build jail later.

The “static” and “cgi” jails which will be hosting the web servers for the static pages and the CGI scripts are created, but not yet configured.

The current main FreeBSD web server (www.FreeBSD.org) is not very fast by today’s standards (dual 800MHz) and for years it has been suffering from some hardware issues (bad RAM), causing it to crash from time to time, so it has been due for replacement for some time.

A few month ago the main FreeBSD CVS server (repoman) was replaced with a faster system and “I” got the old server to use as a new www.FreeBSD.org replacement. Even though the old repoman was not quite fast enough to handle the load put on it as a CVS/Perforce server it’s still a dual 2.4GHz Intel XEON with 2GB RAM and SCSI RAID controller, so it should be plenty fast as a web server for some time.

Since the new www.FreeBSD.org, let’s call it nnwww for now, is many thousand KM away from me (it’s in California, USA and I’m in Denmark) I don’t have physical access to the box, only remote serial console access, so reinstalling the box from scratch is a bit troublesome. Instead I’m upgrading the already installed FreeBSD version on nnwww (FreeBSD 4.10) to FreeBSD 6.1-STABLE. Upgrading from FreeBSD 4 to something newer can be a bit painful and blow up badly, so I was a bit nervous about doing it, but so far it seems to be going without a hitch. As of this writing the system is happily running FreeBSD 5.5-STABLE (you have to go via FreeBSD 5 when upgrading from 4 -> 6).

So far so good, the real work will start when I’m going to prepare the system to run as www.FreeBSD.org… the current setup on www.FreeBSD.org is rather complex, so I expect it to take some time getting everything up and running.

So, this is one of the tasks I expect a good part of my vacation will be spent on :-). Stay tuned of more updates on the install process.

I was recently at BSDCan 2006 which is a great yearly BSD conference in Ottawa, Canada organized by Dan Langille. There were several Yahoo! employees attending which made me think a bit more about why I was using Google as my primary search engine when they run that other operating system, and Yahoo run FreeBSD (and support FreeBSD in various ways).

So, when I got home I decided to find out how to get Opera 8.54 to use Yahoo search in the little permanent search box. It turned out to be rather simple (at least with my quick hack). Opera keeps the search engine configuration in .opera/search.ini and in the default version installed on my system has Google is the first entry and Opera Web as nr. 2. I never use Opera Web so I decided to simply remove Opera Web, bump Google to be the second search engine and add an entry for Yahoo! as the first search engine.

I have been running with Yahoo! search for a bit more than a week now and it hasn’t really made a big difference, in that I still find what I’m searching for, so I have no current plans to switch back.

Before editing any config files remember to make a backup of the files, just in case….

So, to do the same in your Opera just open .opera/search.ini in your favorite editor (which obviously should be Emacs), delete the [Search Engine 2] section, change the Google entry [Search Engine 1] header into [Search Engine 2], and then finally add the new [Search Engine 1] section as shown below.

[Search Engine 1]
Name=&Yahoo
URL=http://search.yahoo.com/search?p=%s&ei=UTF-8
Query=
Key=y
Is post=0
Has endseparator=0
Encoding=utf-8
Search Type=0
Verbtext=17063
Position=-2
Nameid=0