simon’s blog

I finally got tired at looking at the hostname “wikitest”, so I decided to move the FreeBSD wiki to sky.FreeBSD.org. This also means that the wiki can now be fully “official” and has been renamed to wiki.FreeBSD.org. I took the opportunity to familiarize myself some more with how a moinmoin installation works so I did spend a good part of a weekend doing the migration but now there are fewer direct hacks in the wiki and I actually somewhat knows where the files are. The small downside to moving the wiki, and the main reason I haven’t done this before, is that I have a bit less freedom configuring the jails on sky since I now have to be a bit careful not to accidentally break the wiki. The move actually happened over a week ago, I just didn’t get around to writing about it before.

The current FreeBSD.org “monitoring system” consists of running “ruptime | grep down” from cron every hour. This is actually very effective compared to the simplicity, but it doesn’t catch e.g. when squid on www.FreeBSD.org die due to the disk being being full. To better detect this kind of errors I have I have been working on setting up Nagios for FreeBSD.org to be able to find out quickly when stuff crash. The configuration of the Nagios installation still isn’t complete, but at least it does warn me about major outages now. Thanks to the Nagios install by Erwin Lansing I also get mails if the FreeBSD Nagios crash so that part is also covered.

In unrelated news FreeBSD 4.X is no longer supported by the FreeBSD Security Team, so that is very nice that we finally could drop the support since FreeBSD 4.x has diverted quite a lot from FreeBSD 5/6/7 by now (or rather the other way around). It was getting increasingly difficult backporting fixes etc. for Security Advisories. RIP FreeBSD 4.

So, it has been a while since I written about the progress of sky, the next primary FreeBSD.org web server, and the reason is rather simple… there haven’t really been any progress.
Getting sky up and running turned out to be a bigger task than I initially imagined, and things have a tendency of getting in the way.
The move of all the main FreeBSD.org systems took some time, even though I only helped with preparations and fixing things which was broken after the move. Peter Wemm handled all the on-site stuff for the move itself.
When sky will be ready is still not certain, since I don’t have any illusions about fewer things taking up time, but who knows…

I haven’t written an update for the status on sky.FreeBSD.org (the next www.FreeBSD.org) for a while and that’s unfortunately since there isn’t much news. “Things” have a tendency to get in the way… That said, some progress has been made, e.g. I think all CGI scripts except man.cgi have been updated to work with perl 5.8, so that’s one less issue which has to be dealt with.

If anybody is wondering which things I’m talking about that is taking my time it’s e.g. FreeBSD-SA-06:19.openssl, FreeBSD-SA-06:20.bind, and FreeBSD-SA-06:21.gzip which took some of my time in the last couple of weeks (not that I was the only one working on them – far from – but I handle a part of them).

FreeBSD-SA-06:20.bind was a particularly cooperative advisory. It was mainly written by philip@, remko@, and myself with language fixes by cperciva@ and brueffer@. And of cause the thanks to both philip@ and remko@ for that is prodding them to write more. The thanks to brueffer@ was another advisory for him to proofread.

And when you, the reader, find all the typo’s and grammar errors in this blog post you know why I never write an advisory entirely by myself – somebody always need to check my Danglish :-).

My vacation has ended (a few weeks ago now), so progress on sky has slowed down due to less “FreeBSD time” in general and even less time for sky setup since various other things has used up most “FreeBSD time”.

I’m currently trying to setup some of the backend magic required by the CGI scripts. For some scripts that is quite a lot of things that need to be setup behind the scenes, so this takes some time.

It also turned out that nobody has tried to run many of the CGI scripts on Perl 5.8, so it also takes some time to get the minor nits fixed for things changed since Perl 5.0.

So, overall things are progressing with sky, but it will take some time before it’s all done.

On Tuesday I’m giving a presentation at AAUUG in Aarhus about “The FreeBSD Security Officer function” and on Saturday I’m giving the same presentation at BSD-DK in Copenhagen. Since I haven’t made a presentation of this type before I’m a bit excited about how that’s going to turn out… I hope people will find it interesting… time will tell :-).

After some more talk on #bsddocs (@EFnet), with many ideas for names for the new web-server, I decided on sky based on bluesky which erwin@ suggsted. bluesky just seemed so long to type all the time… :-). DNS has been updated so sky.FreeBSD.org is the new official name.

The jail containing the static web pages is now mostly working, though it still need more magic in the apache configuration for all the reverse proxy, aliases etc. being done on the current setup. People have suggested using various other smaller web servers, but due to all the magic configuration we have that’s just not possible.

The cgi jail is also configured and most CGI script now works. That said I’m sure some tweaking is still needed to get all to run. In the process I also found one script which should have been removed long ago, so that has now been removed from CVS.

To take some of the load of the server for CGI scripts we use squid as a reverse proxy / HTTP accelerator. The reverse proxy jail, and the squid proxy within it, is also set up and working. In the latest stable squid version they changed how to set up a HTTP accelerator but they haven’t yet updated most of the documentation, so it took a bit of time find out out how to configure it (and I’m still not entirely sure I did it right, even if it works…).

peter@ got the FreeBSD.org firewall updated to allow traffic to the static and cgi jails, so basically all external requirements is done, and I just need to finish it all.

The work on nnwww is continuing. I’m trying to find a better name of the box, but so far no genius names have been found, though the members of #bsddocs had some… interesting suggestions. iwantapony.freebsd.org and drososucks.freebsd.org were both discarded…

The rest of the FreeBSD 4 -> 6 migration was rather painless and there were basically no problems related to the upgrade itself. That said, I did spent some time doing silly things, which took some hours to track down as just me doing silly things.

I have now started with the jail setup. This will be done with “lightweight” jails where most parts are shared via read-only nullfs mounts to (hopefully) make it less painful to maintain. Basically each “major” service will be put in its own jail.

So far the “cvsup” and “build” jails are partially ready. The cvsup jail continuously keeps the local CVSup mirror updated and build jail simply contains the build of the www/ repository. Other builds (like portaudit-db) will probably be added to build jail later.

The “static” and “cgi” jails which will be hosting the web servers for the static pages and the CGI scripts are created, but not yet configured.

The current main FreeBSD web server (www.FreeBSD.org) is not very fast by today’s standards (dual 800MHz) and for years it has been suffering from some hardware issues (bad RAM), causing it to crash from time to time, so it has been due for replacement for some time.

A few month ago the main FreeBSD CVS server (repoman) was replaced with a faster system and “I” got the old server to use as a new www.FreeBSD.org replacement. Even though the old repoman was not quite fast enough to handle the load put on it as a CVS/Perforce server it’s still a dual 2.4GHz Intel XEON with 2GB RAM and SCSI RAID controller, so it should be plenty fast as a web server for some time.

Since the new www.FreeBSD.org, let’s call it nnwww for now, is many thousand KM away from me (it’s in California, USA and I’m in Denmark) I don’t have physical access to the box, only remote serial console access, so reinstalling the box from scratch is a bit troublesome. Instead I’m upgrading the already installed FreeBSD version on nnwww (FreeBSD 4.10) to FreeBSD 6.1-STABLE. Upgrading from FreeBSD 4 to something newer can be a bit painful and blow up badly, so I was a bit nervous about doing it, but so far it seems to be going without a hitch. As of this writing the system is happily running FreeBSD 5.5-STABLE (you have to go via FreeBSD 5 when upgrading from 4 -> 6).

So far so good, the real work will start when I’m going to prepare the system to run as www.FreeBSD.org… the current setup on www.FreeBSD.org is rather complex, so I expect it to take some time getting everything up and running.

So, this is one of the tasks I expect a good part of my vacation will be spent on :-). Stay tuned of more updates on the install process.