Archive for the ‘wiki’ Category

Varnish to the rescue

Sunday, September 6th, 2009

Some months ago the FreeBSD.org mailman web interface was getting slower and slower, and increasingly taking resources away from mail processing to the point where it started to be a problem.

Without knowing too much about how mailman works it seems like each load of the the listinfo ‘front page’, which lists all 150 or so FreeBSD.org mailing-lists, requires mailman to lock and read the config files for each list. This might not be a problem with few lists or a non-busy site, but for FreeBSD.org this resulted in the list info page taking in best case 2-3 seconds to load (this is still the case), but when the system got busy it was frequently taking > 20 seconds to load the list info page, since there were “many” request at the same time.

It is possible this could be fixed in mailman, but I wasn’t too keen on trying to optimize mailman considering I knew very little about it, and some web searching didn’t reveal any obvious solutions. This was when I remembered Poul-Henning Kamp’s goal for the Varnish reverse caching proxy project: You should be able to drop Varnish in front of an overloaded CMS or similar and be up and running in 5 minutes (OK, could be it wasn’t 5 minutes, but something like that). I had been meaning too look a Varnish for years, but never gotten around to it before.

Without knowing much about how to use Varnish I read a bit of the docs and did a basic install in a fresh jail on sky.FreeBSD.org. The setup was the most basic one which just proxied every requests it received to the ‘real’ mailman web server, and of course cached when possible. The default Varnish configuration is rather conservative and does not cache pages for that long when there is no expire information available from the backend web server. The default expiration time is 2 minutes as I recall.

Even this rather short caching by Varnish of pages made the difference between the FreeBSD.org mailman web interface being very unresponsive and unacceptably slow, to “just working” and responding as fast as can be expected. Before setting up Varnish I was getting regular (several times a day) Nagios mails about the mailman interface not responding in a reasonable time, and after installing Varnish I haven’t had a single one :-). I also haven’t needed to touch the Varnish set up – it has just taken care of itself.

Over the last couple of days I decided to learn more about Varnish and using it more which resulted in that today I switched the FreeBSD wiki over to also running behind Varnish. The wiki (MoinMoin) doesn’t support explicitly purging changed pages from Varnish, so the wiki pages can’t be cached for very long since we would risk returning out of date pages for a long time. This means I don’t expect Varnish to change responsiveness of the wiki much but it does mean that should a wiki page suddenly get a lot of hits, e.g. by a slashdotting or similar, I won’t have to scramble to prevent the wiki from stop working due to the load.

The FreeBSD IPv6 Wiki – now fully public

Wednesday, August 6th, 2008

The main DNS record for the FreeBSD wiki (wiki.FreeBSD.org) has now been updated to include the IPv6 address record.
The server hosting the wiki, sky.FreeBSD.org, has been running with bz’s latest IPv6 jail patch for 10 days without issues.
See the freebsd-jail list for patches: http://lists.freebsd.org/pipermail/freebsd-jail/2008-July/000379.html

The FreeBSD wiki history

Sunday, June 22nd, 2008

As I have been asked several times why we use moinmoin for the FreeBSD Developers Wiki and the answer is mainly “historical reasons” I decided to write the history up so I can just point people at it :-).

The history, as I recall it, can now be found on the WikiHistory wiki page.

The FreeBSD IPv6 Wiki

Saturday, April 12th, 2008

A month or so before EuroBSDCon 2007 conference the FreeBSD.org systems at Yahoo! had gotten IPv6 connectivity with the main web server and mail servers being accessible via IPv6. The FreeBSD wiki was still IPv4 only as was (and still is) is running in a jail.

At the conference I talked to Bjoern A. Zeeb (AKA bz@) about the issue with IPv4 only jails and he was interested in making a patch so FreeBSD jails could support IPv6 and the FreeBSD wiki could be accessible via IPv6.

I should poke Bjoern regularly about making the patch, which I failed miserably at, but he got work done on the patch anyway. A few weeks ago he sent me the IPv6 jail patch for me to try out. Since life should be interesting I didn’t try it on a test server, but on the production web server sky which hosts the FreeBSD wiki and more. Just in case there were any problems I made sure I was around to recover things in case the system blew up, but none of that happened. In fact, since I installed the patch on sky a week ago there haven’t been any problems (that I know of at least). Granted there aren’t much IPv6 traffic, but the IPv4 part have been under its normal load.

So far the main FreeBSD.org DNS record for the wiki has not been updated to include the AAAA records, so people will use IPv6 if they have it, but that expected to come soon. For now people can try out the wiki using IPv6 by accessing http://v6.wiki.nitro.dk/. It has a slight (100%) likeness with the IPv4 wiki, but… IPv6!

For people interested in the patch the work is being done in the FreeBSD Perforce repository at //depot/user/bz/jail/…. I am sure Bjoern will post appropriate public patch when he think it is ready. Credit should also go to Pawel Jakub Dawidek (AKA pjd@) who made the multi IP(v4) jail patch which Bjoern based his patch on. Thanks to Bjoern and Pawel for the work making this possible!

Now I just need to actually get around to setting up IPv6 at home, so I can actually try out the IPv6 wiki myself in anything other than lynx from other hosts… any year now.

wiki goes into the sky, and more

Wednesday, February 7th, 2007

I finally got tired at looking at the hostname “wikitest”, so I decided to move the FreeBSD wiki to sky.FreeBSD.org. This also means that the wiki can now be fully “official” and has been renamed to wiki.FreeBSD.org. I took the opportunity to familiarize myself some more with how a moinmoin installation works so I did spend a good part of a weekend doing the migration but now there are fewer direct hacks in the wiki and I actually somewhat knows where the files are. The small downside to moving the wiki, and the main reason I haven’t done this before, is that I have a bit less freedom configuring the jails on sky since I now have to be a bit careful not to accidentally break the wiki. The move actually happened over a week ago, I just didn’t get around to writing about it before.

The current FreeBSD.org “monitoring system” consists of running “ruptime | grep down” from cron every hour. This is actually very effective compared to the simplicity, but it doesn’t catch e.g. when squid on www.FreeBSD.org die due to the disk being being full. To better detect this kind of errors I have I have been working on setting up Nagios for FreeBSD.org to be able to find out quickly when stuff crash. The configuration of the Nagios installation still isn’t complete, but at least it does warn me about major outages now. Thanks to the Nagios install by Erwin Lansing I also get mails if the FreeBSD Nagios crash so that part is also covered.

In unrelated news FreeBSD 4.X is no longer supported by the FreeBSD Security Team, so that is very nice that we finally could drop the support since FreeBSD 4.x has diverted quite a lot from FreeBSD 5/6/7 by now (or rather the other way around). It was getting increasingly difficult backporting fixes etc. for Security Advisories. RIP FreeBSD 4.