Archive for September, 2009

Varnish to the rescue

Sunday, September 6th, 2009

Some months ago the FreeBSD.org mailman web interface was getting slower and slower, and increasingly taking resources away from mail processing to the point where it started to be a problem.

Without knowing too much about how mailman works it seems like each load of the the listinfo ‘front page’, which lists all 150 or so FreeBSD.org mailing-lists, requires mailman to lock and read the config files for each list. This might not be a problem with few lists or a non-busy site, but for FreeBSD.org this resulted in the list info page taking in best case 2-3 seconds to load (this is still the case), but when the system got busy it was frequently taking > 20 seconds to load the list info page, since there were “many” request at the same time.

It is possible this could be fixed in mailman, but I wasn’t too keen on trying to optimize mailman considering I knew very little about it, and some web searching didn’t reveal any obvious solutions. This was when I remembered Poul-Henning Kamp’s goal for the Varnish reverse caching proxy project: You should be able to drop Varnish in front of an overloaded CMS or similar and be up and running in 5 minutes (OK, could be it wasn’t 5 minutes, but something like that). I had been meaning too look a Varnish for years, but never gotten around to it before.

Without knowing much about how to use Varnish I read a bit of the docs and did a basic install in a fresh jail on sky.FreeBSD.org. The setup was the most basic one which just proxied every requests it received to the ‘real’ mailman web server, and of course cached when possible. The default Varnish configuration is rather conservative and does not cache pages for that long when there is no expire information available from the backend web server. The default expiration time is 2 minutes as I recall.

Even this rather short caching by Varnish of pages made the difference between the FreeBSD.org mailman web interface being very unresponsive and unacceptably slow, to “just working” and responding as fast as can be expected. Before setting up Varnish I was getting regular (several times a day) Nagios mails about the mailman interface not responding in a reasonable time, and after installing Varnish I haven’t had a single one :-). I also haven’t needed to touch the Varnish set up – it has just taken care of itself.

Over the last couple of days I decided to learn more about Varnish and using it more which resulted in that today I switched the FreeBSD wiki over to also running behind Varnish. The wiki (MoinMoin) doesn’t support explicitly purging changed pages from Varnish, so the wiki pages can’t be cached for very long since we would risk returning out of date pages for a long time. This means I don’t expect Varnish to change responsiveness of the wiki much but it does mean that should a wiki page suddenly get a lot of hits, e.g. by a slashdotting or similar, I won’t have to scramble to prevent the wiki from stop working due to the load.