Lotsa FreeBSD

As some of you might now, I’ve been working on a GUI installer for FreeBSD as a Google Summer of Code 2007 project that would one day, with some luck, replace the aging sysinstall. The SoC is now officially over and it’s time to make a public release of what’s been done so far.

But first, I’d like to write a bit about the project itself. Here are some of the more important ideas planned for the project, in no particular order:

• Make a modern installer for a modern FreeBSD system, with support for advanced features not present in sysinstall.


• Make the installer run directly of a FreeBSD Live CD


• Separate the installer (as a tool) into the front-end and the back-end.


• Make both the front-end and the back-end extensible enough so new features can be easily added.


• Make the front-end and the back-end interchangeable so people can write their own replacements.


• Make the back-end network-aware so it can support network (remote) installations. Use a service announcement and configuration technology such as Zeroconf.


• Eventually, make the back-end a part of FreeBSD base to be used for regular system configuration.

I’m happy to say the project is going on nicely, and that I’ve created infrastructure that can support the above features (as well as some other goodies discussed at the recent BSDCan). The project itself is hosted in the FreeBSD’s Perforce depot. It consists of three parts / subprojects:

• installer – the installer application, created in Python with PyGTK as the GUI library


• pybackend – the backend daemon. For speedy coding, it was also implemented in Python, but if the project gains popularity, a C version that can be included in FreeBSD base system is planned.


• makeimage – a Python script that creates a LiveCD ISO image with the installer. It can be customized to produce generic FreeBSD LiveCDs.
  

The back-end is a mostly stateless XML-RPC server that provides two kind of services: synchronous RPC calls and asynchronous “jobs” (that can take a long time and have progress checking infrastructure). The front-end is a modular PyGTK application that provides the user interface and uses the back-end for all “real” work.

I’ve created an ISO image with the installer embedded in it that can be used primarily for testing. The LiveCD is a fully working FreeBSD 7-CURRENT installation (i386) with X.Org 7.0, Xfce 4.2 desktop environment, Firefox, Thunderbird and a couple of supporting utilities. The image was built with mtune=generic CPU optimizations (pentium-m, pentium-4 and others). The installer version included in this LiveCD is a test version, more like a technology preview than a usable application. It can only install the system on a blank, unpartitioned drive and has only been tested on VMWare so far. I’ve disabled features that I know will not work (yet) but there’s still a chance there are bugs in the existing/enabled options.

Speaking of bugs, the overall state of FreeBSD 7-CURRENT is not very stable right now, and there are several known bugs and panics that will hopefully be resolved before 7.0. The system on the LiveCD (that is also used for the new installation) is not the “official” system that can be downloaded from FreeBSD source repositories. It contains several local patches I’ve made (or found from other developers and applied) to resolve some of the bugs and instabilities. I’ve submitted the patches to re@ some time ago, but they have still not been applied to the official source tree. Even so, there are several more-or-less known problems in the kernel I’m using, so you can expect random panics (you can imagine it’s hard do develop something like this with random panics happening all the time ). In particular, the LiveCD kernel might panic during the last phase (configuration), which is a problem I currently can’t solve.

I think that’s all that needs to be said. Download the installer image, try it out and see for yourself how it works. As I’ve said, there’s still a lot to be done and some planned features are not present in this release – but have fun trying it out. I won’t make an official announcement on current@ because I think it’s too early for that, so if you have any suggestions or questions you can post them as comments to this blog post. Please post both successes and failures (but keep the reports short ). (n.b. I’ll delete all comments that don’t have something useful to say, e.g. “I’m downloading it and I’ll try it tomorrow” type of posts).

P.S. If you don’t want to try it yet, I’ve created several screenshots of finstall.

I’ve just had a sort-of shock about how cheap the USB flash drives have become (vs. how pricey they used to be a long time ago). I didn’t look at their prices for a long time (since I didn’t need any new flash drives) so I was pleasantly surprised with the per/MB prices that have become “normal”. On the other hand, capacity of consumer USB drives hasn’t gone up much – it seems 8 GB is the top of the range now, and speed is still not great – it seems 8-10 MB/s is the norm. But it’s relatively cheap technology now.

It seems that now is a good time to start experimenting with flash memory even in production, especially where seek times are important (flash drives have no seek latency that’s present in mechanical drives). Having no seek time has a nice side-effect when drives are used in RAID1 with gmirror, which can be configured to split large read requests over the mirrored drives. With mechanical drives this mode couldn’t produce significant performance because the drive heads still needed to seek through the unread portions (for sequential requests) but the situation is ideal for seek-less drives.

I think the ideal solution here would be RAID 1+0 with four drives. Admittedly, this would only give 16 GB of storage space (if each individual drive is 8 GB), but the (read) performance should scale linearly to ~~ 40 MB/s (once it would double in gmirror/RAID1, then again in gstripe/RAID0), which is decent. 16 GB is relatively small, but flash memory is much cheaper than server memory (e.g. FB-DIMM) and most databases are relatively small, so it might not be affordable to keep the whole database in RAM any more.

Only one question remains now – how many IOPS can be had from flash memory (unconfirmed information says: around 2000), and does having all flash drives on the same host USB bus (e.g. one USB hub with 4 drives, connected to the motherboard port) introduce too much latency?

I’m pleased to announce another release candidate for GEOM_VIRSTOR, A FreeBSD kernel GEOM class providing storage virtualisation facility, regardless of type of underlying devices or its usage. It is a kind of “virtual memory” for disks – you pre-allocate address space (i.e. make a large virtual drive) and worry later about where to find physical space to back it.

This is a bug fix release – a bug that can cause data corruption on large-ish virtual drives (1 TB+) was fixed.

The development snapshot can be found here – everyone (or, to be precise, everyone with 7-CURRENT) is invited to test it.