In this post I write how to install FreeBSD on a remote linux system by creating a root image prepared for mirroring on a local system. There is no (free) access to the remote console, only access via ssh.

Background story

While I was at the university, I worked remotely for an ISP (administration of several FreeBSD systems). During this time I started to use my own domain, and I was allowed to host it directly at the ISP for free. I tried to not use too much space on the harddisk (at the end about 400 MB) and to not provide something which attracted too much people to keep the bandwith on a sane level. After the university I was still available to answer questions, and I was still allowed to host my website there for free. As the number of questions can be counted with IIRC one hand since then, I decided at some point (recently, to be exact — better late than never) to move my website to a different place (I am still available if they have some questions — for free — but I do not expect to get much questions from them).

At the same time my brother decided to move to a new server hoster, as his old one decided to increase the amount of money customers have to pay for the service. So we searched together a new ISP where he either could host his server, or get a rootserver for a good price (the idea was to have my domain in a jail on a server of the company of my brother, and we share the costs for it). We found Manitu (the owner has even a blog about his business), which is even not far away from my place.

Unfortunately they do not provide FreeBSD preinstalled on their rootservers, but they offer a remote rescue system (read: booting a linux image… I assume via PXE or similar) and we knew someone who has some servers there, so I was able to get a rough idea what kind of hardware is used there (the hard facts like PCI IDs and such). The idea was to build a very small disk image, put it on the harddisk over the network via the remote rescue system, and then to configure the remainder of the harddisk(s) to use it. And here is how I did it (my brother thought “who is better suited to install a FreeBSD system remotely without access to the console of the machine (the ISP offers to hook up a KVM switch, but only during business hours and you have to pay for it) than one of the developers of FreeBSD…”).

HOWTO

In the title of this post I wrote “via a tiny disk image”. This is true for a suitable definition of tiny.

What we have in the rootserver are two 160 GB harddisks. They shall be used in a software mirror (via gmirror). The root-FS shall have about 5 GB. This is more than needed, but as this is only 3% of one harddisk, I prefer to keep it a little bit bigger than too small after an remote update or two. The machine has 2 GB of RAM. We do not expect much kernel panics (= crash dumps) there, so we do not really need >2 GB of swap (forget the rule of having twice as much swap than RAM, with the current amount of RAM in a machine you are in “trouble” when you need even the same amount of swap than RAM). I decided to go with 1 GB of swap (mirrored too, to prevent a harddisk failure to take down the machine), this is more than enough. The rest of the harddisk will be used for jails, the distfiles/packages for/of the ports, and as WRKDIRPREFIX when building ports.

Now, pushing/pulling a 160 GB image over the network to install a system is not really something I want to do. I would prefer to transfer less than 500 MB (that is 0.3% of the entire disk) to get this job done, and this is feasible. Due to an error or two I had to transfer the image several times until everything was working, so it was more in the area of maybe 2 GB (~1% of the entire disk).

First let us define some variables in the shell, this way you just need to change the values in one place and the rest is copy&paste:

ROOTFS_SIZE=5G
ROOTFS_NAME=root0
FILENAME=rootfs

Then change your current directory to a place where you have enough space for the image. There we will create a container for the image, and make it ready for partitioning:

truncate -s ${ROOTFS_SIZE} ${FILENAME}
mdconfig -a -t vnode -f ${FILENAME}

Create the rootfs:

# create one active FreeBSD slice with $ROOTFS_SIZE GB
fdisk -e /dev/mdX
gmirror label ${ROOTFS_NAME} /dev/mdXsY
bsdlabel -w /dev/mirror/${ROOTFS_NAME}
# create an "a"-partition for everything
bsdlabel -e /dev/mirror/${ROOTFS_NAME}
newfs -U /dev/mirror/${ROOTFS_NAME}a

Mount the new rootfs to /mnt and install FreeBSD:

mount /dev/mirror/${ROOTFS_NAME}a /mnt
cd /usr/src
make buildworld >&! buildworld.log
make buildkernel >&! build_generic.log
make installworld DESTDIR=/mnt
make distribution DESTDIR=/mnt
make installkernel DESTDIR=/mnt

Now you need to create /mnt/etc/rc.conf (set the defaultrouter, the IP address via ifconfig_IF (and do not forget to use the right IF for it), the hostname, set sshd_enable to yes, add an user (I used “vipw –d /mnt/etc”) which is in the wheel group so you can login remotely, and maybe other things you are interested in), /mnt/etc/resolv.conf, /mnt/etc/hosts. Finally, do not forget to load the gmirror module, it will safe a lot of head-scratching (yes, an echo would be shorter, but WP converts the single-quotes to double-quotes), and add the rootfs to the fstab:

cat > /mnt/boot/loader.conf <<EOT
geom_mirror_load="YES"
EOT
echo "/dev/mirror/root0a / ufs rw,noatime 1 1" >/mnt/etc/fstab

Now we are ready to install, the image can be unmounted:

umount /mnt

The final steps are to login into the rescue console (theoretically you should be able to overwrite even a running system, but then you need to make sure there is a way to power-cycle the system remotely to force a reboot) of the new system and to install via a compressed ssh connection (my remote rescue system is a linux system, so linux-syntax has to be used). The login to the rescue console is not shown here, but the install from the remote system is simple (this assumes the image resides on a system which is accessible from the new system):

ssh -C -o CompressionLevel=9 user@myhost cat /path/to/${FILENAME} | dd of=/dev/hda bs=1m

Alternatively you can compress it with bzip2 on your system and add an bunzip2 into the pipe above. This way you could even use an HTTP server to fetch the file from, but then the command to fetch the file needs to have a way to output the file it downloads on stdout. Both ways of transferring the image depend upon the stability of the transfer. If the connection is cut, the system can not boot anymore. So if you do not have a rescue console which is independent from the content of the harddisk, you better have a plan B.

If I did not make an error here (I did this some months ago, I hope I did not forget to write down some important step and also corrected all steps which where not written down correctly), you did everything correctly too, and the remote system does not need other kernel modules loaded, you can now reboot the new system, cross your fingers for some moments, and then login to the new system.

Post install TODO

Now you should have a basic system up and running, and you can start to configure it.

I added two more FreeBSD partitions, 1 GB for swap, and the rest of the harddisk (I took care to not have the last partition cover also the last sector of the harddisk, else gmirror will think it has to mirror the entire harddisk and not only the last partition) as one big partition. For the second harddisk I made the same partitioning as for the first harddisk.

Then I created two more gmirrors, one mirror for the swap, and one for the rest of the space. The mirror for the swap I created with the option “-F”, to not synchronize the partition after a power failure (not necessary for swap). All two I also created with the “-n” option, to not sync the contents (there is nothing yet, so it does not matter what is written there).

Now just a quick “bsdlabel” on the two mirrors to create a “b”-partition for the swap and a “d”-partition for the large partition. To add the swap it is just an “echo /dev/mirror/swap0b none swap sw 0 0 >/etc/fstab”. Do not forget to add the big partition to the fstab (after doing a “newfs” off course).

To be sure it will survive a reboot, do a quick test of manually mounting the big partition (use the easy manual way via “mount /place/where/it/is/mounted” after adding it to the fstab, not the complete manual way, this is meant to test the fstab entry too), and a “swapon –a” to test the swap. I also made a reboot test, just to be sure everything is ok.

The final step was to add the partition on the second harddisk to the rootfs (“gmirror insert ${ROOTFS_NAME} /dev/…”).