Category Archives: VirtualBox

Growing a VirtualBox disk with ZFS on it

I have a VirtualBox VM on a Windows host with a 32 GB disk. That disk is partitioned with GPT and has four partitions: a boot partition, a swap partition, a smallish UFS root partition, and a ZFS partition. I need more space in the latter, so let’s grow it.

The first step is to shut down the VM and resize the virtual disk. This cannot be done in the GUI—we have to use the command-line utility:

C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9>"\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo "CrashBSD 9.vdi"
UUID:                 4a088148-72ef-4737-aae6-0a39e05aee06
Accessible:           yes
Logical size:         32768 MBytes
Current size on disk: 14484 MBytes
Type:                 normal (base)
Storage format:       VDI
Format variant:       dynamic default
In use by VMs:        CrashBSD 9 (UUID: 06bbe99d-9118-4c11-b29b-4ffd175ad06c)
Location:             C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9\CrashBSD 9.vdi

C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9>"\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "CrashBSD 9.vdi" --resize 65536
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9>"\Program Files\Oracle\VirtualBox\VBoxManage.exe" showhdinfo "CrashBSD 9.vdi"
UUID:                 4a088148-72ef-4737-aae6-0a39e05aee06
Accessible:           yes
Logical size:         65536 MBytes
Current size on disk: 14485 MBytes
Type:                 normal (base)
Storage format:       VDI
Format variant:       dynamic default
In use by VMs:        CrashBSD 9 (UUID: 06bbe99d-9118-4c11-b29b-4ffd175ad06c)
Location:             C:\Users\des\VirtualBox VMs\FreeBSD\CrashBSD 9\CrashBSD 9.vdi

Next, we boot the VM into single-user mode. It will repeatedly complain about the secondary GPT table, which is supposed to be located at the end of the disk but is now in the middle, since we doubled the size of the disk:

GEOM: ada0: the secondary GPT header is not in the last LBA.
# gpart list ada0
Geom name: ada0
modified: false
state: CORRUPT
fwheads: 16
fwsectors: 63
last: 67108830
first: 34
entries: 128
scheme: GPT
Providers:
[...]

Thankfully, this is trivial to fix. In fact, this exact use case is mentioned in the gpart(8) man page:

# gpart ada0 recover
ada0 recovered.
# gpart list ada0
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 134217694
first: 34
entries: 128
scheme: GPT
Providers:
[...]

Next, we resize the ZFS partition to fill all available space:

# gpart list ada0
[...]
4. Name: ada0p4
   Mediasize: 25769803776 (24G)
[...]
# gpart resize -i 4 ada0
ada0p4 resized
# gpart list ada0
[...]
4. Name: ada0p4
   Mediasize: 60129442304 (56G)
[...]

Let’s see what our pool looks like now:

# zpool import crash
# zfs list crash
NAME    USED  AVAIL  REFER  MOUNTPOINT
crash  14.2G  9.31G    31K  /

Hmm, no cigar. The pool hasn’t grown because the underlying vdev hasn’t automatically expanded to fill the resized partition. That’s easy to fix, though:

# zpool online -e crash ada0p4

And there we go:

# zfs list crash
NAME    USED  AVAIL  REFER  MOUNTPOINT
crash  14.2G  40.8G    31K  /

Backing up your VMs

A few weeks ago, I finally got my Drobo (a B800i with eight 2 TB disks) set up correctly so I can back up my Windows 7 computer to it. The only data I really care about on that computer are my VirtualBox VMs—so imagine my surprise when I discovered today that they weren’t being backed up! It turns out that with the default settings (“let Windows choose”), it does not back up your entire home directory, but only AppData, your desktop, your libraries, and a handful of other directories (including Downloads). Since VirtualBox stores VMs in a separate directory under your home directory rather than in AppData\Local or even My Documents, Windows Backup does not include them. If you want it to, you’ll have to either configure backups manually, or create a library that includes your VMs.

In the end, it doesn’t really matter, because backing up a VM’s disk image while it’s running is mostly pointless. Until now, I’ve been backing up my FreeBSD desktop by the simple expedient of rsync’ing ~des to a server with redundant storage; when I get around to it, I’ll set up a Bacula server backed by the Drobo.

Personal story of a ports committer

You all know that sometimes you end up dealing with all the ugly stuff instead of doing useful work. Over the last few months I was kept busy at $dayjob got assimilated by portmgr and had to look after redports. All of those new challenges are nice on it's own and I really enjoyed being part of the FreeBSD community and ecosystem but then 11/11 happened.

At that day quite a lot has changed for me since redports was isolated as a precaution and all ports building clusters of portmgr were effectively shut down. That situation was quite a mess since all automated systems and clusters were gone. No INDEX builds, no QAT, no pointyhat so also no exp-runs anymore. Whenever someone broke the ports tree we didn't even knew. It proved to be quite hard to get back on track again after that incident. INDEX checks and a very very limited QAT are already running again but pointyhat and redports are still dead. :(

The daily frustration and dealing with all that strange decisions that are taken because of the need to get stuff done is hard sometimes. But It's almost Christmas and without redports I have much more spare time so I try to calm down and focus on stuff that I can hack on my own. And that worked out quite nice so far ...

tvheadend:
I've noticed in the XBMC 12.0 release notes that they have included the PVR branch and thus support DVB-S2/C/T in XBMC. Well actually they only provide some backend configuration interfaces and rely on a backend like mythtv or tvheadend to handle the DVB stuff. Mythtv would be okay for that task but It's huge for such a small job. Tvheadend is a nice and small TV streaming server that suites perfectly and only does the bare minimum without a lot of dependencies. Configuration is done in a web based GUI or can be done in XBMC. So I started working on a tvheadend port. A few weeks later I'm at the point now where tvheadend compiles fine and also starts. I've just ripped out all that epoll stuff and linuxisms that I stumbled accross so it doesn't run properly yet. Adding kqueue support is the next step now.

virtualbox:
Due to redports being unavailable the vbox work has also frozen. I tried to collect all that patches and complains in my inbox so that they don't get lost. Since the situation did not improve I temporary created a github repository for the virtualbox ports and committed all the accumulated patches there: https://github.com/decke/freebsd-vbox

This includes almost all patches that were flying around on mailinglists and updates vbox to 4.2.6 / 4.1.24 but testing was very limited so take care if you give them a try. Testing will show us if we can commit it to the portstree by New Year's Eve.

The return of the FreeBSD desktop

I have a confession to make: I haven't used FreeBSD as a desktop OS for years. The reason is twofold:

  1. Since 2005, my work has required me to run Linux (Debian and Ubuntu at Linpro, RedHat at the University of Oslo) and, briefly, Windows at Kongsberg Maritime. I eventually stopped using stationary computers, resorting instead to a (company-provided) laptop running either Ubuntu, or Windows with Ubuntu in VirtualBox.
  2. More importantly, around the time I started at Linpro, it became increasingly difficult to maintain a FreeBSD desktop. The modularization of X.org and the increasing complexity of desktop environments mean that the number of packages required for a complete desktop system has grown from a bit over 100 to well over 600 (in addition to the kernel and base operating system, which is monolithic in FreeBSD). The FreeBSD ports system does not scale well, and the lack of a proper binary update procedure makes it almost impossible to keep that many packages up-to-date.

This is about to change. Thanks to what may very well be the most important innovations in recent FreeBSD history, I am now once again running a FreeBSD desktop.

I am referring, of course, to PKGNG and its companion, Poudrière.

(Actually, I've been running PC-BSD for a while, but I'm not a big fan of PBIs, and they don't really address the updating problem. This is about building a FreeBSD desktop from scratch, and keeping it up-to-date.)

Here is the procedure I followed:

I created a new VirtualBox VM with a 128 GB disk—this may not seem like much, but most of my storage requirements are met by the host system, which has two 2 TB disks, and an older file server with over 2 TB of storage, including ~40 GB of source code.

Instead of the default NAT configuration, I set up a bridged network interface with promiscuous mode enabled; I run my own DNS and DHCP servers (on a soekris, of course), so within my home network, the VM has its own fixed IP address and DNS name.

I then installed FreeBSD 9.0-RELEASE amd64 from the disc1 ISO. I did not select any packages, nor did I install the ports tree. However, as soon as the system booted, I downloaded and extracted an up-to-date ports tree using portsnap and built pkg and poudriere.

I had initially planned to rely entirely on the pkgbeta repository, but there are two problems with this: firstly, all the packages there are built with the default options, which in many cases make no sense at all; and secondly and most importantly, it did not at that time have a full set of Gnome 2 packages.

I therefore set up my own Poudrière. I ran a first build with only ports-mgmt/pkg and ports-mgmt/poudriere, then updgraded those two packages and started over with a slightly larger set of packages, including mail/postfix, security/sudo, shells/zsh and other essentials which don't take long to build. Finally, I added emulators/virtualbox-ose-additions, x11/xorg, x11/gnome2, and a number of desktop applications (e.g. Emacs) and development tools (e.g. Subversion) which I use. Then it was just a matter of

% sudo pkg install x11/xorg x11/gnome2 editors/emacs

Whenever I need an additional package, I add it to the package list and re-run Poudrière. I don't really need to do that—I could just as easily build it straight from the ports tree—but it ensures that I get a “clean” package that I can safely use on another VM or machine, should I ever need to, and that it is rebuilt when updated, along with all the other packages I use:

% sudo poudriere ports -u
% sudo poudriere bulk -f ~/poudriere-packages -j 83amd64 -k

I had some minor configuration trouble. First, X.org's autoconfiguration feature still doesn't work very well, at least in VirtualBox, so I used the xorg.conf from a previous post, with only minor modifications:

Section "InputDevice"
        Identifier      "Generic Keyboard"
        Driver          "kbd"
        Option          "XkbRules"      "xorg"
        Option          "XkbModel"      "pc105"
        Option          "XkbLayout"     "us"
EndSection

Section "InputDevice"
        Identifier      "VBox Mouse"
        Driver          "vboxmouse"
        Option          "CorePointer"
EndSection

Section "Device"
        Identifier      "VBox Video"
        Driver          "vboxvideo"
EndSection

Section "Monitor"
        Identifier      "VBox Monitor"
EndSection

Section "Screen"
        Identifier      "Default Screen"
        Device          "VBox Video"
        Monitor         "VBox Monitor"
EndSection

Section "ServerLayout"
        Identifier      "Default Layout"
        Screen          "Default Screen"
        InputDevice     "Generic Keyboard"
        InputDevice     "VBox Mouse"
EndSection

With the VirtualBox guest additions installed and this xorg.conf in place, everything works beautifully—mouse integration, clipboard integration and dynamic desktop resizing.

The second issue I had was with GDM's greeter, which did not display my user in the user list and would not let me type in my user name. The first part is understandable, as it only displays users who have logged in at least once already. The second is more surprising, but I did not have the energy to try to figure out why. Instead, I worked around it by disabling the user list:

% sudo -u gdm gconftool-2 --type bool \
    --set /apps/gdm/simple-greeter/disable_user_list true

Shared folders are still not implemented for FreeBSD guests, but I rarely need to transfer files between the host and the guest; when I do, I use FileZilla to SFTP them over, and there is always SMB for more complex use.

If I were running on real hardware or a weaker host (this is a four-core i7 with 16 GB RAM), I might use a subset of PC-BSD's tricked-out sysctl.conf, but I don't need audio and I haven't experienced any interactivity issues yet.

That's it—try it yourself, and share your experience below!

The return of the FreeBSD desktop

I have a confession to make: I haven’t used FreeBSD as a desktop OS for years. The reason is twofold:

  1. Since 2005, my work has required me to run Linux (Debian and Ubuntu at Linpro, RedHat at the University of Oslo) and, briefly, Windows at Kongsberg Maritime. I eventually stopped using stationary computers, resorting instead to a (company-provided) laptop running either Ubuntu, or Windows with Ubuntu in VirtualBox.
  2. More importantly, around the time I started at Linpro, it became increasingly difficult to maintain a FreeBSD desktop. The modularization of X.org and the increasing complexity of desktop environments mean that the number of packages required for a complete desktop system has grown from a bit over 100 to well over 600 (in addition to the kernel and base operating system, which is monolithic in FreeBSD). The FreeBSD ports system does not scale well, and the lack of a proper binary update procedure makes it almost impossible to keep that many packages up-to-date.

This is about to change. Thanks to what may very well be the most important innovations in recent FreeBSD history, I am now once again running a FreeBSD desktop.

I am referring, of course, to PKGNG and its companion, Poudrière.

(Actually, I’ve been running PC-BSD for a while, but I’m not a big fan of PBIs, and they don’t really address the updating problem. This is about building a FreeBSD desktop from scratch, and keeping it up-to-date.)

Here is the procedure I followed:

I created a new VirtualBox VM with a 128 GB disk—this may not seem like much, but most of my storage requirements are met by the host system, which has two 2 TB disks, and an older file server with over 2 TB of storage, including ~40 GB of source code.

Instead of the default NAT configuration, I set up a bridged network interface with promiscuous mode enabled; I run my own DNS and DHCP servers (on a soekris, of course), so within my home network, the VM has its own fixed IP address and DNS name.

I then installed FreeBSD 9.0-RELEASE amd64 from the disc1 ISO. I did not select any packages, nor did I install the ports tree. However, as soon as the system booted, I downloaded and extracted an up-to-date ports tree using portsnap and built pkg and poudriere.

I had initially planned to rely entirely on the pkgbeta repository, but there are two problems with this: firstly, all the packages there are built with the default options, which in many cases make no sense at all; and secondly and most importantly, it did not at that time have a full set of Gnome 2 packages.

I therefore set up my own Poudrière. I ran a first build with only ports-mgmt/pkg and ports-mgmt/poudriere, then updgraded those two packages and started over with a slightly larger set of packages, including mail/postfix, security/sudo, shells/zsh and other essentials which don’t take long to build. Finally, I added emulators/virtualbox-ose-additions, x11/xorg, x11/gnome2, and a number of desktop applications (e.g. Emacs) and development tools (e.g. Subversion) which I use. Then it was just a matter of

% sudo pkg install x11/xorg x11/gnome2 editors/emacs

Whenever I need an additional package, I add it to the package list and re-run Poudrière. I don’t really need to do that—I could just as easily build it straight from the ports tree—but it ensures that I get a “clean

User Thank You

Brandon Gooch, a system administrator at Southeastern Oklahoma State University, recently wrote the Foundation to express his gratitude towards FreeBSD developers in general and the recent wireless work in particular. The Foundation is always looking to collect appreciations and success stories, as well as pain points, that if addressed, could assist in FreeBSD adoption. When was the last time you pinged a developer on IRC or sent them a quick email to let them know how much you appreciate their work? It only takes a minute to let someone know that their hard work is being put to good use by others.

With Brandon's permission, his email text is re-posted here:

I'd like to thank Rui Paulo and Bernhard Schmidt, and Sam Leffler SO much for working so hard on the 802.11 stack and driver support. In my observations, wireless networking support (or lack of) from the OS seems to be one of the biggest deal-breakers when people I know try FreeBSD (from Linux or even Windows).

Driver support in general is a Good Thing and when one of my co-workers installs FreeBSD and his or her wireless card "just works" -- it looks really good for the project.

I'm sure there are still many issues left to resolve and still more features to implement. I believe that even better support is on the way. I also know that an OS can't be "all things to all people", but the solid code and great support from the developers and community make it a real pleasure to take part in the FreeBSD ecosystem.

I'm also very proud of the efforts of the FreeBSD Xorg devs (Robert Noland in particular) and the VirtualBox devs (Bernhard Froehlich and Co.). They have helped me implement several FreeBSD-based solutions at work due the the excellent features and performance of both Xorg and VirtualBox.

I look forward to opportunities to evangelize the FreeBSD project both in my professional and personal life. Thank you for making that task much easier.

Call for tester: VirtualBox 3.1.2 for FreeBSD

Hi All, Changelog from VirtualBox is available here: http://www.virtualbox.org/wiki/Changelog Changes in the port: - VirtualBox and the guest additions have been updated to 3.1.2. - Port has been renamed to virtualbox-ose to reflect that we are using the OSE version. Requested by: mm@ - A seperate port for the kernel modules has been created: virtualbox-ose-kmod - A seperate port for guest additions for [...]