And remember to update gptzfsboot if you `zpool upgrade`
or it won’t boot

This is sad because the stuff that works is written in assembler, and the loader, in C should be easier to fix. So there’s PMBR code, gpt*boot, and loader code duplicating the same function of reading the list of partitions in the GPT. The last one is b0rked.

Can someone help me identify a version that is known to work in this configuration so I can hack something together?

Removing the -S and -L, changing the -r to 128 KB (max size for ZFS) I just 400 MBytes/sec write, or just under 20 MBytes/sec per drive.

Running the following on our server (4 CPU cores, 8 GB RAM, 24 SATA drives configured into 3 raidz2 vdevs of 8-drives) gives 15 MB/s per drive writes, without any lockups. FreeBSD 7.2, without kmem tuning, arc limited to 1 GB.

/usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C

“I’m just testing FBSD ZFS on a new 8-core, 8GB ram Xeon 24TB (24×1TB disc) storage system and while I’ve not paniced the system I’m finding that it’s slow (although to be fair, in my experience BSD software RAID tends to be slow, period). I had hoped to go the FBSD/ZFS route on this system, and held out till 7.1-RELEASE hoping for some needed fixes, but now it’s looking like Solaris will be the OS of choice based on the employer’s demand for ZFS.”

We sustain 80 MBps read (with peaks to 120 MBps) and 50 MBps write (with peaks just over 70 MBps) on our storage server, doing rsync backups for 105 systems every night, to a gzip-9 compressed ZFS filesystem. Other than a few hiccups before we got the tuning right for this setup (we started wtih 7-STABLE before 7.1 was released), it’s been rock-solid for us.

“For example, I can’t write to a 6 disc (6×1TB) Raidz2 ZFS pool faster than about 50 MB/s average.”

We’re using a 3Ware 9650SE-16ML (PCIe) and a 3Ware 9550SXU-16ML (PCI-X) connected to 24 WD SATA2 500 GB drives. Each drive is configured on the RAID controller as “Single Drive” ‘arrays’ (not JBOD) which allows us to monitor the drives using the RAID controller, and use the cache on the controller. They show up as separate SCSI disks in the OS.

“It also has a tendency to write/pause/write/pause.”

I’ve seen this on my home system using a simple 3-drive raidz vdev, and it’s kind of annoying. But we haven’t seen this on the work systems.

“ZFS is also very CPU intensive when scrubbing, which could well be the case on Solaris too. For example, all 8 cores are in use (2.33GHz) and it still takes well over an hour to scrub 1.1TB of data on a Raidz2.”

I’ll have to double-check this on our systems. We’ve only run a re-silver once, and that was extremely CPU/disk intensive, but we had a poorly configured pool at that time (single 24-drive raidz2 vdev).

Just started a scrub … each CPU core is listed as 75% idle in top, although “zpool status” does show an ETA of 550 hours (10 TB pool). gstat shows 15 MBps read per drive.

“So scrubbing 24TB is going to take all day using all 8 cores. By comparison, the 24 port Areca 1680 controller in that box can do its RAID6 scrub in about the same time (even at its low priority setting) but the difference is the system CPU cores are idle/usable for something else.”

Well, duh, the RAID controller has it’s own processor for doing that.

“I have no idea how people on Solaris are getting 100s of MB/sec with ZFS.”

It all depends on how the pool is created, how many vdevs it has (the more vdevs, the better the I/O as it stripes across all the vdevs), how each vdev is created (don’t use more than 8-9 drives per raidz vdev), and what the IO/sec rating is for the drives.

“Days of tweaking the knobs on 7.0-RELEASE ZFS didn’t fundamentally change my reported performance.”

ZFS in 7.0 was known to have issues, and the recommendation was to upgrade to 7-STABLE to get the fixes. ZFS in 7.1 worked much better. And things are even better in 7.2.

With the upgrade to ZFSv13 in 7-STABLE (which will be available in 7.3), things are getting even better.

IOW, now is the time to start looking at ZFS in FreeBSD.