Monthly Archives: September 2009

bsdtalk177 – Giorgos Keramidas

Interview with Giorgos Keramidas. We talk about contributing to FreeBSD and creating a welcoming experience for new developers.

File Info: 18Min, 9MB.

Ogg Link:

[CFT] Qt-4.6.0 technology preview ready for beta test

The FreeBSD KDE team is pleased to announce the first public call to test Qt-4.6.0-tp1. It can downloaded from the area51 repo with the following command

svn co

From there you can use kdemerge to overlay the changes to your ports tree. Please note that you will require some patches from the repo to get games/kdegames4 and net/kdenetwork4 to build.

Our initial testing has yielded a binary compatibility between 4.6.0 and 4.5.2. Please be aware that this needs more testing.

One other warning, the py-qt* bindings are still a work in progress. Dima Panov, aka fluffy@, is currently working on it.

Please send feedback to [email protected]

This branch was made possible by the hard work of Alberto Villa and other contributors.

EuroBSDCon Feedback

Reports are starting to trickle in from those who attended last week's EuroBSDCon. Robert Watson notes that the FreeBSD Developer's Summit was the largest yet in the EU with around 70 attendees. And, despite the poor economy, attendance for the conference was up at about 180 attendees, and the dinner sponsored by iXsystems sold out at 150 with a waiting list.

Slides, papers, and audio recordings should be available soon on the UKUUG website. We'll post summaries of the trip reports as they become available.

Rodrigo Osorio has posted pictures here. If you have EuroBSDcon pics, leave a comment containing a link.

pefs crypto primitives

This is a second part of a series of posts devoted to pefs (stacked cryptographic filesystem). First part was about benchmarking encryption overhead

So I’m going to shed some light on cryptographic primitives used in pefs.

Supported data encryption algorithms: AES, Camellia, Salsa20.

Salsa20 is a new stream cypher by Daniel Bernstein from eSTREAM portfolio. Available AES and Camellia key sizes: 128, 192 and 256 bits. Adding another block cipher is trivial (choosing one with 128bit block size would save your from writing few more lines of code). Stream ciphers are a bit different as they do not usually support encryption at arbitrary offset in key stream (what CTR mode does for block ciphers). Most of them also lack support for tweaked encryption. Salsa20 fits here well mostly because it’s designed on top of hash function. Another possible candidate for inclusion is Skein – SHA3 hash function by Bruce Schneier et al. submitted to SHA3 competition. Usage of Skein as a stream cipher is suggested by its authors.

File names are always encrypted using AES-128 in CBC mode. Encrypted file name consists of a unique per file tweak, checksum and name itself:
XBase64(checksum || E(tweak || filename))

Checksum is VMAC of encrypted tweak and file name:
checksum = VMAC(E(tweak || filename))

Main reason for not providing alternatives is to keep it simple: “complexity is insecurity“. Data encryption is entirely different: encrypted data is not parsed in any way by pefs and user expects to be able to use secure-fast-best_name cipher.

Name has such structure to work around some of CBC shortcomings. Random tweak value is placed at the beginning of the first encrypted block. That gives us unique encrypted file names and eliminates the need of dealing with initial IV. (For those who cares: initial IV is zero and name is padded with zeros).

Encrypt-then-Authenticate construction is used. In addition to being most secure variant at the moment it allows checking if the name was encrypted by the given key without performing decryption. VMAC was chosen because of it performance characteristics and its ability to produce 64bit MAC (without truncation of original result, like in HMAC case). 64bit is almost mandatory here because larger MAC would result in much larger file name and bigger MAC can hardly improve security. But the real reason is that no real “authentication” performed. It’s designed to be just a cryptographic checksum (sounds incorrect but I can’t find a better wording), so that breaking VMAC wouldn’t result in breaking encrypted data, and this VMAC doesn’t “authenticate” encrypted data. It’s main purpose is to find a key the file is encrypted with.

Encrypted directory name also contains tweak but it’s used solely to randomize first CBC cipher block and keep name structure uniform.

Data encryption utilizes the tweak to get unique per file ciphertext (for files with same data). Block ciphers (AES, Camellia) operate in counter (CTR) mode with 64bits of a block containing the tweak and other 64bits containing file offset. Salsa20 supports tweaked encryption out of the box. Choosing such mode of operation eliminates necessity of performing cipher block size io. Which significantly simplifies entire filesystem code: io, page mapping, doesn’t change file size, etc (it removes a tone of error prone code, believe me, I’ve tried it other way).

As one has probably guessed there are 3 keys involved: one for name encryption, VMAC and data encryption. These are derived from user supplied key using HKDF algorithm (HMAC based key derivation function, IETF draft). The kernel part expects cryptographically strong key from userspace. This key is generated by PBKDF on top of HMAC-SHA512.

Standard implementations of ciphers are used, but I do not use opencrypto framework, so there is no hardware acceleration available. opencrypto is not used mainly because it lacks support for CTR mode. It’s also unlikely to ever support Salsa20 the way it’s used in pefs. opencrypto is rather heavy weight so using it solely for name encryption would probably worsen performance (hw initialization costs for encrypting short chunks with different keys) and add extra dependency.

Besides pefs supports multiply keys, mixing files encrypted with different keys in single directory, transparent(unencrypted) mode, key chaining (adding a series of keys by entering just one of them) and more. I’m going to write about it soon.

More help for the FreeBSD KDE team

The FreeBSD KDE team is pleased to report that Raphael Kubo da Costa has joined the team. He is a KDE contributor, maintaining the ark tool, and a recent convert to FreeBSD :-D

More recently Raphael has been our man in the middle, getting the FreeBSD Qt patches merged upstream to Nokia via Gitorious.

Congratulations and welcome aboard!

Atom as a web server?

The idea of using a simple, low-profile, low-performance, but also low-cost, low-power consumption and low-maintainance non-mainstream platforms as servers is absolutely nothing new. I have a great fondness for the very idea of using Atom CPUs as servers, and there are already commercial solutions with such a setup available out there. As soon as I can justify using it in a project, I will :)

But for now, I have an Atom-based netbook and I tried seeing what exactly can I expect from such a system in terms of performance.


pefs benchmark

pefs is a stacked cryptographic filesystem for FreeBSD. It has started as a Goggle Summer of Code’2009.

I’ve just come across performance comparison of eCryptfs against plain ext4 filesystem on Ubuntu, benchmark I was going to perform on my own.

I run dbench benchmarks regularly while working on pefs. But use it mostly as a stress test tool. I haven’t reached the point I can start working on improving performance yet. But measuring pefs overhead is going to be interesting.

Unfortunately I fail to interpret dbench results from the article. They’ve used dbench 4, while I’m using dbench 3 from ports. But never the less result of 4-8 Mb/s looks too strange for me.

I’ve benchmarked 4 and 16 dbench clients on zfs, pefs with salsa20 encryption (256 bit key) on top of same zfs partition and pefs with aes encryption (128 bit key, ctr mode). I executed benchmark for 3 times in each setup.

First of all, cipher throughput:
salsa20 ~205.5 Mb/s
aes128 ~81.3 Mb/s

Benchmark results:

dbench zfs/pefs 4 clients

dbench zfs/pefs 16 clients

In both cases (4 and 16 clients) CPU was limiting factor, disks where mostly idle. This explains such divergence in zfs results, I’ve actually benchmarked zfs arc cache performance. Because of unpredictable zfs inner workings one can get the best aes128 result surprisingly close to the worst salsa20 one (salsa20 is ~2.5 times faster than aes128).

The graph comparing average values:
dbench zfs/pefs 4 clients

Conclusion is that pefs is 2x times slower. But that shouldn’t be solely because of encryption. From my previous testing I can conclude that it’s mostly filesystem overhead:

  • Current pefs implementation avoids data caching (to prevent double caching and restrain one’s paranoia). I had version using buffer management for io (bread/bwrite) it’s performance was awful, something like 20-30 Mb/s with salsa20 encryption.
  • Sparse files (add file resizing here too) is implemented very poorly: it requires exclusive lock and fills gap with zeros. While this “gap” is likely to be filled by application really soon.
  • Lookup operation is very expensive. It calls readdir and decrypts name for each directory entry.

eCryptfs IOzone benchmark also shows 2x difference

Ports feature freeze now enforced

As an experiment, there will not be a complete ports freeze ahead of the release of 8.0.  To ensure that the tree is kept in a stable condition, we do need to have more discipline about the feature freeze we are currently using.  To achieve this, I have added a new tag to the commit log Feature safe: which needs to be set to yes explicitly for each commit.  While this is is tedious, it does mean that commits will be allowed without prior explicit approval from portmgr.  Portmgr is still working on a written set of rules for what is and isn’t allowed during a feature freeze, so please use common sense for now, and try to err on the side of caution and contact portmgr when in any doubt.  Sweeping commits touching a large number of ports, shared library updates affect more than a trivial number of ports, you get the picture; anything that would require a rebuild of more than a trivial amount of ports is not allowed without prior approval from portmgr.  With the release only a few weeks off, we cannot afford any major mishaps, so be conservative.

The feature freeze is enforced from right now and current plan is to tag the tree after RC1 is released.  There will be a short freeze while the tag is processed, and after that we will proceed with the ports slush as usual.

Please consider your commits in this light and help make this experiment a success so we can avoid long freezes for future releases.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Welcome wen@

Please welcome Wen Henping as a new ports committer.
miwi@ will be his mentor and I will be, for the first time, his co-mentor.
Wen, good luck for your punishment!

Setting up Calendarserver on FreeBSD 7.2

The last few days I am trying to setup CalendarServer on FreeBSD 7.2. I failed so far :-)

Together with Tom Scholten I got rid of a lot of errors (oh well a lot, just getting forward); but I am stuck with the following error;

Building Zope Interface…

Building select26…

Building PyDirector…

Using python as Python

Starting server…
exec python /usr/local/src/Twisted/bin/twistd -n caldav -f /usr/local/src/CalendarServer/conf/caldavd-dev.plist -o ProcessType=Combined
/usr/local/src/CalendarServer/calendarserver/tap/ DeprecationWarning: mktap and related support modules are deprecated as of Twisted 8.0. Use Twisted Application Plugins with the ‘twistd’ command directly, as described in ‘Writing a Twisted Application Plugin for twistd’ chapter of the Developer Guide.
from twisted.scripts.mktap import getid
Traceback (most recent call last):
File “/usr/local/src/Twisted/bin/twistd”, line 19, in
File “/usr/local/src/Twisted/twisted/scripts/”, line 27, in run, ServerOptions)
File “/usr/local/src/Twisted/twisted/application/”, line 689, in run
File “/usr/local/src/Twisted/twisted/application/”, line 669, in parseOptions
usage.Options.parseOptions(self, options)
File “/usr/local/src/Twisted/twisted/python/”, line 231, in parseOptions
File “/usr/local/src/Twisted/twisted/python/”, line 241, in parseOptions
File “/usr/local/src/CalendarServer/calendarserver/tap/”, line 218, in postOptions
File “/usr/local/src/CalendarServer/calendarserver/tap/”, line 230, in loadConfiguration
File “/usr/local/src/CalendarServer/twistedcaldav/”, line 199, in load
configDict = ConfigDict(self._provider.loadConfig())
File “/usr/local/src/CalendarServer/twistedcaldav/”, line 401, in loadConfig
configDict = readPlist(self._configFileName)
File “/usr/local/src/CalendarServer/twext/python/”, line 80, in readPlist
rootObject = p.parse(pathOrFile)
File “/usr/local/src/CalendarServer/twext/python/”, line 403, in parse
File “/usr/local/src/CalendarServer/twext/python/”, line 415, in handleEndElement
File “/usr/local/src/CalendarServer/twext/python/”, line 467, in end_string
File “/usr/local/src/CalendarServer/twext/python/”, line 428, in addObject
File “/usr/local/src/CalendarServer/twext/python/”, line 298, in __getattr__
raise AttributeError, attr
AttributeError: append

If someone has a suggestion on how to fix this I would be very interested. For now I had a lot of time in this today, and I am not willing to spend more time on it.