Monthly Archive for September, 2009

The Ports Management Team: Partial ports thaw

The ports tree is now tagged and partially thawed.  Until 8.0 is released, sweeping commits still need explicit approval from portmgr to assure that tags can be slipped for potential security issues.  For more information what constitutes a sweeping change, see the portmgr web pages.

Ivan Voras: Firefox HTML 5 Ogg Theora playback

Apparently, playing 320x200 15 FPS native HTML5 (Ogg Theora) video in Firefox on FreeBSD takes less than 1% of a CPU core, and that's with X11 compositing enabled (translucent window), but not AIGLX. Nice. However, the "same" video at 640x480 at 30 FPS takes around 10%. It looks like audio is unsynced with video in both cases.

Read more...

Will Backman: bsdtalk177 – Giorgos Keramidas

Interview with Giorgos Keramidas. We talk about contributing to FreeBSD and creating a welcoming experience for new developers.

File Info: 18Min, 9MB.

Ogg Link:
http://cisx1.uma.maine.edu/~wbackman/bsdtalk/bsdtalk177.ogg

Thomas Abthorpe: [CFT] Qt-4.6.0 technology preview ready for beta test

The FreeBSD KDE team is pleased to announce the first public call to test Qt-4.6.0-tp1. It can downloaded from the area51 repo with the following command

svn co http://area51.pcbsd.org/branches/qt-4.6tp/

From there you can use kdemerge to overlay the changes to your ports tree. Please note that you will require some patches from the repo to get games/kdegames4 and net/kdenetwork4 to build.

Our initial testing has yielded a binary compatibility between 4.6.0 and 4.5.2. Please be aware that this needs more testing.

One other warning, the py-qt* bindings are still a work in progress. Dima Panov, aka fluffy@, is currently working on it.

Please send feedback to kde-freebsd@kde.org.

This branch was made possible by the hard work of Alberto Villa and other contributors.

FreeBSD Foundation: EuroBSDCon Feedback

Reports are starting to trickle in from those who attended last week's EuroBSDCon. Robert Watson notes that the FreeBSD Developer's Summit was the largest yet in the EU with around 70 attendees. And, despite the poor economy, attendance for the conference was up at about 180 attendees, and the dinner sponsored by iXsystems sold out at 150 with a waiting list.

Slides, papers, and audio recordings should be available soon on the UKUUG website. We'll post summaries of the trip reports as they become available.

Rodrigo Osorio has posted pictures here. If you have EuroBSDcon pics, leave a comment containing a link.

Gleb Kurtsou: pefs crypto primitives

This is a second part of a series of posts devoted to pefs (stacked cryptographic filesystem). First part was about benchmarking encryption overhead

So I’m going to shed some light on cryptographic primitives used in pefs.

Supported data encryption algorithms: AES, Camellia, Salsa20.

Salsa20 is a new stream cypher by Daniel Bernstein from eSTREAM portfolio. Available AES and Camellia key sizes: 128, 192 and 256 bits. Adding another block cipher is trivial (choosing one with 128bit block size would save your from writing few more lines of code). Stream ciphers are a bit different as they do not usually support encryption at arbitrary offset in key stream (what CTR mode does for block ciphers). Most of them also lack support for tweaked encryption. Salsa20 fits here well mostly because it’s designed on top of hash function. Another possible candidate for inclusion is Skein – SHA3 hash function by Bruce Schneier et al. submitted to SHA3 competition. Usage of Skein as a stream cipher is suggested by its authors.

File names are always encrypted using AES-128 in CBC mode. Encrypted file name consists of a unique per file tweak, checksum and name itself:
XBase64(checksum || E(tweak || filename))

Checksum is VMAC of encrypted tweak and file name:
checksum = VMAC(E(tweak || filename))

Main reason for not providing alternatives is to keep it simple: “complexity is insecurity“. Data encryption is entirely different: encrypted data is not parsed in any way by pefs and user expects to be able to use secure-fast-best_name cipher.

Name has such structure to work around some of CBC shortcomings. Random tweak value is placed at the beginning of the first encrypted block. That gives us unique encrypted file names and eliminates the need of dealing with initial IV. (For those who cares: initial IV is zero and name is padded with zeros).

Encrypt-then-Authenticate construction is used. In addition to being most secure variant at the moment it allows checking if the name was encrypted by the given key without performing decryption. VMAC was chosen because of it performance characteristics and its ability to produce 64bit MAC (without truncation of original result, like in HMAC case). 64bit is almost mandatory here because larger MAC would result in much larger file name and bigger MAC can hardly improve security. But the real reason is that no real “authentication” performed. It’s designed to be just a cryptographic checksum (sounds incorrect but I can’t find a better wording), so that breaking VMAC wouldn’t result in breaking encrypted data, and this VMAC doesn’t “authenticate” encrypted data. It’s main purpose is to find a key the file is encrypted with.

Encrypted directory name also contains tweak but it’s used solely to randomize first CBC cipher block and keep name structure uniform.

Data encryption utilizes the tweak to get unique per file ciphertext (for files with same data). Block ciphers (AES, Camellia) operate in counter (CTR) mode with 64bits of a block containing the tweak and other 64bits containing file offset. Salsa20 supports tweaked encryption out of the box. Choosing such mode of operation eliminates necessity of performing cipher block size io. Which significantly simplifies entire filesystem code: io, page mapping, doesn’t change file size, etc (it removes a tone of error prone code, believe me, I’ve tried it other way).

As one has probably guessed there are 3 keys involved: one for name encryption, VMAC and data encryption. These are derived from user supplied key using HKDF algorithm (HMAC based key derivation function, IETF draft). The kernel part expects cryptographically strong key from userspace. This key is generated by PBKDF on top of HMAC-SHA512.

Standard implementations of ciphers are used, but I do not use opencrypto framework, so there is no hardware acceleration available. opencrypto is not used mainly because it lacks support for CTR mode. It’s also unlikely to ever support Salsa20 the way it’s used in pefs. opencrypto is rather heavy weight so using it solely for name encryption would probably worsen performance (hw initialization costs for encrypting short chunks with different keys) and add extra dependency.

Besides pefs supports multiply keys, mixing files encrypted with different keys in single directory, transparent(unencrypted) mode, key chaining (adding a series of keys by entering just one of them) and more. I’m going to write about it soon.

FreeBSD News Flash: FreeBSD 8.0-RC1 Available

The first of the Release Candidates for the FreeBSD-8.0 release cycle is now available. ISO images for Tier-1 architectures and a memory stick image for amd64/i386 are now available on most of the FreeBSD mirror sites.

Thomas Abthorpe: More help for the FreeBSD KDE team

The FreeBSD KDE team is pleased to report that Raphael Kubo da Costa has joined the team. He is a KDE contributor, maintaining the ark tool, and a recent convert to FreeBSD :-D

More recently Raphael has been our man in the middle, getting the FreeBSD Qt patches merged upstream to Nokia via Gitorious.

Congratulations and welcome aboard!

Ivan Voras: Atom as a web server?

The idea of using a simple, low-profile, low-performance, but also low-cost, low-power consumption and low-maintainance non-mainstream platforms as servers is absolutely nothing new. I have a great fondness for the very idea of using Atom CPUs as servers, and there are already commercial solutions with such a setup available out there. As soon as I can justify using it in a project, I will :)

But for now, I have an Atom-based netbook and I tried seeing what exactly can I expect from such a system in terms of performance.

Read more...

Dag-Erling Smorgrav: 64-bit quotas

Bonus feature: mostly readable shot of the whiteboard Kirk and I used while reviewing the state of 64-bit quotas:

Dag-Erling Smorgrav: FreeBSD 9.0 Wishlist

Mostly readable shots of the whiteboards used during the 9.0 brainstorming session.

Philip Paeps: BSD Certification at EuroBSDCon

10:25 BST

On Saturday at 15:00 local time, I'm proctoring the BSD Certification exam in Cambridge. So far, fewer people have signed up than the number of papers I have with me. If you're interested in taking the exam and you happen to be in the area for EuroBSDCon, you can still sign up on the website.

First come, first served!

Gleb Kurtsou: pefs benchmark

pefs is a stacked cryptographic filesystem for FreeBSD. It has started as a Goggle Summer of Code’2009.

I’ve just come across performance comparison of eCryptfs against plain ext4 filesystem on Ubuntu, benchmark I was going to perform on my own.

I run dbench benchmarks regularly while working on pefs. But use it mostly as a stress test tool. I haven’t reached the point I can start working on improving performance yet. But measuring pefs overhead is going to be interesting.

Unfortunately I fail to interpret dbench results from the article. They’ve used dbench 4, while I’m using dbench 3 from ports. But never the less result of 4-8 Mb/s looks too strange for me.

I’ve benchmarked 4 and 16 dbench clients on zfs, pefs with salsa20 encryption (256 bit key) on top of same zfs partition and pefs with aes encryption (128 bit key, ctr mode). I executed benchmark for 3 times in each setup.

First of all, cipher throughput:
salsa20 ~205.5 Mb/s
aes128 ~81.3 Mb/s

Benchmark results:

dbench zfs/pefs 4 clients

dbench zfs/pefs 16 clients

In both cases (4 and 16 clients) CPU was limiting factor, disks where mostly idle. This explains such divergence in zfs results, I’ve actually benchmarked zfs arc cache performance. Because of unpredictable zfs inner workings one can get the best aes128 result surprisingly close to the worst salsa20 one (salsa20 is ~2.5 times faster than aes128).

The graph comparing average values:
dbench zfs/pefs 4 clients

Conclusion is that pefs is 2x times slower. But that shouldn’t be solely because of encryption. From my previous testing I can conclude that it’s mostly filesystem overhead:

  • Current pefs implementation avoids data caching (to prevent double caching and restrain one’s paranoia). I had version using buffer management for io (bread/bwrite) it’s performance was awful, something like 20-30 Mb/s with salsa20 encryption.
  • Sparse files (add file resizing here too) is implemented very poorly: it requires exclusive lock and fills gap with zeros. While this “gap” is likely to be filled by application really soon.
  • Lookup operation is very expensive. It calls readdir and decrypts name for each directory entry.

eCryptfs IOzone benchmark also shows 2x difference

Ivan Voras: ZFS !experimental

By the judgement of developers, ZFS is no longer considered experimental in FreeBSD :)

Read more...

Erwin Lansing: Ports feature freeze now enforced

As an experiment, there will not be a complete ports freeze ahead of the release of 8.0.  To ensure that the tree is kept in a stable condition, we do need to have more discipline about the feature freeze we are currently using.  To achieve this, I have added a new tag to the commit log Feature safe: which needs to be set to yes explicitly for each commit.  While this is is tedious, it does mean that commits will be allowed without prior explicit approval from portmgr.  Portmgr is still working on a written set of rules for what is and isn’t allowed during a feature freeze, so please use common sense for now, and try to err on the side of caution and contact portmgr when in any doubt.  Sweeping commits touching a large number of ports, shared library updates affect more than a trivial number of ports, you get the picture; anything that would require a rebuild of more than a trivial amount of ports is not allowed without prior approval from portmgr.  With the release only a few weeks off, we cannot afford any major mishaps, so be conservative.

The feature freeze is enforced from right now and current plan is to tag the tree after RC1 is released.  There will be a short freeze while the tag is processed, and after that we will proceed with the ports slush as usual.

Please consider your commits in this light and help make this experiment a success so we can avoid long freezes for future releases.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

FreeBSD News Flash: New committer: Wen Heping (ports)

Philippe Audeoud: Welcome wen@

Please welcome Wen Henping as a new ports committer.
miwi@ will be his mentor and I will be, for the first time, his co-mentor.
Wen, good luck for your punishment!

Remko Lodder: Setting up Calendarserver on FreeBSD 7.2

The last few days I am trying to setup CalendarServer on FreeBSD 7.2. I failed so far :-)

Together with Tom Scholten I got rid of a lot of errors (oh well a lot, just getting forward); but I am stuck with the following error;

Building Zope Interface…

Building select26…

Building PyDirector…

Using python as Python

Starting server…
exec python /usr/local/src/Twisted/bin/twistd -n caldav -f /usr/local/src/CalendarServer/conf/caldavd-dev.plist -o ProcessType=Combined
/usr/local/src/CalendarServer/calendarserver/tap/caldav.py:48: DeprecationWarning: mktap and related support modules are deprecated as of Twisted 8.0. Use Twisted Application Plugins with the ‘twistd’ command directly, as described in ‘Writing a Twisted Application Plugin for twistd’ chapter of the Developer Guide.
from twisted.scripts.mktap import getid
Traceback (most recent call last):
File “/usr/local/src/Twisted/bin/twistd”, line 19, in
run()
File “/usr/local/src/Twisted/twisted/scripts/twistd.py”, line 27, in run
app.run(runApp, ServerOptions)
File “/usr/local/src/Twisted/twisted/application/app.py”, line 689, in run
config.parseOptions()
File “/usr/local/src/Twisted/twisted/application/app.py”, line 669, in parseOptions
usage.Options.parseOptions(self, options)
File “/usr/local/src/Twisted/twisted/python/usage.py”, line 231, in parseOptions
self.subOptions.parseOptions(rest)
File “/usr/local/src/Twisted/twisted/python/usage.py”, line 241, in parseOptions
self.postOptions()
File “/usr/local/src/CalendarServer/calendarserver/tap/caldav.py”, line 218, in postOptions
self.loadConfiguration()
File “/usr/local/src/CalendarServer/calendarserver/tap/caldav.py”, line 230, in loadConfiguration
config.load(self["config"])
File “/usr/local/src/CalendarServer/twistedcaldav/config.py”, line 199, in load
configDict = ConfigDict(self._provider.loadConfig())
File “/usr/local/src/CalendarServer/twistedcaldav/stdconfig.py”, line 401, in loadConfig
configDict = readPlist(self._configFileName)
File “/usr/local/src/CalendarServer/twext/python/_plistlib.py”, line 80, in readPlist
rootObject = p.parse(pathOrFile)
File “/usr/local/src/CalendarServer/twext/python/_plistlib.py”, line 403, in parse
parser.ParseFile(fileobj)
File “/usr/local/src/CalendarServer/twext/python/_plistlib.py”, line 415, in handleEndElement
handler()
File “/usr/local/src/CalendarServer/twext/python/_plistlib.py”, line 467, in end_string
self.addObject(self.getData())
File “/usr/local/src/CalendarServer/twext/python/_plistlib.py”, line 428, in addObject
self.stack[-1].append(value)
File “/usr/local/src/CalendarServer/twext/python/_plistlib.py”, line 298, in __getattr__
raise AttributeError, attr
AttributeError: append

If someone has a suggestion on how to fix this I would be very interested. For now I had a lot of time in this today, and I am not willing to spend more time on it.

Alexander Leidinger: Back from holiday: mail-stats

After two weeks of holiday I have about 5.400 emails (about 51 MB, downloaded and filtered in 3h). Split up this is:

  • 2763 from FreeBSD mailinglists
  • 320 SPAM (101 false positives, mostly FreeBSD related… I should have a look at this and take appropriate actions)
  • 115 in the normal inbox = personal + not filtered emails (18 false negative SPAM)
  • 16 postmaster related, e.g. bounces from forged SPAM, 4 of them are not SPAM bounces but valid mails…
  • 114 security related (Bugtraq, …)
  • 220 status messages (e.g. FreeBSD periodic mails from various systems)
  • the rest is various other (filtered away) mails

It will take a while until I had a look at them…

Share/Bookmark

Eric Anholt: goodbye internets

This Sunday I'm heading out on a bike trip through Oregon and Washington with my housemate's dad's family and friends. 50-70 miles a day for 7 days. Gear hauled by car, sleeping in hotels, and eating out all the time. The first two ground rules that were sent out:


  • The morning of the ride we draw straws for SAG wagon driver order. Everybody drives a half day. After everybody has driven once we negotiate.

  • Afternoon driver responsible for buying a fifth of Jack Daniels and three or four six packs of good beer, plus chips and salsa. Also some soda.



You probably won't hear from me for a while. I don't exactly intend to be merging patches in the evenings.

I've certainly been busy with patches recently, though. The big update yesterday was http://lists.freedesktop.org/archives/intel-gfx/2009-September/004122.html -- those nasty 8xx hangs with GEM should now be fixed in drm-intel-next, to be landing in a 2.6.31.x near you soon. I've spent a long time on this bug, and we came incredibly close to fixing it with the clflush idea on the 8xx_chipset_flush() back in March, but the fact that we were writing to an uncached page meant we completely bypassed the cache we were trying to flush.

The 2.6.31 kernel is finally out, with a lot of improvements in our graphics driver and only a slight delay in releasing due to us screwing up (uh, let's not do that again). One of the biggest additions is DisplayPort support. DP is like HDMI done right -- more bandwidth, nice connectors, better compatibility story, and a low power design for use inside of laptops. keithp wrote the code and has been using it on his x200s's dock, and I came awfully close to picking up an x301 which has it on the laptop. KMS is also in much better shape once again, with a ton of work from Ma Ling, Zhao Yakui, and Jesse Barnes. It's also nice in looking back at the log to see a lot of fixes for serious issues in from non-Intel folks -- integrating with the community is continuing to pay off for us. My contribution this cycle was generally GEM stability fixes again, though a number of fixes came from Chris Wilson's careful reviews of the code.

I've also fixed what I think was the last major regression for texture tiling, which is around a 30% win on many GPU-bound OpenGL apps on the 965 and newer. That'll be landing on by default in Mesa 7.6.

On the plate for 2.6.32 is framebuffer compression support for around a .5W power savings, automatic downclocking of the GPU when idle (no need for user power management, and another .5W or so), experimental new GPU reset support from Ben Gamari and Owain Ainsworth (That's right, an OpenBSD developer), and of course more KMS fixes and new hardware support. We're also probably going to land execbuf2, which will let us do texture tiling efficiently on the pre-965 hardware.

Most of my time recently, though, I've finally been able to get back to spending serious time with Mesa after being stuck in 2D and the kernel for years. It's where I enjoy working, despite the build system and development model. In the last few weeks, I've added support for ARB_sync, ARB_map_buffer_range, ARB_depth_clamp, ARB_copy_buffer, and ARB_draw_elements_base_vertex for our hardware. All but ARB_depth_clamp (AKA NV_depth_clamp) work on pre-965 hardware as well.

I also fixed up some major performance penalties for applications using OpenGL correctly. I of course found these by writing an application using OpenGL the way it should be used as of the last 5 years or so. Unfortunately most open-source software out there fails in that respect (thinking in particular of OpenArena, blender, and the various TuxRacer forks, being what seem to be the most popular offenders). One fix was a 50-100% fps improvement, and another was a 70% CPU usage win. I'm going to be working on NV_primitive_restart support soon, which will be another CPU usage win, and likely a performance win as well on the 965.

Another big change for us has been the addition of Ben Holmes to our team. He's a fellow student in idr's OpenGL class, who's been writing tests for us. A lot of bugfixes I've done recently have been "hey, Ben, I think there might be a problem if an app does the following thing." He writes a test in piglit, tests that it fails, sends it out, we add it to the tree, then go fix the driver. Today, I got dFdx()/dFdy() fixed in our GLSL support by exactly this method. Unfortunately we'll be losing him back to school soon, but it's been well worth it so far.