Monthly Archives: October 2010

MD5 for distinfo has been deprecated

erwin@ committed http://www.freebsd.org/cgi/query-pr.cgi?pr=149657, based on work by dougb@ and rene@.   It deprecates the use of md5 checksums in distinfo.  So here on in, when you run make makesum, the md5 will no longer be generated, only the sha256 checksum.

Existing distinfo containing md5 info will silently be ignored.  So at this time there is no need re-create distinfo for the sake of removing it, just allow the regular flow of ports updates take care of it.

MD5 checksums deprecated

Last night, I committed a large update to the ports tree that deprecated MD5 checksums based on the work by Doug Barton and Rene Laden in ports/149657. For a long time we’ve had both MD5 and SHA256 checksums in the distinfo file, even though having multiple checksumming algorithms does not add any additional security. From today, MD5 checksums are no longer generated, but existing checksums will silently be ignored. For now, we won’t be doing large sweeps through the tree removing MD5, but let them slowly disappear when individual ports are updated, to avoid the churn on the cvs repository, mirrors, and package build infrastructure such large sweeps will cause.
The ports framework internals were also updated to reflect this change by renaming the MD5_FILE macro to DISTINFO_FILE. A lot of thanks to Dough and Rene!

Correct numbers for EARS and Deskstar

Western Digital WD10EARS:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096      143839     911     911
  131072    1024     512    4096      145876     898     898

   65536    2048       0    8192       96727     677    1355
   65536    2048     512    8192       88182     743    1486
   65536    2048    1024    8192       89126     735    1470

   32768    4096       0   16384       23063    1420    5683
   32768    4096     512   16384       76939     425    1703
   32768    4096    1024   16384       75719     432    1731
   32768    4096    2048   16384       76007     431    1724

   16384    8192       0   32768       16567     988    7911
   16384    8192     512   32768       67676     242    1936
   16384    8192    1024   32768       68772     238    1905
   16384    8192    2048   32768       68422     239    1915
   16384    8192    4096   32768       15728    1041    8333

Western Digital WD20EARS:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096     1963003      66      66
  131072    1024     512    4096     1964781      66      66

   65536    2048       0    8192      897269      73     146
   65536    2048     512    8192      898143      72     145
   65536    2048    1024    8192      897456      73     146

   32768    4096       0   16384       18923    1731    6926
   32768    4096     512   16384     1216071      26     107
   32768    4096    1024   16384     1212785      27     108
   32768    4096    2048   16384     1213512      27     108

   16384    8192       0   32768       13645    1200    9605
   16384    8192     512   32768      804264      20     162
   16384    8192    1024   32768      802154      20     163
   16384    8192    2048   32768      802877      20     163
   16384    8192    4096   32768       13960    1173    9388

Hitach HDS722020ALA330:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096       34849    3761    3761
  131072    1024     512    4096       34807    3765    3765

   65536    2048       0    8192       19341    3388    6776
   65536    2048     512    8192       19332    3389    6779
   65536    2048    1024    8192       19352    3386    6772

   32768    4096       0   16384        8756    3741   14967
   32768    4096     512   16384        8803    3722   14888
   32768    4096    1024   16384        8782    3731   14924
   32768    4096    2048   16384        8744    3747   14989

   16384    8192       0   32768        5036    3253   26025
   16384    8192     512   32768        5035    3253   26029
   16384    8192    1024   32768        4997    3278   26227
   16384    8192    2048   32768        5042    3249   25994
   16384    8192    4096   32768        5039    3251   26010

Slightly different numbers, same conclusion: stay away from WD's Green Series.

Correct numbers for EARS and Deskstar

Western Digital WD10EARS:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096      143839     911     911
  131072    1024     512    4096      145876     898     898

   65536    2048       0    8192       96727     677    1355
   65536    2048     512    8192       88182     743    1486
   65536    2048    1024    8192       89126     735    1470

   32768    4096       0   16384       23063    1420    5683
   32768    4096     512   16384       76939     425    1703
   32768    4096    1024   16384       75719     432    1731
   32768    4096    2048   16384       76007     431    1724

   16384    8192       0   32768       16567     988    7911
   16384    8192     512   32768       67676     242    1936
   16384    8192    1024   32768       68772     238    1905
   16384    8192    2048   32768       68422     239    1915
   16384    8192    4096   32768       15728    1041    8333

Western Digital WD20EARS:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096     1963003      66      66
  131072    1024     512    4096     1964781      66      66

   65536    2048       0    8192      897269      73     146
   65536    2048     512    8192      898143      72     145
   65536    2048    1024    8192      897456      73     146

   32768    4096       0   16384       18923    1731    6926
   32768    4096     512   16384     1216071      26     107
   32768    4096    1024   16384     1212785      27     108
   32768    4096    2048   16384     1213512      27     108

   16384    8192       0   32768       13645    1200    9605
   16384    8192     512   32768      804264      20     162
   16384    8192    1024   32768      802154      20     163
   16384    8192    2048   32768      802877      20     163
   16384    8192    4096   32768       13960    1173    9388

Hitach HDS722020ALA330:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096       34849    3761    3761
  131072    1024     512    4096       34807    3765    3765

   65536    2048       0    8192       19341    3388    6776
   65536    2048     512    8192       19332    3389    6779
   65536    2048    1024    8192       19352    3386    6772

   32768    4096       0   16384        8756    3741   14967
   32768    4096     512   16384        8803    3722   14888
   32768    4096    1024   16384        8782    3731   14924
   32768    4096    2048   16384        8744    3747   14989

   16384    8192       0   32768        5036    3253   26025
   16384    8192     512   32768        5035    3253   26029
   16384    8192    1024   32768        4997    3278   26227
   16384    8192    2048   32768        5042    3249   25994
   16384    8192    4096   32768        5039    3251   26010

Slightly different numbers, same conclusion: stay away from WD's Green Series.

BSD-Day@2010

BSD-Day@2010 (http://wiki.freebsd.org/BSDDay_2010), Eötvös Loránd University, Budapest, Hungary 20 November, 2010. This one-day event is to gather Central European developers of today's open-source BSD systems, popularize their work, and provide a real-life communication interface between developers and users. There are no formalities, no papers, and no registration or participation fee, however the invited developers are encouraged to give a talk on their favorite BSD-related topic. The goal is to motivate potential future developers and users, especially undergraduate university students to work with BSD systems.

The FreeBSD-linuxulator explained (for developers): basics

The last post about the Linuxulator where I explained the Linuxulator from an user point of view got some good amount of attention. Triggered by a recent explanation of the Linuxulator errno stuff to a fellow FreeBSD developer I decided so see if more developers are interested in some more info too…

The syscall vector

In sys/linux/linux_sysvec.c is all the basic setup to handle Linux “system stuff

Correct numbers for EADS

As Pieter de Goeje kindly pointed out, due to an overflow bug, phybs reported incorrect tps numbers. I've corrected the bug and started re-running the benchmarks, but so far, I've only had time to test the WD20EADS. Here are the results:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096       55121    2377    2377
  131072    1024     512    4096      102522    1278    1278

   65536    2048       0    8192       47447    1381    2762
   65536    2048     512    8192       36531    1793    3587
   65536    2048    1024    8192       64753    1012    2024

   32768    4096       0   16384       37204     880    3522
   32768    4096     512   16384       28833    1136    4545
   32768    4096    1024   16384       43464     753    3015
   32768    4096    2048   16384       37968     863    3452

   16384    8192       0   32768       19079     858    6869
   16384    8192     512   32768       25722     636    5095
   16384    8192    1024   32768       27485     596    4768
   16384    8192    2048   32768       21333     768    6144
   16384    8192    4096   32768       27655     592    4739

Remember, this is not an Advanced Format drive, but it's still surprisingly slow and surprisingly inconsistent.

Correct numbers for EADS

As Pieter de Goeje kindly pointed out, due to an overflow bug, phybs reported incorrect tps numbers. I've corrected the bug and started re-running the benchmarks, but so far, I've only had time to test the WD20EADS. Here are the results:

   count    size  offset    step        msec     tps    kBps

  131072    1024       0    4096       55121    2377    2377
  131072    1024     512    4096      102522    1278    1278

   65536    2048       0    8192       47447    1381    2762
   65536    2048     512    8192       36531    1793    3587
   65536    2048    1024    8192       64753    1012    2024

   32768    4096       0   16384       37204     880    3522
   32768    4096     512   16384       28833    1136    4545
   32768    4096    1024   16384       43464     753    3015
   32768    4096    2048   16384       37968     863    3452

   16384    8192       0   32768       19079     858    6869
   16384    8192     512   32768       25722     636    5095
   16384    8192    1024   32768       27485     596    4768
   16384    8192    2048   32768       21333     768    6144
   16384    8192    4096   32768       27655     592    4739

Remember, this is not an Advanced Format drive, but it's still surprisingly slow and surprisingly inconsistent.

EuroBSDCon 2010 Trip Report: Efstratios Karatzas

Efstratios Karatzas, the 2010 Google Summer of Code student who worked on the Audit Kernel Events project, has sent in his trip report for the EuroBSDCon DevSummit. This was his first FreeBSD conference and his first opportunity to meet other FreeBSD developers in person. He writes:

Attending the conference was a great way for me to get more involved with the FreeBSD project. The most significant part of the trip was getting to know all sorts of people actively working on the project, from kernel hackers to bugmeisters and doc people.

The 15 minute length presentations at the Dev Summit were helpful in getting informed about what other people are working on at the moment and also provided an understanding of how different teams operate in the scope of the FreeBSD project. Unfortunately, there weren't any people actively involved with parts of my work besides our pf maintainer, but I still had some very interesting talks with all sorts of people: a dinner with andre@ giving a mini lecture on kernel architecture and a talk with hps@ about memory mapping pop into mind. Another positive impact that the trip had on me was to encourage me to work harder and support the project to the best of my abilities. All in all, it was a great trip indeed.

MeetBSD California unConference and DevSummit

The FreeBSD Foundation is a proud sponsor of MeetBSD California, which will be held in Mountain View, California November 5-6. There will be a Foundation booth at this event with lots of swag and informational brochures. Be sure to stop by to say hi, get your questions answered, suggest ideas for future funded projects, and consider making a donation.

If you are in the California area and have a commit bit (src, ports, or docs) or have participated in a FreeBSD Google Summer of Code project, you're welcome to sign up for the DevSummit the day before the conference. If you need to be sponsored (i.e. don’t have a FreeBSD commit bit), let Kris know and he’ll add you to the wiki page.

FreeBSD at GSoC Mentor Summit

As in previous years, Google held a "Mentor Summit" to bring together representatives from the open source organizations that participated in the Google Summer of Code to share experiences of what worked, what didn't, and generally learn from each other about shepherding students through the program. The mentor summit is always run Unconference-style and it is a great opportunity to meet, learn, and socialize with the many other open source organizations.In addition to several hours of face to face FreeBSD-related catch-up with Brooks Davis over pizza and beer, I particularly enjoyed catching up with old colleagues and learning about the current state of a variety of other open source projects I use such as R, Boost, NTP, and Ganeti.This weekend Brooks and I were the only FreeBSD representatives. Given that I'm local and Google sponsors the travel of 2 representatives from each open source organization it's quite unfortunate we couldn't get another FreeBSD mentor here this year. I would strongly encourage some of the other mentors that have never participated in this forum to volunteer to represent FreeBSD next year. This program has funded approximately 117 students to work on FreeBSD over the past 5 years and the mentor summit is best way I know of to improve the experience for students and open source projects next year.Thanks again to all the FreeBSD mentors that worked with students this summer and hope to see some of you at the post-GSoC Mentor Summit next year...

FreeBSD at GSoC Mentor Summit

As in previous years, Google held a "Mentor Summit" to bring together representatives from the open source organizations that participated in the Google Summer of Code to share experiences of what worked, what didn't, and generally learn from each other about shepherding students through the program. The mentor summit is always run Unconference-style and it is a great opportunity to meet, learn, and socialize with the many other open source organizations.

In addition to several hours of face to face FreeBSD-related catch-up with Brooks Davis over pizza and beer, I particularly enjoyed catching up with old colleagues and learning about the current state of a variety of other open source projects I use such as R, Boost, NTP, and Ganeti.

This weekend Brooks and I were the only FreeBSD representatives. Given that I'm local and Google sponsors the travel of 2 representatives from each open source organization it's quite unfortunate we couldn't get another FreeBSD mentor here this year. I would strongly encourage some of the other mentors that have never participated in this forum to volunteer to represent FreeBSD next year. This program has funded approximately 117 students to work on FreeBSD over the past 5 years and the mentor summit is best way I know of to improve the experience for students and open source projects next year.

Thanks again to all the FreeBSD mentors that worked with students this summer and hope to see some of you at the post-GSoC Mentor Summit next year...

Report from KyivBSD

The Foundation was a sponsor of this year's KyivBSD, held in Kiev, Ukraine on September 25. Alexander Yerenkow, the conference organizer, provided this report on the conference:

KyivBSD was the second installment in a newly created series of BSD-related conferences held in the Ukraine. The conference was attended by people from the Ukraine as well as Russia, Belarus, and Kazakhstan. The Foundation's financial support helped to make both this and last year's conference possible.

This year we were able to attract new partners and sponsors. Last year it was difficult to attract local companies as many were unfamiliar with BSD. This year, having last year's success as an example, was a lot easier. The local branch of D-Link was interested in sponsoring the conference and gave away three brand new WiFi routers. We received proposals from a few companies to place advertisements at the conference for money, but at the moment, we have no need for additional funds. We saw first-hand that many companies, individuals, and users have become more aware of FreeBSD and believe that the conference played a role in raising this awareness.

During the conference we ran a lottery with donated placards, books and routers for prizes. The funds raised from the lottery will be donated back to the Foundation at the end of this year.

The day after the conference we proctored the BSDA certification, which was the nearest certification event this fall for exam candidates from Russia and Kazakhstan. We were happy to provide them with the opportunity to take the exam.

Looking forward to next year, we hope to attract even more companies and attendees.

Jumstart/JET for FreeBSD (brainstorming)

There are some HOWTOs out there in the net which describe some automatic network based install via PXE-booting a machine from a server which has a specific FreeBSD release in the PXE-booting area and a non-interactive config for sysinstall to install this FreeBSD version on the machine which PXE-boots this.

The setup of this is completely manual and only allows to netboot one FreeBSD version. The server-side setup for the clients is also completely manual (and only allows to install one client at a time, it seems). This is not very user-friendly, and far away from the power of Jumpstart/JET for Solaris where you create a template (maybe from another template with automatic value (IP, name, MAC) replacement) and can specify different OS releases for different clients and then just run a command to generate a good config for this.

I thought a little bit how it could be done and decided to write down all the stuff (so far 160 lines, 830 words) to not forget some details. All in all I think this could be done (at least a sensible subset) in a week or two (fulltime) if you have the hardware, motivation, and time. As always, the problems are within the details, so I may be off with my estimation a little bit (also depends upon the knowledge-level (shell, tftp, dhcpd, install–software) of the person doing this).

Unfortunately I do not know if I have the hardware at home to do something like this. I have some unused harddisks which could be used in a machine which is used temporary as a test-install-client (normally I use this machines as my Desktop… if I do not use my little Netbook instead, as I do not do much at home currently), but I’ve never checked if this machine is PXE-booting-capable (VIA KT133 chipset with a 3Com 3c905C-TX Fast Etherlink XL). I also do not have the time to do this (with the current rate of free time I would expect to need about a year), except maybe someone would call my boss and negotiate something.

I can not remember any request to have something like this on the freebsd-current, freebsd-arch or freebsd-hackers list since I read them (and that is since about at least 3.0-RELEASE). Is this because nearly nobody is interested in something like this, or are the current possibilities enough for your needs? Do you work at a place where this would be welcome (= directly used when it would be done)? If you use a simple solution to make a net-install, what is your experience with this (pros/cons)?

ZFS and GPT by accident

A popular warning message on FreeBSD installs is geometry does not match label (255h,63s != 16h,255s), or similar. This is allegedly caused by sysinstall and mostly harmless. Still, I had a number of less interesting but infinitely more important things to do last night, so I decided to see what I could do about this message.

First I tried to "fix" the label. For various reasons bsdlabel and friends had no real effect. Eventually I resorted to a set of larger hammers: dd and hexdump. GEOM tried to protect me from smashing my skull in, but after a sysctl kern.geom.debugflags=17, it let me proceed. Fiddling with this, my system spent a lot of time restarting, but never getting much beyond the bootloader.

Eventually, I had the bright idea of getting rid of this silly MBR nonsense and just going with a GPT label instead. This is reasonably simple:

# gpart create -s GPT ad4
# gpart add -b 34 -s 128 -t freebsd-boot ad4
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ad4

(note if gpart create complains, gpart destroy first may be helpful, if not, dd if=/dev/zero of=/dev/ad4 is a sufficiently large hammer)

I have two disks in this machine and wanted to mirror them, so I did the same for ad6. At this point, it was tempting to just gpart add partitions for swap, root, usr, and home and create a mirror over them:

# gpart add -s 4G  -t freebsd-swap ad4
# gpart add -s 1G  -t freebsd-ufs  ad4
# gpart add -s 20G -t freebsd-ufs  ad4
# gpart add        -t freebsd-ufs  ad4

# gpart add -s 4G  -t freebsd-swap ad6
# gpart add -s 1G  -t freebsd-ufs  ad6
# gpart add -s 20G -t freebsd-ufs  ad6
# gpart add        -t freebsd-ufs  ad6

# gmirror label -vb round-robin root ad4p3 ad6p3
# gmirror label -vb round-robin usr  ad4p4 ad6p4
# gmirror label -vb round-robin home ad4p5 ad6p5

# newfs -L root    /dev/mirror/root
# newfs -L usr  -U /dev/mirror/usr
# newfs -L home -U /dev/mirror/home

# cat <<EOF >/etc/fstab
> /dev/ufs/root     /       ufs     rw  1   1
> /dev/ad4p2        none    swap    sw  0   0
> /dev/ad6p2        none    swap    sw  0   0
> /dev/ufs/usr      /usr    ufs     rw  2   2
> /dev/ufs/home     /home   ufs     rw  2   2
> EOF

This seems to work fine at first glance. The expected device nodes will be created under /dev/mirror and /dev/ufs. Unfortunately, after a reboot, /dev/ufs/home won't be there and the filesystem won't be mounted. I think that may be because the UFS label wants to live in the same space as the gmirror label but I didn't look into it further.

While looking up things related to GPT and gpart, I bumped into lots of great documentation about ZFS on FreeBSD and that reminded me that I wanted to try that out at some point. No time like the present. I blasted away my root, usr and home partitions and replaced them with ZFS partitions:

# gpart delete -i 5 ad4
# gpart delete -i 4 ad4
# gpart delete -i 3 ad4

# gpart delete -i 5 ad6
# gpart delete -i 4 ad6
# gpart delete -i 3 ad6

# gpart add -t freebsd-zfs -l disk0 ad4
# gpart add -t freebsd-zfs -l disk1 ad6

(note that at this point, I also discovered GPT labels, so I also labeled the swap partitions):

# gpart modify -i 2 -l swap0 ad4
# gpart modify -i 2 -l swap1 ad6

I also had to replace the bootcode with something which could boot from ZFS:

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad4
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad6

Creating ZFS filesystems is remarkably easy:

# kldload opensolaris
# kldload zfs

# zpool create hogfather mirror /dev/gpt/disk0 /dev/gpt/disk1
# zpool set bootfs=hogfather hogfather

And at that point, I found the RootOnZFS/GPTZFSBoot/Mirror wiki page which told me what to do next in detail. :-)

So suddenly, hogfather is running ZFS. Truth be told, the warning message that led me down this path is gone, but I'm sure there's a way to get rid of it and stay with UFS too. I would have rather liked to try out the shiny new journaling bits that have recently been bolted on to UFS.

Having ZFS at home is nice too though. It Just Works[tm].