I like big storage, I work with virtualization in its various forms so that all the hype about "cloud computing" sounds like "well, duh!" to me. Still, I think it's wrong to look at it as the possible future of IT in general. We have worked hard to move away from centralized computing and I think we should not go back.
Monthly Archives: October 2010
MD5 for distinfo has been deprecated
erwin@ committed http://www.freebsd.org/cgi/query-pr.cgi?pr=149657, based on work by dougb@ and rene@.  It deprecates the use of md5 checksums in distinfo.  So here on in, when you run make makesum, the md5 will no longer be generated, only the sha256 checksum.
Existing distinfo containing md5 info will silently be ignored.  So at this time there is no need re-create distinfo for the sake of removing it, just allow the regular flow of ports updates take care of it.
Droso » FreeBSD 2010-10-29 08:14:23
Last night, I committed a large update to the ports tree that deprecated MD5 checksums based on the work by Doug Barton and Rene Laden in ports/149657. For a long time we’ve had both MD5 and SHA256 checksums in the distinfo file, even though having multiple checksumming algorithms does not add any additional security. From today, MD5 checksums are no longer generated, but existing checksums will silently be ignored. For now, we won’t be doing large sweeps through the tree removing MD5, but let them slowly disappear when individual ports are updated, to avoid the churn on the cvs repository, mirrors, and package build infrastructure such large sweeps will cause.
The ports framework internals were also updated to reflect this change by renaming the MD5_FILE macro to DISTINFO_FILE. A lot of thanks to Dough and Rene!
FreeBSD Status Report – Q3 2010
A new FreeBSD quarterly status report has been published! Among the many reports, some selected new statuses are:
- Status of Summer of Code projects
- News on Capsicum security capabilities
- ZFS, GELI and HAST news
- Userland DTrace
- CLANG replacing GCC in the base system
- New TCP congestion control algorithms
- Support for USB 3.0
Read the whole report for details!
A new status report!
A new FreeBSD quarterly status report has been published! Among the many reports, some selected new statuses are:
- Status of Summer of Code projects
- News on Capsicum security capabilities
- ZFS, GELI and HAST news
- Userland DTrace
- New TCP congestion control algorithms
Read the whole report for details!
Correct numbers for EARS and Deskstar
Western Digital WD10EARS:
count size offset step msec tps kBps 131072 1024 0 4096 143839 911 911 131072 1024 512 4096 145876 898 898 65536 2048 0 8192 96727 677 1355 65536 2048 512 8192 88182 743 1486 65536 2048 1024 8192 89126 735 1470 32768 4096 0 16384 23063 1420 5683 32768 4096 512 16384 76939 425 1703 32768 4096 1024 16384 75719 432 1731 32768 4096 2048 16384 76007 431 1724 16384 8192 0 32768 16567 988 7911 16384 8192 512 32768 67676 242 1936 16384 8192 1024 32768 68772 238 1905 16384 8192 2048 32768 68422 239 1915 16384 8192 4096 32768 15728 1041 8333
Western Digital WD20EARS:
count size offset step msec tps kBps 131072 1024 0 4096 1963003 66 66 131072 1024 512 4096 1964781 66 66 65536 2048 0 8192 897269 73 146 65536 2048 512 8192 898143 72 145 65536 2048 1024 8192 897456 73 146 32768 4096 0 16384 18923 1731 6926 32768 4096 512 16384 1216071 26 107 32768 4096 1024 16384 1212785 27 108 32768 4096 2048 16384 1213512 27 108 16384 8192 0 32768 13645 1200 9605 16384 8192 512 32768 804264 20 162 16384 8192 1024 32768 802154 20 163 16384 8192 2048 32768 802877 20 163 16384 8192 4096 32768 13960 1173 9388
Hitach HDS722020ALA330:
count size offset step msec tps kBps 131072 1024 0 4096 34849 3761 3761 131072 1024 512 4096 34807 3765 3765 65536 2048 0 8192 19341 3388 6776 65536 2048 512 8192 19332 3389 6779 65536 2048 1024 8192 19352 3386 6772 32768 4096 0 16384 8756 3741 14967 32768 4096 512 16384 8803 3722 14888 32768 4096 1024 16384 8782 3731 14924 32768 4096 2048 16384 8744 3747 14989 16384 8192 0 32768 5036 3253 26025 16384 8192 512 32768 5035 3253 26029 16384 8192 1024 32768 4997 3278 26227 16384 8192 2048 32768 5042 3249 25994 16384 8192 4096 32768 5039 3251 26010
Slightly different numbers, same conclusion: stay away from WD's Green Series.
Correct numbers for EARS and Deskstar
Western Digital WD10EARS:
count size offset step msec tps kBps 131072 1024 0 4096 143839 911 911 131072 1024 512 4096 145876 898 898 65536 2048 0 8192 96727 677 1355 65536 2048 512 8192 88182 743 1486 65536 2048 1024 8192 89126 735 1470 32768 4096 0 16384 23063 1420 5683 32768 4096 512 16384 76939 425 1703 32768 4096 1024 16384 75719 432 1731 32768 4096 2048 16384 76007 431 1724 16384 8192 0 32768 16567 988 7911 16384 8192 512 32768 67676 242 1936 16384 8192 1024 32768 68772 238 1905 16384 8192 2048 32768 68422 239 1915 16384 8192 4096 32768 15728 1041 8333
Western Digital WD20EARS:
count size offset step msec tps kBps 131072 1024 0 4096 1963003 66 66 131072 1024 512 4096 1964781 66 66 65536 2048 0 8192 897269 73 146 65536 2048 512 8192 898143 72 145 65536 2048 1024 8192 897456 73 146 32768 4096 0 16384 18923 1731 6926 32768 4096 512 16384 1216071 26 107 32768 4096 1024 16384 1212785 27 108 32768 4096 2048 16384 1213512 27 108 16384 8192 0 32768 13645 1200 9605 16384 8192 512 32768 804264 20 162 16384 8192 1024 32768 802154 20 163 16384 8192 2048 32768 802877 20 163 16384 8192 4096 32768 13960 1173 9388
Hitach HDS722020ALA330:
count size offset step msec tps kBps 131072 1024 0 4096 34849 3761 3761 131072 1024 512 4096 34807 3765 3765 65536 2048 0 8192 19341 3388 6776 65536 2048 512 8192 19332 3389 6779 65536 2048 1024 8192 19352 3386 6772 32768 4096 0 16384 8756 3741 14967 32768 4096 512 16384 8803 3722 14888 32768 4096 1024 16384 8782 3731 14924 32768 4096 2048 16384 8744 3747 14989 16384 8192 0 32768 5036 3253 26025 16384 8192 512 32768 5035 3253 26029 16384 8192 1024 32768 4997 3278 26227 16384 8192 2048 32768 5042 3249 25994 16384 8192 4096 32768 5039 3251 26010
Slightly different numbers, same conclusion: stay away from WD's Green Series.
BSD-Day@2010
The FreeBSD-linuxulator explained (for developers): basics
The last post about the Linuxulator where I explained the Linuxulator from an user point of view got some good amount of attention. Triggered by a recent explanation of the Linuxulator errno stuff to a fellow FreeBSD developer I decided so see if more developers are interested in some more info too…
The syscall vector
In sys/linux/linux_sysvec.c is all the basic setup to handle Linux “system stuff� in FreeBSD. The “system stuff� is about translating FreeBSD errnos to Linux errnos, about translating FreeBSD signals to Linux signales, about handling Linux traps, and about setting up the FreeBSD system vector (the kernel structure which contains all the data to identify when a Linux program is called and to be able to lookup the right kernel functions for e.g. syscalls and ioctls).
There is not only one syscall vector, there is one for a.out (struct sysentvec linux_sysvec) and one for ELF (struct sysentvec elf_linux_sysvec) binaries (at least on i386, for other architectures it may not make sense to have the a.out stuff, as they maybe never seen any a.out Linux binary).
The ELF AUX args
When an ELF image is executed, the Linuxulator adds some runtime information (like pagesize, uid, guid, …) so that the userland can query this information which is not static at build-time easily. This is handled in the elf_linux_fixup function(). If you see some error messages about missing ELF notes from e.g. glibc, this is the place to add this information to. It would not be bad from time to time to have a look what Linux is providing and missing pieces there. FreeBSD does not has an automated way of doing this, and I am not aware of someone who regularly checks this. There is a little bit more info about ELF notes available in a message to one of the FreeBSD mailing lists, it also has an example how to read out this data.
Traps
Linux and FreeBSD do not share the same point of view how a trap shall be handled (SIGBUS or SIGSEGV), the corresponding decision making is handled in translate_traps() and a translation table is available as _bsd_to_linux_trapcode.
Signals
The values for the signal names are not the same in FreeBSD and Linux. The translation tables are called linux_to_bsd_signal and bsd_to_linux_signal. The translation is a feature of the syscall vector (= automatic).
Errnos
The values for the errno names are not the same in FreeBSD and Linux. The translation table is called bsd_to_linux_errno. Returning an errno in one of the Linux syscalls will trigger an automatic translation from the FreeBSD errno value to the Linux errno value. This means that FreeBSD errnos have to be returned (e.g. FreeBSD ENOSYS=78) and the Linux program will receive the Linux value (e.g. Linux ENOSYS=38, and as the Linux kernel returns negative errnos, the linux program will get –38).
If you see somewhere an “-ESOMETHING� in the Linuxulator code, this is either a bug, or some clever/tricky/dangerous use of the sign-bit to encode some info (e.g. in the futex code there is a function which returns –ENOSYS, but the sign-bit is used as an error indicator and the calling code is responsible to translate negative errnos into positive ones).
Syscalls
The Linux syscalls are defined similar to the FreeBSD ones. There is a mapping table (sys/linux/syscalls.master) between syscall numbers and the corresponding functions. This table is used to generate code (“make sysent� in sys//linux/) which does what is necessary.
Correct numbers for EADS
As Pieter de Goeje kindly pointed out, due to an overflow bug, phybs reported incorrect tps numbers. I've corrected the bug and started re-running the benchmarks, but so far, I've only had time to test the WD20EADS. Here are the results:
count size offset step msec tps kBps 131072 1024 0 4096 55121 2377 2377 131072 1024 512 4096 102522 1278 1278 65536 2048 0 8192 47447 1381 2762 65536 2048 512 8192 36531 1793 3587 65536 2048 1024 8192 64753 1012 2024 32768 4096 0 16384 37204 880 3522 32768 4096 512 16384 28833 1136 4545 32768 4096 1024 16384 43464 753 3015 32768 4096 2048 16384 37968 863 3452 16384 8192 0 32768 19079 858 6869 16384 8192 512 32768 25722 636 5095 16384 8192 1024 32768 27485 596 4768 16384 8192 2048 32768 21333 768 6144 16384 8192 4096 32768 27655 592 4739
Remember, this is not an Advanced Format drive, but it's still surprisingly slow and surprisingly inconsistent.
Correct numbers for EADS
As Pieter de Goeje kindly pointed out, due to an overflow bug, phybs reported incorrect tps numbers. I've corrected the bug and started re-running the benchmarks, but so far, I've only had time to test the WD20EADS. Here are the results:
count size offset step msec tps kBps 131072 1024 0 4096 55121 2377 2377 131072 1024 512 4096 102522 1278 1278 65536 2048 0 8192 47447 1381 2762 65536 2048 512 8192 36531 1793 3587 65536 2048 1024 8192 64753 1012 2024 32768 4096 0 16384 37204 880 3522 32768 4096 512 16384 28833 1136 4545 32768 4096 1024 16384 43464 753 3015 32768 4096 2048 16384 37968 863 3452 16384 8192 0 32768 19079 858 6869 16384 8192 512 32768 25722 636 5095 16384 8192 1024 32768 27485 596 4768 16384 8192 2048 32768 21333 768 6144 16384 8192 4096 32768 27655 592 4739
Remember, this is not an Advanced Format drive, but it's still surprisingly slow and surprisingly inconsistent.
July-September, 2010 Status Report
EuroBSDCon 2010 Trip Report: Efstratios Karatzas
Attending the conference was a great way for me to get more involved with the FreeBSD project. The most significant part of the trip was getting to know all sorts of people actively working on the project, from kernel hackers to bugmeisters and doc people.
The 15 minute length presentations at the Dev Summit were helpful in getting informed about what other people are working on at the moment and also provided an understanding of how different teams operate in the scope of the FreeBSD project. Unfortunately, there weren't any people actively involved with parts of my work besides our pf maintainer, but I still had some very interesting talks with all sorts of people: a dinner with andre@ giving a mini lecture on kernel architecture and a talk with hps@ about memory mapping pop into mind. Another positive impact that the trip had on me was to encourage me to work harder and support the project to the best of my abilities. All in all, it was a great trip indeed.
MeetBSD California unConference and DevSummit
If you are in the California area and have a commit bit (src, ports, or docs) or have participated in a FreeBSD Google Summer of Code project, you're welcome to sign up for the DevSummit the day before the conference. If you need to be sponsored (i.e. don’t have a FreeBSD commit bit), let Kris know and he’ll add you to the wiki page.
Colored snake!
Probably not many users noticed, but some time ago I committed a hopefully useful (but always shiny!) addition to the "snake" console screensaver - now it contains the system load numbers and is color-coded per system load :)
FreeBSD at GSoC Mentor Summit
FreeBSD at GSoC Mentor Summit
In addition to several hours of face to face FreeBSD-related catch-up with Brooks Davis over pizza and beer, I particularly enjoyed catching up with old colleagues and learning about the current state of a variety of other open source projects I use such as R, Boost, NTP, and Ganeti.
This weekend Brooks and I were the only FreeBSD representatives. Given that I'm local and Google sponsors the travel of 2 representatives from each open source organization it's quite unfortunate we couldn't get another FreeBSD mentor here this year. I would strongly encourage some of the other mentors that have never participated in this forum to volunteer to represent FreeBSD next year. This program has funded approximately 117 students to work on FreeBSD over the past 5 years and the mentor summit is best way I know of to improve the experience for students and open source projects next year.
Thanks again to all the FreeBSD mentors that worked with students this summer and hope to see some of you at the post-GSoC Mentor Summit next year...
Report from KyivBSD
KyivBSD was the second installment in a newly created series of BSD-related conferences held in the Ukraine. The conference was attended by people from the Ukraine as well as Russia, Belarus, and Kazakhstan. The Foundation's financial support helped to make both this and last year's conference possible.
This year we were able to attract new partners and sponsors. Last year it was difficult to attract local companies as many were unfamiliar with BSD. This year, having last year's success as an example, was a lot easier. The local branch of D-Link was interested in sponsoring the conference and gave away three brand new WiFi routers. We received proposals from a few companies to place advertisements at the conference for money, but at the moment, we have no need for additional funds. We saw first-hand that many companies, individuals, and users have become more aware of FreeBSD and believe that the conference played a role in raising this awareness.
During the conference we ran a lottery with donated placards, books and routers for prizes. The funds raised from the lottery will be donated back to the Foundation at the end of this year.
The day after the conference we proctored the BSDA certification, which was the nearest certification event this fall for exam candidates from Russia and Kazakhstan. We were happy to provide them with the opportunity to take the exam.
Looking forward to next year, we hope to attract even more companies and attendees.
Jumstart/JET for FreeBSD (brainstorming)
There are some HOWTOs out there in the net which describe some automatic network based install via PXE-booting a machine from a server which has a specific FreeBSD release in the PXE-booting area and a non-interactive config for sysinstall to install this FreeBSD version on the machine which PXE-boots this.
The setup of this is completely manual and only allows to netboot one FreeBSD version. The server-side setup for the clients is also completely manual (and only allows to install one client at a time, it seems). This is not very user-friendly, and far away from the power of Jumpstart/JET for Solaris where you create a template (maybe from another template with automatic value (IP, name, MAC) replacement) and can specify different OS releases for different clients and then just run a command to generate a good config for this.
I thought a little bit how it could be done and decided to write down all the stuff (so far 160 lines, 830 words) to not forget some details. All in all I think this could be done (at least a sensible subset) in a week or two (fulltime) if you have the hardware, motivation, and time. As always, the problems are within the details, so I may be off with my estimation a little bit (also depends upon the knowledge-level (shell, tftp, dhcpd, install–software) of the person doing this).
Unfortunately I do not know if I have the hardware at home to do something like this. I have some unused harddisks which could be used in a machine which is used temporary as a test-install-client (normally I use this machines as my Desktop… if I do not use my little Netbook instead, as I do not do much at home currently), but I’ve never checked if this machine is PXE-booting-capable (VIA KT133 chipset with a 3Com 3c905C-TX Fast Etherlink XL). I also do not have the time to do this (with the current rate of free time I would expect to need about a year), except maybe someone would call my boss and negotiate something.
I can not remember any request to have something like this on the freebsd-current, freebsd-arch or freebsd-hackers list since I read them (and that is since about at least 3.0-RELEASE). Is this because nearly nobody is interested in something like this, or are the current possibilities enough for your needs? Do you work at a place where this would be welcome (= directly used when it would be done)? If you use a simple solution to make a net-install, what is your experience with this (pros/cons)?
ZFS and GPT by accident
A popular warning message on FreeBSD installs is geometry does not match label (255h,63s != 16h,255s), or similar. This is allegedly caused by sysinstall and mostly harmless. Still, I had a number of less interesting but infinitely more important things to do last night, so I decided to see what I could do about this message.
First I tried to "fix" the label. For various reasons bsdlabel and friends had no real effect. Eventually I resorted to a set of larger hammers: dd and hexdump. GEOM tried to protect me from smashing my skull in, but after a sysctl kern.geom.debugflags=17, it let me proceed. Fiddling with this, my system spent a lot of time restarting, but never getting much beyond the bootloader.
Eventually, I had the bright idea of getting rid of this silly MBR nonsense and just going with a GPT label instead. This is reasonably simple:
# gpart create -s GPT ad4 # gpart add -b 34 -s 128 -t freebsd-boot ad4 # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ad4
(note if gpart create complains, gpart destroy first may be helpful, if not, dd if=/dev/zero of=/dev/ad4 is a sufficiently large hammer)
I have two disks in this machine and wanted to mirror them, so I did the same for ad6. At this point, it was tempting to just gpart add partitions for swap, root, usr, and home and create a mirror over them:
# gpart add -s 4G -t freebsd-swap ad4 # gpart add -s 1G -t freebsd-ufs ad4 # gpart add -s 20G -t freebsd-ufs ad4 # gpart add -t freebsd-ufs ad4 # gpart add -s 4G -t freebsd-swap ad6 # gpart add -s 1G -t freebsd-ufs ad6 # gpart add -s 20G -t freebsd-ufs ad6 # gpart add -t freebsd-ufs ad6 # gmirror label -vb round-robin root ad4p3 ad6p3 # gmirror label -vb round-robin usr ad4p4 ad6p4 # gmirror label -vb round-robin home ad4p5 ad6p5 # newfs -L root /dev/mirror/root # newfs -L usr -U /dev/mirror/usr # newfs -L home -U /dev/mirror/home # cat <<EOF >/etc/fstab > /dev/ufs/root / ufs rw 1 1 > /dev/ad4p2 none swap sw 0 0 > /dev/ad6p2 none swap sw 0 0 > /dev/ufs/usr /usr ufs rw 2 2 > /dev/ufs/home /home ufs rw 2 2 > EOF
This seems to work fine at first glance. The expected device nodes will be created under /dev/mirror and /dev/ufs. Unfortunately, after a reboot, /dev/ufs/home won't be there and the filesystem won't be mounted. I think that may be because the UFS label wants to live in the same space as the gmirror label but I didn't look into it further.
While looking up things related to GPT and gpart, I bumped into lots of great documentation about ZFS on FreeBSD and that reminded me that I wanted to try that out at some point. No time like the present. I blasted away my root, usr and home partitions and replaced them with ZFS partitions:
# gpart delete -i 5 ad4 # gpart delete -i 4 ad4 # gpart delete -i 3 ad4 # gpart delete -i 5 ad6 # gpart delete -i 4 ad6 # gpart delete -i 3 ad6 # gpart add -t freebsd-zfs -l disk0 ad4 # gpart add -t freebsd-zfs -l disk1 ad6
(note that at this point, I also discovered GPT labels, so I also labeled the swap partitions):
# gpart modify -i 2 -l swap0 ad4 # gpart modify -i 2 -l swap1 ad6
I also had to replace the bootcode with something which could boot from ZFS:
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad4 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad6
Creating ZFS filesystems is remarkably easy:
# kldload opensolaris # kldload zfs # zpool create hogfather mirror /dev/gpt/disk0 /dev/gpt/disk1 # zpool set bootfs=hogfather hogfather
And at that point, I found the RootOnZFS/GPTZFSBoot/Mirror wiki page which told me what to do next in detail. :-)
So suddenly, hogfather is running ZFS. Truth be told, the warning message that led me down this path is gone, but I'm sure there's a way to get rid of it and stay with UFS too. I would have rather liked to try out the shiny new journaling bits that have recently been bolted on to UFS.
Having ZFS at home is nice too though. It Just Works[tm].