Category Archives: FreeBSD

You need to construct additional pylons. Building wine for #FreeBSD

I’ve been playing Blizzards’s Starcraft 2 on Linux via wine emulation lately and thought I’d see if I can get the same thing working on FreeBSD via the emulators/i386-wine-devel port.  After talking with the fine folks in #bsdports on EFNet, I finally found a recipe that is poudriere friendly and seems to spit out something that sort of works.

David Naylor ([email protected]) has a working method for constructing wine on FreeBSD and this should work in most cases for using current.  The method is really designed for building a binary package for releases, most folks wouldn’t want to go down this route.

In order to begin, get poudriere configured and ready to go.  You’ll need to construct an i386 jail for the first part of this process.  Something like I show in my poudiere blog post

poudriere jail -c -j 11i386 -v head -a i386 -m svn

This will give you a build environment to get the 32bit binaries for wine built and packaged up for step 2.

poudriere builk -j 11i386 emulators/i386-wine-devel

If all goes well, you now have an i386 package of wine that will be consumed as a distfile for the amd64 package build.  I redefine PORTSDIR=/usr/local/poudriere/ports/default in /etc/make.conf.

If you are like me and use poudriere for everything, copy it to /usr/local/poudriere/ports/defaults/distfiles/freebsd:11:x86:64/

Now you’ll need to edit the emulators/i386-wine-devel distfile with the appropriate information generated from a sha256 and ls -l of your packagefile in your local i386 repo:

sha256 i386-wine-devel-1.7.7,1.txz

SHA256 (i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify this with the following to make it compatible with your system:

SHA256 (freebsd:11:x86:64/i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify the Makefile.inc to exclude checks for the OS version:

Index: Makefile.inc
===================================================================
— Makefile.inc    (revision 335346)
+++ Makefile.inc    (working copy)
@@ -41,10 +41,10 @@

.include <bsd.port.pre.mk>

-.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
-IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only
-DISTFILES=
-.endif
+#.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
+#IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only
+#DISTFILES=
+#.endif

.if ${PORT_OPTIONS:MGECKO}
RUN_DEPENDS+=   ${DATADIR}/gecko/wine_gecko-2.24-x86.msi:${PORTSDIR}/emulators/wine-gecko-devel

And now, you can try building the package in your *AMD64* poudriere build with:

poudriere bulk -j 11amd64 emulators/i386-wine-devel

If my instructions have succeeded, you now have a package suitable for installation on your amd64 machine that will now let you do wine things.

Now, I need to figure out what the Blizzard Network Installer is trying to do as it runs, self-updates and hangs.

Experimenting with zero-copy network IO in FreeBSD-HEAD

Back when I started all of this networking hacking, the "big thing" was the overhead of doing poll() and select(). Various operating systems came up with ways of eliminating these - FreeBSD grew the kqueue infrastructure; linux received epoll, Solaris received an epoll-like device and then ended up with some form of kqueue-like event mechanism. Windows has completion ports/overlapped IO which combined the event mechanism with a zero-copy way of doing network IO.

So the Free/Open operating systems have scalable event notification mechanisms for handling large numbers of concurrent sockets but they don't all have some nice, efficient way of doing zero-copy network IO.

Linux has splice()/tee()/vmsplice(). So yes, it effectively does have a way of doing zero-copy socket reading and writing.

OpenBSD does have a splice style syscall to copy data from a source to a destination TCP socket.

FreeBSD, however, has mostly focused on the "disk to network" path for content serving and thus has a lot of time invested in their sendfile() implementation. This is great if you're doing a lot of file to network sending (which Netflix does), but it has some serious shortcomings. The main one I'll address here is the lack of being able to do general zero-copy socket writes from userland. So it can only send data from disk files to the network. You can't implement a zero-copy intermediary proxy server, nor a memory cache that keeps things in pre-allocated memory regions. You have to use disk files (whether that be a real filesystem on disks, or a memory filesystem) and leverage VM hints to control caching.

Recently there was some new sendfile() work to allow sending from POSIX shared memory segments. This intrigued me - it's not the most effective way of doing zero-copy network IO from userland but it's a start. So I set off to write an updated version of my network library from yesteryear to implement some massively parallel network applications with.

The idea is simple - you allocate a POSIX shared memory segment. You then mmap() that region into memory and treat it as a place to allocate write-side network buffers from. Then you use the shared memory filedescriptor and offset to schedule a sendfile() from the shared memory segment to the destination network socket. It's not as elegant as having a write path that wires the memory down and just populates mbufs from that, but that'll come later.

Here's what I found.

Firstly, there's no asynchronous "I'm done!" notification for the sendfile path. So you have no explicit notification that the underlying memory has been freed so you can reuse it. sendfile() has the SF_SYNC flag which causes it to sleep until the transaction is done - primarily so users can be sure they can change the underlying file contents after the syscall completes. This is used by caches such as Varnish that leverage on-disk files as their cache filesystem space.

So I've been adding that. I have a working prototype that is scaling quite well under load and I'll look to commit it to FreeBSD-HEAD soon. It posts a knote to a kqueue filedescrpitor once a transaction has completed.

Once that was done, I started benchmarking the performance of this setup.

The first real roadblock I hit was massive VM contention on the shared memory segment. It turns out that a single POSIX shared memory segment is represented as a single vm_object and this is protected by a single lock. So when 8 threads are actively doing IO from the same shared memory segment it hits massive lock contention. I fixed this in my test suite by allocating one shared memory segment per thread. It's not elegant but it works well enough for benchmarking.

I next hit issues with contention on the VM page lists. Besides the per-object list, there's also a global per-type list (active, inactive, etc.) There's one lock protecting each of these lists. What I found was the VM was shuffling pages between active/inactive and at the traffic rates I was doing (20+gbit/sec) it was a few hundred thousand pages a second being shuffled around. The solution? mlock() the whole region into memory. This prevented the VM from having the pages change state so often and eliminated that overhead.

The code for doing this sendfile() work with posix shared memory is in my libiapp code - http://github.com/erikarn/libiapp . It's terrible and hacky - I'm just experimenting with things for now. But with some tuning, I can get a good 35Gbit/sec out of 70,000 active TCP sockets. There's still a long way to go - I shouldn't be saturating an 8-core CPU with this traffic level when I'm doing no socket data copies. I'll write another update or two about that soon.

Now, what would I like to see? I did some experiments with physical disk IO using the FreeBSD AIO paths doing the same kinds of IO patterns as I am doing with network socket IO (4KiB to 64KiB random disk reads.) It turns out if you do everything correctly, the FreeBSD AIO code will turn physical disk IO into asynchronous disk buffer transactions by wiring the userland buffer into memory and then using that as the backing buffer memory. The overhead of doing the pmap work for this was not too high. So, I wonder if it's worth writing a new transmit path that uses the pmap code (and not the VM!) to wire in a region of memory and then use that for transmit buffers. Combined with an iovec style array of buffers and the above kqueue notification of the network IO completion, I think we can end up with a much more flexible method of doing network IO from userland without the shortcomings by using POSIX shared memory with sendfile().

iTerm2, solarized, ssh, and ls colors

It took me a while to figure this out, but in the end I found out why I couldn’t see ls colors when logging over SSH to a linux box from iTerm2 using the solarized color scheme. It turns out solarized does something that makes the ls colors indistinguishable over ssh. My solution was to have LS_COLORS set to “di=00;34:ln=00;35:so=00;32:pi=01;33:ex=00;31:bd=00;34″ on the Linux box I ssh’ed into. You obviously must have an alias ls –color=auto and the right TERM variable (I use xterm-256color) on that box too. For BSD ls I use LSCOLORS set to “gxfxbEaEBxxEhEhBaDaCaD” to get ls colors.

Posted from Providence, Rhode Island, United States.

The Ports Management Team 2013-12-09 15:19:23

In this week’s interview, we talk to Baptiste Daroussin, who brought us such things as poudriere, pkgng, etc.

Name

Baptiste Daroussin

Committer name

bapt

Inspiration for your IRC nick

Baptiste is too long to type

TLD of origin

.fr

Occupation

Unix system engineer in general

When did you join portmgr@

May 2011

<Editor’s note: here is the video of bapt getting recruited, https://www.youtube.com/watch?v=tZk__K8rqOg >

Blog

http://blog.etoilebsd.net

Inspiration for using FreeBSD

I started using FreeBSD as a student in 2001/2002 and never left it since

Who was your first contact in FreeBSD

According to ports/86401 barner@ was

Who was your mentor(s)

In the ports land: jadawin@ and tabthorpe@
In the src land: des@

What was your most embarrassing moment in FreeBSD

There are too much of them to be able just pick one like this.

Boxers / Briefs / other

Boxers

What is your role in your circle of friends

Depend on the circle

vi(m) / emacs / other

vim

What keeps you motivated in FreeBSD

FreeBSD is a fantastic Operatic System with a crappy package system
The ports needs lots of cleanup and modernization,
The package system still need lots of love.

Favourite musician/band

I like lots of different kind of music, if I have to choose just one band I
would say Metallica from before they mess up (I let fan decide themself when
they started to mess up)

What book do you have on your bedside table

The Restraint of Beasts by Magnus Mills

coffee / tea / other

coffee

Favourite whisky

Old Pultney

Do you have a guilty pleasure

Add french words to the FreeBSD user vocabulary

How would you describe yourself

Hey

sendmail / postfix / other

OpenSMTPD

Do you have a hobby outside of FreeBSD

I do play rugby on a regular basis, I love reading books (real books not
e-something), party with friends etc.

What is your favourite TV show

I do not watch much television

Claim to Fame

pkgng, poudrière, USES, new options framework for the ports tree, staging area
support for the ports tree, more to come :)

What did you have for breakfast today

Chocapics :)

What sports team do you support

I love practicing but I not particularly supporting any team.

What else do you do in the world of FreeBSD

I’m trying to lean pf code and see how complicated it is to sync features from
OpenBSD’s version (I said trying :))

I maintain some contrib code in base: mostly m4 and byacc

What can you tell us about yourself that most people don’t know

I keep refraining myself from forking the ports tree into a portsng thing :)
(not sure most people don’t know that)

Any parting words you want to share

Any volunteer to document my work?

What is your .sig at the moment

regards,
Bapt

Updating FreeBSD 8.2 (or 9.x) to 10 (beta4)

This is a little description how I remotely (no console, booted into multi-user during update, no external services like jails/httpd/… running) updated a FreeBSD 8.2 to 10 (beta4) from source. This should also work when updating from FreeBSD 9.x. Note, I had already switched to ATA_CAM on 8.2, so not instructions for the name change of the ata devices. No IPv6, WLAN or CARP is in use here, so changes which are needed in this area are not covered. Read UPDATING carefully, there are a lot of changes between major releases.

What I did:

  • update /usr/src
  • make buildworld
  • replace “make ” in /usr/src/Makefile.inc1 with ${MAKE} (two times, one for “VERSION”, one for “BRANCH”)
  • verify kernel config for changes needed (running “config MyKernel” in /usr/src/sys/YourArch/conf/ helps to identify syntax problems), sorry I didn’t take notes, but I diffed the old and the new GENERIC config and added/removed according to my interests
  • /usr/obj/…/src/usr.bin/bmake/make buildkernel KERNCONF=MyKernel
  • /usr/obj/…/src/usr.bin/bmake/make installkernel KERNCONF=MyKernel KODIR=/boot/kernel.10
  • mergemaster –p
  • /usr/obj/…/src/usr.bin/bmake/make installworld DESTDIR=/somewhere/test
  • mkdir /root/net10; cp /somewhere/test/rescue/ifconfig /somewhere/test/rescue/route /root/net10
  • create the file /etc/rc.10update with:
    case $(uname –r) in
    8.2*)
            MYIFCONFIG=/sbin/ifconfig
            MYROUTE=/sbin/route
            ;;
    10*)
            MYIFCONFIG=/root/net10/ifconfig
            MYROUTE=/root/net10/route
            ;;
    esac
    export MYIFCONFIG
    export MYROUTE
  • change the files (stupid approach: grep for “ifconfig” and “route” in /etc/rc.d to identify files which need to change, I skipped files which I identified as not needed in my case, if you use pf/IPv6/bridge/…, you may have to change some more files) /etc/rc.d/auto_linklocal /etc/rc.d/defaultroute /etc/rc.d/netif /etc/rc.d/netwait /etc/rc.d/routing: add “. /etc/rc.10update” at the end of the block with “. /etc/rc.subr”, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • change /etc/network.subr: add “. /etc/rc.10update” before the first function, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • make sure that the changes you made are 100% correct, rather triple-check than to not check at all (you will be locked out if they are not 100% OK)
  • stop any jails and make sure they do not restart at boot
  • deactivate the gmirror of the root-fs, if there is one (it is maybe easier to ask a remote hand to swap the boot order in case of problems)
  • here you could just a reboot of the server to come back to your current OS version, so make sure that the modifications in /etc did not cause any problems with the old version (in case you see problems with the v10 kernel), but if you do not have a remote console to single-user mode you have no chance to directly fix the problem (risk mitigation described above), no matter which version of the kernel you boot
  • nextboot –k kernel.10
  • shutdown –r now
  • login
  • check dmesg
  • optional: mv /boot/kernel /boot/kernel.8
  • make installkernel KERNCONF=MyKernel
    to have a v10 /boot/kernel
  • make installworld
  • mergemaster
  • make delete-old
  • rm –r /etc/rc.10update /root/net10
  • change rc.conf: add “inet” in ifconfig-aliases
  • review sysctl.conf for outdated entries
  • shutdown –r now
  • optional: rm –r /boot/kernel.10
  • enable jails again (or later… updating jails is not described here)
  • activate/resync mirror(s)
  • rebuild all ports (attention: new pkg system)
  • make delete-old-libs
  • reboot again to make sure everything is OK after the port-rebuild and removal of old libs (a console.log (syslog.conf) helps here

Bookmark/FavoritesShare/Save

The Ports Management Team 2013-12-02 15:31:58

As port of the ongoing series on getting to know your portmgr@, I interview the portmgr-secretary@, Thomas Abthorpe.  (Yes it is awkward talking about myself in third person)

Name

Thomas Abthorpe

Committer name

tabthorpe

Inspiration for your IRC nick

Uninterestingly, just my uid that I have had in near continuous use since 1989

TLD of origin

CA

Occupation

Regional Technical Support Specialist with the Canadian Grain Commission

When did you join portmgr@

Joined as -secretary in March 2010
Upgraded to full member in March 2011

Blog

http://www.abthorpe.org

Inspiration for using FreeBSD

I wanted to run a webserver, and I wanted something free. I was going
to use something linux, then met up with a former prof from university,
and shared my story with him. He told me FreeBSD was the way to go.

Who was your first contact in FreeBSD

clsung, miwi, rafan

Who was your mentor(s)

clsung/miwi

What was your most embarrassing moment in FreeBSD

I reset a bunch of ports that were name@??.FreeBSD.org, because my
script was only checking for @FreeBSD.org. I was called on it
privately, to which I immediately reverted my commit.

Boxers / Briefs / other

Boxers

What is your role in your circle of friends

Oddly enough, social convener

vi(m) / emacs / other

vim

What keeps you motivated in FreeBSD

I just love that I (and anybody for that matter) can contribute the most
trivial of things, and watch it go live.

Favourite musician/band

Favourite of all time is The Beatles. On the home front, I am huge fan
of the Canadian band Barenaked Ladies. I was so fortunate to even meet
them back stage at a concert in 1997.

What book do you have on your bedside table

The Beardmore Relics. It is a fictionalised story of hoax about
viking relics that were discovered near Beardmore, ON, a little
community I lived in back in the late 70s.

 coffee / tea / other

Coffee, intravenous if possible!

Favourite whisky

Springbank

Do you have a guilty pleasure

Sitting around in comfortable lounge wear, watching Sci-Fi on TV

How would you describe yourself

N-Space, multidimensional nerd.

sendmail / postfix / other

sendmail for internal use, postfix for anything exposed to the Internet

Do you have a hobby outside of FreeBSD

I am a senior apprentice bicycle mechanic with a grass roots movement
called Bicycles for Humanity

What is your favourite TV show

Big Bang Theory. Physics was my other interest I followed in
university. The interaction among the friends, is uncannily similar as
that among my friends. Fortunately we have no Sheldon. And Barenaked
Ladies do the theme song :)

Love almost anything Sci-Fi, particularly the old Stargate franchise(s), Dr. Who
Sanctuary, etc, etc, etc.

Claim to Fame

Don’t really think I have one

What did you have for breakfast today

Coffee and a bagel.  I have that almost everyday, so when I say to people,
“You expect me to remember BLAH? I don’t remember what I had for
breakfast yesterday?” there is an implied irony.

What sports team do you support

Pro hockey – The one and only Montreal Canadiens
Pro lacrosse – The Toronto Rock
University hockey – Lakehead University Thunderwloves

What else do you do in the world of FreeBSD

core@, donations@, formerly kde@

What can you tell us about yourself that most people don’t know

I used to play the bagpipes. Was “fair” at it. My band was invited to
play at the “Toys on Parage” in Santa Ana CA back in 1987. Astronaut
Neil Armstrong was the parade marshall, and I can claim that I was at a
black tie reception with him. I was performing with the band, and I wasn’t
allowed to get near him, but I can at least say I “partied” with him.

Any parting words you want to share

Always remember the Golden Rule, “Do unto others, before others do unto you”,
or something like that :)

What is your .sig at the moment

Thomas Abthorpe | FreeBSD Committer
[email protected] | http://people.freebsd.org/~tabthorpe

Moving forward by going slightly backwards and to the right, UEFI booting on #FreeBSD

The FreeBSD Foundation has been working towards the future of booting in x86 and catching up to our friends in Linux-land by sponsoring work on a UEFI enabled boot loader.  This work was taken on by Benno Rice ([email protected]) and Ed Maste ([email protected]).

So far, it appears that one can indeed boot FreeBSD as I will demonstrate on my Thinkpad T520.

Starting with the UEFI project branch, one must build a 64bit version of libstand in tree.

cd uefi/lib/libstand && make

Modify the makefile in sys/boot/amd64

Index: amd64/efi/Makefile
===================================================================
— amd64/efi/Makefile    (revision 258775)
+++ amd64/efi/Makefile    (working copy)
@@ -77,8 +77,8 @@
LIBEFI=        ${.OBJDIR}/../../efi/libefi/libefi.a
CFLAGS+=    -I${.CURDIR}/../../common

-DPADD=        ${LIBFICL} ${LIBEFI} ${LIBSTAND}
-LDADD=        ${LIBFICL} ${LIBEFI} ${LIBSTAND}
+DPADD=        ${LIBFICL} ${LIBEFI} ../../../../lib/libstand/libstand.a
+LDADD=        ${LIBFICL} ${LIBEFI} ../../../../lib/libstand/libstand.a

.include <bsd.prog.mk>

Now you can build loader.efi and get it to link against the 64bit version of libstand:

cd sys/boot && make

UEFI will look for a FAT formatted partition with the “efi” signature on it.  FreeBSD’s gpart can create this partition for you, so do the following foo:

gpart create -s gpt da0

gpart add -t efi da0

gpart add -t freebsd-ufs da0

$ gpart show da0
=>     34  2013117  da0  GPT  (983M)
34   131072    1  efi  (64M)
131106  1882045    2  freebsd-ufs  (919M)

newfs -t msdosfs /dev/da0p1

newfs /dev/da0p2

Mount the fat formatted partition, create the EFI directory structure(this is mandatory) and copy the loader.efi binary into place as bootx64.efi

mount -t msdosfs /dev/da0p1 /mnt

mkdir -p /mnt/efi/boot

cp uefi/sys/boot/amd64

Because the kernel currently needs to be aware of the new style UEFI memory map, you can’t run stock -current in this configuration.  You’ll need to use a kernel from the projects/uefi branch when constructing your bootable device.  I used a 1G usb thumbdrive for this test, so mount the UFS partition and use it as a DESTDIR for your installworld/installkernel:

make -s buildworld

make -s buildkernel

mount /dev/da0p2 /mnt

DESTDIR=/mnt make -s installworld

DESTDIR=/mnt make -s installkernel

DESTDIR=/mnt make -s distribution

Setup an /etc/fstab on this stick:

/dev/da0p2             /               ufs     rw,              1       1

At this point, your USB disk is ready for its first booting attempt.

EFI_BIOS

EFI_LOADER1

I have to toggle UEFI/Legacy BIOS mode in my laptop.  For your entertainment, here it is.  This has the convient side effect of not booting from my other disk devices in my laptop as they do not have the “proper” fat formatted EFI partition on them.  This actually yeilds a pretty quick boot to the loader.

Amazing!  It did!  Sort of.

Now we have the entertainment of trying to figure out how to get here to multiuser.

 

 

 

EFI_LOADER2

With a “show” we find out that the loader has selected the EFI partition “part6″ as the boot device.  “lsdev” shows us all the partitions that we could boot from, but I have chosen well in this example and can easily see that the one I really want is tagged with a “(removable)”.

In this case executing a “set currdev=part7″ sets up the loader to boot and executing “boot” will get this system into multiuser.

Many thanks to the folks at the FreeBSD Foundation for these initial steps into UEFI.  The project branch in subversion is publicly available and I highly encourage folks to engage the community to get this closer to production grade.

The Ports Management Team 2013-11-25 15:38:23

In this interview, we talk to Martin Wilke, aka miwi, the high quality automated PR and commit bot.

Name

Martin Wilke

Committer name

miwi

Inspiration for your IRC nick

MartIn WIlke

TLD of origin

Ger

Current TLD (if different from above)

My

Occupation

System Administrator

When did you join portmgr@

Jun 4 08:35:57 UTC 2009 (actually a few hours earlier because we did
know that already)

Blog

http://miwi.cc (I did not write anything in ages ^^)

Inspiration for using FreeBSD

Can I use a shout cart?
http://miwi.cc/2012/03/why-freebsd-my-personal-view/ that should explain it
quite well I think.

Who was your first contact in FreeBSD

To be honest, I think many because I spam GNATS quiet heavily :)

Who was your mentor(s)

Krion (wave)

What was your most embarrassing moment in FreeBSD

If you want to know, see my commit history, you will find a lot :(

Boxers / Briefs / other

Boxers

vi(m) / emacs / other

vi(m)/mcedit :)

What keeps you motivated in FreeBSD

Still the community

Favourite musician/band

Gerneral I still like and listen a lot to of German Hip Hop (I still feel
very young!)

What book do you have on your bedside table.

Book? Can I eat that?

coffee / tea / other

black coffee

Do you have a guilty pleasure

Hm..

How would you describe yourself

Shy :)

sendmail / postfix / other

postfix

Do you have a hobby outside of FreeBSD

When you have kids there is not much time left for hobbies :)

What is your favourite TV show

Stargate*, House

Claim to Fame

miwibot and friends ..

What did you have for breakfast today

Rotti Chanai, Ice Milo

What sports team do you support

I keep on following F1 and hope a other German Driver will win ..

What else do you do in the world of FreeBSD

Trying to clean up bapts stuff *gg* (joking), lately I helped a lot
on the cluster setup / package packaging.

What can you tell us about yourself that most people don’t know

I canâ~@~Yt else you will know :)

<Editor note: this is a copy and paste, something may have been lost in translation>

Any parting words you want to share.

Have a nice day!

What is your .sig at the moment

Mess with the best, die like the rest :).

AsiaBSDCon 2013 Videos Posted to YouTube

Sato-san has created a playlist of 15 videos from AsiaBSDCon 2013. Congrats to the organizers for running another successful conference in Tokyo. Some of the FreeBSD-related videos are listed below:

There were also quite a few OpenBSD talks this year. I had a great time at AsiaBSDCon several years back and hope to make it back again some day.

The Ports Management Team 2013-11-18 14:48:17

In this interview we talk to one of the Annoying Reminder Guys, Erwin Lansing, after Joe Marcus Clarke, the second longest serving member of portmgr@

Name

Erwin Lansing

Committer name

erwin (not edwin)

Inspiration for your IRC nick

My research, back in the day, as a post graduate on fruit flies
(Drosophila melanogaster)

ccTLD of origin

.nl

Current ccTLD (if different from above)

.dk

Occupation

See above :-)
Network and system administrator at DK Hostmaster A/S, maintaining the
.dk ccTLD.

When did you join portmgr@

2004 as portmgr secretary, 2005 as a voting member

Blog

http://droso.dk/

Inspiration for using FreeBSD

Quite the usual story, going from Slackware to RedHat to Debian to
Gentoo to FreeBSD after starting at a web hosting startup, and never
looked back.

Who was your first contact in FreeBSD

The one and only web server running the startup I joined after giving up
on academic life.

Who was your mentor(s)

edwin (not erwin)

What was your most embarrassing moment in FreeBSD

Copy/pasting diffs into email into GNATS with all the whitespace fun
that ensued. Many times!

Boxers / Briefs / other

boxers

What is your role in your circle of friends

Who needs friends when one has IRC?

vi(m) / emacs / other

nvi

What keeps you motivated in FreeBSD

It’s the best well-hidden success story in operating systems, by
now in the hands of more people than one can count and used by even more
people, and not one of them knows it! It’s not only the best
operating system currently around, but also the most supportive and
inspiring community.

Favourite musician/band

U2 (yeah, I’m that old)

<editor’s note: tabthorpe is older>

What book do you have on your bedside table

The rough guide to Portugal, though soon to be replaced after returning
from vacation

coffee / tea / other

coffee, strong and black

Do you have a guilty pleasure

see previous

How would you describe yourself

more coffee

sendmail / postfix / other

postfix

bind / nsd / knot

all of the above

Do you have a hobby outside of FreeBSD

There’s life outside FreeBSD?

What is your favourite TV show

Currently replaying Lost, looking forward to the next season of Top Gear

Claim to Fame

Structuring internal procedures inside portmgr as secretary, keeping the
pointyhat cluster alive and building packages for way past its best
before date, and now maintaining Bind.

What did you have for breakfast today

Yogurt

What sports team do you support

The winning team

What else do you do in the world of FreeBSD

Vice-president of the FreeBSD Foundation
Anything DNS

What can you tell us about yourself that most people don’t know

Today’s word is: My turn!

Any parting words you want to share

Goodbye, cruel world

What is your .sig at the moment

None, unless it can’t be avoided

Building old-school TiVo build tools on FreeBSD 9.2-stable

Intro

As long-term readers of my blog know, I've been nursing along a TiVo HR10-250 for the past few years. It works great for the low-quality material that my younger son loves to watch, plus I can harvest video off of it with ease.

Recently, we switched up how it was connected to our TV (and indeed, got a new TV too). During this process, we lost the ability to display subtitles. Since my wife and I like to watch The Daily Show and other similar shows upstairs without going downstairs to the big TV with the HD DirecTV player, and the environment upstairs can be a bit noisy, subtitles are quite useful.

So, after crawling around the DealDatabase forums for a bit, I found a good program called tivovbi. It works really well for displaying closed captioning. However, I used a binary I found on the forum. Nothing is more annoying than a program you download from a forum that randomly core dumps for reasons that are totally mysterious.

I can't even hack it to do anything since I have no TiVo build tools.  Looking for tools online, I can't really find anything that isn't just a Linux binary.  The Linux binaries have issues with the FreeBSD linux ABI implementation, owing in large part to their age (binaries from 10 years ago have some issues, I think with just the packages are needed having a subtle incompatibility).

So what should I do. I could create a virtualbox VM and run linux. But then I'd be running Linux, and copying back and forth to a VM is always a hassle in some way.

So, the other alternative is to build the tools from source. I thought it would be easy to do this, but there's a number of issues with it, so I thought I'd write up my experience. This is on a FreeBSD 9.2-stable system on amd64. That last bit will turn out to be important in a bit, because nothing was easy and simple on this project...

After reading through these instructions, I can't help but marvel at how this lack of integration is tolerated, but to be fair, this is building tools that are nearly a decade old at this point and I did have to kludge around lack of support for 64-bit x86 in gcc... This is nearly 60 separate commands to type. Yuck. Also, looking at the layout in chrome, many of the line breaks have liberties taken with them, so your best bet may be to cut and paste much of what I'm doing here...

Locating the tools

Tivo Utils has a bunch of useful links. Including the linux binaries that I've had issues with. Thankfully, there's a shell script called 'build_mips_x_compiler.sh' which builds all the tools and the basic libraries. I thought I could just run it and have everything built. As with everything else in this project, this wasn't so much the case. So I wound up doing a lot of things by hand.

The first bit of the script fetches all the source tar balls. The gnu stuff has had long-term stable paths, so they fetched. However the TiVo linux didn't transfer, since it was in a new location. In fact, it was also for 4.0, and my TiVo was running 6.4a. So, I had to grab that from TiVo linux downloads page. The Linux 6.4 download was exactly what I needed to grab. Once I had these in place, I was ready to start building. You also need binutils 2.13Gcc 3.0 (yes, 3.0!), glibc 2.2.3, and glibc 2.2.3 linuxthreads support.

Also, I'm installing all the tools in ~/tivo/tools tree, and building them from ~/tivo/toolchain. Fetch all of the above linked tarballs into ~/tivo/toolchain (or whatever you want). Now that we've found the tarballs, we can start with the builds. I'll assume the following environment variables are set. TARGET is set to "mips-TiVo-linux" and PREFIX is set to "$HOME/tivo/tools". I've also added ${PREFIX}/bin to my PATH (I didn't from the start, and got quite far before it mattered). I also had gnu make installed as gmake from ports.

Building Tools

Binutils 2.13

Binutils 2.13 is almost trivial to build:
cd ~imp/toolchain
tar xvfz binutils-2.13.tar.gz
mkdir build-buinutils
cd build-binutils
../binutils-2.13/configure  --target=$TARGET --prefix=$PREFIX
gmake -j 20 all
gmake install
cd ..

gcc 3.0 stage 1

gcc 3.0 stage 1 took a lot of trial and error to create. Turns out, there's no support for FreeBSD 9 in this tarball. That's trivial to add, and I've provided some patches. However, the next problem is that FreeBSD/amd64 isn't supported. This problem turns out to be more difficult to overcome. So you have to configure for FreeBSD/i386 (or backport amd64 support, which I thought would be too hard, so I didn't do it). This means you need to have a 32-bit compiler. After all the talk about making cc -m32 working, I thought I could just do gmake CC="cc -m32". This failed, however. The 32-bit applications dumped core, meaning that gcc couldn't complete its bootstrap process. I wound up having to use FreeBSD's xdev to do it:
pushd ~/FreeBSD/svn/stable/9
sudo make xdev XDEV=i386 XDEV_ARCH=i386 WITHOUT_CLANG=t
popd
Once we have that in place, we can build gcc. Except that it doesn't build quite right. We need to add some additional patches. One to work around a weirdness in the obstack implementation where it relied on an illegal lvalue autoincrement, and one to work around crazy sh that's likely bogus in fixincl. All of these are included in the above patch.
mkdir build-gcc
cd build-gcc
../gcc-3.0/configure --target $TARGET --prefix $PREFIX --without-headers --with-newlib --disable-shared --enable-languages=c --host i386-foo-freebsd9
gmake CC=/usr/i386-freebsd/usr/bin/cc all-gcc
gmake CC=/usr/i386-freebsd/usr/bin/cc install-gcc
If you wanted to see if -m32 worked, you could skip the make xdev step above and substitute CC="cc -m32" in the two gmake commands. I've also tested values up to 20 for -j for at least the first command.

Building the Linux Kernel headers

The following comes pretty much verbatim from the build_mips_x_compiler.sh and have been verified to work. You'll need to grab this patch for these instructions to work. It works around an expr difference, as well a really cheap kludge to get autoconf.h generated. Also, I had to install the bash port, and create a symlink from /bin/bash to /usr/local/bin/bash, which I've not put inline...
tar xf TiVo-6.4-linux-2.4.tar.gz
cd linux-2.4
patch -p0 < ../tivo-linux-2.4.diff
yes "" | gmake ARCH=mips CROSS_COMPILE=mips-TiVo-linux config
gmake ARCH=mips CROSS_COMPILE=mips-TiVo-linux include/linux/version.h
mkdir $PREFIX/$TARGET/include
cp -rf include/linux $PREFIX/$TARGET/include
cp -rf include/asm-mips $PREFIX/$TARGET/include/asm
which will be enough to get us to the next step...  Woof, maybe I should just make a script for all this stuff... With so many fiddly bits, I'm not sure that's a good idea.

Building gmake 3.80!

Newer versions of gmake don't grok the Makefiles that glibc 2.2.3 generates. All kinds of weird errors and odd behavior. So, to make progress, you'll need to build gmake.
fetch ftp://ftp.gnu.org/gnu/make/make-3.80.tar.gz
tar xf make-3.80.tar.gz
cd make-3.80
./configure
make
cp make ~/bin/gmake380
cd ..
I didn't bother installing it, since I just need it for this diversion so I copied into my bin dir, that I have in my path.

Building glibc 2.2.3

Now we're on to glibc. Things have been smooth sailing up to this point. With glibc, we have to extract, configure, build, fix the build oops, build again, fix some info files, then install. Woof! Sure makes for a difficult to reproduce experience. And those are the build on linux instructions... Apparently there's a lot of host leakage that's making things somewhat difficult to reproduce... but so far that appears to be only with gmake 3.82. gmake 3.80 works much better. More patches needed.
tar xf glibc-2.2.3.tar.gz
tar xf glibc-linuxthreads-2.2.3.tar.gz -C glibc-2.2.3
env CC=mips-TiVo-linux-gcc ../glibc-2.2.3/configure --host=$TARGET --prefix=$PREFIX --disable-debug --disable-profile --enable-add-ons --with-headers=${PREFIX}/${TARGET}/include
gmake380
sed -i.old -e 's/elf32-bigmips/elf32-tradbigmips/' elf/rtld-ldscript
gmake380
gmake380 install
cd ..

Munging things around before the second assault  on gcc for its stage 2

Don't know why the original instructions take this time rearrange the deck chairs, but it seems to be necessary.
mv ${PREFIX}/${TARGET}/include/asm ${PREFIX}/include
mv ${PREFIX}/${TARGET}/include/linux ${PREFIX}/include
rm -r ${PREFIX}/${TARGET}/include
rm -r ${PREFIX}/${TARGET}/lib
ln -s ${PREFIX}/include ${PREFIX}/${TARGET}/include
ln -s ${PREFIX}/lib ${PREFIX}/${TARGET}/lib

Build gcc 3.0 stage 2

Now that we have everything else setup, it is time to build the second stage of gcc. Alas, gcc 4.2.1 doesn't allow constructs like (a ? b : c) = d, which gcc 3.0 uses to implement C++. So, no g++ for me. If you want g++, you'd have to build the gcc34 port... The rest is almost boring after all the other hoops we jumped through:
mkdir build-gcc2
cd build-gcc2
../gcc-3.0/configure --target=$TARGET --prefix=$PREFIX --enable-languages=c --host=i386-foo-freebsd9
gmake CC=/usr/i386-freebsd/usr/bin/cc all
gmake CC=/usr/i386-freebsd/usr/bin/cc install
cd ..

Building stuff

Now, it is time build things...  I'll write more blog entries about that...

Running Qemu VMs on FreeBSD

I'm slowly working on getting libvirt more functional on FreeBSD and, thanks to help of a lot of other people who did very valuable reviews of my patches and contributed other portability fixes, at this point libvirt on FreeBSD is capable of running Qemu VMs with some limitations. I'll provide a quick howto on doing that and what limitations exist at this point.

Building libvirt

As I'm playing with the codebase, it's more convenient for me to use direct git checkout instead of port. I'll provide an instuction how to build it (port should work as well, but I haven't tested).

Configure command looks this way:

CFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure \
--without-polkit \
--disable-werror

CFLAGS and LDFLAGS are needed to help it find yajl includes and libs. We're disabling polkit because we don't want to waste extra time configuring it and we disable treating warnings as errors because some third party libs' headers could mess things up.

When it configures successfully, it's traditional gmake && gmake install.

Preparing a VM

Nothing special about VM preparation, just create a Qemu VM like you usually do, configure it for you needs and you're almost ready to go.

There are some things recommended to do though:

  • Make virtio stuff available (see vtnet(4), virtio_balloon(4), virtio_blk(4) for details) on your guest
  • Configure network interface to using DHCP by adding this line to /etc/rc.conf: ifconfig_vtnet0="DHCP"

Defining and running a domain

Before running up libvirtd, make sure you have bridge(4) and tap(4) modules loaded or built into kernel.

Then you need to execute libvirtd and then connect to it using virsh:

virsh -c "qemu:///system"
Now, we need to create an XML file with domain definition. Here's the one I use: https://gist.github.com/novel/7399465

You might notice we're using virtio for network device and for the disk drive. You'll have to modify a path to the image location and adjust mem/cpu amount for your needs.

Once you're done with that, execute 'define /path/to/domain.xml' in virsh, and domain with name 'qemu' (unless you changed it) will be created. You can start it using 'start qemu' command in virsh.

To see what's going on, we could use vnc. To figure out what vnc port our VM is available at, we need to execute 'vncdisplay qemu' and it should print something like:

127.0.0.1:0

Now we can connect using vnc client, for example, if you use net/vnc, the proper command will be:

vncviewer 127.0.0.1:0

If everything goes well, you'll get an access to your VM's console.

Run ifconfig to check if there's an IP address on vtnet0 interface:

vtnet0: flags=8843 metric 0 mtu 1500
options=80028
ether 52:54:00:ae:4c:37
inet 192.168.122.117 netmask 0xffffff00 broadcast 192.168.122.255
media: Ethernet 1000baseT
status: active

And on the host you'll have:

  • bridge(4) device virbr0
  • tap(4) device vnet0 which corresponds to our VM and is member of virbr0 bridge

It's possible to connect from guest to host and vice versa. Unfortunately, it's not possible to provide access to the internet for VM without manual configuration. Normally, libvirt should do that, but its bridge driver needs some work to do that on FreeBSD, and that's on my todo list.

Apart from bridge stuff, there are a lot of other limitations on FreeBSD currently, but firewalling for bridge driver and nwfilter are probably the most important now. It's not quite easy to come up with a good way to implement it. For example, one has to choose what firewall package to use, pf and ipfw being the main candidates for that.

I'll cover my thoughts and concerns in a separate post.

Feel free to poke me if you have any questions/thoughts.

Update 25/01/2014: some people have troubles because of the old qemu version. On FreeBSD, please use qemu-devel port, because qemu port is too old.

The Ports Management Team 2013-11-11 14:11:14

In this interview, we talk to another portmgr-lurker@, Antoine Brodin aka antoine@  He claims his superpower is laughing at tabthorpe’s jokes

<editor’s note: whatever that means>

Name

Antoine Brodin

Committer name

antoine

Inspiration for your IRC nick

You have to ask my parents :)

TLD of origin

FR

Occupation

IS Security Consultant. I mainly perform incident response and provide security expertise.

When did you join portmgr@

Not yet, I am just a lurker

Blog

None

Inspiration for using FreeBSD

I started using FreeBSD in 2002, when I got my own computer. It is the best operating system around.

Who was your first contact in FreeBSD

pav@ handled my first ports PR in 2004

Who was your mentor(s)

rwatson@ was my src mentor in 2008, eadler@ and miwi@ were my ports mentors in 2013

What was your most embarrassing moment in FreeBSD

I don’t remember having one

Boxers / Briefs / other

Boxers

vi(m) / emacs / other

vi

What keeps you motivated in FreeBSD

There are improvements and new features implemented all the time. In the ports area for instance, there is STAGEDIR, pkgng, USES, QA stuff in just a few months

Favourite musician/band

My favorite artists are Admiral Tibett and Glen Washington

coffee / tea / other

Green tea

Do you have a guilty pleasure

Chocolate, fried chicken

sendmail / postfix / other

sendmail (because it is in base)

What is your favourite TV show

I don’t own a TV

What did you have for breakfast today

Hot chocolate, bread and mirabelle jam

What can you tell us about yourself that most people don’t know

I like cutting down trees in Morvan forest

What is your .sig at the moment

Cheers,

Antoine

The Unusual Suspects, #FreeBSD Vendor Summit 2013

I was fortunate enough this year to be able to help the FreeBSD Foundation host the 2013 Fall Vendor Summit at my workplace, Yahoo.  Our facilities in Sunnyvale are very first class and I like to help out with my non-technical resources whenever possible (because, frankly, if you’ve seen my code, you would prefer it that way).

George Neville-Neil of the FreeBSD Project and FreeBSD Foundation had asked if Yahoo could host again this year and we agreed to a one day presentation and get together at the main campus.

Lots of folks who don’t normally go for conferences showed up to this invitation only event, and for once it felt like we had a strong showing.  I had booked a conference room for 55 people and we had close to 70 show up.  It was really close to bordering on overflow into the hallway at one point.

I think my biggest takeaways this year was the fact that “FreeBSD Doesn’t Have Visualization” is now just a myth and doesn’t really match reality.  The Bhyve project has taken a good direction and now can spin up other o/s instances, like Linux, via the ACPI framework implemented during the Google Summer of Code projects.  It was also very good to see VMWare and Google Compute folks showing up and asking for “what we need to help you folks support FreeBSD in our cloud things.”

Instead of the hallway track at normal conferences, we had the “back of the room on the floor” track this year where there was much debating over the validity of git as a FreeBSD source management tool.  The thing is, the project already exports FreeBSD SVN src to a self hosted git repo (http://git.freebsd.org) and a github instance (https://github.com/freebsd).   The debate swirls around the archaic “email patches to mailing lists” mentality instead of the “send pull request” things that the git world now has.

Interesting point from this discussion, perhaps we should now take the time to assign people who are more involved to important sections of kernel and source code.  The FreeBSD ports system has direct maintainers and a system to timeout maintainers who are AFK.  The FreeBSD base system has a more liberal approach as any committer can and does commit to any aspect of the tree.  Its common practice to not do this without review, but its no a true formal review process.  This leads to some cases where patches go to mailing lists and never get picked up and reviewed.

Otherwise, a fine time was had and I certainly look forward to the next conference, AsiaBSDCon 2014.

The Ports Management Team 2013-11-04 14:01:32

In this interview, we take a short break from the usual, and interview a portmgr-lurker@, Mathieu Arnold.  He was one of the founding members of perl@ back in the day

Name

Mathieu Arnold

Committer name

mat

Inspiration for your IRC nick

Well, not very inspirational, but everybody’s been calling me mat since
high school, well, except my grandmother, maybe.

TLD of origin

.fr

Occupation

Sysadmin/Developper/Network Admin/Stuff doer.

When did you join portmgr@

Ah, can we put a time in the future ? :-)

<editor note, this is a stock question>

Inspiration for using FreeBSD

Around 2000/2001 I needed something to replace a Linux firewall, the
Paris team, roberto@, pb@, were advocating FreeBSD, so I went that way,
with ipf. We were so pleased with everything that, well, over the time
we switched over, we’ve been a FreeBSD shop for a bit more than 10
years.

Who was your first contact in FreeBSD

That must have been roberto@ or pb@, don’t know which one :-)

Who was your mentor(s)

demon was.

What was your most embarrassing moment in FreeBSD

I’m not sure there was one, maybe this year in August, when I added perl
5.18 and broke the ports tree in r324662 and fixed it in r324667.

Boxers / Briefs / other

Boxers.

What is your role in your circle of friends

Being the clown.

vi(m) / emacs / other

vim. What Else…
https://github.com/mat813/dotvim

What keeps you motivated in FreeBSD

Hum, I get bored easily, so, things changing and going forward.

Favourite musician/band

My favorite band of all time would be Dire Straits. More recently,
though, Moriarty, Of Monster and Men, Owl City, Passenger (Mike
Rosenberg), so, pop/folk/rock would be my treat.

What book do you have on your bedside table

Right now, I’m at about 3/4 of The Department of Mad Scientists, The
World Without Us, and about half of Cryptography Engineering. On my
daily commute in the subway, I read science fiction or fantasy. Iain
Banks, Terry Pratchett, Peter F. Hamilton, Neil Asher, Lois McMaster
Bujold, Charles Stross, David Weber, John Ringo…

coffee / tea / other

Tea. Mostly green tea, Long Jing, a bit of white  Bai Mu Dan, and in the
mornings, some Far Too Good For Ordinary People Darjeeling teas.

Do you have a guilty pleasure

Like to cook, and to eat. People following me on instagram think it
should be renamed instakilogram because of me.

How would you describe yourself

Quiet, thoughtful, kind, witty, oh, wait, this isn’t a dating site.

sendmail / postfix / other

sendmail when cron’s the only one sending stuff, postfix otherwise.

Do you have a hobby outside of FreeBSD

Photography and hiking. (Mostly at the same time.)
http://www.flickr.com/photos/mathieu-arnold/sets/72157620872036952/

What is your favourite TV show

Doctor Who. Then House, or The Big Bang Theory.

Claim to Fame

Ah, a long time ago, I translated linux HowTos, people are still quoting
me the white lies that I wrote in them, and I still try to correct
them…

What did you have for breakfast today

Earl gray, a toasted slice of whole bread, with a touch of butter and
melon jam with orange zests, and a yoghurt.

What sports team do you support

Ah, so, sport, the thing with the silly people encouraging other silly
people running around… None, really, maybe the French rugby team
during the rugby world cup or the rugby league world cup.

What else do you do in the world of FreeBSD

Beside being part of perl@ team, I try to advocate a bit…

What can you tell us about yourself that most people don’t know

Let me think about that, and get back to you :-)

Any parting words you want to share

I’m not feeling really like lecturing right now, so, no…

What is your .sig at the moment


Mathieu Arnold

Doing arduino development on FreeBSD-HEAD

I'm a sucker for punishment.

Or, I noticed that FreeBSD's pkgng binary package repository ships with a port of the Arduino development environment. It's this java thing that wraps around avr-gcc and avrdude. It's very popular, it's open source, and I figured what the hell.

I plugged in my Arduino Leonardo and .. it was detected as a umodem device. Excellent!

.. and then it wasn't. It went away very quickly and came back as a single interface (OK) with three child interfaces (Hm, okay), but only one uhid (human interface) interface active (Not Ok.) The modem port used to program and talk to the thing wasn't there.

I then went on a bit of a journey. I found that quite some work had already been done to correct issues in the FreeBSD USB stack - however, it still wasn't working. It showed up fine - it identified itself as a generic USB serial port device, and yet umodem didn't bind to it.

Next - the umodem source code. It yes, claimed anything identifying as a USB serial class device - but it only claimed devices that ALSO identified as an AT-class modem. Yes, a serial modem that you speak AT commands to. The Leonardo identifies itself as a USB serial class device but with NO command encoding. umodem didn't like that.

So, to the USB 1.1 standards documention! After reading the relevant bits, I discovered that the rest of the device handling is the same! Ie, it doesn't matter whether the device says "I speak AT commands" or "I speak no commands", it's still serial. This identifier is just for the upper layer application to decide whether to send AT commands or not.

Thus the fix was simple - also claim devices that say "no commands" as well as "AT commands." That fix is in -HEAD and I hope to try and sneak it into 10.0.

And with that - FreeBSD-HEAD is now a viable development environment for the Arduino Leonardo.

How I learned to stop worrying and love the powderkeg. #FreeBSD

FreeBSD has grown up a lot in this release cycle.  The most useful tool from the 10.0/11.0 world in a long time, poudriere (powder keg in French) has made my ports usage almost trivial now.

More or less, poudriere is a tool that allows you to build ports packages compatible with the new PKGNG format without contaminating your working system.  It uses a series of jails and build environments to do what a lot of more savvy FreeBSD developers and engineers have been doing for years.

Even using portmaster to maintain my systems seems archaic in comparison, not to mention error prone.  More or less, my 3 or 4 systems have been converted to use themselves as a repository for packages and they build their own packages.  This is a bit redundant to be honest, and it makes the most sense to use one host as a repository and have your other machines pull in packages from it.  My implementation is due to running 11-current and being having machines I control on very different and restrictive networks.

poudriere setup for 11-current (head builds)

Start by install poudriere from ports or a package that you can get your hands on.  Then command poudriere to setup its basejail on FreeBSD SVN HEAD:

poudriere jail -c -j 11-amd64 -v head -a amd64 -m svn

This will create a jail on your local machine based on SVN head at the time of execution (yes, its going to compile everything from source and will take a while, get a cup of coffee, perhaps a sandwich).  The thing is, your machine is still available for other things while this is going on.  You are not going to crash X or other applications while this is happening.  Its building a separate jail for the purpose of creating packages.

Once its build, you can update your jail world trivially via:

poudriere jail -u -j 11-amd64

Now, grab the ports tree via:

poudriere ports -c

Updates to your ports tree via portsnap are easy with a :

poudriere ports -u

At this point, you are ready to configure poudriere to build your package via the “bulk” command.  I copied /usr/local/etc/poudriere.conf.sample to /usr/local/etc/poudriere.conf and made exactly one change to the default settings.  I use ZFS ( which I highly recommend, see my post on the Bacon of Filesystems ) and my ZPOOL is a different name than the default.

Creating your list of ports for your builds is a trial and error endeavor to be honest.  I suspect, there are easier ways to do it, but I determined my list below based on the list I had installed already and some questions to various mailing lists.  I created a /usr/local/etc/myports file with the following in it as a list of ports that I want built.  Poudriere will build all required dependencies for me, build-time and run-time and create nice little packages for me.

x11/xorg
x11/xdm
x11/xsm
x11-wm/xfce4
x11/xfce4-screenshooter-plugin
x11/xscreensaver
shells/bash
www/firefox
www/linux-f10-flashplugin11
www/nspluginwrapper
graphics/evince
net-im/finch
editors/vim
sysutils/tmux
comms/amtterm
ports-mgmt/dialog4ports
ports-mgmt/pkg
ports-mgmt/poudriere
java/openjdk7
editors/vim-lite
sysutils/synergy-devel
devel/git
emulators/qemu-devel

At this point, I was read to do the build run via:

poudriere bulk -f /usr/local/etc/myports -j 11-amd64

This builds all the things for me, caching packages when needed for reuse.  Very handy for me to be honest.

Setting up the pkg repo couldn’t be simpler either.  I copied /usr/local/etc/pkg.conf.sample to /usr/local/etc/pkg.conf and made a single change to point the system to use the locally build packages in a locally generated repo:

PACKAGESITE        : file:///usr/local/poudriere/data/packages/11-amd64-default

The final step was to initialize my repository via:

pkg repo /usr/local/poudriere/data/packages/11-amd64-default

I then updated my system via the newly built packages:

pkg update

pkg upgrade -f

This refreshed all the packages on my system with ones that are cleanly built by poudriere.  This allowed me to now audit what I had installed and to see what I could remove or what else I needed to have built:

pkg version -R

Anything with a “=” means that it comes from the repository and is up to date.  Anything with a “?” means it comes from an unknown source.   I learned I had a lot of dependencies installed for builds that I didn’t need for runtime cases:

pkg autoremove

Many, many, many thanks to the FreeBSD portmgr team ([email protected]), Baptiste Daroussin, Bryan Drewery and the others who have deadlifted the FreeBSD ports system into the future. Now I can look at whats left and I have never been more content with FreeBSD ports.  *boom*

*edit* reference to poudriere official docs and such:

https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/index.wiki

*edit* after pkg-1.2.1 release.

The pkg.conf config and locations have moved around and become incompatible with this blog post.  You’ll want to do two things if you are using this as a guide for updates:

1.  Disable the FreeBSD repo configuration in /etc/pkg/FreeBSD.conf

2. Move your local repo config to /etc/pkg/my_repo.conf and give it the following syntax:

me: {
url: file:///usr/local/poudriere/data/packages/11-amd64-default,
signature_type: none,
enabled: yes
}

The Ports Management Team 2013-11-01 03:43:05

The FreeBSD Ports Management team is pleased to announce a pilot project
called portmgr-lurkers@.  Over the course of the next two years, volunteers
from our group of ports committers will participate in portmgr@ activites
by being added to our mailing list.

At four month intervals, two committers at a time will be brought in to
work on various projects and learn the inner workings of the team.

The first two -lurkers will start on November 1, 2013. They are Mathieu
Arnold (mat@) and Antoine Brodin (antoine@).

The Ports Management Team 2013-10-31 12:55:52

We are pleased to announce that official binary packages are now
available for pkg, the next generation package management tool for FreeBSD.

Pkg allows you to either use ports with portmaster/portupgrade or to
have binary remote packages without ports.

We have binary packages available for i386 and amd64 on
8.3,8.4,9.1,9.2,10.0 and 11 (head).

More at http://lists.freebsd.org/pipermail/freebsd-pkg/2013-October/000107.html