Author Archives: Sean

The Short List #8: Using #lldb with a core file on #FreeBSD

Debugging qemu this evening and it took me a minute or two to figure out the syntax for debugging a core file with lldb.

lldb mips-bsd-user/qemu-mips -c /mipsbuild/qemu-mips.core

Make sure you have permissions to access both the binary and the core, else you get a super unhelpful error of:

error: Unable to find process plug-in for core file ‘/mipsbuild/qemu-mips.core’

But, after that, you can start poking around:

Core file ‘/mipsbuild/qemu-mips.core’ (x86_64) was loaded.

Process 0 stopped

* thread #1: tid = 0, 0x00000000601816fa qemu-mips`_kill + 10, name = ‘qemu-mips’, stop reason = signal SIGILL

frame #0: 0x00000000601816fa qemu-mips`_kill + 10

qemu-mips`_kill + 10:

-> 0x601816fa: jb 0x60182f5c ; .cerror

0×60181700: ret

0×60181701: nop

0×60181702: nop

(lldb) bt

* thread #1: tid = 0, 0x00000000601816fa qemu-mips`_kill + 10, name = ‘qemu-mips’, stop reason = signal SIGILL

* frame #0: 0x00000000601816fa qemu-mips`_kill + 10

frame #1: 0x000000006003753b qemu-mips`force_sig(target_sig=<unavailable>) + 283 at signal.c:352

frame #2: 0x00000000600376dc qemu-mips`queue_signal(env=<unavailable>, sig=4, info=0x00007ffffffe8878) + 380 at signal.c:395

frame #3: 0×0000000060035566 qemu-mips`cpu_loop [inlined] target_cpu_loop(env=<unavailable>) + 1266 at target_arch_cpu.h:239

frame #4: 0×0000000060035074 qemu-mips`cpu_loop(env=<unavailable>) + 20 at main.c:201

frame #5: 0x00000000600362ae qemu-mips`main(argc=1623883776, argv=0x00007fffffffd898) + 2542 at main.c:588

frame #6: 0x000000006000030f qemu-mips`_start + 367

 

Sometimes you have to sit down and write #FreeBSD documentation

When working on new projects or hacks, sometimes you just have to stop, think and start writing down your processes and discoveries. While working on bootstrapping the DLink DIR-825C1, I realized that I had accumulated a lot of new (to me) knowledge from the FreeBSD Community (namely, Adrian Chadd and Warner Losh).

There is a less than clear way of constructing images for these embedded devices that has an analogue in the Linux community under the OpenWRT project. Many of the processes are the same, but enough are different that I thought it wise to write down some of the processes into the beginning of a hacker’s guide to doing stuff and/or things in this space.

The first document I came up with was based on the idea that we can netboot these little devices without touching the on-board flash device. This is what you should use to get the machine bootstrapped and figure out where all the calibration data for the wireless adapters exist. This is crucial to not destroying your device. The wireless calibration data (ART) is unique to each device, destroying it will mean you have to RMA this device.

The second document I’ve created is a description of how to construct the flash device hints entries in the kernel hints file for FreeBSD. I found the kernel hints file to be cumbersome in comparison to the linux kernel way of using device specific C files for unique characteristics.

Its interesting stuff if you have the hankering to dig a bit deeper into systems that aren’t PC class machines.

The Short List #6: Make the CD drive do something useful on #FreeBSD

Noted today that while grip could access the CD drive on my machine, clemetine-player and xfburn could not.

Figure out which device node your CD drive is with camcontrol:

camcontrol devlist | grep cd
at scbus4 target 0 lun 0 (cd0,pass2)

Simply add the following to /etc/devfs.conf and restart devfs to get access to the CD device:

perm /dev/cd0 0666
perm /dev/xpt0 0666
perm /dev/pass2 0666

Now bear in mind, that this means any user of your machine has access to the device now. Hopefully, on a desktop computer, you know all the users of your machine.

Burning all the bridges. Cleaning up jails with ezjail-admin on #FreeBSD

I noted that my updates on my jail host didn’t actually do a delete-old/delete-old-libs during the basejail process:

ezjail-admin update -i

I tend to update my jails with my base host svn updates to -current, so there’s a bit of churn and burn with regards to old files and such. This came to a head today as my src.conf on the base host declares WITHOUT_NIS to conserve my limited space.

The python port checks for the existence of the yp binaries to determine whether or not to build NIS support. So, if the old binaries are lying around and support for NIS is removed from your system, python’s build will abort with something like the following:

Install them as needed.
====
====> Compressing man pages (compress-man)
===> Installing for python27-2.7.6_2
===> Checking if lang/python27 already installed
===> Registering installation for python27-2.7.6_2 as automatic
pkg-static: lstat(/var/ports/basejail/usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/nis.so): No such file or directory
*** Error code 74

I realized that even though my host system was fairly clean (I do port rebuilds after each upgrade and delete-old delete-old-libs following that), the basejail was still filled with obsoleted files.

A super dangerous and super effective way to clean that up is the following:
yes | make delete-old DESTDIR=/usr/jails/basejail
yes | make delete-old-libs DESTDIR=/usr/jails/basejail

Dangerous, because you have to realize that your deleting binaries and libraries that might still be in use if you haven’t recompiled your ports packages. Effective, because it will cleanup and purge a lot of things if you haven’t done it in a while.

This also led me to understand that the /etc/src.conf tuneables WITHOUT_* don’t *stop* the buildsystem from creating the binaries and libraries. It doesn’t seem to shorten your build time. It *will* allow you to purge them from your system at install time with the delete-old make targets.

httperf tuning for #FreeBSD testing

Was playing around with httperf to excercise Apache / stunnel SSl benchmarks on FreeBSD this week and ran into the code that nerfs simultaneous connections down from the environment ulimit of maxfiles to the limit FD_SETSIZE as defined in <select.h>.

One can override this at compile time and push the system harder by passing in some ./configure foo:

env CC=”cc -DFD_SETSIZE=4096″

However, you will then be able to max out the number of ports in use very quickly if you try to use stunnel and apache in this configuration.  I noted that on our systems we raise the low port number and reduce the high port number for connections:

net.inet.ip.portrange.first=20000

net.inet.ip.portrange.last=49151

I set first down to 2000 and last up to 65534 for my testing.  This gives me quite a bit more ports to use in testing.  At this point I can run stunnel on 443 forwarding to apache on localhost:80 and get more than 8k simultaneous connections when using SSL accelerators on FreeBSD 10

 

The short list #5: coredumping with sudo on #FreeBSD

Things I learned from a misbehaving pam module managing our sudo context at work.  sudo, for security, will not dump core files if it hits a segfault.  You need to tell the kernel to allow set uid root binaries to core dump *and* you have to let sudo know that its ok via a sudo.conf entry.

DO NOT LEAVE THESE AS DEFAULTS

kern.sugid_coredump: 1

/etc/sudo.conf –> Set disable_coredump true

ref –> http://www.sudo.ws/sudo.man.html

 

You need to construct additional pylons. Building wine for #FreeBSD

I’ve been playing Blizzards’s Starcraft 2 on Linux via wine emulation lately and thought I’d see if I can get the same thing working on FreeBSD via the emulators/i386-wine-devel port.  After talking with the fine folks in #bsdports on EFNet, I finally found a recipe that is poudriere friendly and seems to spit out something that sort of works.

David Naylor ([email protected]) has a working method for constructing wine on FreeBSD and this should work in most cases for using current.  The method is really designed for building a binary package for releases, most folks wouldn’t want to go down this route.

In order to begin, get poudriere configured and ready to go.  You’ll need to construct an i386 jail for the first part of this process.  Something like I show in my poudiere blog post

poudriere jail -c -j 11i386 -v head -a i386 -m svn

This will give you a build environment to get the 32bit binaries for wine built and packaged up for step 2.

poudriere builk -j 11i386 emulators/i386-wine-devel

If all goes well, you now have an i386 package of wine that will be consumed as a distfile for the amd64 package build.  I redefine PORTSDIR=/usr/local/poudriere/ports/default in /etc/make.conf.

If you are like me and use poudriere for everything, copy it to /usr/local/poudriere/ports/defaults/distfiles/freebsd:11:x86:64/

Now you’ll need to edit the emulators/i386-wine-devel distfile with the appropriate information generated from a sha256 and ls -l of your packagefile in your local i386 repo:

sha256 i386-wine-devel-1.7.7,1.txz

SHA256 (i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify this with the following to make it compatible with your system:

SHA256 (freebsd:11:x86:64/i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify the Makefile.inc to exclude checks for the OS version:

Index: Makefile.inc
===================================================================
— Makefile.inc    (revision 335346)
+++ Makefile.inc    (working copy)
@@ -41,10 +41,10 @@

.include <bsd.port.pre.mk>

-.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
-IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only
-DISTFILES=
-.endif
+#.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
+#IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only
+#DISTFILES=
+#.endif

.if ${PORT_OPTIONS:MGECKO}
RUN_DEPENDS+=   ${DATADIR}/gecko/wine_gecko-2.24-x86.msi:${PORTSDIR}/emulators/wine-gecko-devel

And now, you can try building the package in your *AMD64* poudriere build with:

poudriere bulk -j 11amd64 emulators/i386-wine-devel

If my instructions have succeeded, you now have a package suitable for installation on your amd64 machine that will now let you do wine things.

Now, I need to figure out what the Blizzard Network Installer is trying to do as it runs, self-updates and hangs.

Moving forward by going slightly backwards and to the right, UEFI booting on #FreeBSD

The FreeBSD Foundation has been working towards the future of booting in x86 and catching up to our friends in Linux-land by sponsoring work on a UEFI enabled boot loader.  This work was taken on by Benno Rice ([email protected]) and Ed Maste ([email protected]).

So far, it appears that one can indeed boot FreeBSD as I will demonstrate on my Thinkpad T520.

Starting with the UEFI project branch, one must build a 64bit version of libstand in tree.

cd uefi/lib/libstand && make

Modify the makefile in sys/boot/amd64

Index: amd64/efi/Makefile
===================================================================
— amd64/efi/Makefile    (revision 258775)
+++ amd64/efi/Makefile    (working copy)
@@ -77,8 +77,8 @@
LIBEFI=        ${.OBJDIR}/../../efi/libefi/libefi.a
CFLAGS+=    -I${.CURDIR}/../../common

-DPADD=        ${LIBFICL} ${LIBEFI} ${LIBSTAND}
-LDADD=        ${LIBFICL} ${LIBEFI} ${LIBSTAND}
+DPADD=        ${LIBFICL} ${LIBEFI} ../../../../lib/libstand/libstand.a
+LDADD=        ${LIBFICL} ${LIBEFI} ../../../../lib/libstand/libstand.a

.include <bsd.prog.mk>

Now you can build loader.efi and get it to link against the 64bit version of libstand:

cd sys/boot && make

UEFI will look for a FAT formatted partition with the “efi” signature on it.  FreeBSD’s gpart can create this partition for you, so do the following foo:

gpart create -s gpt da0

gpart add -t efi da0

gpart add -t freebsd-ufs da0

$ gpart show da0
=>     34  2013117  da0  GPT  (983M)
34   131072    1  efi  (64M)
131106  1882045    2  freebsd-ufs  (919M)

newfs -t msdosfs /dev/da0p1

newfs /dev/da0p2

Mount the fat formatted partition, create the EFI directory structure(this is mandatory) and copy the loader.efi binary into place as bootx64.efi

mount -t msdosfs /dev/da0p1 /mnt

mkdir -p /mnt/efi/boot

cp uefi/sys/boot/amd64

Because the kernel currently needs to be aware of the new style UEFI memory map, you can’t run stock -current in this configuration.  You’ll need to use a kernel from the projects/uefi branch when constructing your bootable device.  I used a 1G usb thumbdrive for this test, so mount the UFS partition and use it as a DESTDIR for your installworld/installkernel:

make -s buildworld

make -s buildkernel

mount /dev/da0p2 /mnt

DESTDIR=/mnt make -s installworld

DESTDIR=/mnt make -s installkernel

DESTDIR=/mnt make -s distribution

Setup an /etc/fstab on this stick:

/dev/da0p2             /               ufs     rw,              1       1

At this point, your USB disk is ready for its first booting attempt.

EFI_BIOS

EFI_LOADER1

I have to toggle UEFI/Legacy BIOS mode in my laptop.  For your entertainment, here it is.  This has the convient side effect of not booting from my other disk devices in my laptop as they do not have the “proper” fat formatted EFI partition on them.  This actually yeilds a pretty quick boot to the loader.

Amazing!  It did!  Sort of.

Now we have the entertainment of trying to figure out how to get here to multiuser.

 

 

 

EFI_LOADER2

With a “show” we find out that the loader has selected the EFI partition “part6″ as the boot device.  “lsdev” shows us all the partitions that we could boot from, but I have chosen well in this example and can easily see that the one I really want is tagged with a “(removable)”.

In this case executing a “set currdev=part7″ sets up the loader to boot and executing “boot” will get this system into multiuser.

Many thanks to the folks at the FreeBSD Foundation for these initial steps into UEFI.  The project branch in subversion is publicly available and I highly encourage folks to engage the community to get this closer to production grade.

The Unusual Suspects, #FreeBSD Vendor Summit 2013

I was fortunate enough this year to be able to help the FreeBSD Foundation host the 2013 Fall Vendor Summit at my workplace, Yahoo.  Our facilities in Sunnyvale are very first class and I like to help out with my non-technical resources whenever possible (because, frankly, if you’ve seen my code, you would prefer it that way).

George Neville-Neil of the FreeBSD Project and FreeBSD Foundation had asked if Yahoo could host again this year and we agreed to a one day presentation and get together at the main campus.

Lots of folks who don’t normally go for conferences showed up to this invitation only event, and for once it felt like we had a strong showing.  I had booked a conference room for 55 people and we had close to 70 show up.  It was really close to bordering on overflow into the hallway at one point.

I think my biggest takeaways this year was the fact that “FreeBSD Doesn’t Have Visualization” is now just a myth and doesn’t really match reality.  The Bhyve project has taken a good direction and now can spin up other o/s instances, like Linux, via the ACPI framework implemented during the Google Summer of Code projects.  It was also very good to see VMWare and Google Compute folks showing up and asking for “what we need to help you folks support FreeBSD in our cloud things.”

Instead of the hallway track at normal conferences, we had the “back of the room on the floor” track this year where there was much debating over the validity of git as a FreeBSD source management tool.  The thing is, the project already exports FreeBSD SVN src to a self hosted git repo (http://git.freebsd.org) and a github instance (https://github.com/freebsd).   The debate swirls around the archaic “email patches to mailing lists” mentality instead of the “send pull request” things that the git world now has.

Interesting point from this discussion, perhaps we should now take the time to assign people who are more involved to important sections of kernel and source code.  The FreeBSD ports system has direct maintainers and a system to timeout maintainers who are AFK.  The FreeBSD base system has a more liberal approach as any committer can and does commit to any aspect of the tree.  Its common practice to not do this without review, but its no a true formal review process.  This leads to some cases where patches go to mailing lists and never get picked up and reviewed.

Otherwise, a fine time was had and I certainly look forward to the next conference, AsiaBSDCon 2014.

How I learned to stop worrying and love the powderkeg. #FreeBSD

FreeBSD has grown up a lot in this release cycle.  The most useful tool from the 10.0/11.0 world in a long time, poudriere (powder keg in French) has made my ports usage almost trivial now.

More or less, poudriere is a tool that allows you to build ports packages compatible with the new PKGNG format without contaminating your working system.  It uses a series of jails and build environments to do what a lot of more savvy FreeBSD developers and engineers have been doing for years.

Even using portmaster to maintain my systems seems archaic in comparison, not to mention error prone.  More or less, my 3 or 4 systems have been converted to use themselves as a repository for packages and they build their own packages.  This is a bit redundant to be honest, and it makes the most sense to use one host as a repository and have your other machines pull in packages from it.  My implementation is due to running 11-current and being having machines I control on very different and restrictive networks.

poudriere setup for 11-current (head builds)

Start by install poudriere from ports or a package that you can get your hands on.  Then command poudriere to setup its basejail on FreeBSD SVN HEAD:

poudriere jail -c -j 11-amd64 -v head -a amd64 -m svn

This will create a jail on your local machine based on SVN head at the time of execution (yes, its going to compile everything from source and will take a while, get a cup of coffee, perhaps a sandwich).  The thing is, your machine is still available for other things while this is going on.  You are not going to crash X or other applications while this is happening.  Its building a separate jail for the purpose of creating packages.

Once its build, you can update your jail world trivially via:

poudriere jail -u -j 11-amd64

Now, grab the ports tree via:

poudriere ports -c

Updates to your ports tree via portsnap are easy with a :

poudriere ports -u

At this point, you are ready to configure poudriere to build your package via the “bulk” command.  I copied /usr/local/etc/poudriere.conf.sample to /usr/local/etc/poudriere.conf and made exactly one change to the default settings.  I use ZFS ( which I highly recommend, see my post on the Bacon of Filesystems ) and my ZPOOL is a different name than the default.

Creating your list of ports for your builds is a trial and error endeavor to be honest.  I suspect, there are easier ways to do it, but I determined my list below based on the list I had installed already and some questions to various mailing lists.  I created a /usr/local/etc/myports file with the following in it as a list of ports that I want built.  Poudriere will build all required dependencies for me, build-time and run-time and create nice little packages for me.

x11/xorg
x11/xdm
x11/xsm
x11-wm/xfce4
x11/xfce4-screenshooter-plugin
x11/xscreensaver
shells/bash
www/firefox
www/linux-f10-flashplugin11
www/nspluginwrapper
graphics/evince
net-im/finch
editors/vim
sysutils/tmux
comms/amtterm
ports-mgmt/dialog4ports
ports-mgmt/pkg
ports-mgmt/poudriere
java/openjdk7
editors/vim-lite
sysutils/synergy-devel
devel/git
emulators/qemu-devel

At this point, I was read to do the build run via:

poudriere bulk -f /usr/local/etc/myports -j 11-amd64

This builds all the things for me, caching packages when needed for reuse.  Very handy for me to be honest.

Setting up the pkg repo couldn’t be simpler either.  I copied /usr/local/etc/pkg.conf.sample to /usr/local/etc/pkg.conf and made a single change to point the system to use the locally build packages in a locally generated repo:

PACKAGESITE        : file:///usr/local/poudriere/data/packages/11-amd64-default

The final step was to initialize my repository via:

pkg repo /usr/local/poudriere/data/packages/11-amd64-default

I then updated my system via the newly built packages:

pkg update

pkg upgrade -f

This refreshed all the packages on my system with ones that are cleanly built by poudriere.  This allowed me to now audit what I had installed and to see what I could remove or what else I needed to have built:

pkg version -R

Anything with a “=” means that it comes from the repository and is up to date.  Anything with a “?” means it comes from an unknown source.   I learned I had a lot of dependencies installed for builds that I didn’t need for runtime cases:

pkg autoremove

Many, many, many thanks to the FreeBSD portmgr team ([email protected]), Baptiste Daroussin, Bryan Drewery and the others who have deadlifted the FreeBSD ports system into the future. Now I can look at whats left and I have never been more content with FreeBSD ports.  *boom*

*edit* reference to poudriere official docs and such:

https://fossil.etoilebsd.net/poudriere/doc/trunk/doc/index.wiki

*edit* after pkg-1.2.1 release.

The pkg.conf config and locations have moved around and become incompatible with this blog post.  You’ll want to do two things if you are using this as a guide for updates:

1.  Disable the FreeBSD repo configuration in /etc/pkg/FreeBSD.conf

2. Move your local repo config to /etc/pkg/my_repo.conf and give it the following syntax:

me: {
url: file:///usr/local/poudriere/data/packages/11-amd64-default,
signature_type: none,
enabled: yes
}

The Short List #7: wpa_passphrase on #FreeBSD. Because plaintext passwords are dumb.

If you have configured wireless networks on your FreeBSD laptop/pc. You can use wpa_passphrase to make the password entries more obscure with the use of wpa_passphrase.

For example, given the following network entry in wpa_supplicant.conf:

network={
ssid=”BRUNO”
scan_ssid=1
priority=5
key_mgmt=WPA-PSK
psk=”Super Secret Plain Text Password”
}

wpa_passphrase can give you a psk entry that is more obscure:

$ wpa_passphrase BRUNO
# reading passphrase from stdin
Super Secret Plain Text Password
network={
ssid=”BRUNO”
#psk=”Super Secret Plain Text Password”
psk=536ac6450b5874686e33122f80d39ad633b59613389f123d149c76fef4f3c877
}

Just remember to delete the plain text version when you copy paste it into your config.

Wait

We recently “destroyed” an Dell Poweredge R720 at the office and it was to be retired to the dust bin.  We ran the Dell updates for firmware things and the machine now is completely unbootable.  Too bad, it was a very nice machine.

But, this led me down an interesting road.  I cracked open the server and started removing components one by one.  Lo and behold I noted that there was a very interesting 4 pin connector on the motherboard labeled “DRAC UART”.  Hmmm … I thought this sounded interesting.  After scrounging up a multimeter, I verified that it was indeed a 3.3 volt connection and seemed to have a ground pin as well.

I grabbed a TTL to USB converted off of my coworkers desk and proceeded to hook it up.  I was surprised and amused to find out that the Dell DRAC 7 seems to be nothing more than a fancy OpenWRT instance running on a Hitachi SH4A 32bit microprocessor.

Cadillac ZFS #FreeBSD

I had an opportunity at work to build up a new machine to do our FreeBSD builds at work this quarter and wanted to see how far I could take ZFS on high end OEM hardware.

After evaluating HP and Dell gear, I settled on the Dell r720xd as my platform to move forward.  Primarily, this was due to the lack of *real* JBOD support on the HP line of SAS controllers.  The Dell H310 has a “SYSPD” option in mfi(4) that allows one to use the raw disks and ignore the RAID capabilites.  I went ahead and modified the FreeBSD mfiutil(4) tool to allow run time configuration into this mode.  http://svnweb.freebsd.org/base?view=revision&revision=254906

I ended up with 64G of RAM and 2x CPU: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz (2300.05-MHz K8-class CPU).  I stacked 12x3TB SAS drives (really just SATA drives with SAS firmware, but hey, they cost WAY MORE).

Setup the zpool with 2x raidz1 vdevs on this go around.  There was some debate between myself and other colleagues if I should have gone with 1 raidz2 pool.  It theoretically would have some better failure handling since I would have 2x parity disks in the same pool, but it seemed that I should go with 2x vdevs, each with 1 parity drive in this case because of how much write activity building 7 different FreeBSD distributions simultaneously would generate.

I ended up with a zpool that looks like this:

pool: zroot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Wed Aug 21 15:55:09 2013
config:

NAME              STATE     READ WRITE CKSUM
zroot             ONLINE       0     0     0
raidz1-0        ONLINE       0     0     0
mfisyspd1p3   ONLINE       0     0     0
mfisyspd2p3   ONLINE       0     0     0
mfisyspd3p3   ONLINE       0     0     0
mfisyspd4p3   ONLINE       0     0     0
mfisyspd5p3   ONLINE       0     0     0
mfisyspd0p3   ONLINE       0     0     0
raidz1-1        ONLINE       0     0     0
mfisyspd6p3   ONLINE       0     0     0
mfisyspd7p3   ONLINE       0     0     0
mfisyspd8p3   ONLINE       0     0     0
mfisyspd9p3   ONLINE       0     0     0
mfisyspd10p3  ONLINE       0     0     0
mfisyspd11p3  ONLINE       0     0     0

Sexy.

Performance wise, this machine now spits out our production images in about 95 minutes as opposed to the 255 minutes from before.  Its a complete dead lift of hardware, new cpus, disks, more ram, different F/S, etc.  I’m pretty happy with it, but of course, its Cadillac prices, so your mileage will vary.

The short list: #3 #FreeBSD

Sandybridge/Ivybridge/Haswell boxes take upwards of 10 minutes to just GET TO THE BOOT LOADER now.  When I want to get a box into single user, and not miss the loader prompt when using FreeBSD:
nextboot -o “-s” -k kernel

ZFS based systems will emit a terrifying warning of:

WARNING: loader(8) has only R/O support for ZFS
nextboot.conf will NOT be reset in case of kernel boot failure

Which means you need to pay attention if your system panics on restart.

The short list: #2

Migrating a system from its UFS installation to a new shiny ZFS pool:
cd / && tar –exclude=zroot/ –exclude=dev/ –exclude=proc/ -cvf – * | ( cd /zroot; tar xfp – )

How to fry a router in X-1 steps, The Serial Console complete

If you saw my first post  regarding my attempts to use a DLink DIR-825 with FreeBSD, you noted that I was busily trying to get the serial console working.  I won’t go over the reasons for it in this new post, but get straight into the fun and games.

My motivation here is to make the serial console available without ripping open the case every time there is an issue.  FreeBSD in a OpenWRT/DD-WRT style configuration is a very open universe right now and there will be lots of changes to come.

For now, let’s start with the ingredient list:

  1. DLink DIR-825 rev. B1
  2. Soldering Iron, solder, solder braid, good tweezers, scissors, exacto-knife
  3. Jumper wire leads, F-F
  4. OSEPP FTDI Breakout Board kit
  5. Circuit Board safe superglue

Step 1 of this operation is to open the DIR-825, remove the main board and solder on pins to JP1 on the board.

Once that is secure and you’ve validated that you can use your USB TTL adapter to get a serial console then you can move forward to setup the internal mounting and give yourself an external connector.

The case is used in 2 and 3 antenna models.  This unit leaves us the center antenna mount as a  very convient place to secure our TTL USB adapter.  Get your exacto-knife and cut out the black label area covering this hole.  Then gently cut out the top half of the innner plastic ring as I have shown in this photo.

DLink_USB_Mod_CaseSee how nicely the USB connector presents itself?  Its as though we planned this out!

Next, I modified the TTL adapter itself.  I needed to remove the jumper at the center of the adapter as it collides with the capacitor directly adjacent to the mounting point.

DIR825-USB-solderingI then was able to solder a very small length of wire to short the 3.3v selector permanently.  You’ll need a steady hand and the tweezers to pull out the pins and insert the wire.DIR825-USB-33v

You’ll note that I’ve removed the pin header that comes with the adapter.  This was accomplished with a sharp pair of scissors and then using the soldering iron to extract the remainder of the pins with judicious use of tweezers.

Once that was done, I soldered on individual pins for TX, RX and ground. Note that I’ve bent the pins at a 45 degree angle.  This will avoid any issues with the pins making contact with the mainboard.

Now that you have all the pieces ready to go, its time to break out the superglue and mount the thing.  When choosing a glue, you’ll need to get one that is safe for PCB and plasitc in general, your local electronics and hobby shop can direct you to the proper stuffs.

I mounted the TTL adapter upside down so that the connector can reach the opening on the case and there are two lower points of contact for the glue to set correctly.DIR825-USB-mount

I applied glue to the two corners next to the USB header, flipped it over and brought it into firm contact with the antenna mount.  After about 1 minute, I released pressure and applied a line of glue across the surface of the TTL adapter such that it sealed against the case of the router.

DIR825-USB-GLUE2Now … WALK AWAY for at least 6 hours.  If you’re using the right kind of glue, there’s some science happening here and you need to let the glue/plastic cure to get a secure mount.DIR825-USB-FINAL

Wire up your TX, RX and ground leads, reattach the case housing and connect up your new USB serial interface to a PC.  On FreeBSD, you should see a new serial port appear.

ugen5.2: <FTDI> at usbus5
uftdio0: <FT232R USB UART> on usbus5

Go ahead, open it with your handy, dandy comm utility, cu:

cu -l /dev/ttyU0 -s 115200

If you’ve done everything right, congrats, you’ll now see the embedded linux distribution boot up.  You are now at Step 0 getting your FreeBSD based home router running.

The short list: #1

Like the subtitle says, these should be obvious, but I’m forever looking them up.

#1:  Find files in a perforce checkout that are not checked in or are modified:
for file in `p4 diff -f -sa`; do p4 diff -f $file; done

#2 Crosscompile FreeBSDi386 on amd64, its stupid simple, but I can never remember:
TARGET=i386 TARGET_ARCH=i386 MAKEOBJDIRPREFIX=/var/tmp make -s -j16 buildworld
TARGET=i386 TARGET_ARCH=i386 MAKEOBJDIRPREFIX=/var/tmp make -s -j16 buildkernel KERNCONF=PAE

 

AllMost There

Spent some time this week screwing around with the “serial” port on my laptop.  Serial consoles are one of those things that you seem to always need when doing o/s development.  Its the only reliable way to get debugging information and on PC architectures, its really the best way to find and resolve serious problems in the kernel or drivers.

Laptops dropped their DB-9, RS-232 connection years ago and up until recently, you couldn’t really do much with debugging except take pictures of your laptop’s console when it had crashed.  I discovered that my laptop has AMT features built into it and decided to see if I could make FreeBSD use it as a serial console for debugging and diagnostics.  Turns out, its mostly functional and seems to do what I want.

Turning AMT on is a bit goofy, mainly do to some fairly strict password requirements, but lets start from the beginning.  I’m using my Lenovo T520 as an example, so your BIOS will vary based on your laptop vendor.

Hit your “enter the bios key command” which on my ThinkPad is the blue “ThinkVantage” button, but for the rest of the universe is probably F10, DEL or some other normal key.  This should bring up your bios selection menu and now enter your bios configuration menus.

Turn AMT on in BIOS

AMT_BIOS2 I left the values, more or less, at the defaults here.

AMT_BIOS1Save, reboot.  Now when you hit your bios configuration button:

AMT_Boot

Hit <ctrl-P> now to enter the AMT configuration screens.  AMT_Boot2

The first time you do this, there will only be one option, so hit “1″ to enter the AMT configuration screens.  We’ll come back to this menu list later.

AMT_MainIf this is your first time entering this machine’s AMT device, its going to force you to reset the password from “admin” to something else.  The password requirements on the AMT device are pretty strict, but this should explain it for you.

You’ll need to adjust one or two settings and activate the AMT configuration’s IPv4 settings.

AMT_TimeoutSelect SOL/IDRR/KVM.  We’re going to activate Legacy Serial Redirection here

AMT_Conf0Select SOL

AMT_Conf1Obviously, Enable is what we want here.

AMT_Conf2Select Legacy Redirection Mode, then select Enable

AMT_Conf8

Back to Main Menu and Select Network Setup

AMT_Conf3TCP/IP Settings and Setup Wired Lan IPv4 Configuration

AMT_Conf4I’m going to set DHCP mode for this test.  You can set static IP if you wish.

AMT_Conf5Yet another Enable here.

AMT_Conf6At this point, you’re ready to save/exit/reboot.  To test that the DHCP settings are working, toggle your BIOS button again (F10/DEL/ThinkVantage) and hit <ctrl-P> to get the AMT menu again.

AMT_DHCPHit “2″ to check that the AMT can indeed get an IP address.  Note that this address is the same as your laptop IP, at least it should be.  There is some magic going on here that allows the AMT device to share the MAC address and the IP address of your host.  Its interesting but also causes odd things to happen that we’ll dig into a little bit later in the post.

Now that you’ve gotten an IP address, you can boot normally into FreeBSD.  We now need to modify FreeBSD to boot up onto a non-standard (COM1,2,3,4) and get the serial console working.  This is pretty standard stuff, /etc/ttys, /boot/loader.conf, /etc/make.conf things.

Determine what the pci bus address of the UART is in your system via:
pciconf -l | grep uart
uart0@pci0:0:22:3: class=0×070002 card=0x21cf17aa chip=0x1c3d8086 rev=0×04 hdr=0×00
or
dmesg | grep uart
uart0: <Intel AMT – KT Controller> port 0×6070-0×6077 mem 0xf522c000-0xf522cfff irq 19 at device 22.3 on pci0
uart0: console (115200,n,8,1)
uart0: fast interrupt

make.conf:
BOOT_COMCONSOLE_SPEED=115200

loader.conf:
console=”comconsole”
comconsole_speed=”115200″
comconsole_pcidev=”0:22:3″ #Determine this value from the pciconf -l

device.hints:
If you have a device.hints file, comment out the hints for hw.uart settings.  Leave only the setting for the flags:
#hint.uart.0.at=”isa”
#hint.uart.0.port=”0x3F8″
hint.uart.0.flags=”0×10″
#hint.uart.0.irq=”4″
#hint.uart.1.at=”isa”
#hint.uart.1.port=”0x2F8″
#hint.uart.1.irq=”3″

boot.config:
add the -Dh for dual console support

/etc/ttys:
Enable ttyu0
ttyu0 “/usr/libexec/getty std.115200″ vt100 on secure

Issue a kill -HUP 1 to let init fire up your getty instance.  If all is well, you can now go to the computer you want to use to as a console device for this laptop.  Using FreeBSD ports, install comms/amtterm. In order to connect to your laptop and validate that its working correctly, issue the following command:
amtterm -p <password you set in the AMT> <IP address of laptop>

If you get a tty login, you’ve suceeded!amtterm: NONE -> CONNECT (connection to host)
ipv4 192.168.1.209 [192.168.1.209] 16994 open
amtterm: CONNECT -> INIT (redirection initialization)
amtterm: INIT -> AUTH (session authentication)
amtterm: AUTH -> INIT_SOL (serial-over-lan initialization)
amtterm: INIT_SOL -> RUN_SOL (serial-over-lan active)
serial-over-lan redirection ok
connected now, use ^] to escape

FreeBSD/amd64 (powernoodle) (ttyu0)

login:

 

The last steps are fairly trivial.  You need to rebuild your kernel to pickup the changes to /etc/make.conf.  If you do an installkernel and reboot, you’ll now be able to capture the boot up sequence (no boot0 support) once loader starts.  You’ll get to see the beastie menu in all its glory.

The machine I’m using uses a Intel Ethernet chipset via the em(4) driver, this has some very strange behavior that I will take a look at over the next couple of weeks.

With AMT enabled, the ethernet connection only negotiates to 100Mbit.  This is very amusing to me as it acts a lot like IPMI in this regard.  When em(4) initializes it *will* drop your amtterm connection.  Irritating, but em(4) hasn’t been taught enough logic to deal with the connection to the AMT device.

Rebooting the laptop will disconnect your AMT session.  AMT is not a server grade thing like IPMI, so there’s no garauntee of reliability at all.  AMT is sniffing packets that are addressed to the laptop and snipping them out of the air.  Its a very delicate hardware hack but it does to the trick.

Since AMT presents itself as a non-standard COM port, I’m pretty sure that the boot0 and boot2 loaders can’t use AMT as a serial console.  This means that things like gptzfsboot might have issues that you cannot debug.  As far as I can tell however, this is the one of the most likely to work mechanisms for administration of your laptop and development for things like suspend/resume support.

Vertically Integrating Synergy

Because I couldn’t find any help from the Internet-O-Tron today, let me put this little bit of quality suffering out there for your entertainment.

I just completed murdering my linux box at the office and converted it over to FreeBSD Current with all the lovely Xorg goodness that I need it for.  Really, I only use it to run finch in a screen session and ssh into it from outside, but it does have its value in that.

The last … LAST little bit of my desktop at the office is the synergy tool from ports.  Its a tiny little bit of computer software glue that makes using a laptop as my primary work machine much more useful.  When I dock my T520 (also running FreeBSD Current), the synergy application connects over to my work desktop and I can control/login to it with my mouse and keyboard from my laptop.  Super Effective!

One tiny nit today that really burned out my brain cells.  I had changed the hostname of the box without restarting X.  This caused all kinds of grief that I had never seen before and was super confusing.  I would have been completely lost except for the wonder that is “telnet <hostname> <port>” that I learned way back in the mists of time to troubleshoot firewall and networking issues.

Try it for yourself on a *nix box.  While logged into Gnome, XFCE, KDE or whatever, change the hostname to something other than the one that was set when X started.  <takei>OH MY</takei>

synergy cannot open secondary screen

Breaking down and running the synergy server with -f so that I could see what’s going on, I noted that there appeared to never be a live connection to the host from my client.  Icky.  I assumed initially that there was a firewalling/network problem and started looking at hosts.allow and running truss on synergyc to see what was going on.  I noted, after a long time of grinding through truss output, that synergyc was never even opening a socket to begin with, I fired off a quick “telnet <server> 24800″

When that *DID* connect to the synergys server and I saw the logs it generated, I’ll be honest, I was super confused for like 15 minutes.  The only clue that drug me out of the confusion was the check for .Xauthority stuffs in the truss output.  Geezus, if that hadn’t have been there, I’d have never remembered that I had changed my hostname after I logged in.

The simple answer, not unlike the three finger salute in windows, was to log out of X and back in.  And just like Disco in the 90s, synergy was back.

To whomever in my distant past taught me the trick to telnet as a trouble shooting aid, THANK GOD FOR YOUR EXISTENCE ON THIS PLANET.