Update on ioquake/ARM for FreeBSD: controls support added

I’ve update ioquake binaries and pushed respective changes to github. New version has support for mouse and keyboard so you can actually play Quake3 on FreeBSD/Pi alas without sound. It should run out of the box on normal console. Make sure you have moused running or specify mouse device by setting Q_MOUSE_DEV environment variable, e.g.:

env Q_MOUSE_DEV=/dev/ums0 ioquake3.arm +set s_initsound 0

Using TrueOS as a IPFW based home router

Setting up a FreeBSD / TrueOS home router with IPFW

Over the Christmas holidays I had some spare time and was ready to take the plunge and retire an old Asus router. It had begun getting rather slow, due to the increasing number of devices connected to our network, and of course I wanted peace of mind using a FreeBSD system I could be sure was up to date with security fixes. I used PC-BSD’s server release, TrueOS 10.1, because I wanted to use ZFS with boot-environments to ensure upgrading and replacing disks would be risk-free down the road. The following details how I setup TrueOS on the new box.

 

* Disclaimer *

This is by no means a comprehensive IPFW or NATd tutorial, but hopefully it can serve as a good reference for other users trying to do similar things.

 

The Hardware:

To replace it, I had an old AMD 64 x2 Dual-Core system which hadn’t been used in a while.

CPU: AMD Athlon™ 64 X2 Dual Core Processor 5200+ (2719.21-MHz K8-class CPU)
real memory = 4966055936 (4736 MB)
avail memory = 3974365184 (3790 MB)
80GB IDE disk drives (2x)
Intel PWLA83911GTBLK PRO/1000 GT network adapter (2x) — (Amazon Link)

This box is easily overkill, with more than enough horsepower to handle our internet traffic, which is currently 100Mb/s down, 10MB/s up, however it was something I had on hand which means I was only out the cost of a couple network cards. Also, I expect it will be able to keep up with future speed increases, since 200Mb/s down and 20MB/s up is supposed to be coming to our area in the near future.

The Installation:

To get started, I grabbed a copy of the CD-sized installer for TrueOS from PC-BSD’s website:
http://www.pcbsd.org/en/download.html

The installation was very straight forward. I picked the defaults to use ZFS on root, with GRUB so that boot-environments work properly. The only major change was to go to advanced disk setup and attach the second 80GB disk drive as a mirror of the primary (ada0). Since these disks aren’t super new I wanted the ability to easily replace them when they begin to fail down the road.

Post Install Configuration:

With the installation finished, the first thing I did was to edit /etc/rc.conf with the following lines:

# Disable some PC-BSD / TrueOS services we don’t need to run on a router
syscache_enable=“NO“
php_fpm_enable=“NO“
appcafe_enable=“NO”

This disabled the AppCafe / Warden web-interface, since I don’t plan on running any applications or jails on this box, only using it purely as a router. Of course if you plan on running jails or doing other package work, the web-interface could be useful and you can skip these lines.

Next I started the configuration of the network interfaces:

# Network configuration
hostname=“interdictor“
ifconfig_em0=“inet 192.168.1.1 netmask 255.255.255.0″ # LAN IP / Bottom NIC
ifconfig_em1=“inet XXX.XXX.XXX.XXX netmask XXX.XXX.XXX.XXX” # WAN IP / Top NIC
defaultrouter=“XXX.XXX.XXX.XXX”

# Enable gateway mode
gateway_enable=“YES”

A couple of comments help me keep straight which ports on the back of the system are the WAN / LAN. Of course I replaced the XXX lines with the static IP / gateway from my ISP.

Next, I was going to be using this router to assign local IP’s with DHCP, so I first installed the DHCP server from PC-BSD’s package repo:

# pkg install net/isc-dhcp43-server

Next, I configured my DHCP information in /usr/local/etc/dhcpd.conf

# option definitions common to all supported networks…
option domain-name “XXXXXXXXX”;
option domain-name-servers 8.8.8.8, 8.8.4.4;
default-lease-time 86400;
max-lease-time 90000;

# Use this to enble / disable dynamic dns updates globally.
#ddns-update-style none;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

# Home Network
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.254;
option domain-name-servers 8.8.8.8, 8.8.4.4;
option broadcast-address 192.168.1.255;
option routers 192.168.1.1;
default-lease-time 86400;
max-lease-time 90000;
}

And finally enable the service in /etc/rc.conf

# DHCPD config
dhcpd_enable=“YES“
dhdpd_ifaces=“em0”

With DHCPD now enabled, I next went and created /etc/natd.conf with the following:

interface em1
use_sockets yes
same_ports yes
dynamic yes
log_denied yes
log yes

Optionally if you run services behind NAT and need to forward ports, you can do so with syntax like the following:
# Sample redirect port
# redirect_port tcp 192.168.1.20:80 80

Where 192.168.1.20 is the local IP of your system hosting a service, and “:80 80” being the local server port, and port on your router to open.

For our last config, we now need to setup our firewall rules, which we will do by creating the file /etc/ipfw.rules with a setup something like this:

#!/bin/sh
# Flush out the list before we begin.
ipfw –q –f flush

# Set rules command prefix
cmd=“ipfw –q add“
pif=“em1” # interface name of NIC attached to Internet
iif=“em0” # interface name of NIC attached to LAN
server=“192.168.1.1″
inside=“192.168.1.0/24″
me=“XXX.XXX.XXX.XXX” # Public WAN address
ks=“keep-state” # Laziness
skip=“skipto 600″

# Allow everything through the local NIC
######################################
$cmd 00020 allow all from any to any via em0

######################################
# No restrictions on Loopback Interface
######################################
$cmd 00025 allow all from any to any via lo0

# NAT the inbound stuff
######################################
$cmd 00100 divert natd ip from any to any in via $pif # NAT any inbound packets

######################################
# Allow packet through if it matches existing entry in dynamic rules

######################################
$cmd 00105 check-state

######################################
# Allow all outgoing packets
######################################
$cmd 00110 $skip all from any to any out via $pif $ks

# Deny all inbound traffic from non-routable reserved address spaces
######################################
$cmd 00300 deny all from 192.168.0.0/16 to any in via $pif #RFC 1918 private IP
$cmd 00301 deny all from 172.16.0.0/12 to any in via $pif #RFC 1918 private IP
$cmd 00302 deny all from 10.0.0.0/8 to any in via $pif #RFC 1918 private IP
$cmd 00303 deny all from 127.0.0.0/8 to any in via $pif #loopback
$cmd 00304 deny all from 0.0.0.0/8 to any in via $pif #loopback
$cmd 00305 deny all from 169.254.0.0/16 to any in via $pif #DHCP auto-config
$cmd 00306 deny all from 192.0.2.0/24 to any in via $pif #reserved for doc’s
$cmd 00307 deny all from 204.152.64.0/23 to any in via $pif #Sun cluster interconnect
$cmd 00308 deny all from 224.0.0.0/3 to any in via $pif #Class D & E multicast
######################################

# Deny public pings
######################################
$cmd 00310 deny icmp from any to any in via $pif

# Log all the other troublemakers
$cmd 00550 deny log all from any to any

# Skip location for NATD
$cmd 600 divert natd ip from any to any out via $pif # skipto location for outbound stateful rules
$cmd 610 allow ip from any to any

 

Now that NATD / IPFW are setup, we can set them to run at boot with the following /etc/rc.conf entries:

# IPFW
firewall_enable=“YES“
firewall_logging=“YES“
firewall_script=”/etc/ipfw.rules”

# NATD
natd_enable=“YES“
natd_interface=“em1“
natd_flags=”-m –f /etc/natd.conf”

If you don’t want to reboot the server you can start the services with the following commands:

# service natd start
# service isc-dhcpd start
# service ipfw start

Also, the ipfw.rules file is just a shell-script, so you can always re-apply the firewall rules with “sh /etc/ipfw.rules”.

That’s it! At this point you should be able to replace your consumer router with the new system, and it’ll begin handling NAT traffic, assigning DHCP addresses, etc. Even during my network’s peak usage, I’ve yet to see the system drop below 95% idle.

 

 

Bonus content!

Some users may want to do NAT reflection for public facing services. This basically means you can have clients, such as your phone, connect to the same public address, regardless of being on the local LAN, or roaming outside the home. OpenBSD has a great explanation of this, along with the methods of solving it. I was able to easily do this with IPFW and TCP proxying. To get started, install tcpproxy:

# pkg install tcpproxy

Next, we have to create our /usr/local/etc/tcpproxy.conf file:

listen * 50080
{
  remote: 192.168.1.20 80;
  remote-resolv: ipv4;
};

In the example above, we are going to listen on port 50080 on the local router machine, and proxy this to the remote machine on the LAN: 192.168.1.20 on port 80. Next we will need to add a firewall rule to /etc/ipfw.rules causing it to redirect traffic coming from the LAN NIC to this service:

# Setup NAT reflection
$cmd 00002 fwd 127.0.0.1,50080 tcp from 192.168.1.0/24 to $me 80 in via em0

With those entries complete, you can now start the tcpproxy service, and restart IPFW to enable them.

 

On profiling HTTP, or "god damnit people, why are all the open source tools slow?"

Something that's been a challenge at work (and at other things in the past) has been "how do I generate enough traffic to test this thing?"

If you're running some public facing boxes then sure, you can do A/B testing. But what if you're not able to test it in the real world? What if you need to do testing before you ship, and the traffic levels have to be stupid high?

So, what do you do?

I've done this a few times. When doing squid and other reverse proxy development, I would run tools like apachebench, httperf, even web polygraph - but these things scaled poorly. They didn't handle tens of thousands of concurrent connections and scale to both slow and fast clients - their use of poll() and select() just wouldn't work out well.

Something I did at Netflix was to start building TCP testing tools that more than 65,000 concurrent sockets. My aim is much higher, but one has to start somewhere. I was testing out the network stack rather than specifically doing HTTP testing. Here at my current job, I'm much more interested in real HTTP and all processing.

I looked at what's out there, and it's not very pretty. I need to be able to do 10G of traffic, looking upwards towards 20G and 40G of HTTP in the future. After a little more digging into what was out there - and finding httperf actually reverted my changes to use libevent and went back to poll/select! - I decided it was about time I just started writing something minimal to stress test things and build upon it as the need arose. I want something that eventually ends up like web polygraph - multiple client/server sets with different URL choices from a pool, a variety of client IP addresses, and other things like how often to make the requests and other request pacing.

So, I grabbed libevent, libevhtp from Mark Ellzey and threw them together. It turned out okish - libevent/libevhtp still does a bunch of memcpy()'ing inside the buffer management routines that makes 40G on one box infeasible at the moment, but it's good enough to get a few gigabit of client traffic on one core. There were some hiccups which I'll cover below, but it's good enough to build upon.

What did I learn?
  • Well, it turns out the client code in libevhtp was a bit immature. Mark and I talked a bit about it on IRC and then I found there was an outstanding pull request that found and fixed a bunch of these. So, my code has turned into another thing - a libevhtp client and server test suite.
  • The libevhtp threading model is fine for a couple of CPUs, but it's the standardish *NIX model of "one thread does accept, farms work off to other threads." So it's not going to scale well at high request rates to multiple CPUs. That's cool; that's what the FreeBSD-HEAD RSS work is for.
  •  There's memcpy()'ing in the libevhtp body handling code. It's not a big deal at 1G, but at 10G it's definitely noticeable. I've spoken to Mark about it.
But, it's a good starting point. Once the rest of the bugs get shaken out, it'll be a good high throughput HTTP traffic tester.

What would I do next, after the bugs?

  • the server will eventually grow the ability to generate controllable sized responses. That way the client can control how big a response to send and thus can create a mix of requests/replies.
  • .. and HTTP request body testing would be nice.
  • The client side needs to grow the ability to create client pools, like web polygraph, where certain subsets of clients get certain behaviours (like a pool of IPs to use, separate pool of URLs to fetch from, the time between each HTTP request, etc.)
The other trick is how to simulate lots (and I do mean lots) of IP addresses. I don't want to create separate loopback connections for each - that would be crazy. Instead, it'd be good to use the transparent interception support in FreeBSD IPFW that allows both connections from and connections to arbitrary IP addresses. A little trickery with IP routing so we don't need more than 1 ARP entry for each server and voila!

Oh, and the code?

https://github.com/erikarn/libevhtp-http/

Audio on Raspberry Pi

With stable VCHIQ driver next obvious target was to add VCHIQ-based audio support. So let me introduce to you: vchiq_audio, first take. It’s part of vchiq-freebsd repo so if you use Crochet to build SD card image just enable option VideoCore in config file and module will be automatically included.

From shell run kldload vchiq_audio and you’re good to do. I believe that audio output is picked up automatically by VideoCore so if you have HDMI connected it’s probably going to be HDMI. I do not have device to confirm this. Adding knob to control audio output (auto, headphones, HDMI) is on my ToDo list.

Quality is not ideal though. From quick tests it seems to work fine on system with rootfs on NFS but there are audio drops on SD-based system while playing mp3 over NFS. I’m going to debug and stresstest it more thoroughly next week.

Short instruction on how to install mpg321 package on RPi:

env PACKAGESITE=http://chips.ysv.freebsd.org/packages/11armv6-11armv6/ SIGNATURE_TYPE=none pkg bootstrap

mkdir -p /usr/local/etc/pkg/repos
cd /usr/local/etc/pkg/repos
echo 'FreeBSD: { enabled: no }' > FreeBSD.conf

cat > chips.ysv.conf <<__EOF__
chips.ysv: {
  url: "http://chips.ysv.freebsd.org/packages/freebsd:11:armv6:32:el:eabi:softfp",
  mirror_type: "http",
  signature_type: "none",
  enabled: yes
}
__EOF__

pkg install mpg321

RaspberryPi, FreeBSD and ioquake3

After New Year I got back to hacking the VCHIQ stuff (thanks to adrian@ for prodding). Since last time I touched NetBSD folks got it merged to main tree, syncing with latest upstream code and fixing some stupid bugs in my codebase. So I partially merged things back, spent some time on fixing more bugs introduced by yours truly, merged userland bits from latest Broadcom’s bits (and fixing some bugs introduced by them). And as a result VCHIQ got stable enough to run ioquake3d on raspberry pi. Well, you can’t play it because there is no sound and no mouse support and keyboard support is severely crippled but you can navigate menus and watch demoes.

Here is short summary of how to get it running:

  • Get latest HEAD that includes r276794
  • Get latest crochet-freebsd
  • Create configuration file for RasspberryPi, make sure that it’s configured for 2Gb SD card and has VideoCore enabled. i.e. it contains:
    option ImageSize 1950mb # for 2 Gigabyte card
    option VideoCore
    
  • Build RPi image and flash to SD card
  • mount FreeBSD partition, e.g. mount /dev/mmcsd0s2a /mnt
  • Copy Quake3 PAK files to /baseq3 directory on SD card
  • Download http://people.freebsd.org/~gonzo/arm/ioquake3.arm.tar.gz and copy *.so files to /baseq3 and ioqake3.arm to /usr/bin on SD card
  • Unmount FreeBSD partition and mount boot partiotion, e.g. mount_msdosfs /dev/mmcsd0s1 /mnt
  • Edit config.txt and change gpu_mem value to 64
  • Unmount SD card and boot it on your Pi
  • Load vchiq module: kldload vchiq
  • Start Quake3: ioqake3.arm +set s_initsound 0

Keyboard support is really broken. TAB and ENTER works, so you can navigate menus. But that’s pretty much it.

ioquake3 codebase with my minor changes located here: https://github.com/gonzoua/quake3
I provide pre-compiled binaries because for some reason ioquake3 built with xdev tools crash in qsort (libc incompatibilities?) so I use make buildenv to build it.

And here is photo of demo in action (there are RaspberryPi and ZedBoard on it too, yay!)
Photo Jan 07

AsiaBSDCon 2014 Videos Posted (6 years of BSDConferences on YouTube)

Sato-san has once created a playlist of videos from AsiaBSDCon. There were 20 videos from the conference held March 15-16, 2014 and papers can be found here. Congrats to the organizers for running another successful conference in Tokyo. A full list of videos is included below. Six years ago when I first created this channel videos longer than 10 minutes couldn't normally be uploaded to YouTube and we had to create a special partner channel for the content. It is great to see how the availability of technical video content about FreeBSD has grown in the last six years.

[youtube http://www.youtube.com/watch?v=videoseries?list=PLnTFqpZk5ebAUvwU75KRZ7zwU5YUaPudL&w=560&h=315]

New Update GUI for PC-BSD / Automatic Updates

The PC-BSD team is happy to announce we’ve put the finishing touches on the new Update GUI. Users on edge will be able to download and test out the new update GUI with their next update. The new Update GUI will also enable automatic updates which will happen at boot up or every 24 hours. You will also be able to choose what parts of your system you want to update (i.e. Packages, Security, etc.). If you have any questions, concerns, or to report a bug, please feel free to create a bug report on bugs.pcbsd.org so we can quickly get back to you and help troubleshoot the issue.

snapshot1

Lumina Desktop 0.8.0 Released

The first Qt5-based version of the Lumina desktop environment has just been released!

The source code for this release is available in the Lumina repository on GitHub, the FreeBSD port has been updated, and the new package is getting pushed out to the PC-BSD “Edge” repository here in the next couple days. Version 0.8.0 is mainly just an update to all the backend systems, but there are a few notable changes:

Updates for version 0.8.0:

- Uses Qt5 (tested with 5.2) instead of Qt4. This also brings in a new dependency on the XCB libraries.

- Uses the new QMultimedia framework instead of Phonon for playing multimedia files. This has much better results when playing audio/video files through the Insight file manager.

- New built-in single application framework for the Lumina project (no more external dependency for this, and it works on all OS’s).

- New task manager mode (no groups). This reproduces the traditional task manager functionality of one button per window, and it lists the window title for each button. For ease-of-use, this mode is treated as a distinct/new panel plugin through the Lumina configuration utility.

- The task manager “action” menu (right-click) now has many more options that are auto-generated based on the current window state.

- The lumina-open crash reporter for applications now provides details about the crash from the application (if any are available).

 

Bugs Fixed:

- New windows will no longer appear with the titlebar underneath a Lumina panel.

- Command-line inputs with special localized characters are  now properly used.

- Large clean-up of recursive file operations in the file manager. (Better reporting and error detection, add ability to copy a directory into itself, make sure no files are missed during recursive operations).

- Detect when running in a VirtualBox VM (if OS is FreeBSD) and hide the screen brightness controls since these do not work in a VM.

- Clean up how the special “Exec=” field codes are handled for XDG desktop shortcuts (fixes the issues with starting various KDE applications like Okular).

- Various other small bugs fixed.

 

As always, please test it out and submit any bugs or feature requests through the PC-BSD bug tracker.

FreeBSD + Packer = Vagrant

So I recently discovered a tool to build Vagrant images called Packer. It allows you to script the install via key presses over VNC to automate the install of any OS. I am running on a rather fast machine (Core i7, 16GB of RAM, SSD), so I suspect there might be some lurking problems for people on slower machines due to timing of the commands.

Everything is available from my Github repo: https://github.com/so14k/packer-freebsd

To get started:

  • Install Vagrant and Packer
  • Clone the repo onto your machine
  • Build the Vagrant box: packer build template.json
  • Wait while it builds..
  • Start the Vagrant box: vagrant up
  • Start hacking: vagrant ssh

Give it a spin and let me know what you think!

The Short List #8: fetchmailrc/gmail/ssl … grrr #FreeBSD

Didn’t realize that a fetchmail implementation I was using was actually *not* using SSL for a month.  I had installed security/ca_root_nss but FreeBSD doesn’t assume that you want to use the certificates in this package.  I don’t understand it, but whatever.

So, add this to your fetchmailrc to actually use the certificate authorities in there and really do SSL to your gmail account:

sslcertfile /usr/local/share/certs/ca-root-nss.crt

Upgrade to PC-BSD 10.1 is Now Live!

Hey everyone!

Kris has made the update to 10.1 live on the servers. To upgrade to 10.1 you can simply open the update GUI and start the update from there. You will notice the update takes a little longer to complete, but the good news is it runs in the background and there are no unexpected resets :).

If you are on the EDGE repo you most likely have the newest broken version of pkg which will need to be fixed before upgrading. To fix pkg:

% pkg install –f pkg

After that you should be in business. Please send us your feedback and / or any questions!

Solutions Linux’15

Solutions Linux'15 (http://www.solutionslinux.fr/?lg=en), CNIT, Paris - La Défense, France 20 - 21 May, 2015. Solutions Linux is one of the major corporate events in France about Linux and Open Source solutions. The event will also host an open source community 'village'. A group of BSD enthusiasts lead by BSD France will be there to promote BSD solutions.