Author Archives: brd

Upgrading Graphite

Recently swills@ upgraded Graphite and reconfigured how it works to fit more in to the FreeBSD file system layout.

So if you are upgrading from a graphite installation older than 0.9.12_1, you will need to follow the following instructions:

  1. Stop carbon
  2. Copy the old data from /usr/local/storage/whisper/* to /var/db/carbon/whisper/
  3. Copy the /usr/local/etc/carbon/carbon.conf.example over to carbon.conf
  4. Set the SECRET_KEY to something random in /usr/local/etc/graphite/local_settings.py
  5. Then follow the instructions after the install, including updating the httpd.conf per the message after the install
  6. Restart Carbon and Apache

Be careful that you do not miss any of the steps and you should have a working Graphite install.

Puppet + pkgng/poudriere

First thing we will need a clone of https://github.com/xaque208/puppet-pkgng into /usr/local/etc/puppet/modules/.

This will be pushed out to the clients as long as: pluginsync = true

For me the next step is to create a manifests/init.pp in the new module directory. This is important to me because I want to sync out a /usr/local/etc/pkg.conf to all my machines so that they point to my internal poudriere repos. So I end up with something like this:

file { "/usr/local/etc/pkg.conf":
        mode => 755,
        owner => root,
        content => "packagesite: http://pkg/91-web/
",
}

Once that is done it is easy to use pkgng packages via:

package { "www/apache22":
        ensure => installed,
        provider => pkgng,
        require => File['/usr/local/etc/pkg.conf'],
}

BSDCan 2013 Talk: FreeBSD Birth to Death: Managing the Lifecycle of a FreeBSD Server

This is a bunch of links to the tools I talk about in my presenation

Tools:

Collectd: https://collectd.org/

Graphite: http://graphite.wikidot.com/
Nagios: http://www.nagios.org/

Poudriere: http://fossil.etoilebsd.net/poudriere

Config Management:
Salt Stack: http://saltstack.com/
Chef: http://www.opscode.com/chef/
Puppet: http://puppetlabs.com/

Subversion: http://subversion.apache.org/

LogStash: http://logstash.net/
Audit: http://www.freebsd.org/handbook/audit.html

CARP: http://www.freebsd.org/handbook/carp.html

OATH: http://www.openauthentication.org/

Serial Console: http://www.freebsd.org/handbook/serialconsole-setup.html

Generic Resources:
FreeBSD Handbook: http://freebsd.org/handbook
Everything Sysadmin Blog: http://everythingsysadmin.com/resources.html

The Importance of Serial Console

I have long been a huge fan of having serial console on my servers–it can really save the day when a mistake is made. Yesterday, one of my coworkers botched the sshd_config in an upgrade of a server, so the server came up fine, but without sshd. As a result, the system was not accessible for remote login via the network.

Over the years, I have done serial console in many ways. I began with a single null modem cable between the back of two servers. Next, I utilized a RocketPort multi-port serial card with 8 serial ports on it. These days, I have moved on to employing big serial console servers such as those made by OpenGear, providing up to 48 ports. They also have ancillary features such as providing a Nagios platform and Environmental monitoring.

No matter your physical connectivity, I recommend using Conserver. This helps by logging what is happening on the console, which can be very handy if you need to see what happened in the past whether it be a function of the system, or to see who did what. It also provides multi-user access, so you can watch while someone else is working and both of you can collaborate on fixing a problem.

In order for the previous technologies to be useful, the servers require configuration as well. The first step is to configure the BIOS for serial console redirection. Once this has been performed, the OS will need to be configured to present a console login via the serial port. The FreeBSD Handbook explains how to do this Here.

PXE Booting FreeBSD 9

I have thrown together a quick guide to get FreeBSD 9 to PXE Boot:

http://freebsd.so14k.com/freebsd9_pxe.shtml

In FreeBSD 9, a few things have changed. If you have an old PXE environment from FreeBSD 8, you will want to make note of the following:

  • No more mfsroot.
  • Which means, no more changes to /boot/loader.conf, it should be empty infact.
  • You need the new pxeboot binary from 9, do not try using an old one.

Pushing the Puppet patch for FreeBSD password management upstream

I attended LISA in Boston last week and was able to talk to a few of the Puppet developers. This reminded me I needed to push this patch upstream.

I opened a ticket in the Puppet Bug tracker, 11318. Then I found out that someone by the nick of tdb had already incorporated our changes into another pull request that adds more functionality and some unit tests. So hopefully this will be committed soon and we can have this support upstream.

I just wanted to thank tdb for taking this work and running with it!

brd’s notes

My old anoncvs/cvsup server (anoncvs1.freebsd.org/cvsup14.freebsd.org) finally died and I am working on building up a new one.

I have setup the hardware and I am prepared to ship the server out. Just need to confirm the new IP info and ship it out. Hopefully I will get this done this week prior to heading out to LISA.

brd’s notes

I am attending the FreeBSD Developer Summit for the next two days proceeding BSDCan. It is good to see everyone again and wonderful to sit down and talk with them face to face. Simon and I will be getting together and working on some clusteradm@ topics. I am currently in the Documentation Working Group meeting and we have covered many different subjects, but one of interest to me is.. We are talking about converting from SGML to XML for the Handbook and Articles. There are many benefits, such as making digital publishing easier.

brd’s notes

Describing what I really wanted to happen a coworker of mine, Andrew Hust, was able to help me write up the ruby to get it done. So without further delay:

http://freebsd.so14k.com/puppet/pw_managespasswords.diff

I will be sending this to the FreeBSD puppet port maintainer and submitting it as a patch to the port soon. I wanted to get it out there so we could get some feedback.

Update: See the new patch I posted in the comments.

Changing password hashes on FreeBSD with Puppet

I finally hacked together a Puppet recipe to update password hashes on FreeBSD!

Let me first say that I want to get native support for this to work like it should. According to this page: https://projects.puppetlabs.com/projects/puppet/wiki/Puppet_Free_Bsd. FreeBSD is missing something in the shadow libraries. I have added a note to that comment requesting someone add more info, so that we can tackle that problem.

On to the solution..

I have a custom function called `setupuser’ to manage users/groups/homedirs. I inherited this function from a coworker, so I am not sure if it is the best or right way. It did happen to make tackling the problem easy. Here is the custom function:

define setupuser($realname, $username, $password, $uid, $gid, $groups = false, $shell, $homedir) {

        group { "$username":
                ensure => present,
                gid => $gid
        }

        user { "$username":
                require => Group[$username],
                ensure => present,
                password => $password,
                uid => $uid,
                gid => $gid,
                comment => "$realname",
                groups => $groups,
                home => "$homedir",
                shell => "$shell"
        }

        file { "$homedir":
                require => User[$username],
                ensure => directory,
                owner => $username,
                group => $username,
                mode => 0700
        }

        case $operatingsystem {
                freebsd: {
                        exec { "$username hash":
                                command => "echo '$password' | pw user mod $username -H 0",
                                unless => "grep -q '$username:$password:' /etc/master.passwd",
                                path => "/bin:/usr/sbin:/usr/bin",
                                require => User[$username],
                        }
                }
        }

}

For reference here is what a call to that function looks like:
(the hash has been modified of course)..

setupuser { "brd":
        realname => "Brad Davis",
        username => brd,
        password => '$1$fffffffffffffffffffffffffffffffffffff,
        uid => 2012,
        gid => 2012,
        groups => $admingroup,
        shell => $defaultshell,
        homedir => "/home/brd"
}

Tricky tricky Firewalls

A few weeks ago I installed a the first firewall in the brand new “yet to be named” datacenter that we are building out for The FreeBSD Project. I wrote a quick PF ruleset to get things going and make sure we could ssh back in before I left the datacenter. I started it and immediately lost my ssh from the outside and could not get back in. I flushed the rules and decided it would have to wait for later because I had a plane to catch.

At BSDCan I finally got some time to look at the rules and figure out what was going wrong. A handy thing to do is setup a cronjob to automatically flush the rules every 5 minutes while you work on them to prevent locking yourself out. After setting up the cronjob I noticed that I could get in just fine if I explicitly added the IP to the rule I was using instead of using the following:

pass in on $ext_if proto tcp from any to ($ext_if) port ssh keep state

So I started looking at the interfaces and I noticed I had accidentally put the IPs on the wrong interfaces. I had the CARP IP on the main interface and the main IP on the CARP interface. Once I noticed this I was able to move the IPs to where they should be and everything worked like it should.