On June 17, 2013, Stanford University is offering a free online course on coursera:
https://www.coursera.org/course/startup
===============================================================================================
About the Course
On June 17, 2013, Stanford University is offering a free online course on coursera:
https://www.coursera.org/course/startup
===============================================================================================
These days, a lot of open source projects and even startup companies are using git and Github to host their code and do rapid development. Although the FreeBSD project’s SCM is Subversion,
it is still possible to use Github to work with FreeBSD thanks to the work of FreeBSD developers who have set up scripts which mirror the FreeBSD code in Github (see: https://wiki.freebsd.org/Git ).
Using the Github repository, it should be possible for a GSoC student to:
It would be fun to try this all out!!
The list of approved GSoC projects for FreeBSD should be available in the next few days.
While we are waiting for the list of approved projects to be announced, I am helping Neeraj Rawat set up a VMÂ so that he can do FreeBSD development for Google Summer of Code (GSoC).
I am interested in learning what the “Out of the Box Experience” is like for students and newcomers to FreeBSD. The technologies that students are exposed to today, and the development environments that they are comfortable in on Linux/Unix systems is far different than when I started using Unix systems in 1992 when I was a chemical engineering student at the University of Toronto.
Neeraj likes the Geany IDE. This a simple graphical IDE is written using the GTK2 toolkit. Luckily, there is a FreeBSD port of Geany.
Neeraj’s home computer is a Windows 7 box, and he is using VMWare Player to run FreeBSD in a virtual machine.
Since Neeraj wants to work on file system stuff with FUSE, I am asking him to use FreeBSD-10-CURRENT, instead of FreeBSD 9.1 release.
I told Neeraj to do the following:
http://www.freebsd.org/doc/en/books/handbook/x-config.html
http://www.freebsd.org/doc/en/books/handbook/x11-wm.html
Neeraj is not finished yet. One of the steps where we got stuck was in step (3). Neeraj had typed:
packagesite:http://mirror.exonetric.net/pub/pkgng/freebsd:10:x86:64/latest/
and not
packagesite:Â http://mirror.exonetric.net/pub/pkgng/freebsd:10:x86:64/latest/
Apparently for the “pkg” utility, in the pkg.conf file, the “packagesite:” must have a space after the colon and before the URL, otherwise it will fail. When the FreeBSD ports team gets all the infrastructure in place to build FreeBSD pkgng packages, we will be able to delete this line and not use a 3rd party provider of FreeBSD packages for 10-CURRENT.
I will provide more details as we get set up. This is all very interesting!
Every year Google funds a project called Google Summer of Code . Google works with different open source projects, and actually pays students to hack on open source projects during their summer break. If a student is interested in participating in Google Summer of Code (GSoC), they must write a proposal for a project that they want to work on for a specific open source project, and submit it. These proposals are evaluated by the open source project and Google. If approved, the student can work on their project, but must work with a mentor in the open source project.
FreeBSD has participated in GSoC for many years.
A few months ago, Neeraj Rawat, who is a computer science student at Indian Institute of Technology, Delhi (IITD) sent me an e-mail and asked if I could mentor him on a FUSE file system project for FreeBSD. I have never done this before, so agreed. I am very interested in bringing new blood into the FreeBSD project. I am interested in learning what the pain points are for new developers to start with FreeBSD, so as a project, we in FreeBSD can improve things. Pedro Giffuni has also agreed to mentor this project. Pedro has a lot of experience with FreeBSD file systems and FUSE.
GSoC has a published schedule for when the GSoC work must start, and when it must end. Neeraj has submitted his proposal, and we will learn if it is accepted by the end of May 2013. Hopefully the proposal will be accepted, and we can do some interesting things over the summer with FUSE and FreeBSD!
Several members of the PC-BSD team will be at Texas LinuxFest, to be held at the AT&T Conference Center in Austin, Texas from Friday May 31 to Saturday June 1. Registration is required for this event at a cost of $25 or $55.
There will be a BSD booth in the Expo (booth #21) on June 1 where we’ll be giving out PC-BSD DVDs, FreeNAS CDs and other cool swag. We can also accept donations to the FreeBSD Foundation.
Kris Moore will present “Automating the deployment of FreeBSD & PC-BSD® systems” at 11:25 on June 1. Dru Lavigne will present “TrueOS and Warden: Easy to Deploy FreeBSD Servers” at 13:30 on June 1.
The BSDA certification exam will be held at 13:00 on May 31. The cost for the exam is $75.
From May 15 – 18, I attended BSDCan and FreeBSD DevSummit in Ottawa, Canada.
These were both excellent conferences.
There were many excellent presentations during this conference, but my favorites were:
With these two technologies, the bar has definitely been raised for FreeBSD to be used in applications which require storage (ZFS) and virtualization (BHyve).
There were more presentations that I wanted to attend, but there was just not enough time!
First thing we will need a clone of https://github.com/xaque208/puppet-pkgng into /usr/local/etc/puppet/modules/.
This will be pushed out to the clients as long as: pluginsync = true
For me the next step is to create a manifests/init.pp in the new module directory. This is important to me because I want to sync out a /usr/local/etc/pkg.conf to all my machines so that they point to my internal poudriere repos. So I end up with something like this:
file { "/usr/local/etc/pkg.conf":
mode => 755,
owner => root,
content => "packagesite: http://pkg/91-web/
",
}
Once that is done it is easy to use pkgng packages via:
package { "www/apache22":
ensure => installed,
provider => pkgng,
require => File['/usr/local/etc/pkg.conf'],
}
Kris has an article in the May issue of BSD Magazine, demonstrating some of the new features in the 9.1 version of Warden, the jail management GUI in PC-BSD.
The magazine is available for free download here and Kris’ article starts on page 17.
Josh Smith has announced the initial launch of the PC-BSD hardware store. This resource is meant to make it easier to find hardware that has been tested to work on PC-BSD. The store itself is here. If you have hardware that you know works and which you don’t see listed, please add it to the Hardware section of the wiki so that it can be included.
This is a bunch of links to the tools I talk about in my presenation
Tools:
Collectd: https://collectd.org/
Graphite: http://graphite.wikidot.com/
Nagios: http://www.nagios.org/
Poudriere: http://fossil.etoilebsd.net/poudriere
Config Management:
Salt Stack: http://saltstack.com/
Chef: http://www.opscode.com/chef/
Puppet: http://puppetlabs.com/
Subversion: http://subversion.apache.org/
LogStash: http://logstash.net/
Audit: http://www.freebsd.org/handbook/audit.html
CARP: http://www.freebsd.org/handbook/carp.html
OATH: http://www.openauthentication.org/
Serial Console: http://www.freebsd.org/handbook/serialconsole-setup.html
Generic Resources:
FreeBSD Handbook: http://freebsd.org/handbook
Everything Sysadmin Blog: http://everythingsysadmin.com/resources.html
It's that time of the year again - time for the biggest, bestest gathering of BSD geeks from around the world - BSDCan 2013. It was great to see old friends and new faces, exchange ideas and talk about the bright future.
Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).
Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.
An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:
----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
time |usr sys idl wai hiq siq| recv send| read writ|util
16-05 08:36:15| 2 3 96 0 0 0| 66B 178B| 0 0 | 0
16-05 08:36:16| 42 14 37 0 0 7| 92M 1268k| 0 0 | 0
16-05 08:36:17| 45 11 36 0 0 7| 76M 1135k| 0 0 | 0
16-05 08:36:18| 27 55 8 0 0 11| 67M 754k| 0 99M|79.6
16-05 08:36:19| 29 41 16 5 0 10| 113M 2079k|4096B 63M|59.6
16-05 08:36:20| 28 48 12 4 0 8| 58M 397k| 0 95M|76.0
16-05 08:36:21| 38 37 14 1 0 10| 114M 2620k|4096B 52M|23.2
16-05 08:36:22| 37 54 0 1 0 8| 76M 1506k|8192B 76M|33.6
So if you want to graph one of the columns, it’s useful to convert all the numbers in the same unit. Bytes would be nice in this case.
Separating all columns with ‘|’ characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. ‘C-x r t’ can do that in Emacs, and you end up with something like this:
| time |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-| | time |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util| |16-05 08:36:15| 2| 3| 96| 0| 0| 0| 66B| 178B| 0 | 0 | 0| |16-05 08:36:16| 42| 14| 37| 0| 0| 7| 92M|1268k| 0 | 0 | 0| |16-05 08:36:17| 45| 11| 36| 0| 0| 7| 76M|1135k| 0 | 0 | 0| |16-05 08:36:18| 27| 55| 8| 0| 0| 11| 67M| 754k| 0 | 99M|79.6| |16-05 08:36:19| 29| 41| 16| 5| 0| 10| 113M|2079k|4096B| 63M|59.6| |16-05 08:36:20| 28| 48| 12| 4| 0| 8| 58M| 397k| 0 | 95M|76.0| |16-05 08:36:21| 38| 37| 14| 1| 0| 10| 114M|2620k|4096B| 52M|23.2| |16-05 08:36:22| 37| 54| 0| 1| 0| 8| 76M|1506k|8192B| 76M|33.6|
The leading and trailing ‘|’ characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.
What we would like to do is convert text like “408B” to just “408″, text like “1268k” to the value of (1268 * 1024), and finally text like “67M” to the value of (67 * 1024 * 1024). The first part is easy:
M-x replace-regexp RET \([0-9]+\)B RET \1 RET
This should just strip the “B” suffix from byte values.
For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like “replace \1 with the value of (expression \1)“. This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the “real” substitution text.
So if we match all numeric values that are suffixed by ‘k’, we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:
\,(* 1024 (string-to-number \1))
The full Emacs command would then become:
M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET
This, and the byte suffix removal, yield now the following text in our Emacs buffer:
| time |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-| | time |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util| |16-05 08:36:15| 2| 3| 96| 0| 0| 0| 66| 178| 0 | 0 | 0| |16-05 08:36:16| 42| 14| 37| 0| 0| 7| 92M|1298432| 0 | 0 | 0| |16-05 08:36:17| 45| 11| 36| 0| 0| 7| 76M|1162240| 0 | 0 | 0| |16-05 08:36:18| 27| 55| 8| 0| 0| 11| 67M| 772096| 0 | 99M|79.6| |16-05 08:36:19| 29| 41| 16| 5| 0| 10| 113M|2128896|4096| 63M|59.6| |16-05 08:36:20| 28| 48| 12| 4| 0| 8| 58M| 406528| 0 | 95M|76.0| |16-05 08:36:21| 38| 37| 14| 1| 0| 10| 114M|2682880|4096| 52M|23.2| |16-05 08:36:22| 37| 54| 0| 1| 0| 8| 76M|1542144|8192| 76M|33.6|
Note: Some of the columns are indeed not aligned very well. We’ll fix that later. On to the megabyte conversion:
M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET
Which produces a version that has no suffixes at all:
| time |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-| | time |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util| |16-05 08:36:15| 2| 3| 96| 0| 0| 0| 66| 178| 0 | 0 | 0| |16-05 08:36:16| 42| 14| 37| 0| 0| 7| 96468992|1298432| 0 | 0 | 0| |16-05 08:36:17| 45| 11| 36| 0| 0| 7| 79691776|1162240| 0 | 0 | 0| |16-05 08:36:18| 27| 55| 8| 0| 0| 11| 70254592| 772096| 0 | 103809024|79.6| |16-05 08:36:19| 29| 41| 16| 5| 0| 10| 118489088|2128896|4096| 66060288|59.6| |16-05 08:36:20| 28| 48| 12| 4| 0| 8| 60817408| 406528| 0 | 99614720|76.0| |16-05 08:36:21| 38| 37| 14| 1| 0| 10| 119537664|2682880|4096| 54525952|23.2| |16-05 08:36:22| 37| 54| 0| 1| 0| 8| 79691776|1542144|8192| 79691776|33.6|
Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type “C-c C-c” with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:
| time | cpu | cpu | cpu | cpu | cpu | cpu | eth0 | eth0 | disk | disk | sda- | | time | usr | sys | idl | wai | hiq | siq | recv | send | read | writ | util | | 16-05 08:36:15 | 2 | 3 | 96 | 0 | 0 | 0 | 66 | 178 | 0 | 0 | 0 | | 16-05 08:36:16 | 42 | 14 | 37 | 0 | 0 | 7 | 96468992 | 1298432 | 0 | 0 | 0 | | 16-05 08:36:17 | 45 | 11 | 36 | 0 | 0 | 7 | 79691776 | 1162240 | 0 | 0 | 0 | | 16-05 08:36:18 | 27 | 55 | 8 | 0 | 0 | 11 | 70254592 | 772096 | 0 | 103809024 | 79.6 | | 16-05 08:36:19 | 29 | 41 | 16 | 5 | 0 | 10 | 118489088 | 2128896 | 4096 | 66060288 | 59.6 | | 16-05 08:36:20 | 28 | 48 | 12 | 4 | 0 | 8 | 60817408 | 406528 | 0 | 99614720 | 76.0 | | 16-05 08:36:21 | 38 | 37 | 14 | 1 | 0 | 10 | 119537664 | 2682880 | 4096 | 54525952 | 23.2 | | 16-05 08:36:22 | 37 | 54 | 0 | 1 | 0 | 8 | 79691776 | 1542144 | 8192 | 79691776 | 33.6 |
The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2 … \9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.
Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.