Craig Rodrigues, FreeBSD developer 2013-05-24 20:40:41

On June 17, 2013, Stanford University is offering a free online course on coursera:

https://www.coursera.org/course/startup

 

===============================================================================================

About the Course

Spiritual sequel to Peter Thiel’s CS183 course on startups. Bridges the gap between academic computer science and production software engineering. Fast-paced introduction to key tools and techniques (command line, dotfiles, text editor, distributed version control, debugging, testing, documentation, reading code, deployments), featuring guest appearances by senior engineers from successful startups and large-scale academic projects. Over the course of the class, students will build a command line application, expose it as a web service, and then link other students’ applications and services together to build an HTML5 mobile app. General principles are illustrated through modern Javascript and the latest web technologies, including Node, Backbone, Coffeescript, Bootstrap, Git, and Github.
===============================================================================================
The notes for this course are also available here:  http://startup.stanford.edu/
This course is quite interesting, because it really shows how a lot of startups in Silicon Valley are rapidly bootstrapped at a technology level.  These days a lot of these startups use Linux out of the box, and don’t even bother looking at the alternatives.
Obviously, I like FreeBSD a lot, and think FreeBSD is a great platform for startups.
What are the things that we as a FreeBSD community can do to improve the FreeBSD platform,
and make it a no-brainer decision for startups to use it for their new companies?
The startups of today become the billion dollar companies of tomorrow, so it is important that FreeBSD is a part
of that.
I’d love to hear the ideas of people in the FreeBSD community for how to make FreeBSD better for startups!

 

rodrigc 2013-05-24 19:50:08

These days, a lot of open source projects and even startup companies are using git and Github to host their code and do rapid development.  Although the FreeBSD project’s SCM is Subversion,
it is still possible to use Github to work with FreeBSD thanks to the work of FreeBSD developers who have set up scripts which mirror the FreeBSD code in Github (see: https://wiki.freebsd.org/Git ).

Using the Github repository, it should be possible for a GSoC student to:

  1. Look at the FreeBSD source code at https://github.com/freebsd/freebsd
  2. Create their own copy of the FreeBSD code in Github by following the instructions at “Fork a Repo”, https://help.github.com/articles/fork-a-repo
  3. Check out the FreeBSD code from their Github copy and rebuild the world and kernel by following the instructions at “Rebuilding world” http://www.freebsd.org/doc/handbook/makeworld.html
  4. Modify the code, debug the kernel, and do other fun stuff by reading “Developer’s handbook”, http://www.freebsd.org/doc/en/books/developers-handbook/
  5. Follow some of the ideas in Alfred Perlstein’s blog post for Basing a Commerical Product on FreeBSD Using Git

 

 

It would be fun to try this all out!!

rodrigc 2013-05-24 19:25:33

The list of approved GSoC projects for FreeBSD should be available in the next few days.

While we are waiting for the list of approved projects to be announced, I am helping Neeraj Rawat set up a VM so that he can do FreeBSD development for Google Summer of Code (GSoC).

I am interested in learning what the “Out of the Box Experience” is like for students and newcomers to FreeBSD.  The technologies that students are exposed to today, and the development environments that they are comfortable in on Linux/Unix systems is far different than when I started using Unix systems in 1992 when I was a  chemical engineering student at the University of Toronto.

Neeraj likes the Geany IDE.  This a simple graphical IDE is written using the GTK2 toolkit.  Luckily, there is a FreeBSD port of Geany.

Neeraj’s home computer is a Windows 7 box, and he is using VMWare Player to run FreeBSD in a virtual machine.

Since Neeraj wants to work on file system stuff with FUSE, I am asking him to use FreeBSD-10-CURRENT, instead of FreeBSD 9.1 release.

I told Neeraj to do the following:

  2.   Install that image into a Virtual Machine, or on a dedicated PC computer using the instructions at: http://www.freebsd.org/doc/en/books/handbook/bsdinstall.html
  3. Once the system is booted up, create a configuration file /usr/local/etc/pkg.conf and in this file has one line:packagesite:  http://mirror.exonetric.net/pub/pkgng/freebsd:10:x86:64/latest/
  4. Use the “pkg” utility to install packages, as documented here: http://www.freebsd.org/doc/en/books/handbook/pkgng-intro.html
  5. Install some basic packages with this command:pkg install devel/git-subversion devel/subversion shells/bash shells/zsh
    pkg install x11/xorg x11-drivers/xf86-input-vmmouse x11-drivers/xf86-video-vmware
    pkg install x11/gnome2
    pkg install devel/geany
  6. Edit /etc/rc.conf and add these entries:gnome_enable=”YES”
    gdm_enable=”YES”
    dbus_enable=”YES”
    hald_enable=”YES”This is covered in these links:http://www.freebsd.org/gnome/docs/faq2.html

    http://www.freebsd.org/doc/en/books/handbook/x-config.html

    http://www.freebsd.org/doc/en/books/handbook/x11-wm.html

 

Neeraj is not finished yet.  One of the steps where we got stuck was in step (3).  Neeraj had typed:

packagesite:http://mirror.exonetric.net/pub/pkgng/freebsd:10:x86:64/latest/

and not

packagesite:  http://mirror.exonetric.net/pub/pkgng/freebsd:10:x86:64/latest/

Apparently for the “pkg” utility, in the pkg.conf file, the “packagesite:” must have a space after the colon and before the URL, otherwise it will fail.  When the FreeBSD ports team gets all the infrastructure in place to build FreeBSD pkgng packages, we will be able to delete this line and not use a 3rd party provider of FreeBSD packages for 10-CURRENT.

I will provide more details as we get set up.  This is all very interesting!

 

Craig Rodrigues, FreeBSD developer 2013-05-24 17:43:45

Every year Google funds a project called Google Summer of Code .  Google works with different open source projects, and actually pays students to hack on open source projects during their summer break.  If a student is interested in participating in Google Summer of Code (GSoC), they must write a proposal for a project that they want to work on for a specific open source project, and submit it.  These proposals are evaluated by the open source project and Google.  If approved, the student can work on their project, but must work with a mentor in the open source project.

FreeBSD has participated in GSoC for many years.

A few months ago, Neeraj Rawat, who is a computer science student at Indian Institute of Technology, Delhi (IITD) sent me an e-mail and asked if I could mentor him on a FUSE file system project for FreeBSD.  I have never done this before, so agreed.  I am very interested in bringing new blood into the FreeBSD project.  I am interested in learning what the pain points are for new developers to start with FreeBSD, so as a project, we in FreeBSD can improve things.  Pedro Giffuni has also agreed to mentor this project.  Pedro has a lot of experience with FreeBSD file systems and FUSE.

GSoC has a published schedule for when the GSoC work must start, and when it must end.  Neeraj has submitted his proposal, and we will learn if it is accepted by the end of May 2013.  Hopefully the proposal will be accepted, and we can do some interesting things over the summer with FUSE and FreeBSD!

Foundation at Texas LinuxFest

There will be a FreeBSD booth (booth #21) in the Expo area at Texas LinuxFest, to be held at the AT&T Conference Center in Austin, Texas on Saturday June 1. Registration is required for this event at a cost of $25 or $55.

Stop by the booth to discuss the Foundation's projects, check out the cool swag, or to make a donation to the FreeBSD Foundation.

PC-BSD at Texas LinuxFest

Several members of the PC-BSD team will be at Texas LinuxFest, to be held at the AT&T Conference Center in Austin, Texas from Friday May 31 to Saturday June 1. Registration is required for this event at a cost of $25 or $55.

There will be a BSD booth in the Expo (booth #21) on June 1 where we’ll be giving out PC-BSD DVDs, FreeNAS CDs and other cool swag. We can also accept donations to the FreeBSD Foundation.

Kris Moore will present “Automating the deployment of FreeBSD & PC-BSD® systems” at 11:25 on June 1. Dru Lavigne will present “TrueOS and Warden: Easy to Deploy FreeBSD Servers” at 13:30 on June 1.

The BSDA certification exam will be held at 13:00 on May 31. The cost for the exam is $75.

 

Craig Rodrigues, FreeBSD developer 2013-05-23 21:31:15

From May 15 – 18, I attended BSDCan and FreeBSD DevSummit in Ottawa, Canada.
These were both excellent conferences.

There were many excellent presentations during this conference, but my favorites were:

With these two technologies, the bar has definitely been raised for FreeBSD to be used in applications which require storage (ZFS) and virtualization (BHyve).

There were more presentations that I wanted to attend, but there was just not enough time!

BSDCan Trip Report: Eitan Adler

The FreeBSD Foundation sponsored 7 attendees of BSDCan 2013. The first trip report is from Eitan Adler, a doc committer, who attended the BSDCan Developers Summit. Eitan writes:

I arrived Tuesday night and met Colin Percival at the airport.  After dropping off luggage at the university, I met up with some of the other developers.

The first day, I attended the "Netflix and FreeBSD" session run by Scott Long.  It was interesting to see what kind of problems users of FreeBSD ran into when running at scale.

For the afternoon working group, I chose to attend the "ports and packages" session.  A variety of topics were discussed but the most discussed topic was cross-building ports across both versions and architectures.  This is a topic that came up repeatedly in prior
discussion and that would come up again in other working groups, so it was good to know about the latest work in this area.

The vendor summit came next.  In the past, the vendor summit focused on kernel work but this one revolved around the user land.  This is particularly important to me as I run FreeBSD on my laptop as my primary development machine.

At night I spent some time in the hacking lounge or other shared areas meeting people.  It was very nice to be able to meet the people I've been talking to for the past three years.

On Thursday I spent my morning in the "Desktop" session.  Getting FreeBSD running well on desktops is critical in attracting new developers in the future.  Kris Moore, from PCBSD, spoke a lot about the customizations that they made.  I pressed to share the improvements
that could be committed upstream.  Other issues discussed were packaging for the desktop and a graphical boot loader for FreeBSD/PC-BSD.

The afternoon session for me was "Documentation": a significant portion of the discussion was about the future print edition of the book and what sections need to be updated and improved.  In particular, how we could get more source committers involved in writing documentation.  We also discussed how to work going forward with other teams that need access to the documentation (e.g., portmgr and postmaster).   We also touched on the FAQ, translations, and the new toolchain.  The final topic we discussed was the automated QA and statistics tools we have (and don't have) and how we could improve in that area.

After dinner I did some work at the documentation hackathon.  I spent the remainder of the night at the hacker lounge discussing kernel internals with Peter Wemm, Sean Bruno, and others.

Unfortunately, I had to leave prior to the conference itself, but I felt that meeting people at the developer summit was well worth the time spent.

brd’s notes » FreeBSD 2013-05-21 16:58:17

First thing we will need a clone of https://github.com/xaque208/puppet-pkgng into /usr/local/etc/puppet/modules/.

This will be pushed out to the clients as long as: pluginsync = true

For me the next step is to create a manifests/init.pp in the new module directory. This is important to me because I want to sync out a /usr/local/etc/pkg.conf to all my machines so that they point to my internal poudriere repos. So I end up with something like this:

file { "/usr/local/etc/pkg.conf":
        mode => 755,
        owner => root,
        content => "packagesite: http://pkg/91-web/
",
}

Once that is done it is easy to use pkgng packages via:

package { "www/apache22":
        ensure => installed,
        provider => pkgng,
        require => File['/usr/local/etc/pkg.conf'],
}

PC-BSD Hardware Store

Josh Smith has announced the initial launch of the PC-BSD hardware store. This resource is meant to make it easier to find hardware that has been tested to work on PC-BSD.  The store itself is here. If you have hardware that you know works and which you don’t see listed, please add it to the Hardware section of the wiki so that it can be included.

brd’s notes » FreeBSD 2013-05-17 19:47:19

This is a bunch of links to the tools I talk about in my presenation

Tools:

Collectd: https://collectd.org/

Graphite: http://graphite.wikidot.com/
Nagios: http://www.nagios.org/

Poudriere: http://fossil.etoilebsd.net/poudriere

Config Management:
Salt Stack: http://saltstack.com/
Chef: http://www.opscode.com/chef/
Puppet: http://puppetlabs.com/

Subversion: http://subversion.apache.org/

LogStash: http://logstash.net/
Audit: http://www.freebsd.org/handbook/audit.html

CARP: http://www.freebsd.org/handbook/carp.html

OATH: http://www.openauthentication.org/

Serial Console: http://www.freebsd.org/handbook/serialconsole-setup.html

Generic Resources:
FreeBSD Handbook: http://freebsd.org/handbook
Everything Sysadmin Blog: http://everythingsysadmin.com/resources.html

bsdtalk225 – PC-BSD with Kris Moore

Interview during BSDCan 2013 with Kris Moore from iXsystems.  We talk about some of the new features of PC-BSD.

File Info: 12Min, 6MB.

Ogg Link
http://cis01.uma.edu/~wbackman/bsdtalk/bsdtalk225.ogg

What keramida said… » FreeBSD 2013-05-16 17:04:58

Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).

Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.

An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:

----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
     time     |usr sys idl wai hiq siq| recv  send| read  writ|util
16-05 08:36:15|  2   3  96   0   0   0|  66B  178B|   0     0 |   0
16-05 08:36:16| 42  14  37   0   0   7|  92M 1268k|   0     0 |   0
16-05 08:36:17| 45  11  36   0   0   7|  76M 1135k|   0     0 |   0
16-05 08:36:18| 27  55   8   0   0  11|  67M  754k|   0    99M|79.6
16-05 08:36:19| 29  41  16   5   0  10| 113M 2079k|4096B   63M|59.6
16-05 08:36:20| 28  48  12   4   0   8|  58M  397k|   0    95M|76.0
16-05 08:36:21| 38  37  14   1   0  10| 114M 2620k|4096B   52M|23.2
16-05 08:36:22| 37  54   0   1   0   8|  76M 1506k|8192B   76M|33.6

So if you want to graph one of the columns, it’s useful to convert all the numbers in the same unit. Bytes would be nice in this case.

Separating all columns with ‘|’ characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. ‘C-x r t’ can do that in Emacs, and you end up with something like this:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66B| 178B|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1268k|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1135k|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 754k|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2079k|4096B|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 397k|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2620k|4096B|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1506k|8192B|  76M|33.6|

The leading and trailing ‘|’ characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.

What we would like to do is convert text like “408B” to just “408″, text like “1268k” to the value of (1268 * 1024), and finally text like “67M” to the value of (67 * 1024 * 1024). The first part is easy:

M-x replace-regexp RET \([0-9]+\)B RET \1 RET

This should just strip the “B” suffix from byte values.

For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like “replace \1 with the value of (expression \1)“. This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the “real” substitution text.

So if we match all numeric values that are suffixed by ‘k’, we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:

\,(* 1024 (string-to-number \1))

The full Emacs command would then become:

M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET

This, and the byte suffix removal, yield now the following text in our Emacs buffer:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 772096|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2128896|4096|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 406528|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2682880|4096|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1542144|8192|  76M|33.6|

Note: Some of the columns are indeed not aligned very well. We’ll fix that later. On to the megabyte conversion:

M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET

Which produces a version that has no suffixes at all:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  96468992|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  79691776|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  70254592| 772096|   0 |  103809024|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 118489088|2128896|4096|  66060288|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  60817408| 406528|   0 |  99614720|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 119537664|2682880|4096|  54525952|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  79691776|1542144|8192|  79691776|33.6|

Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type “C-c C-c” with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:

| time           | cpu | cpu | cpu | cpu | cpu | cpu |      eth0 |    eth0 |  disk |      disk | sda- |
| time           | usr | sys | idl | wai | hiq | siq |      recv |    send |  read |      writ | util |
| 16-05 08:36:15 |   2 |   3 |  96 |   0 |   0 |   0 |        66 |     178 |     0 |         0 |    0 |
| 16-05 08:36:16 |  42 |  14 |  37 |   0 |   0 |   7 |  96468992 | 1298432 |     0 |         0 |    0 |
| 16-05 08:36:17 |  45 |  11 |  36 |   0 |   0 |   7 |  79691776 | 1162240 |     0 |         0 |    0 |
| 16-05 08:36:18 |  27 |  55 |   8 |   0 |   0 |  11 |  70254592 |  772096 |     0 | 103809024 | 79.6 |
| 16-05 08:36:19 |  29 |  41 |  16 |   5 |   0 |  10 | 118489088 | 2128896 |  4096 |  66060288 | 59.6 |
| 16-05 08:36:20 |  28 |  48 |  12 |   4 |   0 |   8 |  60817408 |  406528 |     0 |  99614720 | 76.0 |
| 16-05 08:36:21 |  38 |  37 |  14 |   1 |   0 |  10 | 119537664 | 2682880 |  4096 |  54525952 | 23.2 |
| 16-05 08:36:22 |  37 |  54 |   0 |   1 |   0 |   8 |  79691776 | 1542144 |  8192 |  79691776 | 33.6 |

The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2 … \9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.

Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.


Filed under: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Lisp, Open source, Programming, Software Tagged: Computers, Emacs, Free software, FreeBSD, GNU/Linux, Lisp, Open source, Programming, Software

FreeBSD Foundation Announces Ed Maste is New Director of Project Development

The FreeBSD Foundation is pleased to announce Ed Maste's new role as the Foundation's part-time Director of Project Development.  Ed has served on the Foundation's board for two years, and has stepped down in order to accept this new position.

In this position, Ed will manage the Foundation's sponsored work, including projects funded under specific grants, operational support and project development undertaken by the Foundation's permanent technical staff. Working with the Foundation's Board of Directors, Ed will identify and document specific areas of future project work interest. This roadmap planning will include coordination with FreeBSD consumers and the FreeBSD community.

"2012 represented an inflection point in the Foundation's history,'' said Justin T. Gibbs, President of the FreeBSD Foundation. "The Foundation has a stated goal of investing in permanent staff through 2013. With Ed taking on this new position I'm excited by the Foundation's increased capacity to manage our project development and operational support.''

Ed has over ten years of experience in companies building products on FreeBSD, in both technical and managerial roles. He resides in Kitchener, Ontario, Canada.