Archive for the 'Open Source' Category

Alexander Leidinger: Linuxulator D-Trace probes committed to current

Published on May 18, 2012 in current, dtrace, emulation, FreeBSD, gt 500, kernel, kernel hacker, Linuxulator, little bit, Open Source, probes, Software and Source. Closed

A while ago I committed the linuxulator D-Trace probes I talked about earlier. I waited a little bit for this announcement to make sure I have not broken anything. Nobody complained so far, so I assume nothing obviously bad crept in.

The >500 probes I committed do not cover the entire linuxulator, but are a good start. Adding new ones is straight forward, if someone is interested in a junior–kernel–hacker task, this would be one. Just ask me (or ask on emulation@), and I can guide you through it.

Share

Alexander Leidinger: linux_base-c6

Published on May 18, 2012 in c6, current, FreeBSD, infrastructure, kernel, Linux, Linuxulator, Open Source, patches, Ports, Ports Collection, rpms and Software. Closed

Seems I forgot to announce that the linux_base-c6 is in the Ports Collection now. Well, it is not a replacement for the current default linux base, the linuxulator infrastructure ports are missing and we need to check if the kernel supports enough of 2.6.18 that nothing breaks.

TODO:

To my knowledge, nobody is working on anything of this. Anyone is welcome to have a look and provide patches.

Share

Alexander Leidinger: DTrace in GENERIC (-current)

Published on April 25, 2012 in Blog, current, dtrace, FreeBSD, generic kernel, hooks, kernel, Open Source, Software and Source. Closed

In case you have not noticed yet, KDTRACE_HOOKS is now in the GENERIC kernel in FreeBSD-current. This means you just need to load the DTrace modules and can use DTrace with the GENERIC kernel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you invest reading it.

Share

Alexander Leidinger: Linuxulator progress

Published on April 15, 2012 in amd64, dtrace, eol, expiration date, FreeBSD, kernel, kernel module, kernel panic, linux base, linux module, Linuxulator, Open Source, Ports, probes, Software, Source, stable and Userland. Closed

This weekend I made some progress in the linuxulator:

  • I MFCed the reporting of some linux-syscalls to 9-stable and 8-stable.
  • I updated my linuxulator-dtrace patch to a recent -current. I already compiled it on i386 and arundel@ has it compiled on amd64. I counted more than 500 new DTrace probes. Now that DTrace rescans for SDT probes when a kernel module is loaded, there is no kernel panic anymore when the linux module is loaded after the DTrace modules and you want to use DTrace. I try to commit this at a morning of a day where I can fix things during the day in case some problems show up which I did not notice during my testing.
  • I created a PR for portmgr@ to repocopy a new linux_base port.
  • I set the expiration date of linux_base-fc4 (only used by 7.x and upstream way past its EoL) and all dependent ports. It is set to the EoL of the last 7.x release, which can not use a later linux_base port. I also added a comment which explains that the date is the EoL of the last 7.x release.

Share

Alexander Leidinger: Sockets and nullfs: works now in –current

Published on March 17, 2012 in current, FreeBSD, ip, Jails, kernel, mysql, Open Source, sockets, Software, TCP and Userland. Closed

I just updated to a recent -current and tried the new nullfs. Sockets (e.g. the MySQL one) work now with nullfs. No need to have e.g. jails on the same FS and hardlink the socket to not need to use TCP in MySQL (or an IP at all for the jail).

Great work!

Share

Alexander Leidinger: New opportunities in the linuxulator

Published on March 13, 2012 in compat, Email, failure, FreeBSD, fruits, kernel, kernel messages, Linux, linux base, linux man, Linuxulator, man page, new opportunities, Open Source, Software and Source. Closed

Last weekend I committed some dummy-syscalls to the linuxulator in FreeBSD-current. I also added some comments to syscalls.master which should give a hint which linux kernel had them for the first time (if the linux man–page I looked this up in is correct). So if someone wants to experiment with a higher compat.linux.osrelease than 2.6.16 (as it is needed for a CentOS based linux_base), he should now get some kernel messages about unimplemented syscalls instead of a silent failure.

There may be some low-hanging fruits in there, but I did not really verify this by checking what the dummy syscalls are supposed to do in linux and if we can easily map this to existing FreeBSD features. In case someone has a look, please send an email to emulation@FreeBSD.org.

Share

Alexander Leidinger: New CentOS linux_base for testing soonish

Published on March 13, 2012 in centos linux, experimental task, FreeBSD, freebsd releases, howto linux, infrastructure, kernel, kernel messages, Linux, linux base, Linuxulator, Open Source, Ports, Ports Collection, Software and srpms. Closed

It seems my HOWTO create a new linux_base port was not too bad. There is now a PR for a CentOS 6 based linux_base port. I had a quick look at it and it seems that it is nearly usable to include into the Ports Collection (the SRPMs need to be added, but that can be done within some minutes).

When FreeBSD 8.3 is released and the Ports Collection open for sweeping commits again, I will ask portmgr to do a repo-copy for the new port and commit it. This is just the linux_base port, not the complete infrastructure which is needed to completely replace the current default linuxulator userland. This is just a start. The process of switching to a more recent linux_base port is a long process, and in this case depends upon enough support in the supported FreeBSD releases.

Attention: Anyone installing the port from the PR should be aware that using it is a highly experimental task. You need to change the linuxulator to impersonate himself as a linux 2.6.18 kernel (described in the pkg-message of the port), and the code in FreeBSD is far from supporting this. Anyone who wants to try it is welcome, but you have to run FreeBSD-current as of at least the last weekend, and watch out for kernel messages about unsupported syscalls. Reports to [email protected] please, not here on the webpage.

Share

Martin Wilke: BSD in Malaysia

Published on February 22, 2012 in BSD, FreeBSD, FreeBSD_EN, group, Hacking, Kuala Lumpur, Malaysia, Open Source and Ports. Closed

Hi.

Few days back I’ve met up with Mohd Fazil Azran for a small talk about *BSD at Starbucks coffee. I was interested to know why Malaysian *BSD community is so inactive, and from the discussion, I’d say that the reason is more likely caused by too much of politics in the group, financial issues, lack of interest to share knowledge and blablabla..

So now, I would like to suggest for a complete rebuild of a *BSD open group. It will be a group where everyone shares the same right, and the freedom of speech. No politics, no financial problems (go dutch all the way ), no hidden agenda. Just a group where everyone can share their knowledge freely. Hopefully with this group, we could attract more users to *BSD, as well as building back the trust for *BSD .

The kick off of this new group will be on the 3rd March, where I will give my first talk about FreeBSD - what is FreeBSD, why FreeBSD, FreeBSD ports and who use FreeBSD. My talk will be around 30 to 45 minutes, and afterwards I will be free for questions and discussion, and of course, coffee .

Date: Saturday, March 3, 2012
Time: 2 PM till 5 PM
Location: Old Town White Coffee, Bangsar South (KL)

You should be aware that this would be my first experience, so don’t expect for any professional talk. Everyone is welcome :)

So long.

PS: Help me to share & rt it & and follow me on twitter :)

Alexander Leidinger: What you should know about SSH

Published on February 3, 2012 in conditionals, database application, grain of salt, michael w lucas, Open Source, openssh, proof readers, putty ssh, review, Software, ssh command, ssh putty, system database, Tips & Tricks and Userland. Closed

Michael W. Lucas published his new book “SSH Mastery� (no link to an online store, get it from your preferred online or offline one in your part of the world).

Do you think you know a lot about SSH? I thought I did when Michael searched technical proof-readers for this book. I offered to have a look at his work in progress and he gently accepted (while I do not get money for this, I am one of the persons he thanks for  the technical review in the beginning, so I am involved somehow and as such you should take the following with a grain of salt).

I already had user restrictions in place before the review, but now I narrowed down some restrictions based upon some conditionals. I already used SSH tunnels for various things before (where legally applicable), but I learned some additional VPN techniques with SSH. I already used multiple ssh-keys for various things, but Michael provides some interesting ways of handling a large-volume of ssh-keys over multiple machines. … I really hope that my review was as valuable for Michael, as it was for me to do the review.

He ends the book with “You now know more about SSH, OpenSSH and Putty than the vast majority of IT professionals! Congratulations�, and this is true, and all that in his writing style where you can come with a problem, read about it, and leave with a solution (normally with a little bit of entertainment in between).

I know a lot of people which work daily with SSH, and they know only a small part of what is presented in this book. In my opinion this book is a must-have for every System/Database/Application/Whatever Administrator in charge of something on an UNIX-like system, and even “normal users� of SSH (no matter if they use PuTTY, or a ssh command line program on an UNIX-like system (most probably it will be OpenSSH or a clone of it)) will get some helpful information from this book.

I can only recommend it.

Share

Alexander Leidinger: How to fix FreeBSD for corporations (and user with smaller installations), the tools-viewpoint

Published on January 18, 2012 in bugtracking system, clusters, FreeBSD, iirc, interested parties, man power, Open Source, release engineering, Software, subtopic, viewpoint and volunteer project. Closed

There is a huge discussion going on on hackers@ about how FreeBSD is not suitable for large installations (anymore?). As of this writing, the discussion seems to get some discussion-clusters. We have some sub-topics which could lead to some good improvements.

One subtopic is the release engineering. Some changes like a more guided approach of what should be merged to which branch, the frequency of releases and maybe some kind of long-term-branch(es). There is some discussion to get maybe some joined-funding in some way from interested parties to pay someone to take care about long-term-branch(es).

Another subtopic is the way bugs are handled in our old bugtracking software and how patches go unnoticed there.

And both of them are connected (parts more, parts less) by what can be done in a volunteer project.

To me it looks like the proposals “just� need some refinements and some “volunteers� to put value (this means man power and/or money) to what they said.

What I want to discuss here is, how tools could help with making PRs/patches more visible to developers (there is already the possibility to get emails from the small bugbuster-team about patches in PR database, but you have to ask them to get them) and how to make it more easy to get patches into FreeBSD.

Making bugs more visible to developers

The obvious first: We need a different bugtracking system. We already know about it. There is (or was…) even someone working IIRC on an evaluation of what could be done and how easy/hard it would be. I am not aware of any outcome, despite the fact that it is months (or even a year) since this was announced. I do not blame anyone here, I would like to get time to finish some FreeBSD volunteer work myself.

In my opinion this needs to be handled in a commercial way. Someone needs to be officially paid (with a deadline) to produce a result. Unfortunately there is the problem that the requirements are in a way, that people do not have to change their workflows/procedures.

IIRC people ask that they should be able to send a mail to the bugtracker without the need for authentication. Personally I think the bugtracking issue is in a state where we need to change our workflows/procedures. It is convenient to get mails from the bugtracker and only have to reply to the mail to add something. On the other hand, if I report bugs somewhere, and if I really care about the problem resolution, I am willing login to whatever interface to get this damn problem solved.

Sending a problem report from the system where I have the issue in an easy way is a very useful feature. Currently we have send-pr for this and it uses emails. This means it requires a working email setup. As an user I do not care if the tool uses email or HTTP or HTTPS, I just want to have an easy way to submit the problem. I would not mind if I first have to do a “send-problem register me@tld� (once), “send-problem login me@tld� (once per system+user I want to send from) and then maybe a “send-problem template write_template_here.txt� (to get some template text to fill out), edit the template file and then run “send-problem send my_report.txt file1 file2 …�. That would be a different workflow, but still easy.

Email notifications are surely needed, but if I really care about a problem, I can be bothered to register first. So in my opinion, we need a different bugtracker desperately enough that we need to drop our requirements regarding our current workflow/procedures (even if it means we can not get a command line way of submitting bugs at all). The primary goal of the software needs to be to make it easy to track and resolve bugs. The submission of bugs shall be not hard too. If I look at the state of the world as it is ATM, I would say a webinterface with authentication is not a big burden to take if I really want to get my problem fixed. Some command line tool would be nice to have, but regarding the current state of our bugtracker it needs to be optional instead of a hard requirement.

Apart from making it easy to track and resolve problems, the software also needs to be able to make us aware of the biggest problems. Now… you may ask what is a big problem. Well… IMO it does not matter to you what I think is big or small here. The person with a problem needs to decide what is a big problem to him. And people with the same problem need to be able to tell that it is also a big problem for them. So a feature which allows to “vote� or “+1″ or “AOL� (or however you want to call it) would allow to let users with problems voice their opinion upon the relevance of the problem to our userbase. This also means there needs to be a way to see the highest voted problems. An automatic mail would be best, but as above this is optional. If I as a developer really care about this, I can be bothered to login to a webinterface (or maybe someone volunteers to make a copy & paste and send a mail… we need to be willing to rethink our procedures).

Getting patches more easy into a FreeBSD branch

It looks to me that this topic is requires a little bit more involvement from multiple tools. In my opinion we need to switch to a distributed version control system. One which allows to easily create my own branch of FreeBSD on my own hardware, and which allows to let other users use my branch easily (if I want to allow other to branch from my branch). It also needs to be able to let me push my changes towards FreeBSD. Obviously not directly into the official sources, but into some kind of staging area. Other people should be able to have a look at this staging area and be able to review what I did. They need to be able to make some comments for others to see, or give some kind of (multi-dimensional?-)rating for the patch (code quality / works for me / does not work / …). Based upon the review/rating and maybe some automated evaluation (compile test / regression test / benchmark run) a committer could push the patch into the official FreeBSD tree (ideal would be some automated notification system, a push button solution for integration and so on, but as above we should not be afraid if we do not get all the bells and whistles).

If we would have something like this in place, creating some kind of long-term-release branch could be used more easily in a colaborative manner. Companies which use the same long-term-release branch could submit their backports of fixes/features this way. They also could see if similar branches (there could be related but different branches, like 9.4-security-fixes-only <= 9.4-official-errata-only <= 9.4-bugfixes <= 9.4-bugfixes-and-driverupdates <= …) could be merged to their in-house branch (and maybe consequently push-back to the official branch they branched from if the patch comes from a different branch).

It does not matter here if we would create a fixed set of branches for each release, or if we only create some special-purpose branches based upon the phase of the moon (ideally we would create a lot of branches for every release, companies/users can cherry pick/submit what they want, and the status of a long-term-branch is solely based upon the inflow of patches and not by what the security team or release manager or a random developer thinks it should be… but the reality will probably be somewhere in the middle).

I do not know if tools exists to make all this happen, or which tools could be put together to make it happen. I also did not mention on purpose tools I am aware of which already provide (small) parts of this. These are just some ideas to think about. Interested parties are invited to join the discussion on hackers@ (which is far away from discussing specific tools or features), but you are also free to add some comments here.

Share

Alexander Leidinger: X11 in a jail with NVidia hardware

Published on December 30, 2011 in agpgart, driver nvidia, FreeBSD, GNOME, graphic card, home cinema, home server, Jails, kde 4, kernel, nvidia card, nvidia driver, Open Source, radeon 9200se, Software, Source and Userland. Closed

Just before christmas I decided I will spend the “immense� amount of 40 EUR for a graphic card for a system which was without one. The system is supposed to replace my dying home-server. I already moved everything, except my Desktop-in-a-Jail (actually it is my home-cinema-jail).

The old system had a Radeon 9200SE, and it was enough for what I used it for. Now… for a few bucks you can get a lot more horsepower today. After looking around a little bit I decided to buy a NVidia card. I made this decision because it looks like I can get better driver support for it. So I got a GeForce GT 520 with 1 GB of RAM (I doubt I will be able to use that much RAM) and without a fan.

With the Radeon 9200SE I was not able to get the 3D stuff activated (at least in the jail, I did not try without), Xorg complains about a missing agpgart module but I have AGP in the kernel (no /dev/agpgart outside the jail). I did not spend time to investigate this, as the main purpose — playing movies — worked. Now with the NVidia card I decided to give the 3D part a try again.

After adding the NVidia device entries to the jail, and a little bit of fighting with the Xorg-HAL interaction, I got a working desktop. The biggest problem to verify that 3D is working was, that I did not had xdriinfo installed. After installing it, I noticed that it does not work with the NVidia driver.  :-(   Next stop nvidia-settings: runs great, displays a nice FreeBSD+NVidia logo, and … tells me that OpenGL is configured. Hmmm… OK, but I want to see it!

As I decided to switch from Gnome to KDE 4 at  the same time (I was using KDE when it was at V 0.x, switched to Gnome as it looked nicer to me, and now I switch back after reading all the stuff in the net that KDE 4 is “better� than Gnome 3), I was a little bit out of knowledge how to see the 3D stuff in action. So I quickly went to the settings and searched for something which looks like it may use 3D. To my surprise, it was already using 3D stuff. Nice. I fully realized how nice, when playing a video and using Alt-Tab to switch windows: the video was playing full speed scaled down in the window-switcher-thumbnail-view.

That was too easy. I am happy about it.

Now that I have a working setup of X11-in-a-jail for Radeon and GeForce cards, I want to cleanup my changes to the kernel and the config files (devfs.rules) and have a look to get this committed. A big part of this work is probably writing documentation (most probably in the wiki).

I still want to see some fancy 3D stuff now. I tried to install x11-clocks/glclock, but the build fails with an undefined reference to ‘glPolygonOffsetEXT’. :-( Any recommendation for a fancy 3D display? My priority is on “fancy/nice� with as less violence as possible. Most probably I will look at it once and then deinstall it again, so it should be available in the Ports Collection (or included in KDE 4).

Share

Alexander Leidinger: Tuning guide in the wiki

Published on December 22, 2011 in benchmark, benefit, email notification, FreeBSD, improvements, man page, Open Source, Software, Tips & Tricks, tuning guide, volunteer and wiki. Closed

In the light of the recent benchmark discussion, a volunteer imported the tuning man-page into the wiki. Some comments at some places for possible improvements are already made. Please go over there, have a look, and participate please (testing/verification/discussion/improvements/…).

As always, feel free to register with FirstnameLastname and tell a FreeBSD committer to add you to the contributors group for write access (you also get the benefit to be able to register for an email notification for specific pages).

Share

Alexander Leidinger: A phoronix benchmark creates a huge benchmarking discussion

Published on December 21, 2011 in benchmark numbers, Commercial, fingers, FreeBSD, improvement suggestions, Linux, linux server, Open Source, oracle, oracle linux, Problem(s), release candidate, retest, Software and UFS. Closed

The recent Phoronix benchmark which compared a release candidate of FreeBSD 9 with Oracle Linux Server 6.1 created a huge discussion in the FreeBSD mailinglists. The reason was that some people think the numbers presented there give a wrong picture of FreeBSD. Partly because not all benchmark numbers are presented in the most prominent page (as linked above), but only at a different place. This gives the impression that FreeBSD is inferior in this benchmark while it just puts the focus (for a reason, according to some people) on a different part of the benchmark (to be more specific, blogbench is doing disk reads and writes in parallel, FreeBSD gives higher priority to writes than to reads, FreeBSD 9 outperforms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are presented on the first page). Other complaints are that it is told that the default install was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The author of the Phoronix article participated in parts of the discussion and asked for specific improvement suggestions. A FreeBSD committer seems to be already working to get some issues resolved. What I do not like personally, is that the article is not updated with a remark that some things presented do not reflect the reality and a retest is necessary.

As there was much talk in the thread but not much obvious activity from our side to resolve some issues, I started to improve the FreeBSD wiki page about benchmarking so that we are able to point to it in case someone wants to benchmark FreeBSD. Others already chimed in and improved some things too. It is far from perfect, some more eyes — and more importantly some more fingers which add content — are needed. Please go to the wiki page and try to help out (if you are afraid to write something in the wiki, please at least tell your suggestions on a FreeBSD mailinglist so that others can improve the wiki page).

What we need too, is a wiki page about FreeBSD tuning (a first step would be to take the man-page and convert it into a wiki page, then to improve it, and then to feed back the changes to the man-page while keeping the wiki page to be able to cross reference parts from the benchmarking page).

I already told about this in the thread about the Phoronix benchmark: everyone is welcome to improve the situation. Do not talk, write something. No matter if it is an improvement to the benchmarking page, tuning advise, or a tool which inspects the system and suggests some tuning. If you want to help in the wiki, create a FirstnameLastname account and ask a FreeBSD comitter for write access.

A while ago (IIRC we have to think in months or even years) there was some framework for automatic FreeBSD benchmarking. Unfortunately the author run out of time. The framework was able to install a FreeBSD system on a machine, run some specified benchmark (not much benchmarks where integrated), and then install another FreeBSD version to run the same benchmark, or to reinstall the same version to run another benchmark. IIRC there was also some DB behind which collected the results and maybe there was even some way to compare them. It would be nice if someone could get some time to talk with the author to get the framework and set it up somewhere, so that we have a controlled environment where we can do our own benchmarks in an automatic and repeatable fashion with several FreeBSD versions.

Share

Alexander Leidinger: (Free)BSD inside Android

Published on September 2, 2011 in Android, Commercial, FreeBSD, Open Source, Software, Source and Userland. Closed

Today I was looking into the OpenSource licenses which are displayed for Android (2.3.4). There are several files which come with a BSD license.

During looking at it, I noticed that the libm has the copyright of several FreeBSD people. I did not had an in-deep look if this is because they took the FreeBSD libm, or if this is because parts of the FreeBSD libm where adopted by other BSD projects.

What I noticed is, that some special characters are not displayed correctly. For example the name Dag-Erling Smørgrav looks mangled in the display of the license inside the phone (I hope it is displayed better in my blog). His name is not the only problem case, there are also other characters which are not rendered as expected.

This does not really look professional.

Share

Alexander Leidinger: Linuxulator explained (for developers): adding ioctls directly to the kernel

Published on September 2, 2011 in c file, encoder, FreeBSD, hex value, hex values, independent code, ioctl, kernel, linux directory, linux linux, Linuxulator, Open Source, open source software, open source software development, Software and Source. Closed

After giving an overview of the in-kernel basics of the Linuxulator, I want now to describe how to add support for new ioctls to the Linuxulator.

Where are the files to modify?

The platform independent code for the ioctls is in SRC/sys/compat/linux/linux_ioctl.c. The defines to have names for the ioctl values are in SRC/sys/compat/linux/linux_ioctl.h.

How to modify them?

First of all create a new header which will contain all the structures, named values and macros for those new ioctls. As written above, the ioctl values (e.g. #define LINUX_VIDIOC_ENCODER_CMD 0x564d /* 0xc028564d */) do not belong there, they shall be added to linux_ioctl.h. During the course of adding support for ioctls, you will need this new header. Add it in the SRC/sys/compat/linux/ directory, and prefix the name with a linux_. It would be good to decide on a common tag here (referenced as yourtag in the following), and stay with it. Use it wherever you need to have some specific name for the ioctl-set you want to add. In this case it would result in linux_yourtag.h (or even linux_ioctl_yourtag.h, depending if this is used for something very specific to the ioctls, or some generic linux feature) as the name of the header file. This was not done in the past, so do not expect that the names inside the linux_ioctl.c file will be consistent to this naming scheme, but it is never too late to correct mistakes of the past (at least in Open Source software development).

Now add this header to linux_ioctl.c (you need to include compat/linux/linux_yourtag.h). After that add the ioctl values to linux_ioctl.h. As can be seen above, the defines should be named the same as on linux, but with a LINUX_ prefix (make sure they where not defined before somewhere else). The ioctl values need to be the same hex values as in Linux, off course. Sort them according to their hex value. When you added all, you need to add two more defines. The LINUX_IOCTL_yourtag_MIN and LINUX_IOCTL_yourtag_MAX ones. The MIN-one needs to be an alias for the first (sorted according to the hex value) ioctl you added, and MAX needs to be an alias for the last (again, sorted according to the hex value) ioctl you added.

The next step is to let the Linuxulator know that it is able to handle the ioctls in the LINUX_IOCTL_yourtag_MIN to LINUX_IOCTL_yourtag_MAX range. Search the static linux_ioctl_function_t section of linux_ioctl.c and add such a variable for your ioctl set. The name of the variable should be something like linux_ioctl_yourtag.

Similar for the handler-definition for this. Search the static struct linux_ioctl_handler section and add a yourtag_handler. Set it to { linux_ioctl_yourtag, LINUX_IOCTL_yourtag_MIN, LINUX_IOCTL_yourtag_MAX }. To make this handler known to the Linuxulator, you need to add it to the DATA_SET section. Add DATA_SET(linux_ioctl_handler_set, yourtag_handler) there.

Now the meat, the function which handles the ioctls. You already defined it as linux_ioctl_function_t, but now you need to write it. The outline of it looks like this:

static int
linux_ioctl_yourtag(struct thread *td, struct linux_ioctl_args *args)
{
        struct file *fp;
        int error;
        switch (args->cmd & 0xffff) {
        case LINUX_an_easy_ioctl:
                break;
        case LINUX_a_not_so_easy_ioctl:
                /* your handling of the ioctl */
                fdrop(fp, td);
                return (error);
        /* some more handling of your ioctls */
        default:
       return (ENOIOCTL);
        }
        error = ioctl(td, (struct ioctl_args *)args);
        return (error);
}

An easy ioctl in the switch above is an ioctl where you do not have to do something but can pass the ioctl through to FreeBSD itself. The not so easy ioctl case is an ioctl where you need to do e.g. a fget(td, args->fd, &fp). This is just an example, there are also other possibilities where you need to do additional stuff before the return, or where you do not pass the ioctl to FreeBSD. A typical example of what needs to be done here is to copy values from linux structures to FreeBSD structures (and the other way too), or to translate between 64bit and 32bit. Linux programs on amd64 are 32bit executables and 32bit structures/pointers. To make this work on amd64, you need to find a way to map between the two. There are examples in the kernel where this is already the case. The more prominent examples in the 64bit<->32bit regard are the v4l and v4l2 ioctls.

The tedious part is to research if a translation has to be done and if yes what needs to be translated how. When this is done, most of the work is not so hard. The linux_yourtag.h should contain the structures you need for this translation work.

It is also possible to add ioctls in a kernel module, but this is not subject to this description (I will update this posting with a link to it when I get time to write about it).

Share

Alexander Leidinger: (Free)BSD inside Android

Published on September 2, 2011 in Android, Blog, bsd license, bsd projects, Commercial, dag, free bsd, FreeBSD, Open Source, problem case, Software, Source and Userland. Closed

Today I was looking into the OpenSource licenses which are displayed for Android (2.3.4). There are several files which come with a BSD license.

During looking at it, I noticed that the libm has the copyright of several FreeBSD people. I did not had an in-deep look if this is because they took the FreeBSD libm, or if this is because parts of the FreeBSD libm where adopted by other BSD projects.

What I noticed is, that some special characters are not displayed correctly. For example the name Dag-Erling Smørgrav looks mangled in the display of the license inside the phone (I hope it is displayed better in my blog). His name is not the only problem case, there are also other characters which are not rendered as expected.

This does not really look professional.

Share

Alexander Leidinger: HOWTO add linux-infrastructure ports for a new linux_base port

Published on September 1, 2011 in descriptive explanation, FreeBSD, freebsd ports collection, game servers, headless game, linux application, linux base, linux compatibility, linux dist, linux ports, linux release, Linuxulator, Open Source, Ports Collection, Software, Tips & Tricks and Userland. Closed

In my last blog-post I described how to create a new linux_base port. This blog-post is about the other Linux–ports which make up the Linux–infrastructure in the FreeBSD Ports Collection for a given Linux-release.

What are linux-infrastructure ports?

A linux_base port contains as much as possible and at the same time as little as possible to make up a useful Linux-compatibility-experience in FreeBSD. I know, this is not a descriptive explanation. And it is not on purpose. There are no fixed rules what has to be inside or what not. It “matured� into the current shape. A practical example is, that there is no GUI-stuff in the linux_base. While you need the GUI parts like GTK or QT for software like Skype and acroread, you do not need them for headless game servers. While you may need various libraries for game servers, you may not need those for Skype or acroread. As such some standard parts are in separate ports which are named linux–LINUX_DIST_SUFFIX-NAME. For GTK and the Fedora 10 release this results in linux-f10-gtk2. Such generic ports which depend upon a specific Linux-release make up the Linux-infrastructure in the FreeBSD Ports Collection. Those ports are referenced in port-Makefiles via the USE_LINUX_APPS variable, e.g. USE_LINUX_APPS=gtk2.

If you created a new linux_base port, you need most standard infrastructure ports in a version for the Linux-release used in the linux_base port, to have the Linux-application ports in the FreeBSD Ports Collection working (if you are unlucky, some ports do not play well with the Linux-release you have chosen, but this is out of the scope of this HOWTO).

Updating Mk/bsd.linux-apps.mk

 First we need to set the LINUX_DIST_SUFFIX variable to a value suitable to the new Linux-release. This is done in the conditional which checks the OVERRIDE_LINUX_NONBASE_PORTS variable for valid values. Add an appropriate conditional, and do not forget to add the new valid value to the IGNORE line in the last else branch of the conditional.

The next step is to check the _LINUX_APPS_ALL and _LINUX_26_APPS variables. If there are some infrastructure ports which are not available for the new Linux-release, the conditional which checks the availability of a given infrastructure port for a given Linux-release needs to be modified. If at a later step you notice that there are some additional infrastructure ports necessary for the new Linux-release, _LINUX_APPS_ALL and the check-logic needs to be modified too (e.g. add a new variable for your Linux-release, add the content of the variable to _LINUX_APPS_ALL, and change the check to do the right thing).

After that two tedious parts need to be done.

For each infrastructure port there is a set of variables. The name_PORT variable contains the location of the port in the Ports Collection. Typically you do not have to change it (if you really want to change it, do not do it, fix the naming of the infrastructure port instead), because we use a naming convention here which includes the LINUX_DIST_SUFFIX. The name_DETECT variable is an internal variable, do not change it (if you create a new infrastructure port, copy it from somewhere else and make sure the name in value of the variable matches the port name in the name of the variable). Then there are several name_suffix_FILE variables. Leave the existing ones alone, and add a new one with the correct suffix for your new Linux-release. The value of the variable needs to be an important file which is installed by the infrastructure port in question. FYI: The content of the name_suffix_FILE variables are used to set the name_DETECT variables, depending on the Linux-relase the name_DETECT variables are used to check if the port is already installed. Ideally the name_suffix_FILE variable points to a library in the port. The name_DEPENDS variable lists dependencies of this infrastructure port. If the dependencies changed in your Linux-release, you need to add a conditional to change the dependency if LINUX_DIST_SUFFIX is set to your Linux-release.

Normally this is all what needs to be done in PORTSDIR/Mk/bsd.linux-apps.mk, the rest of the file is code to check dependencies and some correctness checks.

The second tedious part is to actually create all those infrastructure ports. Normally you can copy an existing infrastructure port, rename it, adjust the PORTNAME, PORTVERSION, PORTREVISION, MASTER_SITES, PKGNAMEPREFIX, DISTFILES, CONFLICTS (also in all other Linux-release versions of this infrastructure port), LINUX_DIST_VER, RPMVERSION (if set/neccesary) and SRC_DISTFILE variables, generate the distfile checksums (make makesum), and fix the plist. I suggest to script parts of this work (as of this writing Freshports counts 68 ports where the portname starts with linux-f10-).

Adding new infrastructure ports, or removing infrastructure ports for a given Linux-release

If your Linux-release does not come with a package for an existing infrastructure port, just do not create a corresponding name_suffix_FILE line. You still need to do the right thing regarding dependencies of ports which depend upon this non-existing infrastructure port (if your Linux-release comes with packages for them).

To add a new infrastructure port, copy an existing block, rename the variables, set them correctly, add a new variable for your Linux-release in the first _LINUX_APPS_ALL section, add the content of this variable to _LINUX_APPS_ALL, and change the check-logic as described above.

Final words

If you have something which installs and deinstalls correctly, feel free to provide it on [email protected] for review/testing. If you have questions during the porting, feel also free to send a mail there.

Share

Alexander Leidinger: HOWTO create a new linux_base port

Published on August 29, 2011 in easy case, fedora, FreeBSD, linux base, linux ports, linux release, Linuxulator, minor changes, Open Source, plist, Ports Collection, rpms, Software, suppor, target, Tips & Tricks and Userland. Closed

FreeBSD is in need of a new linux_base port. It is on my TODO list since a long time, but I do not get the time to create one. I still do not have the time to work on a new one, but when you read this, I managed to get the time to create a HOWTO which describes what needs to be done to create a new linux_base port.

I will not describe how to create a new linux_base port from scratch, I will just describe how you can copy the last one and update it to something newer based upon the existing infrastructure for RPM packages.

Specific questions which come up during porting a new Linux release should be asked on freebsd-emulation@Fr[email protected],  there are more people which can answer questions than here in my blog. I will add useful information to this HOWTO if necessary.

In the easy case most of the work is searching the right RPMs and their dependencies to use, and to create the plist.

Why do we need a new linux_base port?

The current linux_base port is based upon Fedora 10, which is end of life since December 2009. Even Fedora 13 is already end of life. Fedora 16 is supposed to be released this year. From a support point of view, Fedora 15 or maybe even Fedora 16 would be a good target for the next linux_base port. Other alternatives would be to use an extended lifetime release of another RPM based distribution, like for example CentOS 6 (which seems to be based upon Fedora 12 with backports from Fedora 13 and 14). Using a Linux release which is told to be supported for at least 10 years, sounds nice from a FreeBSD point of view (only minor changes to the linux ports in such a case, instead of creating a complete new linux_base each N+2 releases like with Fedora), but it also means additional work if you want to create the first linux_base port for it.

The mysteries you have to conquer if you want to create a new linux_base port

What we do not know is, if Fedora 15/16, CentOS 6, or any other Linux release will work in a supported FreeBSD release. There are two ways to find this out.

The first one is to take an existing Linux system, chroot into it (either via NFS or after making a copy into a directory of a FreeBSD system), and to run a lot of programs (acroread, skype, shells, scripts, …). The LTP testsuite is not that much useful here, as it will test mostly kernel features, but we do not know which kernel features are mandatory for a given userland of a Linux release.

The second way of testing if a given Linux release works on FreeBSD is to actually create a new linux_base port for it and test it without chrooting.

The first way is faster, if you are only interested in testing if something works. The second way provides an easy to setup testbed for FreeBSD kernel developers to fix the Linuxulator so that it works with the new linux_base port. Both ways have their merits, but it is up to the person doing the work to decide which way to go.

The meat: HOWTO create a new linux_base port

First off, you need a system (or a jail) without any linux_base port installed. After that you can create a new linux_base port (= lbN), by just making a copy of the latest one (= lbO). In lbN you need to add lbO as a CONFLICT, and in all other existing linux_base ports, you need to add lbN as a conflict.

Change the PORTNAME, PORTVERSION, reset the PORTREVISION in lbN, and set LINUX_DIST_VER  to the new Linux-release version in the lbN Makefile (this is used in PORTSDIR/Mk/bsd.linux-rpm.mk and PORTSDIR/Mk/bsd.linux-apps.mk).

If you do not stay with Fedora, there is some more work to do before you can have a look at chosing RPMs for installation. You need to have a look at PORTSDIR/Mk/bsd.linux-rpm.mk and add some cases for the new LINUX_DIST you want to use. Do not forget to set LINUX_DIST in the lbN Makefile to the name of the distribution you use. You also need to augment the LINUX_DIST_VER check in PORTSDIR/Mk/bsd.linux-rpm.mk with some LINUX_DIST conditionals. If you are lucky, the directory structure for downloads is similar to the Fedora structure, and there is not a lot to do here.

When this is done, you can have a look at the BIN_DISTFILES variable in the lbN Makefile. Try to find similar RPMs for the new Linux release you want to port. Some may not be available, and it may also be the case that different ones are needed instead. I suggest to first work with the ones which are available (make makesum, test install and create plist). After that you need to find out what the replacement RPMs for non-existing ones are. You are on your own here. Search around the net, and/or have a look at the dependencies in the RPMs of lbO to determine if something was added as a dependency of something else or not (if not, forget about it ATM). When you managed to find replacement RPMs, you can now have a look at the dependencies of the RPMs in lbN. Do not add blindly all dependencies, not all are needed in FreeBSD (the linux_base ports are not supposed to create an environment which you can chroot into, they are supposed to augment the FreeBSD system to be able to run Linux programs in ports like they where FreeBSD native programs). What you need in the linux_base ports are libraries, config and data files which do not exist in FreeBSD or have a different syntax than in FreeBSD (those config or data files which are just in a different place, can be symlinked), and basic shell commands (which commands are needed or not… well… good question, in the past we made decisions what to include based upon problem reports from users). Now for the things which are not available and where not added as a dependency. Those are things which are either used during install, or where useful to have in the past. Find out by what it was replaced and have a look if this replacement can easily be used instead. If it can be used, add it. If not, well… bad luck, we (the FreeBSD community) will see how to handle this somehow.

If you think that you have all you need in BIN_DISTFILES, please update SRC_DISTFILES accordingly and generate the distfile via  make –DPACKAGE_BUILDING makesum to have the checksums of the sources (for legal reasons we need them on our mirrors).

The next step is to have a look at REMOVE_DIRS, REMOVE_FILES and ADD_DIRS if something needs to be modified. Most of them are there to fall back to the corresponding FreeBSD directories/files, or because they are not needed at all (REMOVE_*). Do not remove directories from ADD_DIRS, they are created here to fix some edge conditions (I do not remember exactly why we had to add them, and I do not take the time ATM to search in the CVS history).

If you are lucky, this is all (make sure the plist is correct). If you are not lucky and you need to make some modifications to files, have a look at the do-build target in the Makefile, this is the place where some changes are done to create a nice user experience.

If you arrive here while creating a new linux_base port, lean back and feel a bit proud. You managed to create a new linux_base port. It is not very well tested at this moment, and it is far from everything which needs to be done to have the complete Linux infrastructure for a given Linux release, but the most important part is done. Please notify [email protected] and call for testers.

What is missing?

The full Linuxulator infrastructure for the FreeBSD Ports Collection has some more ports around a linux_base port. Most of the infrastructure for this is handled in Mk/bsd.linux-apps.mk.

UPDATE: I got some time to write how to update the Linux-infrastructure ports.

Share

Alexander Leidinger: v4l2 support in the linuxulator now in 8-stable

Published on June 10, 2011 in flash, FreeBSD, kernel, Linuxulator, Open Source, Software, Source, stable and webcams. Closed

I MFCed the v4l2 support in the linuxulator to 8–stable. This allows now to use v4l2–webcams in skype/flash on 8-stable too.

Share

Giorgios Keramidas: Concatenating Video Files with MPlayer

Published on May 20, 2011 in Computers, Free software, FreeBSD, hellug, Linux, Open Source and Software. Closed

I found out how to concatenate the parts of a video to a single file with MPlayer. It’s relatively easy, so this is just a mini-post to save it for posterity:

$ cat video_part1.avi video_part2.avi ... > temp.avi
$ mencoder -forceidx -oac copy -ovc copy \
    temp.avi -o final.avi && \
    rm temp.avi

Filed under: Computers, Free software, FreeBSD, Linux, Open source, Software Tagged: Computers, Free software, FreeBSD, hellug, Linux, Open source, Software