Zhao Shuai's blog

In the last two weeks I was working on the performance test for the new FIFO system. I wrote a benchmark program to test FIFO throughput, and the details are as followes.

testing method

The benchmark program employes the producer/consumer model to test the throughput of the FIFO system. In this model, producer sends messages and consumer receives messages using FIFO. The elapsed time will be caculated. Less time we use, more throughput we have.

The test is executed 100 times for all combinations of the following parameters for message size and number of messages transmitted. The avarage time cost is caculated and then printed.

- Message size(bytes): 64, 128, 256, 512, 1024, 2048, 4096
- Number of messages : 100, 1000, 5000, 10000, 100000

results

The performance test was carried out on a Dell Optiplex 960 box with Intel Core 2 Quad 9400 processor, 4G RAM, and 320G hard disk. FreeBSD 7.0-RELEASE is installed on this machine but the kernel has been updated to 8.0-CURRENT.

Here is the result obtained under the new fifo implementation (pipe version):

100     1000     5000    10000    100000

64B    274us    148us    418us    753us    8909us
128B    89us     216us    769us    1545us   17729us
256B    106us    353us    1530us   3700us   35265us
512B    138us    595us    3709us   7310us   70396us
1024B   184us    1183us   7300us   14103us  141104us
2048B   295us    2696us   13929us  29296us  309294us
4096B   502us    5988us   29094us  58351us  583801us

Here is the reuslt obtained under the old fifo implementation (socket version):

100          1000         5000          10000     100000

64B     85us    175us    584us    1126us    11905us
128B    96us    280us    1128us    2171us    23427us
256B    119us    484us    2170us    4559us    53267us
512B    159us    891us    4645us    9276us    109152us
1024B   239us    1732us    9554us    18825us    222931us
2048B 403us    3552us    18939us    38081us    451727us
4096B 723us    7546us    39151us    83753us    905902us

From these results, we can see that the new fifo implementation gains little advantage over the old fifo implementation when transfering small amount of data. e.g. handling 100*128 bytes of data costs 89 microseconds under the new fifo code while it costs 96 microseconds under the old fifo code. But when transferring large amount of data, the new fifo implementation gains significant advantage over the old fifo implementation. e.g. it takes 1126 microseconds handling 10000*64 bytes of data under the old fifo implementation, but under the new fifo implementation, it takes 753 microseconds, saving 30% of time. That is to say, the new fifo system increases the throughput by 30% in this case.

The functionality tests of my FIFO code is coming to an end. You can found these testing programs in my perforce branch. Here is the testing results.

Each of the fifo core functions is tested separately, including read/write/select/poll/kqueue/sigio. All works well except the one testing fifo status change.(See //depot/projects/soc2009/fifo/fifo_tests/functionality/hangup.c). I haven’t fixed this problem yet.

Special attention is put to the previously reported FIFO bugs. They are PR 74242, 76525, 94772, 76144, 116770.

• kern/74242: it’s an issue of mknod. There is some disagreement about whether mknod(S_FIFO) should work for unprivileged users. According to the bug tracker, our developers has come to an agreement and now we allow unpriviledged users creating fifo using mknod.
• kern/76525, kern/94772, kern/76144: they are of the same problem. If you close one endpoint of fifo, select/poll on the other endpoint does not return. The problem is caused by the socket code, which does not handle POLLHUP. FIFO suffers from this because it uses socket code. But pipe actually implements POLLHUP. When we switch FIFO to use pipe code, those bugs disappear.
• kern/116770: fifo with no writers is opened nonblockingly for reading and the resulting file descriptor is added to a kqueue, kevent will immediately report an EOF condition. There are some arguments about whether it’s really a bug. Under my FIFO implementation, we do not suffer from this problem.

Thanks John for pointing out that there are official regression test for FIFO. The regression tests covers more fields and are very easy to use. All the tests are passed under my FIFO implementation.

After fixing an annoying bug of the FIFO code, I am pleased to say that the new FIFO implementation is ready.

What I have done till now:

(1) create a general layout of pipe code. The original pipe code is moved into kern/subr_pipe.c where we deal with pipe internals. Most of the pipe code is kept untouched except some changes to several function prototype. All the pipe routines that are exposed to the outside world come up with a prefix generic_pipe_*.

(2) use those general pipe routines to rewrite pipe and FIFO.

(3) write functionality tests for the new FIFO system.  Currently  read()/write()/select()/poll()/kqueue() is tested and the result shows that the new FIFO system works well. More tests will be added next week to examine ioctl()/stat()/sigio(). I put special attention to the previously reported FIFO bugs, and am happy to see that a well known FIFO bug (see PR 76525, 76144, 94772) is eliminated.

This week I am re-writing the FIFO system. It is not tough work, since VFS has provided very clear interface. The new FIFO implementation is almost there. Any comments or feedback is welcome.

Some testing program have been written, you can find them in my perforce branch: //depot/projects/soc2009/fifo/fifo_test/functionality/…

Test shows that the new FIFO system works well in most cases. But I also observed that the new FIFO system behaves differently under some circumstances comparing with the old FIFO system. It seems that this is caused by the difference between pipe and socket, so it looks like something a feature rather than a bug. I am not sure about that, and will look into those next week.

This week I am busy with school exams and wasn’t able to work on my project until the weekend.

During the weekend I did some cleanup in the pipe code and began to implement FIFO. All goes well until now.

I am working on a general layout of pipe this week. A new file (subr_pipe.c)is added and it contains the code that can be shared between pipe and FIFO.  My original idea is that subr_pipe.c has a clear separation with sys_pipe.c. The former deals with pipe internals and provides routines for sys_pipe.c(also for FIFO in the future);it doesn’t care about the outside world(VFS layer). The latter provides VFS interfaces without knowing the actual pipe implementation. This involves re-defining some function prototypes and some code rewriting.  But I find it hard to achieve this goal due to the bidirectional feature of pipe. This feature makes it hard to define a  clear boundary between subr_pipe.c and sys_pipe.c. Move the pipe routine into subr_pipe.c and keep it as untouched as possible is not a bad idea and this is my solution.

This work is almost done and I will do some cleanup and test next week. Also I have to re-think about how to implements FIFO using pipe code. Last week my mini version of FIFO works, but it lose some features(e.g. bidirectional) and there are also bugs in it.

This week I also spend a few time learning the kqueue mechanism. Thanks to Jonathan. His paper “Kqueue: A generic and scalable event notification facility” is really good material on this subject.

Coding started

This week I work on a mini version of FIFO(committed in my branch, CHANGE 162355). It does nothing but redirects its operation to pipe code. I have tested it using several simple programs and it works. It supports normal read/write but does not support select/poll/kqueue.

In order to make it work, some changes are made in the pipe code but they are trivial. Next week I will split the pipe routines that can be shared by FIFO into a separate file and rewrite sys_pipe.c using these routines.

This week I found myself know little about kqueue mechanism, next week I will also learn how it works.