ivoras’ FreeBSD blog

June 8, 2008

Weekend hack: adfsd, a kqueue-assisted rsync tool

Filed under: FreeBSD — Tags: , , , — ivoras @ 10:44 pm

I created a small program to help me synchronize files in sort-of real-time between two directories (the idea is that one of them is on a NFS server). There are no replicated file systems for FreeBSD and the canonical way to do this is usually to use rsync or something like it. The problem here is that rsync always traverses both directory structures, compares files and then copies them (via a variety of smart algorithms but it’s still very resource-intensive).

So I created a daemon that uses kqueue(2) to monitor which files changed and feed only those files to rsync (it’s not exactly a new idea, I’m sure somebody has also mentioned it somewhere on the FreeBSD lists). This is in many ways a suboptimal solution since it needs to keep an open file descriptor for all the monitored files (which ties in kernel resources and memory) so it won’t scale for really large directories, which could actually benefit the most from this approach. It will work reasonably well for a small number of files (up to several tens of thousands), with modifying kern.maxfiles and kern.maxfilesperproc sysctls and login session limits (if applicable).

Anyone who’s interested can download the adfs daemon and try it. This was hacked together over the weekend so it probably has some problems. I’ll fix those problems that prevent me from using it, but I’ll update the online archive only if there are interested users.

11 Comments

  1. There are 2 problems with using it/
    1. Core dumping when I’m dellete file from source direcory.
    2. If I’m put big file in source directory, adsf sync this file while it do not copy/

    Sorry for my english. )

    Comment by Oleg Pupken — June 9, 2008 @ 2:51 pm

  2. I cannot replicate problems with deleting files but I can for ‘big’ files. I’ll look into it.

    Comment by ivoras — June 13, 2008 @ 11:20 pm

  3. What constitutes a large file?

    Comment by Adam — October 24, 2008 @ 6:15 pm

  4. What constitutes a “large” file? Are we talking 10mb, 100mb, or gigs?

    Comment by Adam — October 24, 2008 @ 6:16 pm

  5. 10 MB. The whole approach of using kqueues for this appears to be flawed since kqueues coalesces requests. I don’t think adfs should be used at all – something completely different will need to be implemented.

    Comment by ivoras — October 24, 2008 @ 8:30 pm

  6. make
    gcc -o adfsd -Wall -g adfsd.c
    adfsd.c:33:22: sys/hash.h: No such file or directory
    adfsd.c:93: warning: ‘debug_level’ defined but not used
    adfsd.c:101: warning: ‘stat_to_mini_stat’ defined but not used
    *** Error code 1

    Stop in /var/ports/distfiles/adfs/adfs.

    Comment by Max — November 27, 2008 @ 2:33 pm

  7. hash.h is in /usr/include/sys – it should be picked up automatically on FreeBSD 6+. Sorry about the warnings, just ignore them.

    Comment by ivoras — November 27, 2008 @ 2:58 pm

  8. Thanks, Ivoras!

    I have no hash.h in /usr/include/sys but fnv_hash.h, so, i use it.
    Error happens in rsync, my rsync want option “–delete” instead of “–del”, i correct that and remake.
    Next error happens wneh i try to delete file:

    adfsd[83300]: Change detected in: adfsd.core
    adfsd[83300]: Change detected in:
    adfsd[83300]: Sent to rsync: adfsd.core
    rsync: link_stat “/var/ports/distfiles/adfs/adfs/source/adfsd.core” failed: No such file or directory (2)
    rsync error: some files could not be transferred (code 23) at main.c(702)
    adfsd[83300]: Sent 1 files to rsync
    adfsd[83300]: Partial rsync terminated with error 23
    adfsd[83300]: Sent to rsync:
    adfsd in free(): error: chunk is already free
    Abort (core dumped)

    Comment by Max — November 28, 2008 @ 11:09 am

  9. Your hash.h problem is very strange – it is certainly present: http://svn.freebsd.org/viewvc/base/head/sys/sys/ .

    From rsync(1):

    –del an alias for –delete-during
    –delete delete extraneous files from dest dirs
    –delete-during receiver deletes during xfer, not before

    The “–delete-during” should also work, but it might also cause your error with deleting the file. It looks like rsync doesn’t wont to sync a “non-existant” file. Maybe your version of rsync doesn’t support it?

    Anyway, see comment #5 :)

    Comment by ivoras — November 28, 2008 @ 11:35 am

  10. i updated rsync to 3.0.4, in which command “–del” present.
    revert in adfsd.c “–delete” to “–del”, recompile it

    but error still happens when i try to delete file ..

    I even download hash.h and recomplie ..

    Comment by Max — November 28, 2008 @ 11:56 am

  11. [...] – bookmarked by 2 members originally found by guevarani on 2008-10-21 Weekend hack: adfsd, a kqueue-assisted rsync tool http://blogs.freebsdish.org/ivoras/2008/06/08/weekend-hack-adfsd-a-kqueue-assisted-rsync-tool/ – [...]

    Pingback by Bookmarks about Rsync — November 30, 2008 @ 7:45 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress