Still, there is some sort of racing condition I cannot reliably
reproduce, but often enough observe. It looks like this:

    $ ./cachedel z && ./nocache cat z > /dev/null && ./cachestats -v z
    pages in cache: 2/16 (12.5%)  [filesize=64.0K, pagesize=4K]

    cache map:
         0: | | | | | | | | | | | | | | |x|x|

Although of course the file should not be cached at all.

Also, fadv_dontneed will do the double syscall now.

There is a promising patch that adds POSIX_FADV_NOREUSE heuristics, see
http://article.gmane.org/gmane.linux.kernel.mm/65564

    > This will mean that there is no way currently to
    > remove a particular file from the cache on linux.

    Correct. There's not a way to do this for a single file (except running
    POSIX_FADV_DONTNEED twice...).

If only one fadvise64 syscall is performed, the *first* previously
unsused page will somehow not be forgotten.

The kernel tries once to lock the page to be freed. If this fails, it won't be
freed. (This happens quite often, actually!)

The strange thing is that subsequent calls will eventually fill the
cache, bit by bit. (This is in the dimension of n < 10 pages per call.
But still I can't figure out where this happens.)