[Dovecot] minimize mbox mdbox fragmentation

[Timo Sirainen]

On Wed, 2010-10-20 at 23:50 -0500, Stan Hoeppner wrote:
> > Oh, interesting. I didn't know that was possible. And even better: Linux
> > has fallocate() that can do it for other filesystems than just XFS. Or
> > looks like it's only XFS and ext4 (ext3 doesn't support it). I don't
> > know if other OSes support this. 
> There exists posix_fallocate() which would widen the platforms that
> would support this Timo. 

I know about posix_fallocate(), but that does a different thing. It
would work only when this works:

> Maybe in future I could make mdbox
> > support writing to files whose size has been preallocated by actually
> > writing NUL bytes, but that requires some extra code.

>  You may also want to look at posix_fadvise()
> as well (if you're not using it already) which might increase Dovecot's
> overall disk performance a bit.

I once wrote a patch for that and asked if anyone could confirm if it
made things worse or better, but no one ever did.
http://dovecot.org/list/dovecot/2007-November/026819.html

> NOTE: I don't believe fallocate() in either posix or linux only form
> will actually accomplish decreased m[d]box file fragmentation.  I don't
> believe it actually increases the file size on disk, i.e. physically
> allocating additional free extents tailing the end of the file.
> fallocate() is _speculative_ preallocation, which isn't what you want.

Why do you believe that? The man pages and everything tell me that it's
about physical preallocation:

       fallocate  is  used  to  preallocate blocks to a file.  For filesystems
       which support the fallocate system call, this is done quickly by  allo‐
       cating blocks and marking them as uninitialized, requiring no IO to the
       data blocks.  This is much faster than creating a file  by  filling  it
       with zeros.

Also "du" shows that the file's size is actually the preallocated size.
That's how I checked which filesystems supported it.

And I think Linux fallocate() with XFS internally does exactly what
XFS_IOC_RESVSP does, which your quoted mail mentioned being about
physical preallocation.

> > > I don't know if it changes anything in the sequence above, but Dovecot
> > > uses mmap i/o. 

mmap is used only for index files, not for mdbox files.

> It would be beneficial I think if you'd sub to the xfs list Timo and
> pick some brains.

Maybe, but there is still so much other stuff to do, like bug fixing. :)