[Dovecot] minimize mbox mdbox fragmentation

[Stan Hoeppner]

Timo Sirainen put forth on 10/21/2010 9:52 AM:
> On Wed, 2010-10-20 at 23:50 -0500, Stan Hoeppner wrote:
>>> Oh, interesting. I didn't know that was possible. And even better: Linux
>>> has fallocate() that can do it for other filesystems than just XFS. Or
>>> looks like it's only XFS and ext4 (ext3 doesn't support it). I don't
>>> know if other OSes support this. 
>> There exists posix_fallocate() which would widen the platforms that
>> would support this Timo. 
> 
> I know about posix_fallocate(), but that does a different thing. It
> would work only when this works:

Ahh, ok, didn't realize that it was different than the Linux specific call.

>> Maybe in future I could make mdbox
>>> support writing to files whose size has been preallocated by actually
>>> writing NUL bytes, but that requires some extra code.
> 
>>  You may also want to look at posix_fadvise()
>> as well (if you're not using it already) which might increase Dovecot's
>> overall disk performance a bit.
> 
> I once wrote a patch for that and asked if anyone could confirm if it
> made things worse or better, but no one ever did.
> http://dovecot.org/list/dovecot/2007-November/026819.html

I wasn't aware of this.  I don't really have the resources, skill set,
or time to test patches, or I'd do so. :(

>> NOTE: I don't believe fallocate() in either posix or linux only form
>> will actually accomplish decreased m[d]box file fragmentation.  I don't
>> believe it actually increases the file size on disk, i.e. physically
>> allocating additional free extents tailing the end of the file.
>> fallocate() is _speculative_ preallocation, which isn't what you want.
> 
> Why do you believe that? The man pages and everything tell me that it's
> about physical preallocation:
> 
>        fallocate  is  used  to  preallocate blocks to a file.  For filesystems
>        which support the fallocate system call, this is done quickly by  allo‐
>        cating blocks and marking them as uninitialized, requiring no IO to the
>        data blocks.  This is much faster than creating a file  by  filling  it
>        with zeros.
> 
> Also "du" shows that the file's size is actually the preallocated size.
> That's how I checked which filesystems supported it.
> 
> And I think Linux fallocate() with XFS internally does exactly what
> XFS_IOC_RESVSP does, which your quoted mail mentioned being about
> physical preallocation.

Well that's cool then.

>>>> I don't know if it changes anything in the sequence above, but Dovecot
>>>> uses mmap i/o. 
> 
> mmap is used only for index files, not for mdbox files.

Ahh, that's really good to know.  IIRC the comments for enabling mmap in
dovecot.conf don't state this.  I thus assumed enabling mmap was for all
files.

>> It would be beneficial I think if you'd sub to the xfs list Timo and
>> pick some brains.
> 
> Maybe, but there is still so much other stuff to do, like bug fixing. :)

Looks like you've got a handle on it.  Like I said, I know just enough
about this stuff to make me dangerous. ;)

-- 
Stan