[Dovecot] Dovecot never release preallocated space in mdbox

[Stéphane BERTHELOT]

Le 02/08/2013 14:30, Timo Sirainen a écrit :
> On Mon, 2013-07-29 at 11:48 +0200, Stéphane BERTHELOT wrote:
>
>> mdbox_rotate_size = 128M
>> mdbox_rotate_interval = 1d
>> mdbox_preallocate_space = yes
>> with virtual users and location like :
>> mail_location = mdbox:~/mdbox
>>
>> I don't think the remaining config is relevant but ask me if you need
>> some other parts.
>>
>> Using test accounts for 2 weeks now I've figured that the 128M
>> preallocated space is never 'hole punched" (to use a similar term than
>> "man fallocate" on Linux), even when rotating m.* files.
> Yeah, those settings weren't really intended to be used together.
I am not sure to understand. Preallocation is based on rotate size 
actually, so you mean I shouldn't use a rotate interval with this kind 
of settings ?
It would effectively "lose" less space since all files would be around 
the preallocated size except the last one (current).
>> There would certainly be smart to use something similar to
>> "FALLOC_FL_PUNCH_HOLE" on rotation (when doing close() ?) so that when
>> we're sure there won't be anymore data appended to file that the
>> allocated space == used space.
> The problem is that there is no "rotation time". In normal operation
> Dovecot would be wasting time (=disk IO) looking at old files and
> figuring out if they would need hole punching. I guess the doveadm purge
> job could do that, but I'm not sure if that's always the best idea
> either. I remember some people using different mdbox settings for normal
> operations and for doveadm purge runs, so this could also
> unintentionally break things..
Ok, I'm sorry I forgot to add that I'm using a cronjob to doveadm purge 
weekly. That effectively may be a good time to remove unneeded 
preallocated sectors.
I didn't have a look at the internals of the devecot process but I meant 
that this could happen when it chooses to switch to another file 
(because of rotation time, not purge). Effectively it may be at a time 
that we would want to avoid adding more IO...

When reading linux docs lately the more appropriate term may be 
"truncation" rather than "hole punching". That's what I did on command 
line (truncate with proper args) and it resetted the allocated size to 
"real" size.

I am pretty sure my understanding on those options was wrong so I 
already changed to 2M/no preallocation (since I didn't find any evidence 
that preallocation is making dovecot perform so much better ; there 
seemed to be a thread once on this list but noone concluded on that I 
think).
My objective was to get a good compromise on performance (since I'm 
using ext4 anyways) and backup time (otherwise I wouldn't bother with 
rotation to "compensate" the potentially full rebuild of doveadm purge)

Maybe a documentation update should do the trick or a configuration 
warning stating that preallocation+rotate time would leave "wasted" 
space on disk.