[Dovecot] Dovecot never release preallocated space in mdbox
Hello everyone,
I am currently evaluating dovecot for our new email production servers (20k+ mailboxes) and found out something strange.
I'm using those settings on Dovecot 2.2.4 (x86_64 / Slackware / compiled from sources)
mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes with virtual users and location like : mail_location = mdbox:~/mdbox
I don't think the remaining config is relevant but ask me if you need some other parts.
Using test accounts for 2 weeks now I've figured that the 128M preallocated space is never 'hole punched" (to use a similar term than "man fallocate" on Linux), even when rotating m.* files. From what I understand those files will never be appended again because of the mdbox_rotate_interval. Then doveadm purge creates new files so old ones would never grow again.
Here is an example of a mdbox storage using ls -ls (which shows allocated VS used space) total 4065176 1884 -rw------- 1 mail mail 1926656 Jul 29 10:55 dovecot.map.index 4 -rw------- 1 mail mail 460 Jul 29 11:26 dovecot.map.index.log 48 -rw------- 1 mail mail 44304 Jul 29 10:55 dovecot.map.index.log.2 131072 -rw------- 1 mail mail 133165066 Jul 19 15:31 m.10 131072 -rw------- 1 mail mail 133507393 Jul 19 15:32 m.13 131072 -rw------- 1 mail mail 134155182 Jul 19 15:33 m.14 131072 -rw------- 1 mail mail 134213403 Jul 19 15:30 m.2 131072 -rw------- 1 mail mail 46464 Jul 21 04:30 m.21 131072 -rw------- 1 mail mail 134215030 Jul 19 15:30 m.3 131072 -rw------- 1 mail mail 25852 Jul 25 01:54 m.32 131072 -rw------- 1 mail mail 2360 Jul 26 00:05 m.34 131072 -rw------- 1 mail mail 169073 Jul 27 23:18 m.35 131072 -rw------- 1 mail mail 31624 Jul 27 01:55 m.36 131072 -rw------- 1 mail mail 134216982 Jul 28 04:30 m.37 131076 -rw------- 1 mail mail 134217804 Jul 28 04:30 m.38 131072 -rw------- 1 mail mail 134217341 Jul 28 04:30 m.39 131072 -rw------- 1 mail mail 134213719 Jul 19 15:30 m.4 131072 -rw------- 1 mail mail 29740970 Jul 28 04:30 m.40 131072 -rw------- 1 mail mail 129175917 Jul 28 04:30 m.41 131072 -rw------- 1 mail mail 133174937 Jul 28 04:30 m.42 131072 -rw------- 1 mail mail 633436 Jul 28 04:30 m.43 131072 -rw------- 1 mail mail 3154623 Jul 28 04:30 m.44 131072 -rw------- 1 mail mail 3676879 Jul 28 04:30 m.45 131072 -rw------- 1 mail mail 468158 Jul 28 04:30 m.46 131072 -rw------- 1 mail mail 26964 Jul 28 04:30 m.47 131072 -rw------- 1 mail mail 3574599 Jul 28 04:30 m.48 131072 -rw------- 1 mail mail 3789133 Jul 28 04:30 m.49 131072 -rw------- 1 mail mail 134215016 Jul 19 15:30 m.5 131072 -rw------- 1 mail mail 1280074 Jul 28 04:30 m.50 131076 -rw------- 1 mail mail 635459 Jul 28 22:47 m.51 131072 -rw------- 1 mail mail 1459418 Jul 29 10:55 m.52 131072 -rw------- 1 mail mail 132941013 Jul 29 11:26 m.53 131072 -rw------- 1 mail mail 134213475 Jul 19 15:30 m.7 131072 -rw------- 1 mail mail 132240074 Jul 19 15:31 m.9
There's a lot of "lost" space since preallocated space would only be reclaimed when *all* emails in m.X file have refcount=0 and after a doveadm purge call, if I read well the dovecot docs.
On mailboxes patterns with low incoming mail (< 100kb / day) this would waste much space. Of course I can decrease rotate size a lot but it would then produce a lot of files and would certainly become similar performance-wise to sdbox/maildir/...
There would certainly be smart to use something similar to "FALLOC_FL_PUNCH_HOLE" on rotation (when doing close() ?) so that when we're sure there won't be anymore data appended to file that the allocated space == used space.
I will disable space preallocation for our next tests since it wastes much storage for us ; did you have any feedback on how much it may affect performance ? I found in this ML archives some messages about the implementation but didn't see anyone clearly stating how much better preallocation is.
Thanks, best regards, Stephane Berthelot.
On Mon, Jul 29, 2013 at 11:48:00AM +0200, Stéphane BERTHELOT wrote:
mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes
On mailboxes patterns with low incoming mail (< 100kb / day) this would waste much space. Of course I can decrease rotate size a lot but it would then produce a lot of files and would certainly become similar performance-wise to sdbox/maildir/...
128MB is quite a large rotate size if you care about disk space.. We use the default 2 MB, which still packs quite a lot of messages per file compared to maildir. Single maildir-files seems to be around 5-30KB (compressed), which should amount to 50-400 messages per m-file. I don't think that should be similar to maildir/sdbox performance-wise.
-jf
On Mon, 2013-07-29 at 11:48 +0200, Stéphane BERTHELOT wrote:
mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes with virtual users and location like : mail_location = mdbox:~/mdbox
I don't think the remaining config is relevant but ask me if you need some other parts.
Using test accounts for 2 weeks now I've figured that the 128M preallocated space is never 'hole punched" (to use a similar term than "man fallocate" on Linux), even when rotating m.* files.
Yeah, those settings weren't really intended to be used together.
There would certainly be smart to use something similar to "FALLOC_FL_PUNCH_HOLE" on rotation (when doing close() ?) so that when we're sure there won't be anymore data appended to file that the allocated space == used space.
The problem is that there is no "rotation time". In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things..
On Mon, 2013-07-29 at 11:48 +0200, Stéphane BERTHELOT wrote:
mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes with virtual users and location like : mail_location = mdbox:~/mdbox
I don't think the remaining config is relevant but ask me if you need some other parts.
Using test accounts for 2 weeks now I've figured that the 128M preallocated space is never 'hole punched" (to use a similar term than "man fallocate" on Linux), even when rotating m.* files. Yeah, those settings weren't really intended to be used together. I am not sure to understand. Preallocation is based on rotate size actually, so you mean I shouldn't use a rotate interval with this kind of settings ? It would effectively "lose" less space since all files would be around
There would certainly be smart to use something similar to "FALLOC_FL_PUNCH_HOLE" on rotation (when doing close() ?) so that when we're sure there won't be anymore data appended to file that the allocated space == used space. The problem is that there is no "rotation time". In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things.. Ok, I'm sorry I forgot to add that I'm using a cronjob to doveadm purge weekly. That effectively may be a good time to remove unneeded
Le 02/08/2013 14:30, Timo Sirainen a écrit : the preallocated size except the last one (current). preallocated sectors. I didn't have a look at the internals of the devecot process but I meant that this could happen when it chooses to switch to another file (because of rotation time, not purge). Effectively it may be at a time that we would want to avoid adding more IO...
When reading linux docs lately the more appropriate term may be "truncation" rather than "hole punching". That's what I did on command line (truncate with proper args) and it resetted the allocated size to "real" size.
I am pretty sure my understanding on those options was wrong so I already changed to 2M/no preallocation (since I didn't find any evidence that preallocation is making dovecot perform so much better ; there seemed to be a thread once on this list but noone concluded on that I think). My objective was to get a good compromise on performance (since I'm using ext4 anyways) and backup time (otherwise I wouldn't bother with rotation to "compensate" the potentially full rebuild of doveadm purge)
Maybe a documentation update should do the trick or a configuration warning stating that preallocation+rotate time would leave "wasted" space on disk.
On 2013-08-02 8:30 AM, Timo Sirainen tss@iki.fi wrote:
The problem is that there is no "rotation time". In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things.
I'm considering migrating my users to mdbox, and thought that I had sent a follow-up to this, but found it in my Drafts folder...
This does give me a little pause for concern...
Shouldn't there be an internal, automatic mechanism in place for compacting (this I think is the more appropriate term, since it is what Thunderbird and Outlook both use, although it is used for local client files, as opposed to server based files) these files?
Personally I'd like it to happen immediately, when a user deletes a(some) message(es). Maybe add a new setting for 'mdbox_compact_trigger' or something like that, where it only does it if it will reclaim at least ## amount of space - although checking for that in and of itself might be as much or more work as just doing it immediately when the email(s) is(are) deleted.
But, even if it wasn't fully automatic, maybe simplest would be just another option to pass to doveadm purge (ie, -C for 'compact)?
Also - what happens currently when *all* messages in any given mdbox file are deleted? Is the mdbox file deleted?
Thanks,
--
Best regards,
*/Charles/*
On 23.8.2013, at 15.32, Charles Marcus CMarcus@Media-Brokers.com wrote:
On 2013-08-02 8:30 AM, Timo Sirainen tss@iki.fi wrote:
The problem is that there is no "rotation time". In normal operation Dovecot would be wasting time (=disk IO) looking at old files and figuring out if they would need hole punching. I guess the doveadm purge job could do that, but I'm not sure if that's always the best idea either. I remember some people using different mdbox settings for normal operations and for doveadm purge runs, so this could also unintentionally break things.
I'm considering migrating my users to mdbox, and thought that I had sent a follow-up to this, but found it in my Drafts folder...
This does give me a little pause for concern...
Shouldn't there be an internal, automatic mechanism in place for compacting (this I think is the more appropriate term, since it is what Thunderbird and Outlook both use, although it is used for local client files, as opposed to server based files) these files?
Personally I'd like it to happen immediately, when a user deletes a(some) message(es). Maybe add a new setting for 'mdbox_compact_trigger' or something like that, where it only does it if it will reclaim at least ## amount of space - although checking for that in and of itself might be as much or more work as just doing it immediately when the email(s) is(are) deleted.
There used to be some automatic purging done if enough mails were deleted, but that implementation made the performance much worse so I removed it. Some other better working automation could be added I guess.
But, even if it wasn't fully automatic, maybe simplest would be just another option to pass to doveadm purge (ie, -C for 'compact)?
You mean for deleting the preallocated space? I guess it could look at the mdbox_rotate_interval and mdbox_preallocate_space settings and based on that remove the preallocated space from the old files.. But I'm still not convinced it's useful to combine those settings in any case, so doesn't seem like worth the trouble to implement it.
Also - what happens currently when *all* messages in any given mdbox file are deleted? Is the mdbox file deleted?
When any messages are deleted from mdbox, the non-deleted messages are moved to another mdbox file and the old mdbox file is deleted.
participants (4)
-
Charles Marcus
-
Jan-Frode Myklebust
-
Stéphane BERTHELOT
-
Timo Sirainen