September 2013 - dovecot

[Dovecot] Dovecot never release preallocated space in mdbox
by Stéphane BERTHELOT 22 Sep '13

22 Sep '13

Hello everyone, I am currently evaluating dovecot for our new email production servers (20k+ mailboxes) and found out something strange. I'm using those settings on Dovecot 2.2.4 (x86_64 / Slackware / compiled from sources) mdbox_rotate_size = 128M mdbox_rotate_interval = 1d mdbox_preallocate_space = yes with virtual users and location like : mail_location = mdbox:~/mdbox I don't think the remaining config is relevant but ask me if you need some other parts. Using test accounts for 2 weeks now I've figured that the 128M preallocated space is never 'hole punched" (to use a similar term than "man fallocate" on Linux), even when rotating m.* files. From what I understand those files will never be appended again because of the mdbox_rotate_interval. Then doveadm purge creates new files so old ones would never grow again. Here is an example of a mdbox storage using ls -ls (which shows allocated VS used space) total 4065176 1884 -rw------- 1 mail mail 1926656 Jul 29 10:55 dovecot.map.index 4 -rw------- 1 mail mail 460 Jul 29 11:26 dovecot.map.index.log 48 -rw------- 1 mail mail 44304 Jul 29 10:55 dovecot.map.index.log.2 131072 -rw------- 1 mail mail 133165066 Jul 19 15:31 m.10 131072 -rw------- 1 mail mail 133507393 Jul 19 15:32 m.13 131072 -rw------- 1 mail mail 134155182 Jul 19 15:33 m.14 131072 -rw------- 1 mail mail 134213403 Jul 19 15:30 m.2 131072 -rw------- 1 mail mail 46464 Jul 21 04:30 m.21 131072 -rw------- 1 mail mail 134215030 Jul 19 15:30 m.3 131072 -rw------- 1 mail mail 25852 Jul 25 01:54 m.32 131072 -rw------- 1 mail mail 2360 Jul 26 00:05 m.34 131072 -rw------- 1 mail mail 169073 Jul 27 23:18 m.35 131072 -rw------- 1 mail mail 31624 Jul 27 01:55 m.36 131072 -rw------- 1 mail mail 134216982 Jul 28 04:30 m.37 131076 -rw------- 1 mail mail 134217804 Jul 28 04:30 m.38 131072 -rw------- 1 mail mail 134217341 Jul 28 04:30 m.39 131072 -rw------- 1 mail mail 134213719 Jul 19 15:30 m.4 131072 -rw------- 1 mail mail 29740970 Jul 28 04:30 m.40 131072 -rw------- 1 mail mail 129175917 Jul 28 04:30 m.41 131072 -rw------- 1 mail mail 133174937 Jul 28 04:30 m.42 131072 -rw------- 1 mail mail 633436 Jul 28 04:30 m.43 131072 -rw------- 1 mail mail 3154623 Jul 28 04:30 m.44 131072 -rw------- 1 mail mail 3676879 Jul 28 04:30 m.45 131072 -rw------- 1 mail mail 468158 Jul 28 04:30 m.46 131072 -rw------- 1 mail mail 26964 Jul 28 04:30 m.47 131072 -rw------- 1 mail mail 3574599 Jul 28 04:30 m.48 131072 -rw------- 1 mail mail 3789133 Jul 28 04:30 m.49 131072 -rw------- 1 mail mail 134215016 Jul 19 15:30 m.5 131072 -rw------- 1 mail mail 1280074 Jul 28 04:30 m.50 131076 -rw------- 1 mail mail 635459 Jul 28 22:47 m.51 131072 -rw------- 1 mail mail 1459418 Jul 29 10:55 m.52 131072 -rw------- 1 mail mail 132941013 Jul 29 11:26 m.53 131072 -rw------- 1 mail mail 134213475 Jul 19 15:30 m.7 131072 -rw------- 1 mail mail 132240074 Jul 19 15:31 m.9 There's a lot of "lost" space since preallocated space would only be reclaimed when *all* emails in m.X file have refcount=0 and after a doveadm purge call, if I read well the dovecot docs. On mailboxes patterns with low incoming mail (< 100kb / day) this would waste much space. Of course I can decrease rotate size a lot but it would then produce a lot of files and would certainly become similar performance-wise to sdbox/maildir/... There would certainly be smart to use something similar to "FALLOC_FL_PUNCH_HOLE" on rotation (when doing close() ?) so that when we're sure there won't be anymore data appended to file that the allocated space == used space. I will disable space preallocation for our next tests since it wastes much storage for us ; did you have any feedback on how much it may affect performance ? I found in this ML archives some messages about the implementation but didn't see anyone clearly stating how much better preallocation is. Thanks, best regards, Stephane Berthelot.

4 5

[Dovecot] Deleted mails in dsync backups
by Wolfgang.Friebel＠desy.de 22 Sep '13

22 Sep '13

Hello we are doing regular backups of the mdbox mail folders using dsync -u username backup mdbox:/somedir/BACKUPS/username (still with version 2.1.10, but 2.2.5 is in testing phase) We do observe that the directory size in the BACKUPS directory is growing with respect to the original mail folders. I believe this is due to deleted mails not being purged in the backup. We are doing regular doveadm purge -A cleanups, but this does (of course) not affect the backups and subsequent dsync backup runs seem not to clean up deleted mails in the backup either. Will the behaviour of the dsync backup command deal with deleted mails in the 2.2.5 release? If not, what could we do to get rid of the deleted mails in the backup (or is there another reason for the growth in size) We have e.g. after 9 month of running with 2.1.10 du -sk username 588520 du -sk /somedir/BACKUPS/username 863024 -- Wolfgang Friebel Deutsches Elektronen-Synchrotron DESY

2 1

[Dovecot] Rfc: Limit the size of an email IN the mailbox
by Ralf Hildebrandt 22 Sep '13

22 Sep '13

Currently we're in the process of migrating Dovecot mailboxes to Exchange. This is all working very well, except for the occasional user, who used his/her Drafts folder as a file storage: They started writing an email containing lots of image attachments (and thus exceeding our maximum_message_size by far). So the mail never gets sent, but stays in the Drafts folder. Once I try moving that mail to exchange, it won't accept the message since it's too big. I think it would be interesting to have a limit on the total size of an IMAP object, just like in SMTP. Thought? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

3 2

[Dovecot] INBOX protected foldername?
by Hajo Locke 22 Sep '13

22 Sep '13

Hello, have a little problem with folders with name INBOX created by Users. Some Clients use a Prefix for their Folders. In our old setup (dovecot 1.2) it was possible to create a Folder INBOX in mbox-space with some Subfolders. These Subfolders of INBOX showed up in Mailclient at toplevel. Now we updatet to 2.1.7. In higher dovecot versions INBOX seems to be a kind of protected foldername. dovecot is not offering this folder in folderlist. If i rename it, the folder is available again. There are only a few users, so i could fix this manually. But how to explain this? I did not find a setting etc. which could explain this behaviour. Our Config is pretty standard: mail_location is unchanged some dovecot versions we use: mail_location = mbox:~/mail:INBOX=/var/mail/%u most of conf is out of the box and also used in older versions. only new is namespace inbox, but should also be default: namespace inbox { inbox = yes } Is there an explaination for this behaviour? Thanks, Hajo

4 10

[Dovecot] . (dot) in maildir folder names
by Christoph Anton Mitterer 21 Sep '13

21 Sep '13

Hi. Uhm... yeah as the topic implies I want to have . (dots) in my folder names... Unfortunately dovecot's maildirmake hasn't a -f switch as the one from courier/maildrop, but that one in turn is buggy[0] and doesn't encode any characters at all. In principle, maildir++ should allow[1] having (encoded) dots in folder names,... but I don't quite understand how the encoding actually works, cause UTF-7 seems to be already the encoded (without padding and such)... but . is . in UTF-7 as well. Does dovecot support this? At least my mail clients (well I only tried Evolution so far which sucks in so many areas)... couldn't create it... but I guess it's rather a stupid client issue. Anyone knows how it would be encoded? Interestingly, and this might be actually an inssue in doveocot when I create maildirs with non-ASCII chars without encoding these, e.g. maildir/.München/ (instead of the correctly encoded (.M&APw-nchen) Dovecot seems to "export" these to the clients,... at least Evolution shows it in the list of subscribable folders (even correctly as "München")... but it doesn't allow me to actually subscribe to it. Cheers & thx, Chris. [0] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=709555 [1] http://www.courier-mta.org/maildir.html#id351803

5 24

[Dovecot] Dovecot and CAPITALISATION = grrrr ;-(
by Ben 21 Sep '13

21 Sep '13

Hello, As far as I'm aware, email is case insensitive ? Dovecot however, seems to be taking offence at the fact that someone is sending mail to user(a)SOMEDOMAIN.XYZ. Postfix has no issues handling it, it when it gets passed over to Dovecot LMTP that I get log lines such as : Sep 17 13:48:22 ukc-vm02-mx01 dovecot: auth: passwd-file(user(a)SOMEDOMAIN.XYZ): unknown user However user(a)somedomain.xyz *DOES* exist in the passwd file. Just not with a capitalised domain ! I am confused. Ben

5 6

[Dovecot] mail-filter plugin v2.2 - corrupts IMAP indexes?
by Charles Cazabon 21 Sep '13

21 Sep '13

Hi, I'm trying out the mail-filter plugin (v2.2, with Dovecot 2.2.5 built from source), and it seems like any nontrivial filter causes some sort of index corruption that breaks IMAP use. i.e. using a null filter (basically `cat -`) for mail-filter.sh and mail-filter-out.sh appears to work fine, and the debug logging shows it's using the filters (though it reports the wrong Unix socket for one of them - not sure if that's an actual bug or just a copy&paste error where it logs it; there's a similar accidental use of the same Unix socket path in the readme for the plugin): Sep 15 14:24:03 imap: Debug: Loading modules from directory: .../dovecot/lib/dovecot Sep 15 14:24:03 imap: Debug: Module loaded: .../dovecot/lib/dovecot/mail_filter_plugin.so Sep 15 14:24:03 imap(charlesc): Debug: Effective uid=1000, gid=1000, home=/home/charlesc Sep 15 14:24:03 imap(charlesc): Debug: mail_filter: Filtering mail_filter via socket .../dovecot/var/run/dovecot/mail-filter Sep 15 14:24:03 imap(charlesc): Debug: mail_filter: Filtering mail_filter_out via socket .../dovecot/var/run/dovecot/mail-filter If I use a slightly more complex pair of filter scripts that just swap the case of all alpha chars in the message body (not headers), so this pair of filters results in the same output as the original message, something seems to get corrupted in the indexes, and the IMAP process fails to respond correctly to FETCH requests, and logs this: Sep 15 14:24:03 imap(charlesc): Debug: maildir++: root=.../Mail-test, index=, indexpvt=, control=, inbox=.../Mail-test, alt= Sep 15 14:24:03 imap(charlesc): Error: Cached message size smaller than expected (0 < 1092) Sep 15 14:24:03 imap(charlesc): Error: Maildir filename has wrong S value, renamed the file from .../Mail-test/new/1379276625.M25480P28541.ludmilla,S=0,W=1122 to .../Mail-test/new/1379276625.M25480P28541.ludmilla,S=1092 Sep 15 14:24:03 imap(charlesc): Error: Corrupted index cache file .../Mail-test/dovecot.index.cache: Broken physical size for mail UID 5 Sep 15 14:24:03 imap(charlesc): Error: read(BODY[]) failed: Invalid argument (FETCH for mailbox INBOX UID 5) Sep 15 14:24:03 imap(charlesc): Info: Disconnected: FETCH failed in=105 out=1176 It reports the size of the message changes, too, which is odd, as running the filters manually confirms the output is identical to the input. mail-filter.sh attached; mail-filter-out.sh is identical except using a different temp file name to avoid clashes. Am I doing something wrong? How can I prevent this "Corrupted index cache file" problem? Thanks, Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ -----------------------------------------------------------------------

1 1

[Dovecot] SELECT: Bug in untagged response
by Michael M Slusarz 21 Sep '13

21 Sep '13

Dovecot 2.2.5 In selected state: C: 1 SELECT &#-&#/# S: 1 NO Mailbox name is not valid mUTF-7 S: * OK [CLOSED] Previous mailbox closed. I can verify that this only seems to happen when the mailbox name is incorrect mUTF-7. Things work fine if I access a non-existent mailbox: C: 1 SELECT non-existent-mailbox S: * OK [CLOSED] Previous mailbox closed. S: a NO Mailbox doesn't exist: non-existent-mailbox FYI: Switching to Dave C's workaround discussed here -- http://mailman2.u.washington.edu/pipermail/imap-protocol/2005-September/000… -- regarding servers that don't support UNSELECT. This would normally never happen on Dovecot since it supports UNSELECT; I'm intentionally ignoring that capability for testing purposes. I'm not pipelining a CLOSE with the above SELECT command, like Dave suggests, because the likelihood this SELECT will be successful is about zero and I'd rather avoid a BAD response to a CLOSE command, since it is plausible that a server could issue a CLIENTBUG response for a client that is trying to CLOSE a mailbox in Authenticated State. (Granted, a server could also plausibly issue a CLIENTBUG for identifying a mailbox in incorrect mUTF-7 also, but whatever.) michael

2 1

[Dovecot] Mountpoints with auto mounter
by Steffen Kram 21 Sep '13

21 Sep '13

Hi, I'm running dovecot on a OmniOS installation and I'm wondering what I can do about all those mount point warnings in my logfile. The problem occurs because of the running auto-mounter which manages my mail directories. Isn't it possible for dovecot just try to access the directories before logging an error message. Shall I remove the corresponding mount points from the list of tracked mount points with the doveadm mount-command? The documentation does say nothing about situation with an active auto mounter. Thanks for any suggestions. Cheers, Steffen

3 4

[Dovecot] zlib_save per-user or per-mailbox?
by /dev/rob0 21 Sep '13

21 Sep '13

We'd like to be able to activate zlib_save per-user or per-mailbox, but it seems to be global, all or nothing. Search of this list revealed a comment from Timo in 2012: http://www.dovecot.org/list/dovecot/2012-March/064909.html where he was thinking that compression per-namespace would be a worthy feature. Was that done? I'm in the process of replacing a 2.0 system with 2.2 EE. The old system had zlib_save activated, but seems to deliver to maildirs without compression. Does a userdb_mail entry override global zlib_save? (I'm about to test that, but duh, I am asking first.) Thanks. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:

2 2