October 2012 - dovecot

[Dovecot] When are search indexes updated?
by David Abrahams 06 Dec '12

06 Dec '12

I noticed that occasionally searching in my huge archive mailbox can be really slow, so I tried doveadm index on it and it seemed to do a lot of work, which seemed strange given, for example, that dovecot-lda says it keeps Dovecot index files up-to-date. Then I thought, "maybe these are different files than the search indices." If so, that's not entirely clear from the docs and Wiki. So, questions: * When are search indexes updated? * Are they updated incrementally? * If not, why not? * If so, why would a mailbox's index drift out-of-date, as mine had? BTW, I'm using the clucene search backend. -- Dave Abrahams BoostPro Computing Software Development Training http://www.boostpro.com Clang/LLVM/EDG Compilers C++ Boost

6 16

[Dovecot] Rebuilding indexes fails on inconsistent mdbox
by Milan Holzäpfel 27 Nov '12

27 Nov '12

Hello all, I have a problem with an incosistent mdbox: Oct 24 10:43:23 two dovecot: imap-login: Login: user=<...>, method=PLAIN, rip=..., lip=..., mpid=4977, TLS Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Error: mdbox map .../mdbox/storage/dovecot.map.index corrupted: Unexpectedly lost INBOX uid=638 map_uid=809891 Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Error: mdbox map .../mdbox/storage/dovecot.map.index corrupted: Unexpectedly lost INBOX uid=638 map_uid=809891 Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Disconnected: Internal error occurred. Refer to server log for more information. [2012-10-24 10:43:23] bytes=115/53726 Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Warning: mdbox .../mdbox/storage: Inconsistency in map index (467,31960 != 467,36768) Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Warning: mdbox .../mdbox/storage: rebuilding indexes Oct 24 10:45:19 two dovecot: imap(listen(a)mjh.name): Panic: file mdbox-storage-rebuild.c: line 773 (rebuild_update_refcounts): assertion failed: (map_uid < msgs[i]->map_uid) Oct 24 10:45:19 two dovecot: imap(listen(a)mjh.name): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x380ca) [0x7f99cf35b0ca] -> /usr/lib/dovecot/libdovecot.so.0(+0x3810e) [0x7f99cf35b10e] -> /usr/li b/dovecot/libdovecot.so.0(i_fatal+0) [0x7f99cf334a67] -> /usr/lib/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10a5) [0x7f99cf5f42d5] -> /usr/lib/dovecot/libdovecot-storage.so.0(mdbox_s torage_rebuild+0x24) [0x7f99cf5f4414] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x53565) [0x7f99cf5f4565] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_close+0x1a) [0x7f99cf5c8caa] -> /usr/lib/dovec ot/libdovecot-storage.so.0(mailbox_free+0x13) [0x7f99cf5c8cf3] -> dovecot/imap(client_destroy+0x109) [0x7f99cfaa69e9] -> dovecot/imap(client_input+0xaa) [0x7f99cfaa6dba] -> /usr/lib/dovecot/libdovecot.so.0(i o_loop_call_io+0x48) [0x7f99cf366c98] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xa7) [0x7f99cf367d27] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f99cf366c28] -> /usr/lib/dovecot /libdovecot.so.0(master_service_run+0x13) [0x7f99cf354e33] -> dovecot/imap(main+0x304) [0x7f99cfa9e554] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f99cef8576d] -> dovecot/imap(+0x95e5) [0 x7f99cfa9e5e5] Oct 24 10:45:19 two dovecot: master: Error: service(imap): child 4977 killed with signal 6 (core dumps disabled) I use: Dovecot 2.0.19-0ubuntu1 Ubuntu 12.04, x86-64, Kernel 3.2.0-32-generic local XFS filesystem for the mdbox The problem appeared out of nowhere. Many messages been continously delivered to this mailbox on this installation since May 2012, and the mdbox was only accessed with deliver and imap/pop3 from dovecot. About four hours after the problem initially appeared, I did a hard reset of the system because it was unresponsive. The error message is exactly the same before and after the hard reset. The problem is triggered by both IMAP access and dovecot deliver access. The whole mdbox is 6.6 GiB large and I guess that it contains about 300k-600k messages. It's an archive of public mailing lists, so I could give access to the files. Can anybody say something about this? May the mdbox be repaired? Regards, Milan Holzäpfel -- Milan Holzäpfel <listen(a)mjh.name>

5 15

[Dovecot] maildir and end-of-line encoding
by Christoph Anton Mitterer 23 Nov '12

23 Nov '12

Hi. I just wondered, the following: My MDA may get mails that use LF or CR/LF end of line encodings and deliver them into maildirs. I couldn't find any information about, whether one should or must convert all into one format, cause AFAIK at least on the IMAP side, CR/LF is always used? How does this work on the maildir/backend side of dovcot? Can it work with both and simply automatically convert LF into CR/LF? Thanks, Chris.

2 3

[Dovecot] v2.1 memory usage
by Timo Sirainen 21 Nov '12

21 Nov '12

v2.1's imap/pop3 processes are linked with OpenSSL by default, because of imapc/pop3c storages. I wonder if you notice any actual (system-global) memory usage reduction when compiling without those and you have thousands of imap processes? ./configure --with-storages=mdbox sdbox maildir mbox cydir I guess I could try it myself also but maybe someone else wanting to reduce memory usage is quicker? :) Related to that, you can look at where imap processes' memory goes with: http://dovecot.org/tools/linux-maps-parse-imap.pl I don't know how much memory those (writable!) file-backed memory areas actually use per-process. Maybe nothing, maybe something..

8 13

[Dovecot] mbox vs. maildir storage block waste
by Christoph Anton Mitterer 13 Nov '12

13 Nov '12

Hi. I recently mentioned in several posts, that I'd tended to use mbox rather than maildir, because you don't loose so much space (due to always allocating full blocks per maildir file and thus per mail). I made some tests of my archive, which consists of some 3,4 million mails at a total of 42GB). Most of these mails are probably normal sized, but there are also some with bigger attachments. For those who are interested here are the results: I used a 53687091200 B image file (via loop device) and tested ext4 only. btrfs is IMHO not yet ready, I have had often issues with XFS (corruptions), reiser4 is more or less dead and reiser3 is said to have issues (see e.g. its wikipedia article, even though it has that mode for small files which would fit nicely). As you see the number of mails increased a bit, cause I tested over several days... but this is only a very small increase so it shouldn't change the numbers a lot. 1) Original mbox archives (right now in Evolution) mbox exact space: 38122676224 (does not include meta-data) mbox guess space: 44625670144 (includes Evolution meta-data which is several GBs) mbox num mails: 3412999 (occurances of From_ lines) In the following: - image file, 1B-blocks, Used_begin, Used_end, Available_begin, Available_end result out of df -B 1 - mdir exact used space is the sum of du -B 1 for each regular file (i.e. each mdir file) - mdir guess used space du -B 1 on the root dir of the filesystem - mdir num mails: find . type -f | wc -l on the root dir of the filesystem 2) EXT4 with 4096 blocks: image file: 53687091200 1B-blocks: 52844687360 Used_begin: 188555264 Used_end: 45198778368 Available_begin: 49971777536 Available_end: 2444972032 mdir exact used space: 44810866688 mdir guess used space: 45010243584 mdir num mails: 3423296 delta: 6.688190464 G delta / mail: 1953 B 3) EXT4 with 2048 blocks: image file: 53687091200 1B-blocks: 50324295680 Used_begin: 82857984 Used_end: 41598846976 Available_begin: 47557083136 Available_end: 6041094144 mdir exact used space: 41323991040 mdir guess used space: 41516007424 mdir num mails: 3425033 delta: 3.201314816 G delta / mail: 934 B 4) EXT4 with 1024 blocks: image file: 53687091200 1B-blocks: 50314834944 Used_begin: 38287360 Used_end: 39909360640 Available_begin: 47592193024 Available_end: 7721119744 mdir exact used space: 39683908608 mdir guess used space: 39871086592 mdir num mails: 3425033 delta: 1.561232384 G delta / mail: 455 B As you can see, the delta per mail is rather close to the statistically expected values of 2048B, 1024B and 512B. In the end I probably changed my opinion. ~7GB of wasted block space for all my mails is actually quite a lot, but in days of cheap disk space it's acceptable. And with mbox one has IMHO the major disadvantage that mailservers (including dovecot) store some meta-data _in_ it (i.e. in the mails themselves) , which I don't like a lot. I still think about reports that mbox is much faster with full text search (which sounds reasonable)... but therefore one needs probably and database backend anyway. HTH, Chris.

7 21

[Dovecot] v2.2.alpha1 released
by Timo Sirainen 08 Nov '12

08 Nov '12

http://dovecot.org/releases/2.2/alpha/dovecot-2.2.alpha1.tar.gz http://dovecot.org/releases/2.2/alpha/dovecot-2.2.alpha1.tar.gz.sig I wanted to start stabilizing v2.2 release some months ago already, but I somehow got stuck adding more and more features. Now it looks like all of the necessary API changes are done, so everything I'm planning on near future can still be added to v2.2 without major changes. The redesigned dsync hasn't had much testing yet, so avoid running it with important mails. Would be nice if people started testing and reporting any bugs. I was going to create some kind of a test suite for testing all the possible syncing combinations and also doing some random stress testing, but I haven't had time for that yet. The new dsync supports doing very fast syncs by saving the resulting state and giving it as command line parameter to the next sync. So for example doveadm sync -s "" > new-state saves the state and doveadm sync -s `cat new-state` continues from the saved state. The replicator code doesn't yet support this. * When creating home directories, the permissions are copied from the parent directory if it has setgid-bit set. For full details, see http://wiki2.dovecot.org/SharedMailboxes/Permissions * "doveadm auth" command was renamed to "doveadm auth test" * IMAP: ID command now advertises server name as Dovecot by default. It was already trivial to guess this from command replies. + Implemented IMAP MOVE and BINARY extensions + Implemented IMAP CATENATE, URLAUTH and URLAUTH=BINARY extensions (by Stephan Bosch). + Implemented IMAP NOTIFY extension. Requires mailbox_list_index=yes to be enabled. + Redesigned and rewritten dsync. The new design makes the syncing faster, more reliable and more featureful. The new dsync protocol isn't backwards compatible with old dsync versions (but is designed to be forwards compatible with future versions). + All mailbox formats now support per-user message flags for shared mailboxes by using a private index. It can be enabled by adding :INDEXPVT=<path> to mail location. This should be used instead of :INDEX also for Maildir/mbox to improve performance. + Improved mailbox list indexes. They should be usable now, although still disabled by default. + Added LAYOUT=index. The mailbox directories are created using their GUIDs in the filesystem, while the actual GUID <-> name mapping exists only in the index. + LMTP proxy: Implemented XCLIENT extension for passing remote IP address through proxy.

2 4

[Dovecot] dovecot-lda not correct folder
by tony.blue.mailinglist＠gmx.de 07 Nov '12

07 Nov '12

Hello, i use dovecot with maildir. The maildir-folder looks like this: vmail/mail/user1/cur vmail/mail/user1/.Sent vmail/mail/user1/.optionalfolder Procmail should put some definded mails in the "optionalfolder" My uses configuration does not do this: ... DELIVERMAIL="/usr/lib/dovecot/dovecot-lda" IMAP="$DELIVERMAIL -e -d $LOGNAME -m INBOX" ZUSATZORDNER="$DELIVERMAIL -e -d $LOGNAME -m .optionalfolder" ... dovecot-lda puts the mails for the optionalfolder always in the .cur (INBOX). What´s the correct dovecot-lda parameter to put the mails in the optionalfolder? Thank you! Tony

2 2

[Dovecot] Error: Internal quota calculation error
by Micah Anderson 07 Nov '12

07 Nov '12

Hello, I'm using 2.1.7 with seive and mysql quotas. We had an outage the other day where the database server where quotas are stored was not available for a short period of time. In dovecot land, the following types of errors occured in that scenario: Oct 26 22:19:01 grosbeak dovecot: lda(example(a)riseup.net): Error: Internal quota calculation error Oct 26 22:19:01 grosbeak dovecot: lda(example(a)riseup.net): Error: sieve: msgid=<20122132765181x.ABCCE457(a)example.com>: failed to store into mailbox 'Trash': Internal error occurred. Refer to server log for more information. [2012-10-26 22:19:01] Oct 26 22:19:01 grosbeak dovecot: lda(example(a)riseup.net): Error: sieve: script /maildir/e/example/.dovecot.sieve failed with unsuccessful implicit keep (user logfile /maildir/e/example/.dovecot.sieve.log may reveal additional details) I expect that there would be quota calculation errors as dovecot could not reach the database server, but what worried me was the 'failed to store into mailbox' message from sieve. The 'Trash' mailbox in this particular seive script is the correct location for the message to be filed into, but the worrisome message is the 'failed with unsuccessful implicit keep'. I looked through all the message-ids that reported this error and I found that the messages were properly delivered in the end, so this might be some issue interacting between sieve, dovecot and quota and just causing a scary message that can be ignored? thanks for any information you can provide, dovecot is great! micah --

2 1

[Dovecot] backtrace for non-existant %{ldap:attr} on login
by Steffen Kaiser 07 Nov '12

07 Nov '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I'm fetching the user and auth data from LDAP, this is the string: pass_attrs = uid=user,userPassword=password,homeDirectory=userdb_home,mailUidNumber=userdb_uid,mailGidNumber=userdb_gid,mailLocationDovecot=userdb_mail,uid=userdb_user,=userdb_quota_rule=*:bytes=%{ldap:mailQuotaBytes}, =userdb_quota_rule2=Trash:bytes=+%{ldap:mailQuotaBytesTrash} If mailQuotaBytesTrash or mailQuotaBytes is not present, the LOGIN process does not work: 1 login user pwd 1 NO [UNAVAILABLE] Temporary authentication failure. [mailsrv2:2012-10-31 08:56:51] * OK Waiting for authentication process to respond.. If I add those two attributes, the user can login successfully. dovecot-2.1.10/sbin/dovecot --version 2.1.10 (9cdeab12f3e1) The log entries: 2012-10-31 09:56:51 auth: Panic: pool_data_stack_realloc(): stack frame changed 2012-10-31 09:56:51 auth: Error: Raw backtrace: /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(+0x4857a) [0x7f2c0528c57a] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(+0x485c6) [0x7f2c0528c5c6] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f2c0525feaf] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(+0x58f2e) [0x7f2c0529cf2e] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(+0x442f5) [0x7f2c052882f5] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(buffer_get_space_unsafe+0x68) [0x7f2c05288728] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(str_vprintfa+0x6d) [0x7f2c052a796d] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(str_printfa+0x88) [0x7f2c052a7ac8] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb]() [0x42682e] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(var_expand_with_funcs+0x6fb) [0x7f2c052ac48b] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb](db_ldap_result_iterate_next+0x12f) [0x42734f] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb]() [0x428974] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb]() [0x428f5f] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb]() [0x4282a1] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f2c05298756] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f2c0529978f] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f2c052986f8] -> /usr/local/dovecot-2.1.10/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f2c052840d3] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb](main+0x376) [0x41bba6] -> /lib/libc.so.6(__libc_start_main+0xfd) [0x7f2c04449c8d] -> dovecot2.1/auth [0 wait, 1 passdb, 0 userdb]() [0x40cf89] 2012-10-31 09:56:51 auth: Fatal: master: service(auth): child 15865 killed with signal 6 (core dumps disabled) 2012-10-31 09:56:51 imap-login: Warning: Auth connection closed with 1 pending requests (max 0 secs, pid=15869, EOF) - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBUJDqe2oxLS8a3A9mAQI+YQf/Qd4IIeM35Hmmpl1IMcQwJFK4854G5Dku yK+GsWhE2gxI6KaLO6DSI/kpN79qhQRkHsUAHzoPiZ7kQpZprNaEP/CIPkTzw//i HyC2Odpfa8fWUOqtH5Cp6X5spF6hQa4mmQyzgXguF9bdjZkNu4vQ78wRuQGG4eHi BOkNL0b93DsN3NSIoXDpiAiJn1aleTLe7mYkfsGewjb+AN+FpR4hLbf32yRRn8J0 Fkn8agEijixGMXEBD4ZprTbX9NbUr92YPfRycMnA2A00MUUlv/iKlqXRpMLafRjL bnHL/QE80xHoKKJUR96/RjruciIolAtlyYvhTU9ibRpLYm5Hcd9bZg== =iPvg -----END PGP SIGNATURE-----

2 1

[Dovecot] copymail deleted
by Christian Rößner 07 Nov '12

07 Nov '12

Hi, I had enabled an option in dovecot. mail_attachment_dir = /var/mail/virtual/copymail/attachments After a while I checked /var/mail/virtual and did some cleanup. I did not remember that copymail was specified in dovecot and erased it. Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: file_istream.stat(/var/mail/virtual/copymail/attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6) failed: No such file or directory Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: istream-concat: Failed to get size of stream /var/mail/virtual/copymail/attachments/6a/50/6a506530265ef7c9feb396410eaf6946036e9a79-b034401e794009503a0400002cb72ff6 Oct 30 10:56:05 mx0 dovecot: imap(hidden): Error: read() failed: Invalid argument (FETCH for mailbox INBOX UID 196) Oct 30 10:56:05 mx0 dovecot: imap(hidden): Disconnected: Internal error occurred. Refer to server log for more information. [2012-10-30 10:56:05] in=150 out=950 I have Bacula and have restored most of the stuff, but obviously not all files. That is not too important. But I do not know, how to tell dovecot that it may "forget" about files that produce a "No such file or directory" error. Can I do some "rescan/rebuild" in dovecot? Kind regards -Christian Rößner -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich

2 5