September 2012 - dovecot

[Dovecot] Found bug on doing IMAP search with UTF8 chars
by Weinreich, Nico 12 Sep '12

12 Sep '12

Hi! I'm using Roundcube 0.9git as mail frontend and have installed Dovecot 2.1.7 on Debian Squeeze. When searching for mail via Roundcube and the keyword contains "special" chars (like german umlauts), then Dovecot seems to die when the folder does not contain a message which contains this keyword, i.e. folder contains message /w subject "Dies ist eine Nachricht" => search for "Nachricht" => Roundcube shows that messages folder contains message /w subject "Dies ist eine Nachricht" => search for "Nachricht123" => Roundcube shows error that no messages where found folder contains message /w subject "Auflösung der Frage" => search for "Lösung" => Roundcube shows that messages folder contains message /w subject "Auflösung der Frage" => search for "Lösung123" => Roundcube shows error saying: "Connection to storage server failed" Looking into /var/log/mail.log shows: Sep 5 11:43:32 mail dovecot: imap(abc(a)def.de): Panic: search key not utf8: L.sung123 dsr_sub_allocation Sep 5 11:43:32 mail dovecot: imap(abc(a)def.de): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x4478a) [0 x7f065ba8e78a] -> /usr/lib/dovecot/libdovecot.so.0(+0x447d6) [0x7f065ba8e7d6] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f065ba655ef ] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9a3cb) [0x7f065bd6f3cb] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9a8aa) [0x7f065bd6f8 aa] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x78641) [0x7f065bd4d641] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_fore ach+0x3c) [0x7f065bd4d7bc] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header+0x29) [0x7f065ba78eb9] -> /usr/lib/dovecot/libdovecot-st orage.so.0(+0x9a0cf) [0x7f065bd6f0cf] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x9b160) [0x7f065bd70160] -> /usr/lib/dovecot/libdovecot- storage.so.0(index_storage_search_next_nonblock+0x48) [0x7f065bd706e8] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonbl ock+0x27) [0x7f065bd51a07] -> dovecot/imap() [0x41668f] -> dovecot/imap(imap_search_start+0xd8) [0x416ad8] -> dovecot/imap(cmd_sort+0x1e6) [ 0x40e636] -> dovecot/imap(command_exec+0x3d) [0x41157d] -> dovecot/imap() [0x4104fe] -> dovecot/imap(client_handle_input+0x135) [0x410805] - > dovecot/imap(client_input+0x5f) [0x41112f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x36) [0x7f065ba9a926] -> /usr/lib/dovecot/ libdovecot.so.0(io_loop_handler_run+0x9f) [0x7f065ba9b95f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f065ba9a8c8] -> /usr/li b/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f065ba87053] -> dovecot/imap(main+0x2a4) [0x419354] -> /lib/libc.so.6(__libc_start_ma in+0xfd) [0x7f065b706c8d] -> dovecot/imap() [0x408e79] Huh? This error only occurs when I search for a keyword, that contains probably non-ASCII-US chars (like umlauts, euro sign EUR ...) and no message was found in that folder. When there is any message which contains this keyword, then Roundcube is fine and there error above cannot be found in mail.log. Can anybody confirm this behaviour? Regards

3 10

[Dovecot] Resynchronization of indexes (or Warning: Maildir Scanning <dir> took <X> seconds)
by Angel L. Mateo 12 Sep '12

12 Sep '12

Hello, We have a dovecot servers farm with storage in NFS and indexes in local files. Under heavy load conditions we a log of messages in the form: Sep 3 12:35:49 myotis32 dovecot: imap(<imapuser>): Warning: Maildir: Scanning <maildir cur directory> took 62 seconds (9516 readdir()s, 0 rename()s to cur/, why=0x80) We have check that messages are directed with lmtp to the same server the user (scenario 3 at http://wiki2.dovecot.org/NFS with director in front of backend servers). So my question is why is dovecot using these readdirs operations? why it is scanning the whole cur directory of the user? I guess this is an index resynchronization, but I don't know could it be the cause. PS: I have attached my dovecot -n output

2 4

[Dovecot] dsync issue: "Server sent invalid input: Error parsing input: Unexpected ')'"
by Gábor Lénárt 11 Sep '12

11 Sep '12

Hi, I'm trying to use dsync (dovecot version 2.1.9) for IMAP only per-user migration from other IMAP servers to my shiny new one. One worked quite well, but there is a problem with an old IMAP server: dsync(migtest@office): Debug: imapc(imap-sunw-old.servers.intra:143): Looking up IP address dsync(migtest@office): Debug: imapc(imap-sunw-old.servers.intra:143): Connecting to 192.168.10.100:143 dsync(migtest@office): Error: imapc(imap-sunw-old.servers.intra:143): Server sent invalid input: Error parsing input: Unexpected ')' dsync(migtest@office): Debug: imapc(imap-sunw-old.servers.intra:143): Disconnected dsync(migtest@office): Error: imapc: Command failed: Disconnected from server dsync(migtest@office): Error: user migtest@office: Initialization failed: Initializing mail storage from mail_location setting failed: imapc: LIST failed: Internal error occurred. Refer to server log for more information. [2012-09-10 11:49:02] dsync(migtest@office): Fatal: User init failed What can cause this? "By hand" test with telnet'ing says: dovecot-test:~$ telnet imap-sunw-old.servers.intra 143 Trying 192.168.10.100... Connected to imap-sunw-old.servers.intra. Escape character is '^]'. * OK [CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT LANGUAGE XSENDER X-NETSCAPE XSERVERINFO X-SUN-SORT X-SUN-IMAP X-ANNOTATEMORE X-UNAUTHENTICATE XUM1 AUTH=PLAIN] imap-sunw-old.servers.intra IMAP4 service (Sun Java(tm) System Messaging Server 6.3-5.02 (built Oct 12 2007; 32bit)) (this is one line of course) Is it possible that the problem about ')' is caused by this line? Or is it tottaly different issue? I just think this, because the the major difference between this and the working server on migration is to have ')' and '(' at the imap "greeting", the working server does not have those, and the error message is about something with ')'. The command was the following (no configuration in dovecot for this, only this command is given at shell level - currently mainly for testing if it works or not); doveadm -D -v -o imapc_features=rfc822.size \ -o imapc_host=192.168.10.100 \ -o imapc_user=migtest@office -o imapc_password=SECRET backup -R -f \ -u migtest@office imapc: This was OK with a dovecot v1 as the "source" (as I've written) but has the problem I've desribed with that "old" sun messaging server as the source. I've also used tcpdump then wireshark's "follow TCP stream" on it: the communication seems to be just that "greeting" line, then my (target) dovecot sends FIN. That "old" server works otherwise since years with various IMAP clients. The log of my server does not contain too much just the db lookups, which works nicely (the same for working and non-working source IMAP servers): Sep 10 11:49:02 dovecot-test dovecot: auth: Debug: prefetch(migtest@office): passdb didn't return userdb entries, trying the next userdb Sep 10 11:49:02 dovecot-test dovecot: auth: Debug: ldap(migtest@office): user search: base=cn=mail,dc=office,dc=intra scope=subtree filter=(&(objectClass=mailUser)(uid=migtest@office)) fields=uid Sep 10 11:49:02 dovecot-test dovecot: auth: Debug: ldap(migtest@office): result: uid=migtest@office Any help is welcome, - Gábor

2 2

[Dovecot] [PATCH] Generalize HMAC implementation
by Florian Zeitz 11 Sep '12

11 Sep '12

Hello everyone and Timo in particular, about a year ago I implemented a SHA-1 variant of the HMAC(-MD5) present in Dovecot. I had always disliked this a bit, because it replicates a lot of code. This patch generalizes the HMAC function to take a hash_method struct as parameter, and changes existing code which uses the "old" HMAC function to use this new one. I'm not really sure this is actually a good idea, but I still felt I should provide the code in case you would want to merge it upstream. Attached is the patch as a hg export based on the revision of dovecot-2.2 current at the time of writing. Regards, Florian Zeitz

2 2

[Dovecot] FTS indexing
by Ralf Hildebrandt 11 Sep '12

11 Sep '12

I'm reacreating all FTS indices on all my users' INBOX folders: mkdir ~%s/Maildir/lucene-indexes && chown %s:users ~%s/Maildir/lucene-indexes doveadm index -u %s -q INBOX In "ps auxwww" I'm seeing this: tbudkevy 29186 3.7 0.0 6616 3056 ? S 18:11 0:15 dovecot/indexer-worker [tschuetz INBOX] how can a dovecot/indexer-worker actually access the INBOX of the user "tschuetz" (where "tbudkevy" doesn't have access, due to the different UID)...? Should ~%s/Maildir/lucene-indexes be filled after having run the command sequence for all users? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 5

[Dovecot] Quota less mailbox with mdbox
by Héctor Moreno Blanco 11 Sep '12

11 Sep '12

Good morning everyone, I have a little doubt about setting a quotaless mailbox with mdbox. Currently we have dovecot 2.0.10 with Maildir as mailbox system. We want to migrate to mdbox. The migration is simple and easy, but there are some configuration settings we cannot set to work properly. In our dovecot with Maildir we have the namespace INBOX with certain quota and another one without quota. This is working like a charm. Dovecot.conf: ... namespace { separator = . prefix = inbox = yes type = private } namespace { inbox = no location = prefix = INBOX. separator = . hidden = yes type = private list = no } namespace { location = maildir:%h/Maildir/_backup prefix = _backup. separator = . type = private } ... plugin/quota = maildir:_backup:ns= ... Dovecot-ldap.conf: ... pass_filter = (&(objectClass=qmailUser)(uid=%u)(accountActive=TRUE)) pass_attrs = uid=user,userPassword=password, mailQuotaSize=quota_rule=*:bytes=%$ user_attrs = mailMessageStore=home, mailQuotaSize=quota_rule=*:bytes=%$ user_filter = (&(objectClass=qmailUser)(|(mailAlternateAddress=%u)(mail=%u)(uid=%u))(accountActive=TRUE)) ... When migrated to mdbox and changing the configuration, we don't know how to get it work. We have also tried to follow the instructions from the wiki of dovecot2 without success. Could anyone give us a clue please? If additional information is needed, please let me know. Thank you very much in advanced. Kind regards. ________________________________ [Descripción: C:\Documents and Settings\hhmb\Datos de programa\Microsoft\Firmas\logo_gmv.gif] Héctor Moreno Blanco División de Seguridad e Infraestructuras / Security and Infrastructures Division GMV Isaac Newton, 11 P.T.M. Tres Cantos E-28760 Madrid Tel. +34 91 807 21 00 Fax +34 91 807 21 99 www.gmv.com <http://www.gmv.com/> [Descripción: C:\Documents and Settings\hhmb\Datos de programa\Microsoft\Firmas\icon_blog.gif]<http://www.gmv.com/b2_gmv> [Descripción: C:\Documents and Settings\hhmb\Datos de programa\Microsoft\Firmas\icon_facebook.gif]<http://www.facebook.com/infoGMV> [Descripción: C:\Documents and Settings\hhmb\Datos de programa\Microsoft\Firmas\icon_twitter.gif]<http://www.twitter.com/infoGMV_es> [Descripción: C:\Documents and Settings\hhmb\Datos de programa\Microsoft\Firmas\icon_youtube.gif]<http://www.youtube.com/infoGMV> P Please consider the environment before printing this e-mail. ______________________ This message including any attachments may contain confidential information, according to our Information Security Management System, and intended solely for a specific individual to whom they are addressed. Any unauthorised copy, disclosure or distribution of this message is strictly forbidden. If you have received this transmission in error, please notify the sender immediately and delete it. ______________________ Este mensaje, y en su caso, cualquier fichero anexo al mismo, puede contener informacion clasificada por su emisor como confidencial en el marco de su Sistema de Gestion de Seguridad de la Informacion siendo para uso exclusivo del destinatario, quedando prohibida su divulgacion copia o distribucion a terceros sin la autorizacion expresa del remitente. Si Vd. ha recibido este mensaje erroneamente, se ruega lo notifique al remitente y proceda a su borrado. Gracias por su colaboracion. ______________________

3 2

[Dovecot] Memory leak by getting the virtual size of a IMAP folder
by Dafan Zhai 11 Sep '12

11 Sep '12

Hi everyone, I am writing a dovecot statistic plugin, which calls the 'mailbox_get_metadata' function with MAILBOX_METADATA_VIRTUAL_SIZE as the 2nd parameter. enum mailbox_status_items metadata_items = MAILBOX_METADATA_VIRTUAL_SIZE; struct mailbox_metadata metadata; mailbox_get_metadata(mailbox, metadata_items, &metadata); but Valgrind finds a memory leak when this function is called: ---------------------------------------------------snip-------------------------------------------------------------- ==10304== 12,288 bytes in 3 blocks are definitely lost in loss record 74 of 76 ==10304== at 0x40222A4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==10304== by 0x416EE9F: block_alloc (mempool-alloconly.c:237) ==10304== by 0x416F16C: pool_alloconly_create (mempool-alloconly.c:140) ==10304== by 0x409C07D: mail_search_build_init (mail-search-build.c:187) ==10304== by 0x40C77F1: index_mailbox_get_metadata (index-status.c:200) ==10304== by 0x4067102: maildir_mailbox_get_metadata (maildir-storage.c:486) ==10304== by 0x409F78B: mailbox_get_metadata (mail-storage.c:1298) ==10304== by 0x490492D: mailbox_status_update (statistic-plugin.c:310) ==10304== by 0x4904E2E: statistic_mail_save (statistic-plugin.c:426) ==10304== by 0x41A054D: notify_contexts_mail_save (notify-plugin.c:61) ==10304== by 0x41A104F: notify_save_finish (notify-storage.c:143) ==10304== by 0x409EE94: mailbox_save_finish (mail-storage.c:1673) ---------------------------------------------------snip-------------------------------------------------------------- I have looked into the source code, and found that from the 'mail_search_build_init' function call in lib-storage/index/index-status.c:200 a pool is created, but the pool is not freed in the mailbox_search_deinit function call in lib-storage/index/index-status.c:218. This may be the reason of the memory leak. But I do not know how to free the pool. It seems to me that doveadm is having the same memory leak because it is using 'mail_search_build_init' too. ---------------------------------------------------snip-------------------------------------------------------------- # valgrind --leak-check=full doveadm mailbox status -u testuser1 vsize INBOX ==10457== Memcheck, a memory error detector ==10457== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==10457== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info ==10457== Command: doveadm mailbox status -u testuser1 vsize INBOX ==10457== INBOX vsize=41643319 ==10457== ==10457== HEAP SUMMARY: ==10457== in use at exit: 4,356 bytes in 3 blocks ==10457== total heap usage: 440 allocs, 437 frees, 539,124 bytes allocated ==10457== ==10457== 4,096 bytes in 1 blocks are definitely lost in loss record 3 of 3 ==10457== at 0x40222A4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==10457== by 0x41AEE9F: ??? (in /usr/lib/dovecot/libdovecot.so.0.0.0) ==10457== by 0x41AF16C: pool_alloconly_create (in /usr/lib/dovecot/libdovecot.so.0.0.0) ==10457== by 0x40DC07D: mail_search_build_init (in /usr/lib/dovecot/libdovecot-storage.so.0.0.0) ==10457== by 0x8057EB8: doveadm_mail_mailbox_search_args_build (in /usr/bin/doveadm) ==10457== by 0x805855A: ??? (in /usr/bin/doveadm) ==10457== by 0x8053A7B: doveadm_mail_single_user (in /usr/bin/doveadm) ==10457== by 0x8053D2B: ??? (in /usr/bin/doveadm) ==10457== by 0x80541CF: doveadm_mail_try_run (in /usr/bin/doveadm) ==10457== by 0x805B863: main (in /usr/bin/doveadm) ==10457== ==10457== LEAK SUMMARY: ==10457== definitely lost: 4,096 bytes in 1 blocks ==10457== indirectly lost: 0 bytes in 0 blocks ==10457== possibly lost: 0 bytes in 0 blocks ==10457== still reachable: 260 bytes in 2 blocks ==10457== suppressed: 0 bytes in 0 blocks ==10457== Reachable blocks (those to which a pointer was found) are not shown. ==10457== To see them, rerun with: --leak-check=full --show-reachable=yes ==10457== ==10457== For counts of detected and suppressed errors, rerun with: -v ==10457== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 122 from 35) ---------------------------------------------------snip-------------------------------------------------------------- Dovecot version: 2.1.9 OS: Linux 3.0.30-dist i686 maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX I have applied the following patches from the dovecot 2.1 branch fixing memory leaks: http://hg.dovecot.org/dovecot-2.1/rev/7bdbca7b0913 http://hg.dovecot.org/dovecot-2.1/rev/963482677c0b It would be nice if you could give me some hints how I can address this issue? Thanks in advance! Greetings, Dafan

2 2

[Dovecot] dovecot proxy ssl-parameter from database?
by Daniel Funke 11 Sep '12

11 Sep '12

Hi List, I want to migrate my IMAP-users from an old Courier-IMAP-Server to a new dovecot-Server. The old server supports IMAP without and with TLS. The new one only with TLS. To users should be migrated one by one with changing some client parameters. To leave the servername of the imap server I want to use the dovecot proxy during migration. the proxy uses a sql db to look for the old or new host for every user. So far all works fine. My question is the TLS or noTLS connection from the client to the proxy. How can I make the ssl-parameter user-dependant? clients to the old courier server should work without or with tls through the proxy and clients to the new dovecot server should only work WITH TLS between Client and proxy. (After migration of all clients the proxy should be switched off an all clients connect directly to the dovecot server.) I tried with http://wiki2.dovecot.org/UserDatabase/ExtraFields and returned a field "userdb_ssl" ("required" for host dovecot, "yes" for host courier). But it doesn't work. A migrated user can still login without tls to the proxy and the dovecot backend. Any ideas? Daniel Here is my doveconf -n # 2.1.9: /opt/dovecot/etc/dovecot/dovecot.conf doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1503) # OS: Linux 2.6.32-279.2.1.el6.x86_64 x86_64 CentOS release 6.3 (Final) auth_mechanisms = plain login default_process_limit = 500 disable_plaintext_auth = no listen = * managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /opt/dovecot/etc/dovecot/sql.conf driver = sql } protocols = imap pop3 service imap-login { inet_listener imaps { port = 0 } } service pop3-login { inet_listener pop3s { port = 0 } } ssl_cert = </etc/pki/tls/certs/server.crt ssl_key = </etc/pki/tls/private/server.key userdb { driver = prefetch } verbose_proctitle = yes

2 1

[Dovecot] Alerts when process limits are met
by David Anderson 11 Sep '12

11 Sep '12

Hi, I've not found the answer to this question anywhere - please forgive me if I overlooked. I'd like to be able to be automatically alerted if process limits are hit (e.g. max POP3 logins). Is there a way that I can configure a script to be run, in the same way that I can with quota warnings? I can of course use logwatch, but this alerts me the next day, and logwatch can be noisy and it's easy to overlook. My motivation: for some reason my POP3 listener was hanging, and I had to restart dovecot (this is 2.0.9). But I was not aware - the process was still there, it was accepting connections (but not doing anything useful with them) and was actually managing to log the fact that the process limit was being hit (as more and more connections came in). If I could have been alerted I could have restarted dovecot earlier. Many thanks, David -- WordShell - WordPress fast from the CLI - www.wordshell.net

3 2

[Dovecot] ACL MySQL dict: to_user==from_user entries
by Roland Rosenfeld 11 Sep '12

11 Sep '12

Hi! We run dovecot 2.1.6 with ACL support and MySQL dict for shared/shared-boxes/user/$to/$from. When a user does a SETACL (via IMAP) to himself he is added as $to and $from to the MySQL dict. Se for example this IMAP conversation: 5 GETACL INBOX * ACL "INBOX" "user@domain" lrwstipekxacd 5 OK Getacl completed. 6 SETACL INBOX user@domain lrwstipekxacd 6 OK Setacl complete. 7 GETACL INBOX * ACL "INBOX" "user@domain" akxeilprwtscd 7 OK Getacl completed. As you can see, user@domain sets the ACL of his INBOX to the same value it was set before (reading it from the server after this idempotent write changed the order but none of the flags). But after "SETACL INBOX user@domain lrwstipekxacd" I see a new entry in my imap_acl_shares table where to_user and from_user both contain the value "user@domain". As far as I can see, this entry does not make much sense in the database, because every user should be able so see its own folders. But adding this line may blow up the database especially according to the full table scan problem for acl dict rebuild mentioned in http://www.dovecot.org/list/dovecot/2012-August/067548.html. Does it make sense to run a daily cron job, that removes all entries with from_user=to_user or may this cause other inconsistencies? If you need to have a look at my complete setup, it's in http://www.dovecot.org/list/dovecot/2012-July/067130.html, so I don't have to clone this here. Tschoeeee Roland

2 1