February 2013 - dovecot

[Dovecot] Possible sort optimization (?)
by Michael M Slusarz 25 Feb '13

25 Feb '13

Maybe this is just noise... but I can reproduce this fairly reliably. Mailbox with 21,000+ messages This query: a UID SORT RETURN (ALL COUNT) (DATE) UTF-8 SUBJECT "foo" is always about 10 percent slower than this split query (I've done this 4-5 times, and the numbers are similar): a UID SEARCH RETURN (SAVE) CHARSET UTF-8 SUBJECT "foo" b UID SORT RETURN (ALL COUNT) (DATE) UTF-8 UID $ (The particular query I used matched 5 messages out of the 21,000+) My not-very-scientific benchmarking process: 1.) Stop dovecot process 2.) Delete all dovecot index files for that mailbox 3.) Flush linux paging cache (sync && echo 3 > /proc/sys/vm/drop_caches) 4.) Restart dovecot 5.) Access dovecot via command-line (PREAUTH) 6.) SELECT mailbox 7.) Issue command(s) Could be a potential area for performance improvement or could simply be lazy benchmarking. michael

2 2

[Dovecot] Problem with the command hooks api
by Tony 24 Feb '13

24 Feb '13

I have been trying to use the command hook api in an experimental plugin I have written. The hook is registered using the command_hook_register function, and for most commands my hook functions get called once as expected. But when the client uses one of the "UID" commands (e.g. "UID FETCH" ) the hook functions are executed twice. I am using dovecot version 2.1.12. Am I correct in thinking this is a bug? If I am wrong and this is the intended behaviour, how should I detect whether the hook function call is the first or second for a client's command (in particular for the 'post command' hook)? Assuming it is a bug, I modified the code in the command_exec function to stop the second hook call, a patch for this is below. This appears to apply to later dovecot releases as well. I am happy for this code to be included in future releases of dovecot. Thanks, Tony. diff -ur -x .deps -x Makefile dovecot-2.1.12-orig/src/imap/imap-commands.c dovecot-2.1.12-mod/src/imap/imap-commands.c --- dovecot-2.1.12-orig/src/imap/imap-commands.c 2012-03-22 12:59:17.000000000 +0000 +++ dovecot-2.1.12-mod/src/imap/imap-commands.c 2013-02-20 17:15:50.000000000 +0000 @@ -38,7 +38,7 @@ { "FETCH", cmd_fetch, COMMAND_FLAG_USES_SEQS }, { "SEARCH", cmd_search, COMMAND_FLAG_USES_SEQS }, { "STORE", cmd_store, COMMAND_FLAG_USES_SEQS }, - { "UID", cmd_uid, 0 }, + { "UID", cmd_uid, COMMAND_FLAG_HOOKS_DELAY }, { "UID COPY", cmd_copy, COMMAND_FLAG_BREAKS_SEQS }, { "UID FETCH", cmd_fetch, COMMAND_FLAG_BREAKS_SEQS }, { "UID SEARCH", cmd_search, COMMAND_FLAG_BREAKS_SEQS }, @@ -142,12 +142,20 @@ { const struct command_hook *hook; bool ret; + bool exec_hooks = FALSE; - array_foreach(&command_hooks, hook) - hook->pre(cmd); + if ( !(cmd->cmd_flags & COMMAND_FLAG_HOOKS_DELAY)) { + exec_hooks = TRUE; + array_foreach(&command_hooks, hook) + hook->pre(cmd); + } ret = cmd->func(cmd); - array_foreach(&command_hooks, hook) - hook->post(cmd); + + if (exec_hooks != FALSE) + { + array_foreach(&command_hooks, hook) + hook->post(cmd); + } return ret; } diff -ur -x .deps -x Makefile dovecot-2.1.12-orig/src/imap/imap-commands.h dovecot-2.1.12-mod/src/imap/imap-commands.h --- dovecot-2.1.12-orig/src/imap/imap-commands.h 2011-12-13 11:35:28.000000000 +0000 +++ dovecot-2.1.12-mod/src/imap/imap-commands.h 2013-02-20 17:15:44.000000000 +0000 @@ -30,7 +30,9 @@ Dovecot internally returns it for all kinds of commands, but unfortunately RFC 5530 specifies it only for "delete something" operations. */ - COMMAND_FLAG_USE_NONEXISTENT = 0x10 + COMMAND_FLAG_USE_NONEXISTENT = 0x10, + /* Set when need to delay command hook function calls due to recursive command_exec() */ + COMMAND_FLAG_HOOKS_DELAY = 0x20 }; struct command {

2 1

[Dovecot] LMTP logging
by Peter Rindfuss 24 Feb '13

24 Feb '13

Hi, Is there a way (in Dovecot 2.1.14) to sent LMTP log entries and IMAP log entries to different files? I know I could achieve this with syslog-ng, but as LDA has a log file of its own, I am wondering if this is possible with LMTP as well. I apologize if I missed something in the wiki. Thanks, Peter

3 5

[Dovecot] Problem importing mail from maildir directory
by Thomas Koeller 24 Feb '13

24 Feb '13

Hi, I am trying to import mail from a maildir directory into dovecot using the 'doveadm import' command: ----------- bash-4.0$ doveadm -D import -u marion maildir:${HOME}/Mail '' ALL doveadm(marion): Debug: Loading modules from directory: /usr/lib64/dovecot doveadm(marion): Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so doveadm(marion): Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm doveadm(marion): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(marion): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) doveadm(marion): Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so doveadm(marion): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(marion): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_backend_rescan (this is usually intentional, so just ignore this message) doveadm(marion): Debug: Effective uid=10001, gid=10000, home=/home/marion doveadm(marion): Debug: quota: No quota setting - plugin disabled doveadm(marion): Debug: maildir++: root=/home/marion/Mail, index=, control=, inbox=/home/marion/Mail, alt= doveadm(marion): Debug: Effective uid=10001, gid=10000, home=/home/marion doveadm(marion): Debug: quota: No quota setting - plugin disabled doveadm(marion): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:~/mail:INBOX=/var/spool/mail/marion doveadm(marion): Debug: fs: root=/home/marion/mail, index=, control=, inbox=/var/spool/mail/marion, alt= ----------- What I get is - no mails are imported - no error messages are output - nothing is written to the system log in other words, nothing happens at all. Any hints? tk

2 1

[Dovecot] make install fails on Solaris 10
by Henrik Johansson 24 Feb '13

24 Feb '13

Hello, I am trying to build dovecot-2.1.15 on Solaris 10 update 10 with Solaris Studio 12.3 and get this error: /bin/bash ../../libtool --tag=CC --mode=link /opt/SUNWspro/bin/cc -fast -xipo=2 -I/usr/sfw/include -I/usr/sfw/include -no-undefined -o doveadm doveadm-mail.o doveadm-mail-altmove.o doveadm-mail-expunge.o doveadm-mail-fetch.o doveadm-mail-import.o doveadm-mail-index.o doveadm-mail-iter.o doveadm-mail-mailbox.o doveadm-mail-mailbox-status.o doveadm-mail-copymove.o doveadm-mailbox-list-iter.o doveadm-mail-search.o doveadm-mail-server.o doveadm-print.o doveadm-settings.o doveadm-util.o server-connection.o doveadm.o doveadm-auth.o doveadm-director.o doveadm-dump.o doveadm-dump-dbox.o doveadm-dump-index.o doveadm-dump-log.o doveadm-dump-mailboxlog.o doveadm-dump-thread.o doveadm-instance.o doveadm-kick.o doveadm-log.o doveadm-master.o doveadm-mount.o doveadm-mutf7.o doveadm-penalty.o doveadm-print-flow.o doveadm-print-pager.o doveadm-print-tab.o doveadm-print-table.o doveadm-proxy.o doveadm-pw.o doveadm-sis.o doveadm-stats.o doveadm-who.o dsync/libdsync.a ../auth/libpassword.a ../lib-ntlm/libntlm.a ../lib-otp/libotp.a ../../src/lib-storage/libdovecot-storage.la -R/usr/sfw/lib -L/usr/sfw/lib -lssl -lcrypto -lsocket -lnsl -ldl ../../src/lib-dovecot/libdovecot.la -export-dynamic -lnsl -lsocket -lrt -lsendfile libtool: link: /opt/SUNWspro/bin/cc -fast -xipo=2 -I/usr/sfw/include -I/usr/sfw/include -o .libs/doveadm doveadm-mail.o doveadm-mail-altmove.o doveadm-mail-expunge.o doveadm-mail-fetch.o doveadm-mail-import.o doveadm-mail-index.o doveadm-mail-iter.o doveadm-mail-mailbox.o doveadm-mail-mailbox-status.o doveadm-mail-copymove.o doveadm-mailbox-list-iter.o doveadm-mail-search.o doveadm-mail-server.o doveadm-print.o doveadm-settings.o doveadm-util.o server-connection.o doveadm.o doveadm-auth.o doveadm-director.o doveadm-dump.o doveadm-dump-dbox.o doveadm-dump-index.o doveadm-dump-log.o doveadm-dump-mailboxlog.o doveadm-dump-thread.o doveadm-instance.o doveadm-kick.o doveadm-log.o doveadm-master.o doveadm-mount.o doveadm-mutf7.o doveadm-penalty.o doveadm-print-flow.o doveadm-print-pager.o doveadm-print-tab.o doveadm-print-table.o doveadm-proxy.o doveadm-pw.o doveadm-sis.o doveadm-stats.o doveadm-who.o dsync/libdsync.a ../auth/libpassword.a ../lib-ntlm/libntlm.a ../lib-otp/libotp.a ../../src/lib-storage/.libs/libdovecot-storage.so -L/usr/sfw/lib /home/henrikj/Build/dovecot-2.1.15/src/lib-dovecot/.libs/libdovecot.so -lssl -lcrypto -ldl ../../src/lib-dovecot/.libs/libdovecot.so -lnsl -lsocket -lrt -lsendfile -R/opt/HNKSdovecot/lib/dovecot -R/usr/sfw/lib ld: warning: file ../../src/lib-dovecot/.libs/libdovecot.so: linked to /home/henrikj/Build/dovecot-2.1.15/src/lib-dovecot/.libs/libdovecot.so: attempted multiple inclusion of file ld: warning: file ../../src/lib-dovecot/.libs/libdovecot.so: linked to /home/henrikj/Build/dovecot-2.1.15/src/lib-dovecot/.libs/libdovecot.so: attempted multiple inclusion of file Undefined first referenced symbol in file $XBKaiwbjOCHRV5w.array_append_array_i..L93 dsync/libdsync.a(dsync-brain.o) $XBKaiwbjOCHRV5w.array_append_array_i..L94 dsync/libdsync.a(dsync-brain.o) $XBKaiwbjOCHRV5w.array_append_array_i..L95 dsync/libdsync.a(dsync-brain.o) ld: fatal: symbol referencing errors. No output written to .libs/doveadm *** Error code 2 make: Fatal error: Command failed for target `doveadm' Current working directory /home/henrikj/Build/dovecot-2.1.15/src/doveadm *** Error code 1 Any suggestions? Thanks Henrik

1 0

Re: [Dovecot] problems with imaps
by Joseph Tam 23 Feb '13

23 Feb '13

On 2013-02-22, Matthias Leopold wrote: > with thunderbird 10.0.12 i can't connect to port 993 and get errors in > the logs like > > TLS: SSL_read() failed: error:14094412:SSL > routines:SSL3_READ_BYTES:sslv3 alert bad certificate > > (certificate generated by dovecot mkcert.sh) I haven't come across any problems with our use of self-signed certificates, but I run the latest 2.1.x dovecot so maybe there's some SSL changes between our version and your's. The fact that the same certificate works for other clients, and also for TLS on Thunderbird seem to suggest Thunderbird is fumbling it. But maybe you can try the command diagnostic from the command line "openssl s_client -connect yourserver:993 ..." or use one of the online certificate checkers to get some useful diagnostics. > TLS: SSL_read() failed: error:14094418:SSL > routines:SSL3_READ_BYTES:tlsv1 alert unknown ca This error entry pops up in my logs once in a while. I think the error might be misleading since the error message happen in the middle of a long sequence of successful connections. Also check that the client is actually using the right security mode (not TLS or clear), perhaps by doing a network snoop. Joseph Tam <jtam.home(a)gmail.com>

2 1

[Dovecot] doveadm user error messages
by Thomas Leuxner 22 Feb '13

22 Feb '13

I use 'doveadm user' in some scripts and the exit codes are fine for my purposes. During tests it transpired that Dovecot 2.1 would give an error message when querying for non-existent users. Dovecot 2.2 just exits without a word. Not sure this is intended behaviour: $ doveadm user foo $ echo $? 67 $ id foo id: foo: No such user Regards Thomas

2 2

[Dovecot] assert in mail-transaction-log-file.c
by Anton Yuzhaninov 22 Feb '13

22 Feb '13

I setup new server with dovecot 2.1.12 and mboxes from backup (moboxes was used with dovecot 1.1.6). OS: FreeBSD 8.3-STABLE, amd64. All index files from dovecot 1.1.6 has been deleted. Some times dovecot processes aborted. E. g. Jan 25 03:01:52 ost dovecot: lda(user(a)example.ru): Panic: file mail-transaction-log-file.c: line 1694 (mail_transaction_log_file_map): assertion failed: (file->buffer != NULL) (gdb) bt #0 0x0000000800bd610c in kill () from /lib/libc.so.7 #1 0x0000000800bd4f0b in abort () from /lib/libc.so.7 #2 0x000000080099b339 in default_fatal_finish (type=72898, status=0) at failures.c:191 #3 0x000000080099b3be in i_syslog_fatal_handler (ctx=<value optimized out>, format=<value optimized out>, args=<value optimized out>) at failures.c:413 #4 0x000000080099a8a3 in i_panic (format=<value optimized out>) at failures.c:263 #5 0x000000080081bdb6 in mail_transaction_log_file_map (file=0x801670e00, start_offset=40, end_offset=18446744073709551615) at mail-transaction-log-file.c:1694 #6 0x000000080081da41 in mail_transaction_log_view_set (view=0x801654a80, min_file_seq=1, min_file_offset=40, max_file_seq=2, max_file_offset=18446744073709551615, reset_r=0x7fffffffbedf) at mail-transaction-log-view.c:228 #7 0x000000080081519d in mail_index_sync_map (_map=0x801670b08, type=MAIL_INDEX_SYNC_HANDLER_HEAD, force=false) at mail-index-sync-update.c:950 #8 0x0000000800805be0 in mail_index_map (index=0x801670a80, type=MAIL_INDEX_SYNC_HANDLER_HEAD) at mail-index-map-read.c:415 #9 0x0000000800800bde in mail_index_refresh (index=<value optimized out>) at mail-index.c:735 #10 0x00000008007b2344 in mbox_sync_header_refresh (mbox=0x801680840) at mbox-sync.c:1652 #11 0x00000008007b24e2 in mbox_sync_has_changed_full (mbox=0x801680840, leave_dirty=true, empty_r=0x7fffffffe2e8) at mbox-sync.c:1729 #12 0x00000008007ae1a7 in mbox_save_begin (_ctx=0x80161a500, input=0x801669660) at mbox-save.c:296 #13 0x00000008018025ae in notify_save_begin (ctx=0x80161a500, input=0x801669660) at notify-storage.c:130 #14 0x00000008014261fd in quota_save_begin (ctx=0x80161a500, input=0x801669660) at quota-storage.c:251 #15 0x00000008007cfecc in mailbox_save_begin (ctx=0x7fffffffe3d0, input=0x801669660) at mail-storage.c:1652 #16 0x00000008007ca48d in mail_storage_copy (ctx=0x80161a500, mail=0x801637040) at mail-copy.c:68 #17 0x000000080180269d in notify_copy (ctx=0x80161a500, mail=0x801637040) at notify-storage.c:105 #18 0x00000008014262f6 in quota_copy (ctx=0x80161a500, mail=0x801637040) at quota-storage.c:200 #19 0x00000008007cfe04 in mailbox_copy (_ctx=<value optimized out>, mail=0x801637040) at mail-storage.c:1721 #20 0x0000000801b4b706 in act_store_execute (action=<value optimized out>, aenv=0x8016a3070, tr_context=<value optimized out>) at sieve-actions.c:551 #21 0x0000000801b428de in sieve_result_execute (result=0x8016a3040, keep=0x0) at sieve-result.c:1117 #22 0x0000000801b52d5a in sieve_execute (sbin=<value optimized out>, msgdata=<value optimized out>, senv=<value optimized out>, ehandler=<value optimized out>, flags=<value optimized out>, keep=0x0) at sieve.c:508 #23 0x0000000801a0aec6 in lda_sieve_deliver_mail (mdctx=0x7fffffffe970, storage_r=0x7fffffffe8e8) at lda-sieve-plugin.c:424 #24 0x0000000800651735 in mail_deliver (ctx=0x7fffffffe970, storage_r=0x7fffffffe8e8) at mail-deliver.c:390 #25 0x0000000000402ffa in main (argc=6, argv=0x7fffffffea78) at main.c:434 (gdb) f 5 (gdb) p *file $1 = {log = 0x8016463d0, next = 0x0, refcount = 5, filepath = 0x801620420 "(in-memory index).log", fd = -1, st_ino = 13919248, st_dev = 109, last_mtime = 1359082912, last_size = 0, hdr = {major_version = 1 '\001', minor_version = 2 '\002', hdr_size = 40, indexid = 1359082912, file_seq = 2, prev_file_seq = 0, prev_file_offset = 0, create_stamp = 1359082912, initial_modseq = 0, compat_flags = 1 '\001', unused = "\000\000", unused2 = 0}, mmap_buffer = {data = 0x0, used = 0, priv = {0x0, 0x0, 0x0, 0x0, 0x0}}, buffer = 0x0, buffer_offset = 0, mmap_base = 0x0, mmap_size = 0, sync_offset = 40, sync_highest_modseq = 0, saved_tail_offset = 0, max_tail_offset = 0, saved_tail_sync_offset = 0, index_deleted_offset = 0, index_undeleted_offset = 0, modseq_cache = {{offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}, { offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}, { offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}, {offset = 0, highest_modseq = 0}}, file_lock = 0x0, lock_created = 0, locked = 0, locked_sync_offset_updated = 0, corrupted = 0} Also same aborts can happen in imap processes: Jan 24 16:35:04 ost dovecot: imap(user(a)example.ru): Panic: file mail-transaction-log-file.c: line 1694 (mail_transaction_log_file_map): assertion failed: (file->buffer != NULL) -- Anton Yuzhaninov

2 2

[Dovecot] Ubuntu-12.04 packages for latest Dovecot release
by 3.listas＠adminlinux.com.br 22 Feb '13

22 Feb '13

Hi, Does anyone here know a trusted package repository for Ubuntu LTS which has always the latest version of Dovecot? Thanks ! -- Thiago Henrique adminlinux.com.br

3 4

[Dovecot] problems with imaps
by Matthias Leopold 22 Feb '13

22 Feb '13

hi, i'm using dovecot 1.2.15 with self signed certificates using starttls on ports 110/143 works ok with thunderbird 10.0.12 (and i guess most other clients) using imaps on port 993 works with outlook 2002. with thunderbird 10.0.12 i can't connect to port 993 and get errors in the logs like TLS: SSL_read() failed: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate (certificate generated by dovecot mkcert.sh) or TLS: SSL_read() failed: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca (certificate generated by own openssl cmdline) i know that a lot of information is missing, but since this looks like a more general problem i dare to ask for help before posting more config details thx matthias

3 3