- dovecot - dovecot.org

Initial development work on fts-elasticsearch
by Joshua Atkins 18 Jan '15

18 Jan '15

Hi all, I have started work on an ElasticSearch FTS plugin for Dovecot. It's available now on GitHub at: https://github.com/ascendantcom/fts-elasticsearch Please note that it is still in heavy development right now. It supports indexing, searching and expunging with some caveats (no multi-mailbox search or attachments). Pull requests, suggestions and bug reports very welcome! Cheers, Josh

2 1

dsync SSL fails since 2.2.15
by Martin Carlé 16 Jan '15

16 Jan '15

Hello list, dovecot ran rock-solid on OSX Mavericks for about 1 year replicating my mail between 2 servers via dsync with SSL as that is well described here: http://wiki2.dovecot.org/Replication After upgrading to 2.2.15, dsync gets stuck with the Error: "Received invalid SSL certificate" even though neither any of the dovecot configs nor the certs, keys or the CA have changed! When I simply outcomment SSL and switch dsync to use tcp (instead of tcps) everthing replications still works like a charm. Please help me to get SSL back working! I did a lot of testing and come up with a concrete QUESTION below, hopefully leading the way out of this trap. What happend ============= 2 days before I upgraded one of the machines to OSX Yosemite. Along with this, I also upgraded to dovecot 2.2.15 via homebrew (unfortunately on both machines at once). During this process, also openssl was updated to "OpenSSL 1.0.1k 8 Jan 2015". If checking the unchanged certs against the CA, however, the results are still "OK". 1st check: OK ========== sudo /usr/bin/openssl verify -CAfile /etc/ssl/ca/dovecotCA.pem /etc/ssl/certs/dovecot_on27_signed_cert.pem Password: /etc/ssl/certs/dovecot_on27_signed_cert.pem: OK 2nd check: OK (providing the CAfile and connecting to the doveadm_port) =========== openssl s_client -CAfile /etc/ssl/ca/dovecotCA.pem -connect on27.linkpc.net:8082 CONNECTED(00000003) depth=1 CN = dovecotCA2, O = dovecot, OU = dovecot, ST = dovecot, C = AF, L = dovecot, emailAddress = mc(a)aiguphonie.com verify return:1 depth=0 CN = on27.linkpc.net, O = dovecot, OU = dovecot, ST = dovecot, C = AF, L = dovecot, emailAddress = mc(a)aiguphonie.com verify return:1 --- Certificate chain 0 s:/CN=on27.linkpc.net/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/emailA… i:/CN=dovecotCA2/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/emailAddress=mc@aiguphonie.com --- Server certificate -----BEGIN CERTIFICATE----- dmVjb3RDQTIxEDAOBgNVBAoMB2RvdmVjb3QxEDAOBgNVBAsMB2RvdmVjb3QxEDAO [...] +g== -----END CERTIFICATE----- subject=/CN=on27.linkpc.net/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/… issuer=/CN=dovecotCA2/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/emailAddress=mc(a)aiguphonie.com --- No client certificate CA names sent --- SSL handshake has read 1709 bytes and written 487 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: zlib compression Expansion: zlib compression SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: C4DDBA1FA50039FA5D94EF2359BA037B3903D66B6B637CA0733A9216BFCC3996 Session-ID-ctx: Master-Key: 0495D21CA11AA54856D78B48C3DBE9B70EFFB65F13224B430D2B4B2F80F12BE5A89F31454F9577F22F5DDC26FDBAAFAC Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: [...] 0090 - 2d 97 37 15 bd a9 be 68-c1 79 fa dd d8 75 76 3f -.7....h.y...uv? Compression: 1 (zlib compression) Start Time: 1421443766 Timeout : 300 (sec) Verify return code: 0 (ok) --- - Yet, testing dsync yields: ERROR ========================================================================================== sudo -u _vmail doveadm -v sync -u test tcps:on27.linkpc.net Password: doveadm(test): Info: Received invalid SSL certificate: certificate signature failure: /CN=on27.linkpc.net/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/emailAddress=mc(a)aiguphonie.com doveadm(test): Error: doveadm server disconnected before handshake: Received invalid SSL certificate: certificate signature failure: /CN=on27.linkpc.net/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/emailAddress=mc(a)aiguphonie.com doveadm(test): Fatal: Disconnected from remote: Received invalid SSL certificate: certificate signature failure: /CN=on27.linkpc.net/O=dovecot/OU=dovecot/ST=dovecot/C=AF/L=dovecot/emailAddress=mc(a)aiguphonie.com QUESTION ========= So the question clearly is, how does dovecot check the cert against the CA exactly? Is there a call to the openssl cmd or is the library linked into dovecotadm? If liked, what version is used and how can I possibly change it? or: What's wrong with my CA and cert(s) all of a sudden? How can I create new CA for two certs fitting the (new) needs of dovecotadm? THANK YOU! ========================================================================================== Here are my full but rather simple configs of both machines: ========================================================================================== 1st machine: Yosemite # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # OS: Darwin 14.0.0 x86_64 base_dir = /var/run/dovecot/ default_internal_user = _dovecot default_login_user = _dovenull doveadm_password = secret doveadm_port = 8082 log_path = /usr/local/var/log/dovecot/error mail_home = /var/vmail/%n mail_location = maildir:~/mail mail_plugin_dir = /usr/local/lib/dovecot mail_plugins = " notify replication" namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = scheme=CRYPT username_format=%u /usr/local/etc/dovecot_authDBs/authDBs_on27/passwd.dovecot driver = passwd-file } plugin { mail_replica = tcps:nephelism.linkpc.net replication_full_sync_interval = 1 hour } protocols = imap service aggregator { fifo_listener replication-notify-fifo { user = _vmail } unix_listener replication-notify { user = _vmail } } service auth { unix_listener auth-userdb { group = _vmail mode = 0666 user = _vmail } } service doveadm { inet_listener { port = 8082 ssl = yes } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = _vmail } } ssl = required ssl_cert = </etc/ssl/certs/dovecot_on27_signed_cert.pem ssl_client_ca_file = /etc/ssl/ca/dovecotCA.pem ssl_key = </etc/ssl/private/dovecot_on27_signed_key_noenc.pem userdb { args = username_format=%u /usr/local/etc/dovecot_authDBs/authDBs_on27/userdb.dovecot driver = passwd-file } protocol imap { mail_max_userip_connections = 40 } ========================================================================================== ========================================================================================== 2nd machine: Mavericks # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # OS: Darwin 13.1.0 x86_64 base_dir = /var/run/dovecot/ default_internal_user = _dovecot default_login_user = _dovenull doveadm_password = secret doveadm_port = 8082 log_path = /usr/local/var/log/dovecot/error mail_home = /var/vmail/%n mail_location = maildir:~/mail mail_plugin_dir = /usr/local/lib/dovecot mail_plugins = " notify replication" namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = scheme=CRYPT username_format=%u /usr/local/etc/dovecot_authDBs/authDBs_nephelism/passwd.dovecot driver = passwd-file } plugin { mail_replica = tcps:on27.linkpc.net replication_full_sync_interval = 1 hour } protocols = imap service aggregator { fifo_listener replication-notify-fifo { user = _vmail } unix_listener replication-notify { user = _vmail } } service auth { unix_listener auth-userdb { group = _vmail mode = 0666 user = _vmail } } service doveadm { inet_listener { port = 8082 ssl = yes } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 user = _vmail } } ssl = required ssl_cert = </etc/ssl/certs/dovecot_nephelism_signed_cert.pem ssl_client_ca_file = /etc/ssl/ca/dovecotCA.pem ssl_key = </etc/ssl/private/dovecot_nephelism_signed_key_noenc.pem userdb { args = username_format=%u /usr/local/etc/dovecot_authDBs/authDBs_nephelism/userdb.dovecot driver = passwd-file } protocol imap { mail_max_userip_connections = 40 } -- Fetch my gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 7E3CA33F

1 0

dsync-remote: Error: Failed to set attribute
by Alexander Moisseev 16 Jan '15

16 Jan '15

1. I am looking for backup solution for mdbox with SIS and experimenting with dsync. I am planning to do it like that: doveadm backup [deSISing] => transmitting to remote host => dsync-server [SISing] Is there more convenient method? 2. I have dovecot 2.2.15 on both hosts. The local host is a mail server. Dovecot on the remote host neither running nor configured (I have created an empty dovecot.conf in the config dir). When I tried to back up mailbox from local to remote host with command doveadm backup -P -u user(a)domain.tld \ ssh -c blowfish -i /tmp/vmail/.ssh/id_dsa vmail@remotehost \ doveadm \ -o mail_location=mdbox:/tmp/vmail/domain.tld/user/mdbox \ -o mail_attachment_dir=/tmp/vmail/attachments \ dsync-server I got error messages: dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/files/roundcube: Internal attributes cannot be changed directly dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/files/main: Internal attributes cannot be changed directly dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/files/test: Internal attributes cannot be changed directly dsync-remote(vmail): Error: Mailbox INBOX: Failed to set attribute vendor/vendor.dovecot/pvt/server/sieve/default: Internal attributes cannot be changed directly What do that errors mean? Does dsync try to replicate sieve scripts? -- Alexander

1 0

[SERVERBUG] failed to send mail with SA and antispam plugin
by ML mail 16 Jan '15

16 Jan '15

Hello, I am using the antispam plugin of Dovecot with SpamAssassin and in some cases when users move back mails from the Spam folder to their INBOX (false positive) they get the following error message: [SERVERBUG] failed to send mail Does anyone have an idea what could be the issue here? It is quite annoying as the mail then stays in the Spam folder and can not been re-learnt has ham. Best regards ML

4 6

File locking issues
by Oliver Welter 16 Jan '15

16 Jan '15

Hi All, I upgrade my mail server from doveot 1.1.7 to 2.2.13 and encounter problems with file locking issues. The server has around 400 clients using IMAP and I get tons of "Warning: fscking index file .. dovecot.index" and "Error: Corrupted transaction log". Sometimes the transaction log problem is followed by "Panic: file mail-transaction-log.c: line 350 (mail_transaction_log_set_mailbox_sync_pos): assertion failed: (file_offset >= log->head->saved_tail_offset)" I tried all offered locking methods (fcntl, flock, dotlock) without success. The server runs inside a virtual guest (linux-vserver which is similar to LXC) using an overlay filesystem. I suspect the overlayfs but have no idea how to test this, besides - as I understand - dotlock should not rely on the fs locking at all :( I even wonder why the old dovecot worked for nearly 10 years without any problems, even under higher loads. The config is pretty straightforward: # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.4.57-vs2.3.3.9-g67e6589-dirty x86_64 Gentoo Base System release 2.2 auth_cache_negative_ttl = 0 auth_cache_size = 50 M auth_mechanisms = plain cram-md5 debug_log_path = /var/log/dovecot-debug.log default_client_limit = 900 default_process_limit = 250 disable_plaintext_auth = no first_valid_gid = 89 first_valid_uid = 89 info_log_path = /var/log/dovecot.log last_valid_gid = 89 last_valid_uid = 89 listen = * lock_method = dotlock login_greeting = Serverpilot.net Mail Service mail_gid = 89 mail_location = maildir:~/.maildir mail_uid = 89 namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocols = imap pop3 service imap-login { process_min_avail = 5 service_count = 0 } ssl_cert = </var/qmail/control/servercert.pem ssl_cipher_list = ALL:!EXPORT:!LOW:!MEDIUM:!aNULL:+RC4:@STRENGTH ssl_dh_parameters_length = 2048 ssl_key = </var/qmail/control/servercert.pem ssl_protocols = !SSLv2 !TLSv1.2 userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol imap { mail_max_userip_connections = 25 } Any ideas are welcome. Oliver -- Protect your environment - close windows and adopt a penguin!

1 0

Questions regarding imaptest
by rub zorghy 15 Jan '15

15 Jan '15

Hi, The measurements were created under the following conditions : • operating system : Red Hat Enterprise Linux Server release 6.6 (Santiago) kernel in version 2.6.32-504.el6.x86_64 • virtual server (VMware) with an Intel(R) Xeon(R) 4vCPU E5649 x 2,526 GHz (2 cores per virtual socket) and 4 GB RAM • 7,200 RPM SATA 1TB (FC SAN IBM System Storage N3400) • all file systems had been formated in ext4 • Dovecot v.2.0.9 • imaptest v.1.0.0 • LDAP 389 Directory Server v.1.2.2 • free pagecache, dentries and inodes: echo 3 > /proc/sys/vm/drop_caches and services restarted before the test "Tuning" based upon the book « *Dovecot: POP3/IMAP servers for enterprises and ISPs* » Peer Heinlein : ---%<------------------------------------------ imap {} (10-master.conf). service imap { [...] process_limit = 8192 [...] } auth {} service auth { [...] client_limit = 9516 [...] } process_min_avail = 16 ---%<------------------------------------------ mail_max_userip_connections = 100 The test results shows : Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% 30% 5% 324 142 128 263 299 405 92 153 320 336 335 800/800 212 92 92 182 166 233 59 87 178 201 202 800/800 - 4051 stalled for 16 secs in command: 8 APPEND "INBOX" () {3251+} 315 160 134 300 275 400 85 136 300 317 317 800/800 - 4051 stalled for 17 secs in command: 8 APPEND "INBOX" () {3251+} 247 130 163 292 251 366 99 115 239 240 240 800/800 222 122 117 234 279 383 112 147 263 222 222 800/800 188 93 94 177 168 232 65 87 162 196 195 800/800 290 126 119 260 262 383 102 126 283 282 283 800/800 240 125 107 243 259 353 102 130 253 256 256 800/800 276 158 144 291 280 400 101 149 266 261 261 800/800 254 95 89 189 235 313 78 118 243 253 253 800/800 82 829 926 861 722 593 455 628 1695 2252 4 ms/cmd avg Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe Logo 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% 100% 30% 5% 148 59 55 126 119 174 43 64 130 147 147 800/800 - 8770 stalled for 16 secs in command: 4 FETCH 1:2 (INTERNALDATE UID INTERNALDATE) 261 128 139 247 252 369 101 125 258 269 269 800/800 302 159 158 307 282 398 109 126 296 298 297 800/800 234 96 118 239 270 369 98 138 357 428 427 800/800 361 213 220 450 383 596 151 197 240 165 166 800/800 192 93 92 206 256 352 104 118 297 289 290 800/800 - 9806 stalled for 16 secs in command: 4 FETCH 1 (RFC822.SIZE) 245 85 72 162 128 190 52 73 157 174 173 800/800 - 9806 stalled for 17 secs in command: 4 FETCH 1 (RFC822.SIZE) 294 152 165 348 360 526 136 179 313 308 309 800/800 272 87 79 174 164 229 54 85 218 230 230 800/800 218 152 152 310 325 467 117 152 265 236 221 798/800 119 830 852 859 707 566 511 702 1672 2161 6 ms/cmd avg I have three questions : 1) how to interpret each row except the last one ? 2) how to explain the stalled commands ? 3) why benchmark shows too few read disk I/O ? Thanks so much, rub

1 1

Re: dovecot.org slow & imap search now blazing fast
by Joseph Tam 15 Jan '15

15 Jan '15

From: Laeeth Isharc <laeeth(a)laeeth.com> > In case you were not aware, I wanted to let you know that dovecot.org > has been very slow (often almost unusable) for at least several days now. Not for me as of the writring of this Email. Seems OK to me. Check your browser, network, etc. Joseph Tam <jtam.home(a)gmail.com>

1 0

HG e3640ccaa76d crashes upon IMAP SEARCH
by Thomas Leuxner 15 Jan '15

15 Jan '15

$ doveconf -n | head # 2.2.15 (e3640ccaa76d): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 # OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8 $ /usr/lib/dovecot/imap -u tlx(a)leuxner.net * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL RIGHTS=texk] Logged in as tlx(a)leuxner.net a select "Shops/Amazon" * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk) * OK [PERMANENTFLAGS (\Answered \Flagged \Seen \Draft $NotJunk NotJunk \*)] Flags permitted. * 491 EXISTS * 0 RECENT * OK [UNSEEN 491] First unseen. * OK [UIDVALIDITY 1213986801] UIDs valid * OK [UIDNEXT 568] Predicted next UID * OK [HIGHESTMODSEQ 1029] Highest a OK [READ-WRITE] Select completed (0.003 secs). b search charset utf-8 body "Sleeve" imap(tlx(a)leuxner.net): Panic: file message-decoder.c: line 346 (message_decode_body): assertion failed: (ctx->translation_size <= sizeof(ctx->translation_buf)) imap(tlx(a)leuxner.net): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x7749f) [0x7ffe68e6d49f] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7ffe68e6d57a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ffe68e1df65] -> /usr/lib/dovecot/libdovecot.so.0(message_decoder_decode_next_block+0x688) [0x7ffe68e5c858] -> /usr/lib/dovecot/libdovecot.so.0(message_search_more+0xa1) [0x7ffe68e60881] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0x5b) [0x7ffe68e60a4b] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa9b7b) [0x7ffe69187b7b] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) [0x7ffe6915d675] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa951a) [0x7ffe6918751a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xab76a) [0x7ffe6918976a] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0xe2) [0x7ffe69189e32] -> /usr/lib/dovecot/modules/lib90_stats_plugin.so(+0x2daa) [0x7ffe66dc3daa] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x27) [0x7ffe691627a7] -> /usr/lib/dovecot/imap() [0x41da2f] -> /usr/lib/dovecot/imap() [0x41e0a2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xaa) [0x7ffe68e7f6ca] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x69) [0x7ffe68e804e9] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7ffe68e7f549] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7ffe68e7f5c8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7ffe68e23433] -> /usr/lib/dovecot/imap(main+0x2a1) [0x40c591] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7ffe68a88ead] -> /usr/lib/dovecot/imap() [0x40c6f5] I tried to make it dump a core but failed... Regards Thomas

2 2

attn rick van rein (midget)
by Laeeth Isharc 14 Jan '15

14 Jan '15

From 11 Oct 2013 (Rick van Rein) "Hello, I love Dovecot, but when developing a small IMAP tool, I ran into searching behaviour can easily be optimised. Please forgive a rather detailed suggestion. This was on Dovecot 1.2.15 on Debian Squeeze. My tool? It's called "midget" and retrieves documents from an IMAP box based on their mid: or cid: identifier, as per RFC 2392. I thought this would be useful to retrieve email attachments into a remote shell environment. When using Kerberos, the credentials and a strong hint of one's mail address are already present anyway, DNS SRV does the rest. If you want to see the early code, I'll be happy to share it here." Hi Rick. Thanks for the suggested optimisation - I am not sure if it is you who I have to think for contributing towards making IMAP search fast again. In any case, if you see this I would be interested to see the source for midget if you are willing to share. If you don't want to post to the list then my email is laeeth at laeeth.com Thanks. Laeeth

1 0

dovecot.org slow & imap search now blazing fast
by Laeeth Isharc 14 Jan '15

14 Jan '15

Hi. In case you were not aware, I wanted to let you know that dovecot.org has been very slow (often almost unusable) for at least several days now. Also, I upgraded my dovecot from 2.2.10 to 2.2.15. Thank you for fixing IMAP search. I have c. 2mm messages, and it had become unusable before but it is now very fast. (I saw a reference to an optimisation on the mailing list from back in Oct 2013, so maybe that was it). Regards, Laeeth

1 0