- dovecot - dovecot.org

sieve duplicate locking
by Alexander 'Leo' Bergolth 20 Oct '16

20 Oct '16

Hi! Does the duplicate sieve plugin do any locking to avoid duplicate parallel delivery of the same message? I sometimes experience duplicate mail delivery of messages with the same message-id, despite the use of a sieve duplicate filter. According to the log files, those messages are delivered in the same second by two parallel dovecot-lda processes. (Duplicate filtering works fine in all other cases.) RFC7352 states that the ID of a message may only be committed to the duplicate tracking list at the _end_ of a successful script execution, which may lead to race conditions. Maybe I am running into this? Is there an easy way to serialize mail delivery using some locking inside sieve? Or do I have to serialize per-user dovecot-lda delivery? Any experiences with that? I am using dovecot-2.2.25 and pidgeonhole-0.4.15. Mail is delivered using postfix-2.10 and dovecot-lda as mailbox_command. Mailbox format is maildir with LAYOUT=fs. Cheers, --leo -- e-mail ::: Leo.Bergolth (at) wu.ac.at fax ::: +43-1-31336-906050 location ::: IT-Services | Vienna University of Economics | Austria

2 2

Re: Massive LMTP Problems with dovecot
by Ralf Hildebrandt 20 Oct '16

20 Oct '16

> We'll need the `dovecot -n` output first. Here we go: > # 2.2.devel (933d16f): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.4.devel (63f9b42) > # OS: Linux 3.13.0-98-generic x86_64 Ubuntu 14.04.5 LTS > default_vsz_limit = 2 G > lmtp_user_concurrency_limit = 10000 > mail_attachment_dir = /home/copymail/attachments > mail_location = mdbox:~/mdbox > mail_plugins = zlib fts fts_lucene > mdbox_rotate_size = 128 M > namespace inbox { > inbox = yes > location = > mailbox Drafts { > special_use = \Drafts > } > mailbox Junk { > special_use = \Junk > } > mailbox Sent { > special_use = \Sent > } > mailbox "Sent Messages" { > special_use = \Sent > } > mailbox Trash { > special_use = \Trash > } > prefix = > } > passdb { > args = username_format=%u /etc/dovecot/passwd > driver = passwd-file > } > plugin { > fts = lucene > fts_autoindex = yes > fts_languages = de,en > fts_lucene = whitespace_chars=@. > sieve = file:~/sieve;active=~/.dovecot.sieve > zlib_save = gz > zlib_save_level = 5 > } > protocols = " imap lmtp" > service imap-login { > inet_listener imap { > address = 127.0.0.1 > port = 143 > } > inet_listener imaps { > port = 993 > ssl = yes > } > } > service lmtp { > inet_listener lmtp { > address = 141.42.1.208 > port = 1025 > } > unix_listener /var/spool/postfix/private/dovecot-lmtp { > group = postfix > mode = 0660 > user = postfix > } > } > ssl_ca = /etc/ssl/certs/ca-certificates.crt > ssl_cert = </etc/ssl/certs/mail-cbf.pem > ssl_cipher_list = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 > ssl_key = # hidden, use -P to show it > ssl_prefer_server_ciphers = yes > ssl_protocols = !SSLv2 !SSLv3 > userdb { > args = username_format=%u /etc/dovecot/passwd > driver = passwd-file > } > protocol lmtp { > mail_plugins = zlib fts fts_lucene > } Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 6

Lmtp Fatal Error
by Sven Roellig 19 Oct '16

19 Oct '16

Hi,dovecot is create an Fatal Panik Error. <AM5DMwaCBlgpDQAAWm89Cw>: Fatal: master: service(lmtp): child 3369 killed with signal 6 (core dumps disabled) <1jx3DhuCBlg1DQAAWm89Cw>: Panic: file lda-sieve-plugin.c: line 447 (lda_sieve_execute_scripts): assertion failed: (script != NULL) <1jx3DhuCBlg1DQAAWm89Cw>: Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x93fae) [0x7f547f6a3fae] -> /usr/lib/dovecot/libdovecot.so.0(+0x9409c) [0x7f547f6a409c] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f547f63d56e] -> /usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3ae8) [0x7f547d765ae8] -> /usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7f547fc709a9] -> dovecot/lmtp(+0x7201) [0x7f54800a1201] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x5f) [0x7f547f6b88bf] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10a) [0x7f547f6b9d8a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f547f6b8965] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x30) [0x7f547f6b8b00] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f547f643ac3] -> dovecot/lmtp(main+0x1c9) [0x7f548009f2c9] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f547f286b45] -> dovecot/lmtp(+0x53ae) [0x7f548009f3ae] Normal Mails are not deliverd but filterd mails are deliverd. System is Debian 8.6 Dovecot Core is 2%3a2.3.0-alpha0-1-auto+385 AMD64 can anyone help. Sven

3 4

Iteration base for LDAP
by Taso N. Devetzis 19 Oct '16

19 Oct '16

Greetings, The iteration machinery uses the LDAP search base set with the "base" directive (typically from dovecot-ldap.conf.ext); the same base used during nominal operations (e.g., passdb/userdb searches). Consider a directory: dc=ROOT |_ dc=foo,dc=com,dc=ROOT (foo.com subtree) |_ dc=bar,dc=net,dc=ROOT (bar.net subtree) A search base setting appropriate for mail operations might be: base = dc=%Dd,dc=ROOT # e.g. dc=foo,dc=com,dc=ROOT for user(a)foo.com This fails when iterating, as the variable substitution is meaningless in this context (and even a static subtree search base would only cover a portion of the overall directory during iterative searches). Setting the base to "dc=ROOT" obviously solves the issue at the expense of searching the entire directory for all operations. This is less than optimal. I could not find a way to override this setting at runtime via a doveadm option or similar. Ideally, a separate "iterate_base" setting would solve this issue. Any other solutions? Thanks, /taso

1 0

RE: First steps in Dovecot; IMAP not working
by Joseph Tam 19 Oct '16

19 Oct '16

> > doveadm user mailtest > field value > uid 1002 > gid 8 > home /home/mailtest > mail mbox:~/mail:INBOX=/var/mail/mailtest > system_groups_user mailtest Looks fine. > [log files] I'll try to locate them. Your best hope. Maybe this helps doveconf -n | grep log_ Or just force it somewhere by settings the *log_path configuration values. Also your mail server's log when trying to deliver. Joseph Tam <jtam.home(a)gmail.com>

1 0

Dict proxy client returning empty string instead of multiline string
by Pierre Jaury 18 Oct '16

18 Oct '16

Hello, I am using a dict proxy for my sieve extdata plugin to access some fields from an SQLite database (autoreply text and other database-configured items). All tests are performed against version 2.2.25. $ dovecot --version 2.2.25 (7be1766) My configuration looks like: dict { sieve = sqlite:/etc/dovecot/pigeonhole-sieve.dict } [...] sieve_extdata_dict_uri = proxy::sieve I am able to read pretty much any attribute without any issue, except when the value contains a special character like "\r" or "\n". By using the doveadm dict client, I narrowed it down to the dictionary management part (either server, protocol or client). I was suspecting escaping functions from "lib/strescape.c" (mostly str_tabescape and its counterpart, used by "lib-dict/client.c"), so I monitored socket communications. It seems that escaping is done properly on the server and the socket is not an issue either. The following strace dump results from running "doveadm dict get" against the dict socket: connect(8, {sa_family=AF_UNIX, sun_path="..."}, 110) = 0 fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0 [...] write(8, "H2\t0\t0\tadmin(a)domain.tld\tsieve\n", 30) = 30 [...] read(8, "Otest\1r\1ntest\n", 8192) = 14 Indeed "\1r" and "\1n" are the escape sequences used by "lib/strescape.c". I went deeped and debugged the call to "dict_lookup" performed by doveadm. Indeed the client gets the proper string from the socket and to my surprise, it is properly unescaped. Then, in "client_dict_lookup" ("lib-dict/dict-client.c"), the call to "p_strdup" returns an empty string (null byte set at the target address). Before the call to the dict "->lookup" attribute (client_dict_lookup): RAX: 0x7ffff73a37c0 (push r14) RBX: 0x6831b8 ("priv/reply_body") RCX: 0x7fffffffe240 --> 0x682a60 --> 0x6831b8 ("priv/reply_body") RDX: 0x6831b8 ("priv/reply_body") RSI: 0x683288 --> 0x7ffff7653120 --> 0x7ffff73ea620 ([...]) RDI: 0x690ad0 --> 0x7ffff7400713 --> 0x75250079786f7270 ('proxy') 0x7ffff73a1f10 <dict_lookup+32>: mov rcx,r11 (value_r) 0x7ffff73a1f13 <dict_lookup+35>: mov rdx,r8 (key) 0x7ffff73a1f16 <dict_lookup+38>: mov rsi,r10 (pool) 0x7ffff73a1f19 <dict_lookup+41>: mov rdi,r9 (dict) 0x7ffff73a1f1c <dict_lookup+44>: add rsp,0x8 => 0x7ffff73a1f20 <dict_lookup+48>: jmp rax Before the call to p_strdup in "client_dict_lookup": RSI: 0x6832d8 ("test\r\ntest") (lookup.result.value) RDI: 0x683288 --> 0x7ffff7653120 --> [...] (pool) RAX: 0x0 (result) 0x7ffff73a384f: nop 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] 0x7ffff73a3855: mov rdi,r14 => 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup@plt> 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 0x7ffff73a386f: mov eax,ebx After the call: 0x7ffff73a3850: mov rsi,QWORD PTR [rsp+0x8] 0x7ffff73a3855: mov rdi,r14 0x7ffff73a3858: call 0x7ffff736d3c0 <p_strdup@plt> => 0x7ffff73a385d: mov QWORD PTR [r13+0x0],rax 0x7ffff73a3861: mov rsi,QWORD PTR [rsp+0x18] 0x7ffff73a3866: xor rsi,QWORD PTR fs:0x28 0x7ffff73a386f: mov eax,ebx 0x7ffff73a3871: jne 0x7ffff73a38da RSI: 0x0 RDI: 0x6832d8 --> 0x0 RAX: 0x6832d8 --> 0x0 (result) It is worth noting that I can reproduce the exact same execution flow with a non-multiline result string (lookup.result.value) that is properly copied by "p_strdup" and returned in RAX, then displayed by doveadm. I am not familiar with the pooling mechanism hidden behind the call to p_strdump and not quite sure why this behaviour is emerging. Maybe I am even miles away from an understanding of the issue here, but it sounds to me like something is wrong in the way "p_strdup" performs the copy. Hope this helps, kaiyou.

2 11

Re: First steps in Dovecot; IMAP not working
by Joseph Tam 18 Oct '16

18 Oct '16

Marnaud <anic297(a)mac.com> writes: > "mailtest", the new user, is in group mail(8). In addition, I've added > write permission for "others" to /var/mail. Now, I'm trying to send a > message to "mailtest" from another, working, e-mail account and nothing > happens. This time, "doveadm log errors" is empty. > > In short, I don't get any error but no mail either. Two things to check: # What does dovecot think user "mailtest" has? doveadm user mailtest # The sticky bit should be set on /var/mail (you didn't mention # setting it. It probably doesn't have bearing on this problem, # but it will make it a little more secure. chmod 1777 /var/mail > Since "doveadm log errors" returns an empty result, where should I look > for the problem? I usually don't use this command, I look at the log file which seems to have more details. Try looking there for more diagnostics. Also, look at your MTA's logs as well. Joseph Tam <jtam.home(a)gmail.com>

4 4

Warning: Sent SIGKILL to 100 imap-login processes
by Ximo Mira 18 Oct '16

18 Oct '16

Hi, Last night we tried a traffic bypass in an existing mail environment before migration to new Dovecot backend platfom using Dovecot proxy. We are using an LDAP value for checking the proxy host of the user. POP traffic was running flawlessly, but IMAP connections started to drop when few clients connected: Oct 17 22 :26:51 master: Warning: Sent SIGKILL to 100 imap-login processes There are hundreds of these lines, always between 99 and 100 processes. Looks like some kind of limit, but not sure if its related to the Dovecot proxy machines (pool of 3 in total) or the final destination (the same IMAP server for +10000 users). Concurrect IMAP connections may rise to around 2000. Thanks.

2 1

dovecot 2.2.25 BUG: local_name is not matching correctly
by Arkadiusz Miśkiewicz 18 Oct '16

18 Oct '16

Bug report: When using dovecot 2.2.25 SNI capability it doesn't always match proper vhost config. For example if we have such config: local_name imap.example.com { ssl_cert = </etc/certs/cert.imap.example.com.pem ssl_key = </etc/certs/cert.imap.example.com.pem } but imap client sends mixedcase SNI hostname like "IMAP.example.com" then dovecot won't match above local_name imap.example.coml config section. IMO dovecot should do case insensitive comparison. Case sensitive matching for DNS names makes little sense. -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

2 2

Massive LMTP Problems with dovecot
by Ralf Hildebrandt 17 Oct '16

17 Oct '16

Currently I'm having massive problems with LMTP delivery into dovcot. dovecot/lmtp processes are piling up, eas using considerable amounts of CPU: # ps auxwww|fgrep dove root 20537 0.0 0.0 18124 1196 ? Ss 15:18 0:00 /usr/sbin/dovecot -c /etc/dovecot/dovecot.conf dovecot 20541 0.0 0.0 9620 1084 ? S 15:18 0:00 dovecot/anvil root 20542 0.0 0.0 9752 1264 ? S 15:18 0:00 dovecot/log root 20544 0.0 0.0 21168 2276 ? S 15:18 0:00 dovecot/config copymail 20580 72.8 0.0 39556 7036 ? R 15:18 2:00 dovecot/lmtp dovecot 20582 0.0 0.0 18568 1756 ? S 15:18 0:00 dovecot/auth copymail 20597 77.2 0.0 35688 5136 ? R 15:18 2:06 dovecot/lmtp copymail 20598 39.3 0.0 38060 5596 ? R 15:18 1:04 dovecot/lmtp copymail 20613 62.3 0.0 38036 5600 ? R 15:18 1:41 dovecot/lmtp copymail 20619 56.4 0.0 37732 7448 ? R 15:18 1:31 dovecot/lmtp copymail 20620 75.9 0.0 35872 5336 ? R 15:18 2:03 dovecot/lmtp copymail 20627 37.8 0.0 36480 5892 ? R 15:18 1:01 dovecot/lmtp copymail 20838 60.5 0.0 35640 5036 ? R 15:19 0:59 dovecot/lmtp copymail 20840 66.3 0.0 35920 5296 ? R 15:19 1:04 dovecot/lmtp copymail 20841 66.0 0.0 37456 6852 ? R 15:19 1:04 dovecot/lmtp copymail 20842 64.5 0.0 36424 5808 ? R 15:19 1:02 dovecot/lmtp copymail 20843 67.6 0.0 39612 7064 ? R 15:19 1:05 dovecot/lmtp doveadm stop won't stop these, I have to use kill -9 on them. I already tried disabling fts (entirely), still things won't speed up. I can't strace: # strace -p 20841 Process 20841 attached (and that's it) # dpkg -l|grep dovecot ii dovecot-core 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - core files ii dovecot-imapd 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - IMAP daemon ii dovecot-lmtpd 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - LMTP server ii dovecot-lucene 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - Lucene support ii dovecot-sieve 2:2.2.25-1~auto+57 amd64 secure POP3/IMAP server - Sieve filters support I also tried deleting the mdboxes, that also didn't change anything. Ideas? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 1