April 2018 - dovecot - dovecot.org

Panic when using doveadm fts lookup
by André Rodier 29 Apr '18

29 Apr '18

Hello, I am actually testing dovecot fts search, using doveadm fts lookup. I run this command: doveadm fts lookup -u andre subject "Test" And this generate this error: doveadm(andre): Panic: file mail-storage.c: line 1694 (mailbox_get_open_status): assertion failed: (box->opened) doveadm(andre): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7fa7ac9e8e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f19) [0x7fa7ac9e8f19] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fa7ac97ea91] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_open_status+0x68) [0x7fa7accb5f68] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x494d) [0x7fa7ab90394d] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_backend_lookup+0x4d) [0x7fa7abf6861d] -> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so(+0x2821) [0x7fa7aabff821] -> doveadm(+0x28fcd) [0x5617545e8fcd] -> doveadm(+0x29ba8) [0x5617545e9ba8] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21c) [0x5617545ea9ec] -> doveadm(doveadm_cmd_run_ver2+0x555) [0x5617545f9d25] -> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x5617545f9d77] -> doveadm(main+0x1da) [0x5617545d9afa] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa7ac5d42e1] -> doveadm(_start+0x2a) [0x5617545d9eda] Do I have to do anything special to test the dovecot fts index with doveadm lookup? I have not found the documentation on the dovecot web site. Thanks for your advices. -- André Rodier. https://github.com/progmaticltd/homebox

1 0

Listening on multiple ports
by Derek Harding 28 Apr '18

28 Apr '18

Hi, I want dovecot to listen for imaps (ssl) on both 993 and a non-standard port 8839. I've tried using variations on the inet_listener entries in 10-master.conf as shown in a couple of threads on here but none of the attempts have worked. Can someone provide me with a cut-and-paste solution and confirm into which file it should be placed (i.e. is the dovecot.conf format using the inet_listener = {address} appropriate)? -- Best wishes, Derek

3 6

Virtual mailbox doesn't refresh without deleting indexes
by Bret Martin 26 Apr '18

26 Apr '18

Hello, I have a virtual mailbox with the following dovecot-virtual file: mdbox/* -mdbox/Archive/* -mdbox/Junk -mdbox/Notes all younger 604800 not deleted This works at first, but after messages are older than 7 days, they continue to appear in this mailbox forever. Even if I close all IMAP clients and reconnect, I can't get the mailbox to omit the older-than-7-days messages without deleting its indexes. I have other virtual mailboxes that update as expected, but I don't think any of those are date-based. For example, this one "refreshes" as I'd expect: mdbox/* -mdbox/Archive/* -mdbox/Junk flagged Am I doing something wrong with the first virtual mailbox I mentioned above? I'm using dovecot 2.2.13 on Debian GNU/Linux. Thanks! --Bret

1 0

stripping dovecot...
by Rupert Gallagher 26 Apr '18

26 Apr '18

Hello, I do not need the following, and have configured both source and runtime accordingly: pam, gssapi, zlib, bzlib, lzma, pop3, quota, fts, welcome, lda. Therefore, is it *safe* to manually remove the following from /usr/lib/dovecot/? lib05_pop3_migration_plugin.a lib05_pop3_migration_plugin.la lib05_pop3_migration_plugin.so lib20_fts_plugin.a lib20_fts_plugin.la lib20_fts_plugin.so lib21_fts_squat_plugin.a lib21_fts_squat_plugin.la lib21_fts_squat_plugin.so libdovecot-fts.0.dylib libdovecot-fts.a libdovecot-fts.dylib libdovecot-fts.la lib20_doveadm_fts_plugin.a lib20_doveadm_fts_plugin.la lib20_doveadm_fts_plugin.so lib10_quota_plugin.a lib10_quota_plugin.la lib10_quota_plugin.so lib11_imap_quota_plugin.a lib11_imap_quota_plugin.la lib11_imap_quota_plugin.so lib20_quota_clone_plugin.a lib20_quota_clone_plugin.la lib20_quota_clone_plugin.so lib10_doveadm_quota_plugin.a lib10_doveadm_quota_plugin.la lib10_doveadm_quota_plugin.so lib99_welcome_plugin.a lib99_welcome_plugin.la lib99_welcome_plugin.so libdovecot-lda.0.dylib libdovecot-lda.a libdovecot-lda.dylib libdovecot-lda.la

1 1

director stuck in inifite loop on 2.2.35
by Jan-Pieter Cornet 26 Apr '18

26 Apr '18

Occasionally, all of our directors get stuck in an endless loop. Likely triggered by some network event, but I haven't been able to isolate exactly what triggers it, and am unable to reproduce at the moment. When this happens, it makes the dovecot/director process on all directors unresponsive, making login impossible. Restarting dovecot "fixes" it. Here's the info collected from the stuck process. I hope that a dovecot dev could use this to either fix a bug in the async IO code or point to a possible external source that could have triggered this. Would it help to compile dovecot with -g and run a version with debugging info, to be able to inspect more when the crash happens again? "strace" showed this endless loop: 09:27:52.837960 epoll_ctl(14, EPOLL_CTL_DEL, 213, 0x7ffe8e642cdc) = 0 09:27:52.837993 epoll_ctl(14, EPOLL_CTL_ADD, 213, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=3317538000, u64=94320799358160}}) = 0 09:27:52.838035 epoll_ctl(14, EPOLL_CTL_DEL, 213, 0x7ffe8e642cdc) = 0 09:27:52.838070 epoll_ctl(14, EPOLL_CTL_ADD, 213, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=3317538000, u64=94320799358160}}) = 0 "ltrace" showed the same loop as: 10:38:01.773867 timeout_add(0x7530, 0x55c8c351df75, 965, 0x55c8c3515a00) = 0x55c8c5c658b0 10:38:01.774060 timeout_remove(0x55c8c5acf7a8, 0, 0xffffffff, 0) = 0 10:38:01.774239 io_add(213, 1, 0x55c8c351df75, 1070) = 0x55c8c5c658b0 10:38:01.774429 io_set_pending(0x55c8c5c658b0, 1, 0x55c8c54b8aa0, 0x7fb4ab11b06a) = 0x55c8c54b8aa0 10:38:01.774598 p_strarray_dup(0x7fb4ab703660, 0x55c8c5786e00, 5, 0x55c8c3723740) = 0x55c8c56d7530 10:38:01.774771 io_remove(0x55c8c5acf788, 0x55c8c3516d60, 5, 48) = 0 10:38:01.774957 timeout_add(0x7530, 0x55c8c351df75, 965, 0x55c8c3515a00) = 0x55c8c5c658b0 10:38:01.775129 timeout_remove(0x55c8c5acf7a8, 0, 0xffffffff, 0) = 0 10:38:01.775301 io_add(213, 1, 0x55c8c351df75, 1070) = 0x55c8c5c658b0 10:38:01.775492 io_set_pending(0x55c8c5c658b0, 1, 0x55c8c54b8aa0, 0x7fb4ab11b06a) = 0x55c8c54b8aa0 10:38:01.775662 p_strarray_dup(0x7fb4ab703660, 0x55c8c56d7530, 5, 0x55c8c3723740) = 0x55c8c5786e00 10:38:01.775833 io_remove(0x55c8c5acf788, 0x55c8c3516d60, 5, 48) = 0 That fd 213 is the next-highest FD opened, and is bound to /var/run/dovecot/director-admin. In fact the last few FDs are all connections to director-admin. Last few lines of "lsof" say: director 16109 dovecot 207u unix 0xffff940d13c93800 0t0 1106341075 type=STREAM director 16109 dovecot 208u unix 0xffff9409f5280000 0t0 1106337217 /var/run/dovecot/director-userdb type=STREAM director 16109 dovecot 209u unix 0xffff940d580fa000 0t0 1106337218 type=STREAM director 16109 dovecot 210u unix 0xffff940ac0f61000 0t0 1106343439 /var/run/dovecot/director-userdb type=STREAM director 16109 dovecot 211u unix 0xffff940bed105400 0t0 1106343440 type=STREAM director 16109 dovecot 212u unix 0xffff940b76d31c00 0t0 1106342408 /var/run/dovecot/director-admin type=STREAM director 16109 dovecot 213u unix 0xffff940ab241d000 0t0 1106340510 /var/run/dovecot/director-admin type=STREAM director 16109 dovecot 214u unix 0xffff9409c317ec00 0t0 1106340511 /var/run/dovecot/director-admin type=STREAM I attached with "gdb" and did "bt" at two different moments, resulting in: (gdb) bt #0 0x00007fb4ab11b06a in epoll_ctl () from target:/lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fb4ab48964d in io_loop_handle_add () from target:/usr/lib/dovecot/libdovecot.so.0 #2 0x00007fb4ab4876b9 in ?? () from target:/usr/lib/dovecot/libdovecot.so.0 #3 0x00007fb4ab48776d in io_add () from target:/usr/lib/dovecot/libdovecot.so.0 #4 0x000055c8c35179d3 in doveadm_connections_ring_synced () #5 0x00007fb4ab488166 in io_loop_handle_timeouts () from target:/usr/lib/dovecot/libdovecot.so.0 #6 0x00007fb4ab489963 in io_loop_handler_run_internal () from target:/usr/lib/dovecot/libdovecot.so.0 #7 0x00007fb4ab48839c in io_loop_handler_run () from target:/usr/lib/dovecot/libdovecot.so.0 #8 0x00007fb4ab488548 in io_loop_run () from target:/usr/lib/dovecot/libdovecot.so.0 #9 0x00007fb4ab40c0a3 in master_service_run () from target:/usr/lib/dovecot/libdovecot.so.0 #10 0x000055c8c350b46d in main () (gdb) c Continuing. ^C Program received signal SIGINT, Interrupt. 0x00007fb4ab11b06a in epoll_ctl () from target:/lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007fb4ab11b06a in epoll_ctl () from target:/lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fb4ab489873 in io_loop_handle_remove () from target:/usr/lib/dovecot/libdovecot.so.0 #2 0x00007fb4ab487adf in ?? () from target:/usr/lib/dovecot/libdovecot.so.0 #3 0x000055c8c35158b7 in ?? () #4 0x000055c8c3516b4a in ?? () #5 0x000055c8c3516d78 in ?? () #6 0x000055c8c35179e4 in doveadm_connections_ring_synced () #7 0x00007fb4ab488166 in io_loop_handle_timeouts () from target:/usr/lib/dovecot/libdovecot.so.0 #8 0x00007fb4ab489963 in io_loop_handler_run_internal () from target:/usr/lib/dovecot/libdovecot.so.0 #9 0x00007fb4ab48839c in io_loop_handler_run () from target:/usr/lib/dovecot/libdovecot.so.0 #10 0x00007fb4ab488548 in io_loop_run () from target:/usr/lib/dovecot/libdovecot.so.0 #11 0x00007fb4ab40c0a3 in master_service_run () from target:/usr/lib/dovecot/libdovecot.so.0 #12 0x000055c8c350b46d in main () For completeness, doveconf -n output attached. -- Jan-Pieter Cornet <johnpc(a)xs4all.net> Systeembeheer XS4ALL Internet bv www.xs4all.nl

2 1

Alt Storage and dsync
by Matt Bryant 26 Apr '18

26 Apr '18

Stupid question but if dovecot mdbox is configured using alt path storage on a bi directional dysnc replicated pair of servers ... will the doveadm altmove need to be run on both halves of the cluster ??? rgds Matt -- <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campai…>

1 0

replication and .dovecot.lda-dupes
by Patrick Cernko 25 Apr '18

25 Apr '18

Hi list, this question was already posted a few years ago (https://www.dovecot.org/list/dovecot/2014-November/098585.html) I already asked the original queriest and he told me, that he never got an solution or workaround but it was not important enough for him. When using replication in conjunction with sieve vacations, the .dovecot.lda-dupes file is not synced with the other server. So when delivering to both servers (round-robin or randomized), senders might get more vacation mails than configured as the other server does not know, that the first one already sent a vacation message. Is this a bug or intentional? If it is a bug, I hereby ask for a fix, please. We are using Dovecot version 2.2.27 on Debian/stretch. This is dovecot -n (hostnames anonymized): # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.76.1.amd64-smp x86_64 Debian 9.3 auth_verbose = yes default_vsz_limit = 2 G doveadm_password = # hidden, use -P to show it doveadm_port = 12345 listen = * login_log_format_elements = pid=%p user=<%u> method=%m rip=%r lip=%l mpid=%e %c mail_attachment_dir = /IMAP/mail/attachments mail_attachment_fs = sis-queue /IMAP/mail/attachments/queue:posix mail_home = /IMAP/mail/mailboxes/%u mail_location = mdbox:~/mdbox mail_log_prefix = "%s(%u[%p]): " mail_max_userip_connections = 0 mail_plugins = " notify replication zlib fts fts_squat" maildir_stat_dirs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/ldap.conf driver = ldap } plugin { fts = squat fts_autoindex = yes fts_squat = partial=4 full=10 mail_replica = tcp:other-server sieve = file:~/sieve;active=~/.dovecot.sieve zlib_save = gz zlib_save_level = 3 } postmaster_address = <> protocols = " imap lmtp sieve" service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service anvil { client_limit = 2250 } service auth { client_limit = 2447 } service doveadm { inet_listener doveadm-server { port = 12345 } } service imap-login { inet_listener imap { port = 0 } process_limit = 2047 } service imap { process_limit = 2047 } service lmtp { inet_listener lmtp { port = 24 } } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 0 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl_cert = </etc/ssl/certificate.pem ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/userdb.overrides driver = passwd-file } userdb { args = /etc/dovecot/ldap.conf driver = ldap } verbose_proctitle = yes protocol lmtp { auth_username_format = %n mail_plugins = " notify replication zlib fts fts_squat sieve" } protocol lda { mail_plugins = " notify replication zlib fts fts_squat sieve" } And this is /etc/dovecot/ldap.conf (hostnames anonymized): uris = ldaps://ldap1/ ldaps://ldap2/ base = dc=domain user_filter = (&(objectClass=posixAccount)(uid=%u)) user_attrs = \ =user=%{ldap:uid}, \ =uid=%{ldap:uidNumber}, \ =gid=%{ldap:gidNumber} pass_filter = (&(objectClass=posixAccount)(uid=%u)) iterate_filter = (&(objectClass=posixAccount)(istMailHomeServer=servername)) auth_bind = yes /etc/dovecot/userdb.override is currently an empty file. P.S.: Although I do not think, that this has something to do with our problem, I should mention that we run SIS with system uids. This needs some permission and posixacl tweaks on /IMAP/mail/attachments to work: # force all files created in this dir recursively) to have mode 0666 setfacl -d -m group::rwx -m o:rwx /IMAP/mail/attachments (you have to enable posixacls for the corresponding filesystem!) # trick dovecot to inherit parent dir's permissions (recursively) chmod 2777 /IMAP/mail/attachments Feel free to add these tweaks to the SIS documentation. ;-) Best Regards, -- Patrick Cernko <pcernko(a)mpi-klsb.mpg.de> +49 681 9325 5815 Joint Administration: Information Services and Technology Max-Planck-Institute fuer Informatik & Softwaresysteme

2 2

dovecot HA setup across large geographic distances
by Sophie Loewenthal 24 Apr '18

24 Apr '18

Hi, I have a working postfix/dovecot config on Debian located in Europe[EU]. I would like to provide either an HA/load balanced setup with a second site in Australia[AU] because of users located between EU/Asia/AU. The site to site traffic would be tunneled through either SSH or a VPN * Has this been done with Doevcot over such great distances? * How can the delay between mailstore synchonisation be handled? ( Currently looking for an opensource product like Netex's HyperIP) I'd be grateful for your thoughts. Best wishes, Sophie Sent from a phone, thus brief.

1 1

Pass UniqueID to sieve scripts
by Peter Blok 24 Apr '18

24 Apr '18

Hi, If a user moves a mail to a Junk folder, this e-mail is going thru imap_sieve and ends up in sa-learn. I wrote some tools to immediately propagate that to the other node of the cluster as well as locally. Everything works fine, but sometimes I see a storm of requests hogging the CPU and I would like to collect some debugging information, Preferably I would like the permanent unique id of the mail in the maildir, but am looking for some kind of unique id that will help me identify if an e-mail somehow passes the sieve filter multiple times. Basically I have changed the pipe to sa-learn to my proxy that will send it across to the other node as well as call sa-learn locally. So something I can get in a sieve script and pass along. Peter

2 1

imap-login segfaulting on 2.3.1
by Grant Keller 24 Apr '18

24 Apr '18

Hello, I have a new director ring I am setting up on centos 7 with dovecot 2.3.1. I haven't been able to replecate this in testing, but as soon as I start pushing production traffic to the new ring I see dozens of these in the logs: Apr 18 00:34:00 d.director.imapd.sonic.net kernel: imap-login[163107]: segfault at 10 ip 00007ff625698dd5sp 00007ffe4b77bb28 error 4 in libdovecot.so.0.0.0[7ff6255bf000+16e000] My config: # 2.3.1 (c5a5c0c82): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-693.21.1.el7.x86_64 x86_64 CentOS Linux release 7.4.1708 (Core) # Hostname: c.director.imapd.sonic.net auth_master_user_separator = * auth_username_format = %Ln auth_verbose = yes default_client_limit = 20000 director_mail_servers = 64.142.111.75 64.142.111.43 64.142.111.76 64.142.111.39 64.142.111.81 director_servers = c.director.imapd.sonic.net d.director.imapd.sonic.net disable_plaintext_auth = no doveadm_port = 1842 login_trusted_networks = 64.142.18.0/24 64.142.105.2 64.142.111.67 64.142.111.36 64.142.105.3 64.142.111.35 64.142.111.68 namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = proxy=y nopassword=y driver = static } service anvil { client_limit = 40103 } service auth { client_limit = 41704 } service director { fifo_listener login/proxy-notify { mode = 0666 } inet_listener { port = 9321 } unix_listener login/director { mode = 0666 } } service imap-login { executable = imap-login director process_limit = 20000 process_min_avail = 32 } service imap { process_limit = 20480 } service ipc { unix_listener ipc { user = dovecot } } service pop3-login { executable = pop3-login director process_limit = 20000 process_min_avail = 32 } ssl_cert = </etc/dovecot/ssl/imap.sonic.net.pem ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it syslog_facility = local0 protocol imap { imap_max_line_length = 128 k ssl_cert = </etc/dovecot/ssl/imap.sonic.net.pem ssl_key = # hidden, use -P to show it } protocol pop3 { ssl_cert = </etc/dovecot/ssl/pop.sonic.net.pem ssl_key = # hidden, use -P to show it } backtrace: Stack trace of thread 56084: #0 0x00007fe529be2dd5 i_stream_get_root_io (libdovecot.so.0) #1 0x00007fe529be2e39 i_stream_set_input_pending (libdovecot.so.0) #2 0x00007fe527415a59 openssl_iostream_bio_sync (libssl_iostream_openssl.so) #3 0x00007fe527415c2a openssl_iostream_more (libssl_iostream_openssl.so) #4 0x00007fe527415c6b openssl_iostream_destroy (libssl_iostream_openssl.so) #5 0x00007fe529e8be72 client_unref (libdovecot-login.so.0) #6 0x00007fe529e90441 login_proxy_free_full (libdovecot-login.so.0) #7 0x00007fe529e90e07 login_proxy_finished (libdovecot-login.so.0) #8 0x00007fe529becfb5 io_loop_call_io (libdovecot.so.0) #9 0x00007fe529bee89f io_loop_handler_run_internal (libdovecot.so.0) #10 0x00007fe529bed0b2 io_loop_handler_run (libdovecot.so.0) #11 0x00007fe529bed2d8 io_loop_run (libdovecot.so.0) #12 0x00007fe529b69673 master_service_run (libdovecot.so.0) #13 0x00007fe529e93532 login_binary_run (libdovecot-login.so.0) #14 0x00007fe529767c05 __libc_start_main (libc.so.6) #15 0x00005567b256050a _start (imap-login) Grant Keller System Operations grant.keller(a)sonic.com

2 2