March 2020 - dovecot - dovecot.org

Problem with tika
by Arjen Heidinga 19 May '20

19 May '20

Hello all, For some time now we've bin using solr as a search engine (working great). I have added tika for searching inside documents, however it keeps crashing when indexing. When just indexing mails that arrive works good, but a reindex on all mail constantly crashes with below stacktrace. When I observe the packets with wireshark I see a HTTP-flow going to tika and suddenly, midway a document dovecot (or the server) sends a RST. Sometimes after thousands of documents, sometimes after a few. Usually after a few hundred. It appears there are less errors using http than https. Relevant config: OS: CentOS6, fully updated plugin { fts = solr batch_size = 1 fts_solr = url=https://username:password@solr-01.vevida.net:443/solr/dovecot/ #fts_tika = https://solr-01.vevida.net:443/tika/ batch_size = 1000 fts_autoindex=yes soft_commit=no } # dovecot --version 2.3.9.3 (9f41b88fa) # Configure options: --docdir=%{_docdir}/dovecot \ --disable-static \ --with-nss \ --with-shadow \ --with-pam \ --with-gssapi=plugin \ --with-ldap=plugin \ --with-sql=plugin \ --with-pgsql \ --with-sqlite \ --with-zlib \ --with-bzlib \ --with-lzma \ --with-libcap \ --with-ssl=openssl \ --with-ssldir=%{ssldir} \ --with-solr \ --with-docs # It is compiled agains the latest openssl # Tika and Solr: Both latest versions. #Stacktrace: doveadm(info(a)samenmetrenske.nl): Info: Sent: Caching mails seq=1..161 doveadm(info(a)samenmetrenske.nl): Panic: file http-client-request.c: line 1173 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) doveadm(info(a)xxxxxxxxxxxxxxxxxxx.x): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x2f) [0x7f95d805acbf] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x26) [0x7f95d805add6] -> /usr/lib64/dovecot/libdovecot.so.0(+0xe90ba) [0x7f95d80660ba] -> /usr/lib64/dovecot/libdovecot.so.0(+0xe9161) [0x7f95d8066161] -> /usr/lib64/dovecot/libdovecot.so.0(+0x41158) [0x7f95d7fbe158] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_more+0x424) [0x7f95d8005094] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_connection_output+0x11a) [0x7f95d800a24a] -> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x8f6a) [0x7f95d57a2f6a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x114483) [0x7f95d8091483] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x61) [0x7f95d807e581] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xdc) [0x7f95d808076c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x5c) [0x7f95d807e67c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f95d807e8c8] -> /usr/lib64/dovecot/libdovecot.so.0(+0x89105) [0x7f95d8006105] -> /usr/lib64/dovecot/libdovecot.so.0(http_client_request_send_payload+0x1f) [0x7f95d80063cf] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xd31d) [0x7f95d6ad931d] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_parser_more+0x1a) [0x7f95d6ad83ca] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_build_mail+0x761) [0x7f95d6ad6401] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0x114ca) [0x7f95d6add4ca] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_precache+0x2a) [0x7f95d835ab4a] -> doveadm(+0x31e75) [0x55e7c1052e75] -> doveadm(+0x321fb) [0x55e7c10531fb] -> doveadm(+0x2c321) [0x55e7c104d321] -> doveadm(+0x2c577) [0x55e7c104d577] -> doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x1e8) [0x55e7c104ec38] -> doveadm(doveadm_cmd_run_ver2+0x52e) [0x55e7c105fafe] -> doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55e7c105fb97] -> doveadm(main+0x21a) [0x55e7c1062aca] -> /lib64/libc.so.6(__libc_start_main+0x100) [0x7f95d7983d20] -> doveadm(+0x1c479) [0x55e7c103d479]

1 1

Dovecot Load balancing - Please Please Please help me
by Kishore Potnuru 24 Apr '20

24 Apr '20

I need your help on my issue very badly. I have the dovecot version 2.0.9 installed on RHEL linux 6.10 version. It's a single Live server. I need to add a load balancer by adding another server for resilience. I cannot upgrade the dovecot version at this time as it is a single live server. If something goes wrong, I will be in trouble. Basically the requirement/issue is: a) it's a single live dovecot POP3/IMAP server b) people are accessing their emails from microsoft outlook (by giving the same server name as IMAP and SMTP server). But more than 100000 emails will be travelled/routed. c) This server is not rebooted from couple of years. d) Now they are worried about, just in case if the server is having any issues during the reboot or if something goes wrong with the server in any case, there will be huge issues. e) They would like to create resilience by adding another server and emails will be travelled from both the servers. If one server is having any issues due to any technical issue and if it doesn't come back easily, then the emails will be travelled through the other server and there will no disturbance to the emails. atleast it travels from the other server. I have very less knowledge on this. Could you please help me providing the best solution for this. I will wait for your reply. Please please please help me. Thanks, Kishore Potnuru

4 8

Still interested in SMTPUTF8 support
by David Bürgin 22 Apr '20

22 Apr '20

I haven’t seen this term on the mailing list for a while. I don’t know if you’re tracking this somewhere, in any case I’m curious if you are still planning to add support for SMTPUTF8. Cheers,

2 1

Headsup on feature removal
by Aki Tuomi 17 Apr '20

17 Apr '20

Hi! Dovecot is now a nearly 20 year old product, and during that time it has accumulated many different features and plugins in its core repository. We are starting to gradually remove some of these parts, which are unused, untested or deprecated. We will provide advance notification before removing anything. To start, the following features are likely to be removed in next few releases of Dovecot. - Authentication drivers: vpopmail, checkpassword, bsdauth, shadow, sia - Password schemes: HMAC-MD5, RPA, SKEY, PLAIN-MD4, LANMAN, NTLM, SMD5 - Authentication mechanisms: ntlm, rpa, skey - Dict drivers: memcached, memcached-ascii (use redis instead) - postfix postmap support - autocreate & autosubscribe plugins (use built-in auto=create/subscribe setting instead) - expire plugin (use built-in autoexpunge setting) - fts-squat plugin - mailbox alias plugin - mail-filter plugin - snarf plugin - xz compression algorithm For the authentication drivers that are being removed, we suggest using Lua as a replacement. See https://doc.dovecot.org/configuration_manual/authentication/lua_based_authe… For information about converting between password schemes, see https://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes If you are using any of these features, please start preparing for their removal in the near future. Features will begin to be dropped as of v2.3.11. Additionally, the mbox format will no longer receive new development. It will still be maintained, however its use beyond migrations and other limited use cases will be discouraged. Please contact us via the mailing list if you have any questions. Regards, Dovecot Team

15 23

doveadm backup from gmail with imapc
by Ben Mulvihill 11 Apr '20

11 Apr '20

I am trying to backup a gmail account (not the one I am writing from) to dovecot, using doveadm-backup and imapc, but am having ssl connection problems. ted@expectation:~# doveadm backup -D -R -u ted imapc: dsync(ted): Info: imapc(imap.gmail.com:993): Connected to 74.125.71.108:993 (local 10.7.1.179:53852) dsync(ted): Warning: imapc(imap.gmail.com:993): Server disconnected unexpectedly: SSL_connect() failed: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert number 40 - reconnecting (delay 0 ms) dsync(ted): Info: imapc(imap.gmail.com:993): Connected to 74.125.71.109:993 (local 10.7.1.179:59052) dsync(ted): Error: imapc(imap.gmail.com:993): Server disconnected unexpectedly: SSL_connect() failed: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure: SSL alert number 40 - disconnecting dsync(ted): Error: User initialization failed: imapc: Login to imap.gmail.com failed: Disconnected from serv I am using dovecot version 2.2.33.2 on ubuntu, with the configuration below. I have also enabled "allow access from unsecure apps" in my gmail settings. My first thought looking at the error messages was has that perhaps doveadm-backup was trying to connect with ssl3, which is no longer supported by gmail or anyone else nowadays. But apparently the ssl3_read_bytes function in openssl also handles tls, so the reference to ssl3 in the message is misleading, and the real problem must be elsewhere. If anyone can help me debug this I'd be grateful. Many thanks, Ben root@expectation:/etc/dovecot# doveconf -N # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: Linux 4.15.0-91-generic x86_64 Ubuntu 18.04.4 LTS auth_mechanisms = plain imapc_features = rfc822.size gmail-migration fetch-headers imapc_host = imap.gmail.com imapc_password = # hidden, use -P to show it imapc_port = 993 imapc_ssl = imaps imapc_ssl_verify = no imapc_user = MY_GMAIL_ADDRESS mail_location = maildir:~/Maildir mail_prefetch_count = 20 mail_privileged_group = mail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam name = } protocols = " imap" ssl = yes ssl_cert = </etc/dovecot/private/dovecot.pem ssl_cipher_list = ALL:!LOW:!SSLv2:!SSLv3:!EXP:!aNULL ssl_client_ca_dir = /etc/ssl/certs ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv3 userdb { driver = passwd name = }

6 13

Warning: Failed to do incremental sync
by Laura Smith 08 Apr '20

08 Apr '20

Setup dovecot sync along the lines of (https://wiki2.dovecot.org/Replication) I am doing one way replication. The initial full replication happened without issue, but now I'm seeing these errors on the slave server: doveadm: Warning: /data/mail/foo/bar/Maildir/dovecot-uidlist: Duplicate file entry at line 26397: 1562173159.M215923P17350.mxp,S=2290,W=2339 (uid 143128 -> 143142) Warning: Failed to do incremental sync for mailbox Sent Messages, retry with a full sync (Modseq 1766 no longer in transaction log (highest=17617, last_common_uid=17559, nextuid=17560)) Warning: Failed to do incremental sync for mailbox INBOX, retry with a full sync (Modseq 2540 no longer in transaction log (highest=13870, last_common_uid=19912, nextuid=19913)) I guess dovecot automatically tries a full replication because eventually the messages get pushed and "sync failed" status changes from 'y' to '-'

2 1

Query change mdbox_rotate_size & mdbox_rotate_interval after sometime
by Durga Prasad Malyala 06 Apr '20

06 Apr '20

Hello All, If I change the mdbox_rotate_size to 100 MB & mdbox_rotate_interval to say 3 days after a year of usage what would be the impact on existing mails and new mails? Thanks/DP

1 1

replication and spam removal ("doveadm expunge")
by Olaf Hopp 06 Apr '20

06 Apr '20

Hello everybody, since now I did no replication and spam is delivered into users folder "spambox" Every night there is a cronjob which deletes spam older than 30 days via something like "find .... -ctime +30 -delete" Now I'm going to set up replication (two way) and I thought that doing "rm" is not a good idea. So I modified the job to something like /usr/bin/doveadm expunge -u test1 mailbox INBOX.spambox BEFORE 30d which works like intended, but I see that on the replication partner the spam isn't deleted. Even if I do a doveadm force-resync -u test1 '*' or log in via imap to the replication partner I still see the old spam. So my question is: is this the intended behaviour and I have to run the "doveadm expunge" on both replication partners or should the deletion via "expunge" being replicated to the partner and I have a misconfiguration or maybe I hit a bug ? Regards, Olaf /etc/dovecot/conf.d/12-replication.conf: mail_plugins = $mail_plugins notify replication service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } service doveadm { inet_listener { port = 1109 } } doveadm_password = XXXXX plugin { mail_replica = tcp:X.Y.Z.X:1109 } -- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik Dipl.-Geophys. Olaf Hopp - Leitung IT-Dienste - Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp(a)kit.edu www.atis.informatik.kit.edu www.kit.edu KIT - Die Forschungsuniversität in der Helmholtz-Gemeinschaft Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.

2 4

net_connect_unix(imap) failed
by Philipp Ewald 06 Apr '20

06 Apr '20

Hello everyone, we have a huge problem with dovecot and IMAP connections. we got the following errors: 39665 today dovecot: imap-login: Error: master(imap): net_connect_unix(imap) failed: Resource temporarily unavailable - http://wiki2.dovecot.org/SocketUnavailable (client-pid=29066, client-id=1755, rip=IP, created 562 msecs ago, received 0/4 bytes) we thinks this may be a problem with authorization take too long? Authorization is not local and with SQL. i found the following in source code of Dovecot: #define SOCKET_CONNECT_RETRY_MSECS 500 #define SOCKET_CONNECT_RETRY_WARNING_INTERVAL_SECS 2 [...] i_error("master(%s): %s (client-pid=%u, client-id=%u, rip=%s, created %u msecs ago, received %u/%zu bytes)", This is no process limit problem: ps auxf | grep -c "[d]ovecot/imap$" 688 ps auxf | grep -c "[d]ovecot/imap-login$" 100 cat /proc/`pidof dovecot`/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 64053 64053 processes Max open files 65535 65535 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 64053 64053 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us ulimit -n 1024 dovecot --version 2.3.4.1 (f79e8e7e4) protocols = imap pop3 service imap-login { process_min_avail = 4 service_count = 0 } service imap { process_limit = 4096 } service pop3-login { process_min_avail = 4 service_count = 0 } service pop3 { process_limit = 4096 } Can someone explain why we got this error and how to fix? If you need another information please tell me. -- Philipp Ewald Administrator DigiOnline GmbH, Probsteigasse 15 - 19, 50670 Köln E-Mail: philipp.ewald(a)digionline.de Web: www.digionline.de, www.webweaver.de AG Köln HRB 27711, St.-Nr. 5215 5811 0640 Geschäftsführer: Werner Grafenhain Informationen zum Datenschutz: www.digionline.de/ds

2 2

Disable Dovecot LDA
by Adam Raszkiewicz 02 Apr '20

02 Apr '20

Hi, Is there any way to disable Dovecot LDA? I want to always send email via postfix and relay server even it will be a local delivery within the Dovecot server Thanks, Adam

6 9