- dovecot - dovecot.org

2.3.1 Replication is throwing scary errors
by Andy Weal 06 May '18

06 May '18

Hi all, New to the mailing lists but have joined up because of above */2.3.1 Replication is throwing scary errors /*Brief system configuration MX1 - Main Freebsd 11.1-Release-p9 Hosted on a Vultr VM in Sydney AU MTA = Postfix 3.4-20180401 Dovecot = 2.3.1 File system = ufs MX2 - Backup Freebsd 11.1-Release-p9 Running on bare metal - no VM or jails MTA = Postfix 3.4-20180401 Dovecot = 2.3.1 File system = ufs ( on SSD) */ /*Brief sequence of events 1. apx 10 days back upgraded both mx1 and mx2 to dovecot 2.3.1_2 from 2.3.0 (service dovecot stop, portmaster upgrade, service dovecot start) 2. both systems ran ok with no errors for 10 days. 3. Last night I shutdown mx2 and restarted it a few hours later 4. within minutes i was getting the following types of errors on mx2 May 06 12:56:29 doveadm: Error: Couldn't lock /var/mail/vhosts/example.net/user1/.dovecot-sync.lock: fcntl(/var/mail/vhosts/example.net/user1/.dovecot-sync.lock, write-lock, F_SETLKW) locking failed: Timed out after 30 seconds (WRITE lock held by pid 1960) Before i venture down the rabbit hole of fault finding and excess coffee consumption I was wondering if any of you had any updates on the problems discussed below. Cheers for now, Andy Hi, [Formatting is a bit rough, replying from a trimmed digest email] >/Message: 1 />/Date: Fri, 6 Apr 2018 15:04:35 +0200 />/From: Michael Grimm <trashcan at ellael.org <https://dovecot.org/mailman/listinfo/dovecot>> />/To: Dovecot Mailing List <dovecot at dovecot.org <https://dovecot.org/mailman/listinfo/dovecot>> />/Subject: Re: 2.3.1 Replication is throwing scary errors />/Message-ID: <E7E7A927-68F8-443F-BA59-E66CED8FE878 at ellael.org <https://dovecot.org/mailman/listinfo/dovecot>> />/Content-Type: text/plain; charset=utf-8 />//>/Reuben Farrelly wrote: />>/From: Michael Grimm <trashcan at ellael.org <https://dovecot.org/mailman/listinfo/dovecot>> />//>>>/[This is Dovecot 2.3.1 at FreeBSD STABLE-11.1 running in two jails at distinct servers.] />>>/I did upgrade from 2.2.35 to 2.3.1 today, and I do become pounded by error messages at server1 (and vice versa at server2) as follows: />>>/| Apr 2 17:12:18 <mail.err> server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ />>>/no activity for 600 seconds (last sent=mail_change, last recv=mail_change (EOL)) />>>/| Apr 2 17:12:18 <mail.err> server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ />>>/(send=changes recv=mail_requests) />>>/[?] />>>/| Apr 2 18:59:03 <mail.err> server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ />>>/no activity for 600 seconds (last sent=mail, last recv=mail (EOL)) />>>/| Apr 2 18:59:03 <mail.err> server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ />>>/(send=mails recv=recv_last_common) />>>/I cannot see in my personal account any missing replications, *but* I haven't tested this thoroughly enough. I do have customers being serviced at these productive servers, *thus* I'm back to 2.2.35 until I do understand or have learned what is going on. />//>/In my reply to this statement of mine I mentioned that I have seen those timeouts quite some times during the past year. Thus, I upgraded to 2.3.1 again, and boom: after some hours I ended up in hanging processes [1] like (see Remko's mail in addition) ... />//>/doveadm-server: [IP4/6 <USER1> SOME/MAILBOX import:0/0] (doveadm-server) />//>/? at server2 paired with a file like ? />//>/-rw------- 1 vmail dovecot uarch 0 Apr 3 16:52 /home/to/USER1/.dovecot-sync.lock />//>/Corresponding logfile entries at server2 are like ? />//>/Apr 3 17:10:49 <mail.err> server2.lan dovecot: doveadm: Error: Couldn't lock /home/to/USER1/.dovecot-sync.lock: \ />/fcntl(/home/to/USER1/.dovecot-sync.lock, write-lock, F_SETLKW) locking failed: Timed out after 30 seconds \ />/(WRITE lock held by pid 51110) />//>/[1] Even stopping dovecot will not end those processes. One has to manually kill those before restarting dovecot. />//>/After one day of testing 2.3.1 with a couple of those episodes of locking/timeout, and now missing mails depending with server your MUA will connect to, I went back to 2.2.35. After two days at that version I never had such an episode again. />//>>/It's not just you. This issue hit me recently, and it was impacting />>/replication noticeably. I am following git master-2.3 . />/[...] />>/There is also a second issue of a long standing race with replication />>/occurring somewhere whereby if a mail comes in, is written to disk, is />>/replicated and then deleted in short succession, it will reappear />>/again to the MUA. I suspect the mail is being replicated back from />>/the remote. A few people have reported it over the years but it's not />>/reliable or consistent, so it has never been fixed. />>/And lastly there has been an ongoing but seemingly minor issue />>/relating to locking timing out after 30s particularly on the remote />>/host that is being replicated to. I rarely see the problem on my />>/local disk where almost all of the mail comes in, it's almost always />>/occurring on the replicate/remote system. />//>/It might be time to describe our setups in order to possibly find common grounds that might trigger this issue you describe and Rimko and myself ran into as well. />//>/Servers: Cloud Instances (both identical), around 25ms latency apart. />/Intel Core Processor (Haswell, no TSX) (3092.91-MHz K8-class CPU) />/Both servers are connected via IPsec/racoon tunnels />/OS: FreeBSD 11.1-STABLE (both servers) />/Filesystem: ZFS />/MTA: postfix 3.4-20180401 (postfix delivers via dovecot's LMTP) />/IMAP: dovecot running in FreeBSD jails (issues with 2.3.1, fine with 2.2.35) />/Replication: unsecured tcp / master-master />/MUA: mainly iOS or macOS mail.app, rarely roundcube / For me: Servers: Main: VM on a VMWare ESXi local system (light load), local SSD disks (no NFS) Redundant: Linode VM in Singapore, around 250ms away Also no NFS. Linode use SSDs for IO. There is native IPv6 connectivity between both VMs. As I am using TCPs I don't have a VPN between them - just raw IPv6 end to end. OS: Gentoo Linux x86_64 kept well up to date Filesystem: EXT4 for both MTA: Postfix 3.4-x (same as you) IMAP: Dovecot running natively on the machine (no jail/chroot) Replication: Secured TCP / master-master (but only one master is actively used for SMTP and MUA access) MUA: Thunderbird, Maildroid, MS Mail, Gmail client, Roundcube, LG Mail client on handset etc >/I believe it is worthwhile to mention here that I run a poor man's fail-over approach (round-robin DNS) as follows: />//>/DNS: mail.server.tld resolves to one IP4 and one IP6 address of each server, thus 4 IP addresses in total / For me I have a simple pair of servers that replicate mail to each other. Clients connect to the main server only, which is where mail comes in. The second one that experiences the problem with locking is only a redundant server which is a secondary MX, that only gets occasional SMTP deliveries from time to time. So the amount of replication back from the "secondary" is very minimal; the vast bulk of the replication is the primary replicating changes out. Reuben

2 1

Re: expiring mail from root's Maildirs?
by LuKreme 05 May '18

05 May '18

(Sorry about previous mail) > On May 4, 2018, at 07:45, Aki Tuomi <aki.tuomi(a)dovecot.fi> wrote: > Dovecot has hardcoded root prevention. For delivery, too. So dovecot cannot expire, reindex, clean, repair, etc root mail? The mail is crontab mail that is sent out to users via mutt. What would be the "right” way to do this? (The cron tasks run as root because they are scanning system logs on behalf of certain users and those scans cannot be run as the user.) -- My main job is trying to come up with new and innovative and effective ways to reject even more mail. I'm up to about 97% now.

6 14

dovecot fts hangs on search
by André Rodier 05 May '18

05 May '18

Hello, I am trying to use Doevecot fts, with solr the script provided. To rebuild the index, I use the command: doveadm -D index -u mirina 'inbox' To rescan, I use: doveadm -D fts rescan -u mirina But when I do a search, with doveadm, the program hangs: doveadm -D search -u mirina text Madagascar > Debug: Loading modules from directory: /usr/lib/dovecot/modules > Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so > Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so > Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so > Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm > Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_lookup_dict_iterate_visible_next (this is usually intentional, so just ignore this message) > Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) > Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so > Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so > Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this m$ > ssage) > Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so > Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore thi$ message) > doveadm(mirina): Debug: auth USER input: mirina home=/home/users/mirina uid=1002 gid=1001 mail=mirina(a)homebox.space > doveadm(mirina): Debug: Added userdb setting: mail=mirina(a)homebox.space doveadm(mirina): Debug: Effective uid=1002, gid=1001, home=/home/users/mirina > doveadm(mirina): Debug: Quota root: name=User quota backend=maildir args= > doveadm(mirina): Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 > doveadm(mirina): Debug: Quota grace: root=User quota bytes=214748364 (10%) > doveadm(mirina): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/mails/maildir:INDEX=~/mails/indexes/ > doveadm(mirina): Debug: maildir++: root=/home/users/mirina/mails/maildir, index=/home/users/mirina/mails/indexes, indexpvt=, control=, inbox=/home/users/mirina/mails/maildir, alt= > doveadm(mirina): Debug: quota: quota_over_flag check: STORAGE ret=1 value=134 limit=2097152 > doveadm(mirina): Debug: quota: quota_over_flag check: MESSAGE ret=0 value=3 limit=0 > doveadm(mirina): Debug: quota: quota_over_flag=0((null)) vs currently overquota=0 > doveadm(mirina): Debug: Namespace : Using permissions from /home/users/mirina/mails/maildir: mode=0700 gid=default > doveadm(mirina): Debug: http-client: host localhost: Host created > doveadm(mirina): Debug: http-client: host localhost: DNS lookup successful; got 2 IPs > doveadm(mirina): Debug: http-client: peer [::1]:8080: Peer created > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Setting up connection to [::1]:8080 (1 requests pending) > doveadm(mirina): Debug: http-client: peer [::1]:8080: Linked queue http://localhost:8080 (1 queues linked) > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Started new connection to [::1]:8080 > doveadm(mirina): Debug: http-client: request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu… > ina]: Submitted > doveadm(mirina): Debug: http-client: Waiting for 1 requests to finish > doveadm(mirina): Debug: http-client: peer [::1]:8080: Creating 1 new connections to handle requests (already 0 usable, connecting to 0, closing 0) > doveadm(mirina): Debug: http-client: peer [::1]:8080: Making new connection 1 of 1 > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: HTTP connection created (1 parallel connections exist) > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Connected > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Ready for requests > doveadm(mirina): Debug: http-client: peer [::1]:8080: Successfully connected (connections=1) > doveadm(mirina): Debug: http-client: peer [::1]:8080: Using 1 idle connections to handle 1 requests (1 total connections ready) > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Connection to peer [::1]:8080 claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu… > :Madagascar)&fq=%2Bbox:838a1701af1ae65a3e12000032c62018+%2Buser:mirina] > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Claimed request [Req1: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu… > a3e12000032c62018+%2Buser:mirina] > [...] > doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu…] Sent header > doveadm(mirina): Debug: http-client: peer [::1]:8080: No more requests to service for this peer (1 connections exist) > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Got 200 response for request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu…] (took 2 ms + 0 ms in queue) > doveadm(mirina): Debug: http-client: conn [::1]:8080 [0]: Response payload stream destroyed (0 ms after initial response) > doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu…] Finished > doveadm(mirina): Debug: http-client: queue http://localhost:8080: Dropping request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu…] > doveadm(mirina): Debug: http-client: host localhost: Host is idle (timeout = 1799974 msecs) > doveadm(mirina): Debug: http-client: request [Req5: GET http://localhost:8080/solr/select?fl=uid,score&rows=1&sort=uid+asc&q=%7b!lu…] Free (requests left=1) > doveadm(mirina): Debug: http-client: All requests finished When I do the search with an imap client, I have this error after a long timeout: > May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Panic: file imap-client.c: line 854 (client_check_command_hangs): assertion failed: (client->io != NULL || (client->output_cmd_lock != NULL && client->output_cmd_lock != client->input_lock)) > May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f216b6f6e92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f216b6f6f8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f216b68ca91] -> dovecot/imap(client_continue_pending_input+0x1fc) [0x55f44a7eb15c] -> dovecot/imap(client_input+0x120) [0x55f44a7eb460] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f216b70b9f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f216b70d029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f216b70ba8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f216b70bc38] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f216b692fd3] -> dovecot/imap(main+0x328) [0x55f44a7dde68] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f216b2e22e1] -> dovecot/imap(_start+0x2a) [0x55f44a7ddfea] > May 01 07:08:58 homebox dovecot[5683]: imap(mirina): Fatal: master: service(imap): child 6149 killed with signal 6 (core dumps disabled) > May 01 07:08:58 homebox in.imapproxyd[20305]: Raw_Proxy(): IMAP server unexpectedly closed the connection on sd 11 My 10-master.conf: service decode2text { executable = script /usr/local/bin/decode2text.sh user = dovecot unix_listener decode2text { mode = 0666 } } And my 90-plugins.cof: # Add dovecot fts config fts = solr fts_autoindex = no fts_solr = url=http://localhost:8080/solr/ debug fts_decoder = decode2text Am I doing anything wrong? Thanks for your help. André

2 6

Warning message
by forums＠mehl-family.fr 05 May '18

05 May '18

Hi, In my maillog I see this warning message at every reception of e-mail: _STATUS=SENT (DELIVERED VIA DOVECOT SERVICE (DOVECONF: WARNING: /ETC/DOVECOT/CONF.D/90-QUOTA.CONF LINE 39: GLOBAL SETTING MAIL_PLUGINS WON'T CHAN....._ It is just a warning, the e-mail was delivered. I don't find where is the problem. Thank's.

3 2

v2.2.36 release candidate released
by Timo Sirainen 04 May '18

04 May '18

https://dovecot.org/releases/2.2/rc/dovecot-2.2.36.rc1.tar.gz https://dovecot.org/releases/2.2/rc/dovecot-2.2.36.rc1.tar.gz.sig v2.2.36 is hopefully going to be the last v2.2.x release. Please test this RC well, so we'll have a good final release! v2.3.2 is still going to take a couple of months before it's ready. * login-proxy: If ssl_require_crl=no, allow revoked certificates. Also don't do CRL checks for incoming client certificates. * stats plugin: Don't temporarily enable PR_SET_DUMPABLE while opening /proc/self/io. This may still cause security problems if the process is ptrace()d at the same time. Instead, open it while still running as root. + doveadm: Added mailbox cache decision&remove commands. See doveadm-mailbox(1) man page for details. + doveadm: Added rebuild attachments command for rebuilding $HasAttachment or $HasNoAttachment flags for matching mails. See doveadm-rebuild(1) man page for details. - dsync: Lock file generation failed if home directory didn't exist - Snippet generation for HTML mails didn't ignore &entities inside blockquotes, producing strange looking snippets. - imapc: Fix assert-crash if getting disconnected and after reconnection all mails in the selected mailbox are gone. - pop3c: Handle unexpected server disconnections without assert-crash - fts: If mails contained NUL characters, the text around it wasn't indexed. - Obsolete dovecot.index.cache offsets were sometimes used. Trying to fetch a field that was just added to cache file may not have always found it.

3 4

expiring mail from root's Maildirs?
by ＠lbutlr 04 May '18

04 May '18

I would like to delete old mail from root's sent mailbox but doveadm throws errors when I try to do this, not wanting to process root's mail at all (I am running the commands as root, naturally) # doveadm expunge -u root mailbox Sent SENTBEFORE 1-Apr-2018 doveadm(root): Error: Invalid settings in userdb: userdb returned 0 as uid doveadm(root): Error: User lookup failed: Invalid user settings. Refer to server log for more information. The error makes me think this is intentional, and I can see that makes a sort of sense, but it still leaves me with a Sent box that grow forever until I manually go in and remove old mails. I am using passed for userdb and pam for passdb: passdb { username_filter = "!*@*" driver = pam } userdb { driver = passwd } Can I do this with doveadm or am going to have to resort to either manually removing the mail (hah!) or just crontabbing a task to remove the files more than 30 days old?

2 1

Use of NO-NOSELECT in mail_location
by Joseph Tam 03 May '18

03 May '18

I enabled NO-NOSELECT in mail_location as it seemed to be an optimization without negative consequences. The wiki states NO-NOSELECT : Automatically delete any \NoSelect mailboxes that have no children. These mailboxes are sometimes confusing to users. Also if a \NoSelect mailbox is attempted to be created with "CREATE box/", it's created as selectable mailbox instead. The last sentence is important: if you allow this, you won't be able to create folders (i.e. directories); this caught me offguard. If you use mbox, this is something you don't want to enable. Joseph Tam <jtam.home(a)gmail.com>

1 0

v2.3.1: Event leaked (parent=(nil)): http-client.c:137 and 439
by Milan Knížek 03 May '18

03 May '18

Hi, After upgrade to dovecot 2.3.1, many IMAP clients cause errors in dovecot log: May 01 20:15:51 tiger dovecot[21985]: imap: Warning: Event 0x555fd3c01630 leaked (parent=(nil)): http-client.c:137 May 01 20:15:51 tiger dovecot[21985]: imap: Warning: Event 0x555fd3c00140 leaked (parent=(nil)): http-client.c:439 Namely: * roundcube 1.3.3 * offlineimap * nextcloud mail app Despite the errors, the basic functionality seems to work - mails are received by the client. However, the GUI clients display an "internal server error" message window on each refresh. Some other clients do not create any issues: * MailDroid (Android) * getmail Anybody got a similar behaviour? -- Milan Knížek | knizek.confy(a)gmail.com

2 2

Get mailbox name per guid
by Harald Leithner 02 May '18

02 May '18

Hi, maybe I'm unable to use google or the wiki or the man pages but how do I get the mailbox name if I only have the guid for this Mailbox? thx Harald -- Harald Leithner ITronic Wiedner Hauptstraße 120/5.1, 1050 Wien, Austria Tel: +43-1-545 0 604 Mobil: +43-699-123 78 4 78 Mail: leithner(a)itronic.at | itronic.at

2 2

Corrupted dbox file around offset= EOF reading msg header
by DurgaPrasad - DatasoftComnet 02 May '18

02 May '18

Hello Dovecoters, I am using dovecot-2.2.10-8.el7.x86_64 on RHEL 7.4 When I backup mdboxes (using SIS) using this command doveadm -Dv -o plugin/quota= sync -1f -u user1(a)user.com mdbox:/mailbackup/user1 mails get backedup but all m files are not copied. Last file copied is m.4 If I run the command again I get below error I tried with doveadm backup command as well (many times). Error: Corrupted dbox file / mailbackup/user1/storage/m.4 (around offset=3732536): EOF reading msg header (got 0/30 bytes) Result is all mails are not backedup. Destination is a folder on a NAS with NFS mapped to the server. Thanks/DP --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

1 2