October 2017 - dovecot

Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33)
by Aki Tuomi 15 Oct '17

15 Oct '17

We'll take a look ---Aki TuomiDovecot oy -------- Original message --------From: Reuben Farrelly <reuben-dovecot(a)reub.net> Date: 12/10/2017 07:50 (GMT+02:00) To: Aki Tuomi <aki.tuomi(a)dovecot.fi> Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33) Hi Aki, I think I posted a bt full of this in the original email I sent to the list: https://www.dovecot.org/pipermail/dovecot/2017-October/109583.html I've got heaps of coredumps if we need to pick apart one specific one now though. Reuben On 12/10/2017 3:36 PM, Aki Tuomi wrote: > can you provide bt full output from core? > > > > --- > Aki Tuomi > Dovecot oy > > -------- Original message -------- > From: Reuben Farrelly <reuben-dovecot(a)reub.net> > Date: 12/10/2017 06:22 (GMT+02:00) > To: Aki Tuomi <aki.tuomi(a)dovecot.fi>, dovecot(a)dovecot.org > Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33) > > Hi, > > Just to confirm this is still a major problem for me with the 2.2.33.1 > release (did not exist in previous releases). > > Reuben > > > On 8/10/2017 6:30 AM, Aki Tuomi wrote: >> Thanks! We'll look into it. >> >> >> >> --- >> Aki Tuomi >> Dovecot oy >> >> -------- Original message -------- >> From: Reuben Farrelly <reuben-dovecot(a)reub.net> >> Date: 07/10/2017 13:08 (GMT+02:00) >> To: dovecot(a)dovecot.org >> Subject: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33) >> >> This appears to be breaking dsync at the moment and is caused by >> something committed in the last week or so. This is with the current >> -git of dovecot-2.2 (eaf276b33). >> >> The crash happens all the time - the logs are full of this message. >> >> Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Warning: I/O >> leak: 0x7f8bb6581b4c (ostream-file.c:350, fd 14) >> Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Panic: file >> ioloop.c: line 126 (io_remove_full): assertion failed: (io->callback != >> NULL) >> Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Error: Raw >> backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6] >> -> /usr/lib64/dovecot/li >> bdovecot.so.0(+0xd5e26) [0x7f8bb654be26] -> >> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] -> >> /usr/lib64/dovecot/libdovecot.so.0(+0xf5cb0) [0x7f8bb6 >> 56bcb0] -> /usr/lib64/dovecot/libdovecot.so.0(io_remove+0x1d) >> [0x7f8bb656be31] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10b5f8) >> [0x7f8bb65815f8] -> /usr/lib64/dovec >> ot/libdovecot.so.0(o_stream_cork+0x5a) [0x7f8bb657e289] -> >> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x6406) [0x7f8bb12d6406] >> -> /usr/lib64/dovecot/libssl_iost >> ream_openssl.so(openssl_iostream_bio_sync+0x18) [0x7f8bb12d6af6] -> >> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa52e) >> [0x7f8bb12da52e] -> / >> usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa6ef) [0x7f8bb12da6ef] >> -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7) >> [0x7f8bb657e765] -> >> /usr/lib64/dovecot/libdovecot.so.0(o_stream_send+0x61) [0x7f8bb657e688] >> -> /usr/lib64/dovecot/libdovecot.so.0(+0x10dc2c) [0x7f8bb6583c2c] -> >> /usr/lib64/dovecot/libdovecot.so.0(+0x10ddd4) [0x7f8bb6583dd4] -> >> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7) [0x7f8bb657e765] >> -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsendv+0x55) >> [0x7f8bb657e93f] -> >> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsend+0x61) [0x7f8bb657e8d3] >> -> dovecot/doveadm-server() [0x444a35] -> >> /usr/lib64/dovecot/libdovecot.so.0(i_warning+0xce) [0x7f8bb654afdc] -> >> /usr/lib64/dovecot/libdovecot.so.0(io_loop_destroy+0x2ca) >> [0x7f8bb656d8cd] -> dovecot/doveadm-server() [0x4454d1] -> >> dovecot/doveadm-server() [0x445793] -> dovecot/doveadm-server() >> [0x445dd6] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0xd6) >> [0x7f8bb656d241] -> >> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x225) >> [0x7f8bb656fe58] -> >> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3e) >> [0x7f8bb656d422] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0xbe) >> [0x7f8bb656d345] >> Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Fatal: master: >> service(doveadm): child 27186 killed with signal 6 (core dumped) >> >> >> thunderstorm reuben # file core >> core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, >> from 'dovecot/doveadm-server', real uid: 0, effective uid: 0, real gid: >> 0, effective gid: 0, execfn: '/usr/libexec/dovecot/doveadm-server', >> platform: 'x86_64' >> >> thunderstorm reuben # gdb /usr/libexec/dovecot/doveadm-server core >> GNU gdb (Gentoo 8.0.1 vanilla) 8.0.1 >> Copyright (C) 2017 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later >> <http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-pc-linux-gnu". >> Type "show configuration" for configuration details. >> For bug reporting instructions, please see: >> <https://bugs.gentoo.org/>. >> Find the GDB manual and other documentation resources online at: >> <http://www.gnu.org/software/gdb/documentation/>. >> For help, type "help". >> Type "apropos word" to search for commands related to "word"... >> Reading symbols from /usr/libexec/dovecot/doveadm-server...done. >> [New LWP 27186] >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `dovecot/doveadm-server'. >> Program terminated with signal SIGABRT, Aborted. >> #0 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6 >> (gdb) bt full >> #0 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6 >> No symbol table info available. >> #1 0x00007f8bb60e9aa7 in abort () from /lib64/libc.so.6 >> No symbol table info available. >> #2 0x00007f8bb654a8f5 in default_fatal_finish (type=LOG_TYPE_PANIC, >> status=0) at failures.c:201 >> backtrace = 0x228e4d8 >> "/usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6] -> >> /usr/lib64/dovecot/libdovecot.so.0(+0xd5e26) [0x7f8bb654be26] -> >> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] -> /usr"... >> #3 0x00007f8bb654be26 in i_internal_fatal_handler (ctx=0x7ffef81daa50, >> format=0x7f8bb65ba898 "file %s: line %d (%s): assertion failed: (%s)", >> args=0x7ffef81daa70) >> at failures.c:670 >> status = 0 >> #4 0x00007f8bb654ac39 in i_panic (format=0x7f8bb65ba898 "file %s: line >> %d (%s): assertion failed: (%s)") at failures.c:275 >> ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, >> timestamp_usecs = 0} >> args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = >> 0x7ffef81dab50, reg_save_area = 0x7ffef81daa90}} >

3 6

Updating keywords on copy/move
by Daniele Ricci 14 Oct '17

14 Oct '17

Hello, I'm working on my virtual keywords plugin [1] to add an apparently simple feature: add a keyword to a message when it is moved or copied into a virtual folder (let's forget about what keyword to add, I'm starting with a hard-coded one to understand Dovecot plugin development). So I've added a mail_copy hook by the notify plugin [2] and did a simple call to mail_update_keywords [3]. The current source code shows a call to mail_get_backend_mail, this is me experimenting, I've also tried calling mail_update_keywords directly on "dst" mail. The result is an imap crash: Oct 14 21:22:26 notebook dovecot: imap(daniele): Panic: file mail-log-plugin.c: line 461 (mail_log_mail_transaction_commit): assertion failed: (msg->pretext == NULL) Oct 14 21:22:26 notebook dovecot: imap(daniele): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x9f0a2) [0x7f81af52f0a2] -> /usr/lib/dovecot/libdovecot.so.0(+0x9f19a) [0x7f81af52f19a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f81af4bfcf8] -> /usr/lib/dovecot/modules/lib20_mail_log_plugin.so(+0x19e8) [0x7f81aeae89e8] -> /usr/lib/dovecot/modules/lib15_notify_plugin.so(notify_contexts_mail_transaction_commit+0x58) [0x7f81aecec548] -> /usr/lib/dovecot/modules/lib15_notify_plugin.so(+0x2105) [0x7f81aeced105] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x52) [0x7f81af807ca2] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit+0x21) [0x7f81af807d71] -> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_transaction_commit+0x78) [0x7f81ae8e2fb8] -> /usr/lib/dovecot/modules/lib15_notify_plugin.so(+0x20ec) [0x7f81aeced0ec] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_transaction_commit_get_changes+0x52) [0x7f81af807ca2] -> dovecot/imap(+0x10084) [0x563d0ff2b084] -> dovecot/imap(command_exec+0x5c) [0x563d0ff37d9c] -> dovecot/imap(+0x1b322) [0x563d0ff36322] -> dovecot/imap(+0x1b3bc) [0x563d0ff363bc] -> dovecot/imap(client_handle_input+0x18d) [0x563d0ff3677d] -> dovecot/imap(client_input+0xac) [0x563d0ff36ccc] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f81af544742] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x12e) [0x7f81af545d8e] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x36) [0x7f81af5447d6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f81af544988] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f81af4ca353] -> dovecot/imap(main+0x329) [0x563d0ff29139] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f81af1132e1] -> dovecot/imap(_start+0x2a) [0x563d0ff292ca] This happens after my hook returns. Now, before activating core dumps and starting to do some serious debugging and reverse engineering, I ask: do I have to do something extra such as opening a transaction or something? Isn't the notify plugin supposed to do that already? Thanks for your help! [1] http://github.com/daniele-athome/dovecot-virtual-keywords-plugin [2] https://github.com/daniele-athome/dovecot-virtual-keywords-plugin/blob/keyw… [3] https://github.com/daniele-athome/dovecot-virtual-keywords-plugin/blob/keyw… -- Daniele

1 0

Last Login example
by Jorge Bastos 14 Oct '17

14 Oct '17

Howdy, Where can i find a configuration example for the last login plugin, that fits for latest dovecot version? Thanks in advanced! Jorge,

1 0

Re: Solr/Basic Auth?
by Aki Tuomi 14 Oct '17

14 Oct '17

You can include it in the url, like http://user:pass@host/---Aki TuomiDovecot oy -------- Original message --------From: Larry Rosenman <larryrtx(a)gmail.com> Date: 13/10/2017 23:32 (GMT+02:00) To: Dovecot Mailing List <dovecot(a)dovecot.org> Subject: Solr/Basic Auth? Does the fts-solr plugin support solr's basic auth plugin? If not, can it? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281

1 0

Solr/Basic Auth?
by Larry Rosenman 13 Oct '17

13 Oct '17

Does the fts-solr plugin support solr's basic auth plugin? If not, can it? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281

1 0

Re: Convert from mbox to Maildir
by Joseph Tam 13 Oct '17

13 Oct '17

Nikolai Lusan <nikolai(a)lusan.id.au> writes: >> Just went through this one recently using CentOS with SendMail, Procmail >> and IMAP conversion from mbox to Maildir - I needed to make sure I could >> run both formats while I migrated users one-by-one. > > When I have migrated solutions from mbox to maildir (I have done more than > one) I found it easier to get a mail client to do the hard work for me (if > you're on a LAN with the server it's faster). I created new usernames with > maildir as their base, and copied the folder over. Reading over David's and various schemes, the tricky part is consistency: avoiding race conditions where you have the user manipulating their mailboxes and your MTA/LDA delivering mail, while simultaneously doing mbox->Maildir conversion. To guarantee consistency, it's necessary to lock out other avenues of mbox changes while the conversion takes place. Otherwise, you'll have the situation where mail can get duplicated, or worse, deleted. For example, think of this sequence: convert B, user moves message A->B, convert A. I think the safest is method is 1) Deny the user access to IMAP/POP: deny userdb/passdb, or lock their password and flush their auth cache entry. 2) TEMPFAIL MTA delivery to user during conversion 3) Kick the user 4) Do the conversion 5) Undo 1+2 If you have a smart conversion script (one that recognizes changes and only do delta updates) you can greatly speed up step 4 and minimize outage time. 0) Do an initial conversion that will do the bulk of the work but does not guranatee consistency. Joseph Tam <jtam.home(a)gmail.com>

1 0

Re: v2.2.33 released
by Aki Tuomi 13 Oct '17

13 Oct '17

try with 2.2.33.1 ---Aki TuomiDovecot oy -------- Original message --------From: Odhiambo Washington <odhiambo(a)gmail.com> Date: 13/10/2017 10:42 (GMT+02:00) To: Dovecot Mailing List <dovecot(a)dovecot.org> Cc: dovecot-news(a)dovecot.org Subject: Re: v2.2.33 released On 10 October 2017 at 18:28, Timo Sirainen <tss(a)iki.fi> wrote: > https://dovecot.org/releases/2.2/dovecot-2.2.33.tar.gz > https://dovecot.org/releases/2.2/dovecot-2.2.33.tar.gz.sig > > We're getting close to the last v2.2.x releases. Hopefully we'll have the > first v2.3 beta releases out soon. > Something is wrong with 2.2.33, but I haven't found time to catch it. I have been running 2.2.32 without issues. I use dovecot-lda with Exim. The moment I installed 2.2.33, it was all failures - mail somehow gets delivered, but a failure too is generated. I have backed off from 2.2.33 for now. Maybe I will find time to debug later. Maybe someone already found a way around, but I missed the post? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft."

2 3

v2.2.33 released
by Timo Sirainen 13 Oct '17

13 Oct '17

https://dovecot.org/releases/2.2/dovecot-2.2.33.tar.gz https://dovecot.org/releases/2.2/dovecot-2.2.33.tar.gz.sig We're getting close to the last v2.2.x releases. Hopefully we'll have the first v2.3 beta releases out soon. * doveadm director commands wait for the changes to be visible in the whole ring before they return. This is especially useful in testing. * Environments listed in import_environment setting are now set or preserved when executing standalone commands (e.g. doveadm) + doveadm proxy: Support proxying logs. Previously the logs were visible only in the backend's logs. + Added %{if}, see https://wiki2.dovecot.org/Variables#Conditionals + Added a new notify_status plugin, which can be used to update dict with current status of a mailbox when it changes. See https://wiki2.dovecot.org/Plugins/NotifyStatus + Mailbox list index can be disabled for a namespace by appending ":LISTINDEX=" to location setting. + dsync/imapc: Added dsync_hashed_headers setting to specify which headers are used to match emails. + pop3-migration: Add pop3_migration_ignore_extra_uidls=yes to ignore mails that are visible in POP3 but not IMAP. This could happen if new mails were delivered during the migration run. + pop3-migration: Further improvements to help with Zimbra + pop3-migration: Cache POP3 UIDLs in imapc's dovecot.index.cache if indexes are enabled. These are used to optimize incremental syncs. + cassandra, dict-sql: Use prepared statements if protocol version>3. + auth: Added %{ldap_dn} variable for passdb/userdb ldap - acl: The "create" (k) permission in global acl-file was sometimes ignored, allowing users to create mailboxes when they shouldn't have. - sdbox: Mails were always opened when expunging, unless mail_attachment_fs was explicitly set to empty. - lmtp/doveadm proxy: hostip passdb field was ignored, which caused unnecessary DNS lookups if host field wasn't an IP - lmtp proxy: Fix crash when receiving unexpected reply in RCPT TO - quota_clone: Update also when quota is unlimited (broken in v2.2.31) - mbox, zlib: Fix assert-crash when accessing compressed mbox - doveadm director kick -f parameter didn't work - doveadm director flush <host> resulted flushing all hosts, if <host> wasn't an IP address. - director: Various fixes to handling backend/director changes at abnormal times, especially while ring was unsynced. These could have resulted in crashes, non-optimal behavior or ignoring some of the changes. - director: Use less CPU in imap-login processes when moving/kicking many users. - lmtp: Session IDs were duplicated/confusing with multiple RCPT TOs when lmtp_rcpt_check_quota=yes - doveadm sync -1 fails when local mailboxes exist that do not exist remotely. This commonly happened when lazy_expunge mailbox was autocreated when incremental sync expunged mails. - pop3: rawlog_dir setting didn't work

2 1

Released Pigeonhole v0.4.21 for Dovecot v2.2.33.1.
by Stephan Bosch 13 Oct '17

13 Oct '17

Hello Dovecot users, Here's the definitive 0.4.21 release. There were no changes since the release candidate. Changelog v0.4.21: * redirect action: Always set the X-Sieve-Redirected-From header to sieve_user_email if configured. Before, it would use the envelope recipient instead if available, which makes no sense if the primary e-mail address is available. + vacation extension: Allow ignoring the envelope sender while composing the "To:" header for the reply. Normally, the "To:" header is composed from the address found in the "Sender", "Resent-From" or "From" headers that is equal to the envelope sender. If none is then found, the bare envelope sender is used. This change adds a new setting "sieve_vacation_to_header_ignore_envelope". With this setting enabled, the "To:" header is always composed from those headers in the source message. The new setting thus allows ignoring the envelope, which is useful e.g. when SRS is used. + vacation extension: Compose the "To:" header from the full sender address found in the first "Sender:", "From:" or "Resent-From:" header. Before, it would create a "To:" header without a phrase part. The new behavior is nicer, since the reply will be addressed to the sender by name if possible. - LDA Sieve plugin: Fixed sequential execution of LDAP-based scripts. A missing LDAP-based script could cause the script sequence to exit earlier. - sieve-filter: Removed the (now) duplicate utf8 to mutf7 mailbox name conversion. This caused problems with mailbox names containing UTF-8 characters. The Dovecot API was changed years ago, but apparently sieve-filter was never updated. The release is available as follows: https://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.21.t… https://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.21.t… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this release and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

1 0

Solr 7
by Ricardo Branco 12 Oct '17

12 Oct '17

Before I go down the route of upgrading our main Solr instance from 6 to 7, just checking if anyone is using Solr 7 with Dovecot.

4 4