October 2017 - dovecot

v2.2.33.1 released
by Timo Sirainen 12 Oct '17

12 Oct '17

https://dovecot.org/releases/2.2/dovecot-2.2.33.1.tar.gz https://dovecot.org/releases/2.2/dovecot-2.2.33.1.tar.gz.sig Oops, one a bit too bad bug was left in the release, so fixing it here. - dovecot-lda was logging to stderr instead of to the log file.

2 1

Connection closed reason
by Miloslav Hůla 12 Oct '17

12 Oct '17

Hi, we have one user using the old Alpine client with IMAP. Time to time (3 times per day or 3 times per week) he get error: "MAIL FOLDER INBOX CLOSED DUE TO ACCESS ERROR" and he complains, that inbox stops to refresh with new emails. I don't know Alpine but I can imagine, that Alpine creates TCP connection to IMAPS and uses IDLE. I read wiki page about Timeouts [1] and it seems OK. Alpine is connected 5 or more hours without interrupt. But I found 'Connection closed' at approximate times of user reports in mail.log. Am'I undestand it correctly, that 'Connection closed' is some kind of incorrect connection end? I can find 'Logged out' messages which seems to me as correct ones. I can imagine that such situations may happend for moving clients, such a smartphone with losing signal. But mentioned user is on local network connected by cable. I'm interested, what exactly 'Connection closed' in mail log means. We have following lines in log: dovecot: imap(username): Connection closed in=63 out=288442 dovecot: imap(username): Connection closed (IDLE running for 0.001 + waiting input for 0.001 secs, 2 B in + 10+0 B out, state=wait-input) in=9668 out=17375 imap(username): Connection closed: read(size=3751) failed: Connection reset by peer in=4441 out=51431 dovecot: imap(username): Connection closed (UID fetch running for 0.008 + waiting input for 0.001 secs, 0.001 in locks, 101 B in + 202+202 B out, state=wait-input) in=3330 out=436043 Could it be problem in our network or Dovecot configuration issue? Can I somehow debug all connections for single client? Thank you, Milo [1] https://wiki.dovecot.org/Timeouts

2 2

Re: Connection closed reason
by Joseph Tam 12 Oct '17

12 Oct '17

Miloslav H?la <miloslav.hula(a)gmail.com> > we have one user using the old Alpine client with IMAP. Time to time (3 > times per day or 3 times per week) he get error: "MAIL FOLDER INBOX > CLOSED DUE TO ACCESS ERROR" and he complains, that inbox stops to > refresh with new emails. Hmm, I see when using direct file access, but not when using IMAP, but I don't often connect this way with alpine). > Am'I undestand it correctly, that 'Connection closed' is some kind of > incorrect connection end? I can find 'Logged out' messages which seems > to me as correct ones. > ... > dovecot: imap(username): Connection closed in=63 out=288442 > dovecot: imap(username): Connection closed (IDLE running for 0.001 + > waiting input for 0.001 secs, 2 B in + 10+0 B out, state=wait-input) > in=9668 out=17375 I was able to simulate this using netcat -- this log message comes when the client closes the connection during IDLE. > imap(username): Connection closed: read(size=3751) failed: Connection > reset by peer in=4441 out=51431 I couldn't recreate this, but this happens when the client sends an immediate TCP RST connection teardown packet, rather than the usual FIN/ACK/RST orderly shutdown. I don't know what would cause this -- maybe some firewall state session timeout? Joseph Tam <jtam.home(a)gmail.com>

2 1

How to map email addresses to usernames?
by Andreas Born 12 Oct '17

12 Oct '17

Hello, I have a general question about userdb and passdb lookups, in my case related to driver=sql. The documentation and most examples in the internet seem to assume that a user's login username to access a mailbox is identical to his email-address, or at least to the local-part of this email-address. This setup works fine. But I don't understand how to setup dovecot, when login username and email adresses differ. Or to say it in other words, how to map email-adresses to mailbox users. I need a sql password_query for passdb to validate the password for a given login name during SMTP or SASL authentication. Lets say the credentials are "user-1"/"anypassword", then the query is similar to "select password from mailboxes where username='%u'". The next step, user_query for userdb is executed to get storage information for user "user-1". This query must also search for the username "user-1", not for an email-adress: "select home, mail from mailboxes where username='%u'". So far, so good. But: For incoming mails via LMTP, my user_query from userdb is being executed with the email-address as search parameter (instead of the login name), so it fails. I miss a third query that maps one user to each email address for delivery, prior to the userdb lookup. For Example: bob(a)example.org => user-1 eve(a)example.org => user-1 ted(a)example.org => user-2 @catchall.org => user-3 "select username from addr_maps where addr='%n@%d'". What is the correct setup for this szenario? I've found two solutions that seem to work, but i'm not sure if they are really failsafe. I) I use driver=prefetch as a first userdb, so for authentication, only the password_query from passdb is relevant and returns all mailbox details. A second userdb does a user_query lookup against email-addresses and is only called for incoming mails. passdb { driver = sql args = /etc/dovecot/sql-check-user.conf } userdb { driver = prefetch } userdb { driver = sql args = /etc/dovecot/sql-check-addr.conf } II) The second userdb performs lookups against usernames (as in the passwd query), but I place a third userdb {} within the lmtp's protocol section, which does a address lookup instead of a username lookup: protocol lmtp { userdb { driver = sql args = /etc/dovecot/sql-check-addr.conf } } passdb { driver = sql args = /etc/dovecot/sql-check-user.conf } userdb { driver = sql args = /etc/dovecot/sql-check-user.conf } What do you think, is there a better solution? - greetings Andreas Dovecot Version: 2.2.27 (c0f36b0)

2 1

Newbie question...
by Michael Segel 12 Oct '17

12 Oct '17

Hi, I’ve downloaded and built dovecot from the source. Now I’ve run in to a couple of issues. Like namespaces. Is there an example of the namespaces settings? Nothing fancy, just the basics… These are private mailboxes, but still need to know what I have to do to set this correctly. Thanks In Advance, -Mike

2 1

Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33)
by Aki Tuomi 12 Oct '17

12 Oct '17

can you provide bt full output from core? ---Aki TuomiDovecot oy -------- Original message --------From: Reuben Farrelly <reuben-dovecot(a)reub.net> Date: 12/10/2017 06:22 (GMT+02:00) To: Aki Tuomi <aki.tuomi(a)dovecot.fi>, dovecot(a)dovecot.org Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33) Hi, Just to confirm this is still a major problem for me with the 2.2.33.1 release (did not exist in previous releases). Reuben On 8/10/2017 6:30 AM, Aki Tuomi wrote: Thanks! We'll look into it. --- Aki Tuomi Dovecot oy -------- Original message -------- From: Reuben Farrelly <reuben-dovecot(a)reub.net> Date: 07/10/2017 13:08 (GMT+02:00) To: dovecot(a)dovecot.org Subject: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33) This appears to be breaking dsync at the moment and is caused by something committed in the last week or so. This is with the current -git of dovecot-2.2 (eaf276b33). The crash happens all the time - the logs are full of this message. Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Warning: I/O leak: 0x7f8bb6581b4c (ostream-file.c:350, fd 14) Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Panic: file ioloop.c: line 126 (io_remove_full): assertion failed: (io->callback != NULL) Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6] -> /usr/lib64/dovecot/li bdovecot.so.0(+0xd5e26) [0x7f8bb654be26] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf5cb0) [0x7f8bb6 56bcb0] -> /usr/lib64/dovecot/libdovecot.so.0(io_remove+0x1d) [0x7f8bb656be31] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10b5f8) [0x7f8bb65815f8] -> /usr/lib64/dovec ot/libdovecot.so.0(o_stream_cork+0x5a) [0x7f8bb657e289] -> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x6406) [0x7f8bb12d6406] -> /usr/lib64/dovecot/libssl_iost ream_openssl.so(openssl_iostream_bio_sync+0x18) [0x7f8bb12d6af6] -> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa52e) [0x7f8bb12da52e] -> / usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa6ef) [0x7f8bb12da6ef] -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7) [0x7f8bb657e765] -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_send+0x61) [0x7f8bb657e688] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10dc2c) [0x7f8bb6583c2c] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10ddd4) [0x7f8bb6583dd4] -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7) [0x7f8bb657e765] -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsendv+0x55) [0x7f8bb657e93f] -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsend+0x61) [0x7f8bb657e8d3] -> dovecot/doveadm-server() [0x444a35] -> /usr/lib64/dovecot/libdovecot.so.0(i_warning+0xce) [0x7f8bb654afdc] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_destroy+0x2ca) [0x7f8bb656d8cd] -> dovecot/doveadm-server() [0x4454d1] -> dovecot/doveadm-server() [0x445793] -> dovecot/doveadm-server() [0x445dd6] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0xd6) [0x7f8bb656d241] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x225) [0x7f8bb656fe58] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3e) [0x7f8bb656d422] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0xbe) [0x7f8bb656d345] Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Fatal: master: service(doveadm): child 27186 killed with signal 6 (core dumped) thunderstorm reuben # file core core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'dovecot/doveadm-server', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/libexec/dovecot/doveadm-server', platform: 'x86_64' thunderstorm reuben # gdb /usr/libexec/dovecot/doveadm-server core GNU gdb (Gentoo 8.0.1 vanilla) 8.0.1 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/dovecot/doveadm-server...done. [New LWP 27186] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `dovecot/doveadm-server'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6 (gdb) bt full #0 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f8bb60e9aa7 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f8bb654a8f5 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:201 backtrace = 0x228e4d8 "/usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6] -> /usr/lib64/dovecot/libdovecot.so.0(+0xd5e26) [0x7f8bb654be26] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] -> /usr"... #3 0x00007f8bb654be26 in i_internal_fatal_handler (ctx=0x7ffef81daa50, format=0x7f8bb65ba898 "file %s: line %d (%s): assertion failed: (%s)", args=0x7ffef81daa70) at failures.c:670 status = 0 #4 0x00007f8bb654ac39 in i_panic (format=0x7f8bb65ba898 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:275 ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, timestamp_usecs = 0} args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7ffef81dab50, reg_save_area = 0x7ffef81daa90}}

1 0

Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33)
by Reuben Farrelly 12 Oct '17

12 Oct '17

Hi, Just to confirm this is still a major problem for me with the 2.2.33.1 release (did not exist in previous releases). Reuben On 8/10/2017 6:30 AM, Aki Tuomi wrote: > Thanks! We'll look into it. > > > > --- > Aki Tuomi > Dovecot oy > > -------- Original message -------- > From: Reuben Farrelly <reuben-dovecot(a)reub.net> > Date: 07/10/2017 13:08 (GMT+02:00) > To: dovecot(a)dovecot.org > Subject: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33) > > This appears to be breaking dsync at the moment and is caused by > something committed in the last week or so. This is with the current > -git of dovecot-2.2 (eaf276b33). > > The crash happens all the time - the logs are full of this message. > > Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Warning: I/O > leak: 0x7f8bb6581b4c (ostream-file.c:350, fd 14) > Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Panic: file > ioloop.c: line 126 (io_remove_full): assertion failed: (io->callback != > NULL) > Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Error: Raw > backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6] > -> /usr/lib64/dovecot/li > bdovecot.so.0(+0xd5e26) [0x7f8bb654be26] -> > /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] -> > /usr/lib64/dovecot/libdovecot.so.0(+0xf5cb0) [0x7f8bb6 > 56bcb0] -> /usr/lib64/dovecot/libdovecot.so.0(io_remove+0x1d) > [0x7f8bb656be31] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10b5f8) > [0x7f8bb65815f8] -> /usr/lib64/dovec > ot/libdovecot.so.0(o_stream_cork+0x5a) [0x7f8bb657e289] -> > /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x6406) [0x7f8bb12d6406] > -> /usr/lib64/dovecot/libssl_iost > ream_openssl.so(openssl_iostream_bio_sync+0x18) [0x7f8bb12d6af6] -> > /usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa52e) > [0x7f8bb12da52e] -> / > usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa6ef) [0x7f8bb12da6ef] > -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7) > [0x7f8bb657e765] -> > /usr/lib64/dovecot/libdovecot.so.0(o_stream_send+0x61) [0x7f8bb657e688] > -> /usr/lib64/dovecot/libdovecot.so.0(+0x10dc2c) [0x7f8bb6583c2c] -> > /usr/lib64/dovecot/libdovecot.so.0(+0x10ddd4) [0x7f8bb6583dd4] -> > /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7) [0x7f8bb657e765] > -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsendv+0x55) > [0x7f8bb657e93f] -> > /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsend+0x61) [0x7f8bb657e8d3] > -> dovecot/doveadm-server() [0x444a35] -> > /usr/lib64/dovecot/libdovecot.so.0(i_warning+0xce) [0x7f8bb654afdc] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_destroy+0x2ca) > [0x7f8bb656d8cd] -> dovecot/doveadm-server() [0x4454d1] -> > dovecot/doveadm-server() [0x445793] -> dovecot/doveadm-server() > [0x445dd6] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0xd6) > [0x7f8bb656d241] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x225) > [0x7f8bb656fe58] -> > /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3e) > [0x7f8bb656d422] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0xbe) > [0x7f8bb656d345] > Oct 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Fatal: master: > service(doveadm): child 27186 killed with signal 6 (core dumped) > > > thunderstorm reuben # file core > core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, > from 'dovecot/doveadm-server', real uid: 0, effective uid: 0, real gid: > 0, effective gid: 0, execfn: '/usr/libexec/dovecot/doveadm-server', > platform: 'x86_64' > > thunderstorm reuben # gdb /usr/libexec/dovecot/doveadm-server core > GNU gdb (Gentoo 8.0.1 vanilla) 8.0.1 > Copyright (C) 2017 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-pc-linux-gnu". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <https://bugs.gentoo.org/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /usr/libexec/dovecot/doveadm-server...done. > [New LWP 27186] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `dovecot/doveadm-server'. > Program terminated with signal SIGABRT, Aborted. > #0 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6 > (gdb) bt full > #0 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6 > No symbol table info available. > #1 0x00007f8bb60e9aa7 in abort () from /lib64/libc.so.6 > No symbol table info available. > #2 0x00007f8bb654a8f5 in default_fatal_finish (type=LOG_TYPE_PANIC, > status=0) at failures.c:201 > backtrace = 0x228e4d8 > "/usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6] -> > /usr/lib64/dovecot/libdovecot.so.0(+0xd5e26) [0x7f8bb654be26] -> > /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] -> /usr"... > #3 0x00007f8bb654be26 in i_internal_fatal_handler (ctx=0x7ffef81daa50, > format=0x7f8bb65ba898 "file %s: line %d (%s): assertion failed: (%s)", > args=0x7ffef81daa70) > at failures.c:670 > status = 0 > #4 0x00007f8bb654ac39 in i_panic (format=0x7f8bb65ba898 "file %s: line > %d (%s): assertion failed: (%s)") at failures.c:275 > ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, > timestamp_usecs = 0} > args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = > 0x7ffef81dab50, reg_save_area = 0x7ffef81daa90}}

1 0

Re: Mail Alias Stores in mySQL Not working
by jason hirsh 12 Oct '17

12 Oct '17

"With that query, it appears everyone is sharing the same mail directory.? It, of course, depends on the value of `maildir`.? It doesn't look right.” stored by domain or at least it was until i started messing with mysql "Have you checked your MySQL log file to see the actual query?” hmm found out i had never enabled that log…. another tangent I need to follow "We don't know your mailbox format.? Post your 'doveconf -n’.” My bad # OS: FreeBSD 11.0-RELEASE-p9 amd64 ufs auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = yes base_dir = /var/run/dovecot/ disable_plaintext_auth = no mail_location = maildir:/var/mail/vhosts/%d/%n mail_privileged_group = vmail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } pop3_uidl_format = %v.%u postmaster_address = postmaster(a)theoceanwindow.com service auth-worker { user = vmail } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { mode = 0600 user = vmail } user = dovecot } service director { unix_listener director-userdb { mode = 0600 } } service imap-login { inet_listener imaps { port = 993 ssl = yes } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service pop3-login { inet_listener pop3s { port = 995 ssl = yes } } ssl_ca = </usr/local/etc/keys/certs/ca.crt ssl_cert = </usr/local/etc/keys/certs/domain.crt ssl_key = # hidden, use -P to show it userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol imap { mail_max_userip_connections = 30 } We don't know what your table layout looks like.? Post your 'SHOW COLUMNS FROM `mailbox`’. +------------+--------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +------------+--------------+------+-----+---------------------+-------+ | username | varchar(255) | NO | PRI | NULL | | | password | varchar(255) | NO | | NULL | | | name | varchar(255) | NO | | NULL | | | maildir | varchar(255) | NO | | NULL | | | quota | bigint(20) | NO | | 0 | | | local_part | varchar(255) | NO | | NULL | | | domain | varchar(255) | NO | MUL | NULL | | | created | datetime | NO | | 2000-01-01 00:00:00 | | | modified | datetime | NO | | 2000-01-01 00:00:00 | | | active | tinyint(1) | NO | | 1 | | +------------+--------------+------+-----+---------------------+-------+ What is the on-disk directory layout for mail? mail is stored at /var/mail/vhosts

2 1

Aborted dsync backup and resulting fallout
by Andreas Thienemann 11 Oct '17

11 Oct '17

Hi, I am in the middle of setting up a dovecot imap server to replace an aging cyrus imapd. The new server is using mdbox storage for better filesystem performance and is running dovecot-2.2.32. To migrate email over and keep UIDs etc. intact I used the backup feature of doveadm. During testing this generally worked fine. So thanks for that functionality, it really makes migration much easier. I did another sync yesterday which was unintentionally aborted in the middle which left some corrupted files around. Not a problem generally, that's what doveadm force-resync should fix. It did move a m.2 to m.2.broken mailbox file, so that is good. But it seemed to have issues with the m.3 file, which was an empty file of 0 bytes. Even running force-resync a repeatedly always showed the same message: # doveadm force-resync -u cd INBOX doveadm(cd): Warning: fscking index file /var/spool/imap/user/cd/storage/dovecot.map.index doveadm(cd): Warning: mdbox /var/spool/imap/user/cd/storage: rebuilding indexes doveadm(cd): Error: Corrupted dbox file /var/spool/imap/user/cd/storage/m.3 (around offset=0): EOF while reading file header doveadm(cd): Error: mdbox rebuild: Failed to fix file /var/spool/imap/user/cd/storage/m.3 This looks like incorrect behavior. If the file header on a file expected to be a mailbox header is absent, I would expect it to be either best-effort parsed and the header added or at least be moved out of the way as e.g. m.3.broken. After fixing the m.3 storage manually I encountered another kind of error running dsync again: dsync(cd): Error: lucene index /var/spool/imap/user/cd/lucene-indexes: IndexReader::open() failed (#1): IndexInput read past EOF Lucene index broken should be easily fixed by running doveadm fts rescan. Is that correct? The other error thought had to do with the single instance store for attachments. There were a bunch of errors about a broken pipe and three errors stating success instead: dsync(cd): Error: fs-sis: Couldn't read hash file /var/spool/imap/attachments/c3/0c/hashes/c30cb535927c8bcecb0d45d374624942d0f4e56e: Broken pipe dsync(cd): Error: fs-sis: Couldn't read hash file /var/spool/imap/attachments/0f/0b/hashes/0f0b5351e09ebb487660f23c9b617568093b93d1: Broken pipe dsync(cd): Error: fs-sis: Couldn't read hash file /var/spool/imap/attachments/10/f9/hashes/10f9a4ffd14a5c31f9938c9d8a3ce3f2095fab36: Success dsync(cd): Error: fs-sis: Couldn't read hash file /var/spool/imap/attachments/9b/9a/hashes/9b9a449477882dfca86e80157616cadbb21bb3b8: Success dsync(cd): Error: fs-sis: Couldn't read hash file /var/spool/imap/attachments/df/69/hashes/df698a8bda301a7a474ae2b11414c6f532f4461e: Broken pipe As with the m.3 storage file, all these files had 0 byte size. What is the best way of fixing these up? Delete and be done? Thanks, Andreas

1 0

FTS Dealing badly with Tika failures
by Andreas Thienemann 11 Oct '17

11 Oct '17

Hi, I've seen that latest Dovecot has the ability to use the Apache Tika Server to extract metadata from attachments. That is a pretty useful feature and looks to be an improvement over piping the attachment through a shellscript as done with fts_decoder. Having this feature enabled though I sometimes do see the situation that the tika service does not reply in time and it looks like the whole FTS transaction gets aborted: dovecot: indexer-worker(eggs@localhost): Error: fts_tika: PUT http://localhost:9998/tika/ failed: Request timed out (Request queued 60.145 secs ago, 1 attempts in 60.134 secs, 60.034 in http ioloop, 0.000 in other ioloops, connected 1739.235 secs ago) dovecot: indexer-worker(eggs@localhost): Error: Mailbox INBOX.junk: Mail search failed: Internal error occurred. Refer to server log for more information. [2017-10-09 03:46:09] dovecot: indexer-worker(eggs(a)bawue.de): Error: Mailbox INBOX.junk: Transaction commit failed: FTS transaction commit failed: transaction context (attempted to index 12084 messages (UIDs 2..17850)) Is that actually sensible behavior? I would consider attachment metadata to be nice to have, but not necessarily mandatory. So if the attachment indexing fails, I would hope that at least the mailtext got indexed. But it seems like we abort the whole transaction. Am I reading the error log correctly? cheers, Andreas

1 0