August 2014 - dovecot

LMTP re-reading all messages
by Florent Bautista 20 Aug '14

20 Aug '14

Hi all, I have a little problem with Dovecot 2.1.17 and MailDir accounts. I was using a shared file system (MooseFS), and all was pretty stable. Now I'm switching to CephFS, and Dovecot LMTP have problem to deliver mails. Very often, on large accounts, LMTP processes are running for a few hours before delivering the message. When I strace it, I can see that they are reading (open and stat operations) ALL the messages of the account. I was thinking to an index problem, but I think that after hundreds of delivered mails, indexes are rebuilded and problem should be solved. But not. The same way, If user connects to IMAP, a lot of IMAP processes get stuck on "getquotaroot" command (multiple processes for a single account). The previous FS didn't support mmap, so I disabled it. The new one supports it, so I enabled it, but it does not solve the problem. Do you have an idea of why Dovecot (LMTP or IMAP) spend a lot of time on large accounts ? Thank you a lot. Florent

2 2

antispam plugin crash
by Thomas Witt 20 Aug '14

20 Aug '14

Hi, the dovecot antispam plugin crahses here on a nullpointer dereference in antispam_copy() at mailbox.c The crash tracks back to a call of antispam_classify_copy(asms->box_class, asmb->box_class) At that point in time asmb is set to 0x0, causing the process to crash with SIGSEGV. Regards, Thomas GDB output: Program received signal SIGSEGV, Segmentation fault. antispam_copy (ctx=0x14ed9f0, mail=0x14c6cc0) at mailbox.c:107 107 enum mailbox_copy_type copy_type = (gdb) bt #0 antispam_copy (ctx=0x14ed9f0, mail=0x14c6cc0) at mailbox.c:107 #1 0x00007fdadb627fcd in mailbox_copy () from /usr/lib64/dovecot/libdovecot-storage.so.0 #2 0x00007fdada6cdba0 in act_store_execute () from /usr/lib64/dovecot/libdovecot-sieve.so.0 #3 0x00007fdada6c59d9 in sieve_result_execute () from /usr/lib64/dovecot/libdovecot-sieve.so.0 #4 0x00007fdada6d4a2a in sieve_execute () from /usr/lib64/dovecot/libdovecot-sieve.so.0 #5 0x00007fdada9212cb in lda_sieve_deliver_mail () from /usr/lib64/dovecot/lib90_sieve_plugin.so #6 0x00007fdadb8cae1a in mail_deliver () from /usr/lib64/dovecot/libdovecot-lda.so.0 #7 0x0000000000405ebd in client_input_data_handle () #8 0x00007fdadb34e9bf in io_loop_call_io () from /usr/lib64/dovecot/libdovecot.so.0 #9 0x00007fdadb34f8b7 in io_loop_handler_run_internal () from /usr/lib64/dovecot/libdovecot.so.0 #10 0x00007fdadb34ea29 in io_loop_handler_run () from /usr/lib64/dovecot/libdovecot.so.0 #11 0x00007fdadb34eaa8 in io_loop_run () from /usr/lib64/dovecot/libdovecot.so.0 #12 0x00007fdadb2fe803 in master_service_run () from /usr/lib64/dovecot/libdovecot.so.0 #13 0x0000000000404995 in main () (gdb) p asms $3 = (struct antispam_mailbox *) 0x0

3 9

Panic: file istream.c: line 167 with 2.2.13 + istream.c/.h from dovecot-2-2-dffc6dd2be0d
by Toni Mattila 20 Aug '14

20 Aug '14

Hi, Getting following panics with traces: Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Cached message size smaller than expected (1782 < 1872) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Maildir filename has wrong S value, renamed the file from /path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1782:2,S to /path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1872:2,S Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Corrupted index cache file /path/cencored/Maildir/dovecot.index.cache: Broken physical size for mail UID 5758 Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1782:2,S) failed: Invalid argument Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1782:2,S) failed: Invalid argument (uid=5758) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Cached message size smaller than expected (1781 < 1869) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Maildir filename has wrong S value, renamed the file from /path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1781:2,S to /path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1869:2,S Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Corrupted index cache file /path/cencored/Maildir/dovecot.index.cache: Broken physical size for mail UID 5767 Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1781:2,S) failed: Invalid argument Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1781:2,S) failed: Invalid argument (uid=5767) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Cached message size smaller than expected (9252 < 9346) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Maildir filename has wrong S value, renamed the file from /path/cencored/Maildir/cur/1205131753.12346.host.cencored.fi,S=9252:2,RS to /path/cencored/Maildir/cur/1205131753.12346.host.cencored.fi,S=9346:2,RS Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Corrupted index cache file /path/cencored/Maildir/dovecot.index.cache: Broken physical size for mail UID 6296 Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Panic: file istream.c: line 167 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0x575c1f] -> /usr/lib/dovecot/libdovecot.so.0 [0x575c9d] -> /usr/lib/dovecot/libdovecot.so.0 [0x575514] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x13b) [0x57f99b] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x2b) [0x57fbfb] -> /usr/lib/dovecot/libdovecot.so.0 [0x565b5e] -> /usr/lib/dovecot/libdovecot.so.0 [0x567469] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x43) [0x565cd3] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_body+0x2c) [0x565e4c] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x944dd2] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_parts+0x6a) [0x945c7a] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_parts+0x16) [0x91ad76] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x94aa7c] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x94ba89] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblo Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Fatal: master: service(imap): child 31268 killed with signal 6 (core dumps disabled) These can of course be fixed with maildir-size-fix.pl and deleting dovecot index files, but there are thousands like these. Dovecot 1.2 didn't have any issues with these old maildirs transfered over from courierimap. Best Regards, Toni Mattila

2 9

Dovecot v2.2 FTS is not indexing "text/html" emails...
by Akash 20 Aug '14

20 Aug '14

Hi, I am not sure its intended or a fault in the newest Dovecot versions. I have been using Dovecot v1.2.15 on Debian squeeze and FTS is working as expected. When I search a quoted string "very good", I get 107 results including plain and HTML emails which have this phrase. In order to compare the benefits of lucene over squat, I recently started testing dovecot v2.2.13 on Debian Sid with the same maildir content. But now the same search "very good" yielded just 8 results. I thought it could be some problem with lucene so I tried switching to squat and got 107 results again. After this I deleted the old squat search index files created by v1.2.15 and re-indexed the mail-box by using doveadm index command. Now the same squat search is giving 8 results just as lucene. So I have realized that its not a problem with just lucene but FTS in newer dovecot isn't indexing those emails which have Content-type as text/html. Thus if a mail is like this: Content-Type: text/html <b>He is very good.</b> It isn't shown in search by the squat indexes created using dovecot v2.2.13. I have done further testing on some sample emails which confirmed this behavior. Why is this so? -Regards, Akash

2 2

2.2.13 + hg: Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (fd >= 0)
by Toni Mattila 20 Aug '14

20 Aug '14

Hi, Found this in logfiles: Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Error: UIDL: File name lookup failed: Message was expunged Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Disconnected: POP3 UIDLs couldn't be listed top=187/356348, retr=0/0, del=0/1207, size=789346199 Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (fd >= 0) Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0xbdfc1f] -> /usr/lib/dovecot/libdovecot.so.0 [0xbdfc9d] -> /usr/lib/dovecot/libdovecot.so.0 [0xbdf514] -> /usr/lib/dovecot/libdovecot.so.0 [0xbf162d] -> /usr/lib/dovecot/libdovecot.so.0(io_add_istream+0x3f) [0xbf272f] -> dovecot/pop3 [0x804d149] -> /usr/lib/dovecot/libdovecot.so.0 [0xbff17a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x48) [0xbf1d88] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xff) [0xbf375f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x1e) [0xbf1e2e] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0xbf1eb8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x2d) [0xb9842d] -> dovecot/pop3(main+0x2a9) [0x804b3b9] -> /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xdc) [0x6f7dec] -> dovecot/pop3 [0x804aeb1] Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Fatal: master: service(pop3): child 27510 killed with signal 6 (core dumped) (gdb) bt #0 0x00b1a402 in __kernel_vsyscall () #1 0x0070af30 in raise () from /lib/i686/nosegneg/libc.so.6 #2 0x0070c911 in abort () from /lib/i686/nosegneg/libc.so.6 #3 0x00bdfc14 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:193 #4 0x00bdfc9d in i_internal_fatal_handler (ctx=0xbf82cd34, format=0xc0c784 "file %s: line %d (%s): assertion failed: (%s)", args=0xbf82cd54 "\202\232�") at failures.c:657 #5 0x00bdf514 in i_panic (format=0xc0c784 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:267 #6 0x00bf162d in io_add_file (fd=-1, condition=IO_READ, source_linenum=816, callback=0x804d150 <client_input>, context=0x83ba628) at ioloop.c:40 #7 0x00bf272f in io_add_istream (input=0x83ba960, source_linenum=816, callback=0x804d150 <client_input>, context=0x83ba628) at ioloop.c:86 #8 0x0804d149 in client_output (client=0x83ba628) at pop3-client.c:815 #9 0x00bff17a in stream_send_io (fstream=0x83baa48) at ostream-file.c:468 #10 0x00bf1d88 in io_loop_call_io (io=0x83baaf8) at ioloop.c:441 #11 0x00bf375f in io_loop_handler_run_internal (ioloop=0x83a5498) at ioloop-epoll.c:220 #12 0x00bf1e2e in io_loop_handler_run (ioloop=0x83a5498) at ioloop.c:488 #13 0x00bf1eb8 in io_loop_run (ioloop=0x83a5498) at ioloop.c:465 #14 0x00b9842d in master_service_run (service=0x83a53c0, callback=0x804b4e0 <client_connected>) at master-service.c:566 #15 0x0804b3b9 in main (argc=Cannot access memory at address 0x0 ) at main.c:277 (gdb) Thanks, Toni

2 1

[Patch] mail_log plugin supports remote, local ip addresses
by Vitaliy T 20 Aug '14

20 Aug '14

Hello, In attached file you will find patch for mail_log plugin to output local, remote ip addresses. It may be usefull for somebody. To enable support for dovecot 2.2.x versions change the line 'include "net.h"' instead of 'include "network.h"'. Thank you. P.S. Sorry, for previous empty message. -- With Best Regards, Vitaliy T.

2 1

auth_username_translation and LTMP problem
by Arkadiusz Miśkiewicz 20 Aug '14

20 Aug '14

auth_username_translation seems to be appliet on RCPT TO address of LTMP transport. Why dovecot is doing that? And better question - is there a way to disable auth_username_translation for LTMP but leave enabled for the rest (imap, pop3 etc) ? Background: I'm doing auth_username_translation = @= to allow logins like aaa(a)bbb.pl to be internally translated to aaa=bbb.pl. That works fine. Now my exim delivers mail to dovecot using LTMP and it does translation or it own, so it does: RCPT TO: <aaa=bbb.pl(a)mymbox.pl> Unfortunately looks like dovecot it doing translation one more time and looking in user database for aaa=bbb.pl=mymbox.pl where such user doesn't exist. Only aaa=bbb.pl user exists. Thanks, -- Arkadiusz Miśkiewicz, arekm / maven.pl

2 2

Mail Archiving/Lucene Indexes/mbox
by Larry Rosenman 20 Aug '14

20 Aug '14

I have a script (available on demand) that moves all my mail for a given month to a separate NAMESPACE (#ARCHIVE). When I do that, the lucene-indexes folder in the PRIMARY namespace seems to get cleaned out, and NOTHING gets put in the lucene-indexes folder (although it gets created) in the #ARCHIVE namespace. Is this to spec? the archiving is done via doveadm move commands. Thanks! -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

2 3

Segfault with indexer-worker using Tika/Solr
by G H 20 Aug '14

20 Aug '14

Using Dovecot 2.2.13 with Tika 1.5 on Centos 5 64bit Tika is throwing an exception for some reason, which seems to be causing Dovecot's indexer to segfault. I'm using mdbox/gzipped compressed mail and private indexes for the shared folder. I'm importing mail via formail/dovecot-lda: formail -s /usr/libexec/dovecot/dovecot-lda -d admin -m Public/TestSubFolder < /tmp/Inbox.mbox dovecot -n and the GDB backtrace are attached. # find /var/mail/ -maxdepth 2 /var/mail/_public /var/mail/_public/dovecot.mailbox.log /var/mail/_public/dovecot-uidvalidity /var/mail/_public/dovecot-uidvalidity.53ecade1 /var/mail/_public/storage /var/mail/_public/subscriptions /var/mail/_public/mailboxes /var/mail/admin /var/mail/admin/dovecot-uidvalidity.53ecade2 /var/mail/admin/dovecot.mailbox.log /var/mail/admin/public /var/mail/admin/dovecot-uidvalidity /var/mail/admin/storage /var/mail/admin/subscriptions /var/mail/admin/mailboxes

2 1

BUG: Solr FTS engine appends domain name to index records
by G H 20 Aug '14

20 Aug '14

The Solr FTS engine appends domain names to the "user" and "id" fields inside Solr when indexing. However, when an IMAP search is performed, the domain name is not appended, resulting in 0 results, 100% of the time. You can reproduce this by setting up dovecot to have user names like "admin", "james", etc (as opposed to admin(a)example.com) Then import some mail (ie: "dovecot-lda -d admin < /tmp/mymail.txt"). Now attempt to search for that mail and it will not be returned as a result. You can see in the Solr GUI (http://xxxx:8983/solr/) that record is indexed as: { "uid": 1, "box": "7dff0b0d4f84df53f25300003d823887", "user": "admin(a)example.com", "id": "1/7dff0b0d4f84df53f25300003d823887/admin(a)example.com", "_version_": 1475511406809841700 } However, the IMAP search is sent to Solr performed with user:"admin" instead of user:"admin(a)example.com". Either the Solr FTS plugin needs to not append the domain name or the search does need to do so.

2 2