August 2014 - dovecot

Panic: file istream.c: line 167 with 2.2.13 + istream.c/.h from dovecot-2-2-dffc6dd2be0d
by Toni Mattila 20 Aug '14

20 Aug '14

Hi, Getting following panics with traces: Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Cached message size smaller than expected (1782 < 1872) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Maildir filename has wrong S value, renamed the file from /path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1782:2,S to /path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1872:2,S Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Corrupted index cache file /path/cencored/Maildir/dovecot.index.cache: Broken physical size for mail UID 5758 Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1782:2,S) failed: Invalid argument Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201713258.3136.host.cencored.fi,S=1782:2,S) failed: Invalid argument (uid=5758) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Cached message size smaller than expected (1781 < 1869) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Maildir filename has wrong S value, renamed the file from /path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1781:2,S to /path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1869:2,S Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Corrupted index cache file /path/cencored/Maildir/dovecot.index.cache: Broken physical size for mail UID 5767 Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1781:2,S) failed: Invalid argument Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: read(/path/cencored/Maildir/cur/1201767850.29173.host.cencored.fi,S=1781:2,S) failed: Invalid argument (uid=5767) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Cached message size smaller than expected (9252 < 9346) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Maildir filename has wrong S value, renamed the file from /path/cencored/Maildir/cur/1205131753.12346.host.cencored.fi,S=9252:2,RS to /path/cencored/Maildir/cur/1205131753.12346.host.cencored.fi,S=9346:2,RS Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Corrupted index cache file /path/cencored/Maildir/dovecot.index.cache: Broken physical size for mail UID 6296 Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Panic: file istream.c: line 167 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0x575c1f] -> /usr/lib/dovecot/libdovecot.so.0 [0x575c9d] -> /usr/lib/dovecot/libdovecot.so.0 [0x575514] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x13b) [0x57f99b] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x2b) [0x57fbfb] -> /usr/lib/dovecot/libdovecot.so.0 [0x565b5e] -> /usr/lib/dovecot/libdovecot.so.0 [0x567469] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x43) [0x565cd3] -> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_body+0x2c) [0x565e4c] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x944dd2] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_get_parts+0x6a) [0x945c7a] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_parts+0x16) [0x91ad76] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x94aa7c] -> /usr/lib/dovecot/libdovecot-storage.so.0 [0x94ba89] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblo Jul 6 08:11:52 dovecot: imap(user(a)domain.tld): Fatal: master: service(imap): child 31268 killed with signal 6 (core dumps disabled) These can of course be fixed with maildir-size-fix.pl and deleting dovecot index files, but there are thousands like these. Dovecot 1.2 didn't have any issues with these old maildirs transfered over from courierimap. Best Regards, Toni Mattila

2 9

Dovecot v2.2 FTS is not indexing "text/html" emails...
by Akash 20 Aug '14

20 Aug '14

Hi, I am not sure its intended or a fault in the newest Dovecot versions. I have been using Dovecot v1.2.15 on Debian squeeze and FTS is working as expected. When I search a quoted string "very good", I get 107 results including plain and HTML emails which have this phrase. In order to compare the benefits of lucene over squat, I recently started testing dovecot v2.2.13 on Debian Sid with the same maildir content. But now the same search "very good" yielded just 8 results. I thought it could be some problem with lucene so I tried switching to squat and got 107 results again. After this I deleted the old squat search index files created by v1.2.15 and re-indexed the mail-box by using doveadm index command. Now the same squat search is giving 8 results just as lucene. So I have realized that its not a problem with just lucene but FTS in newer dovecot isn't indexing those emails which have Content-type as text/html. Thus if a mail is like this: Content-Type: text/html <b>He is very good.</b> It isn't shown in search by the squat indexes created using dovecot v2.2.13. I have done further testing on some sample emails which confirmed this behavior. Why is this so? -Regards, Akash

2 2

2.2.13 + hg: Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (fd >= 0)
by Toni Mattila 20 Aug '14

20 Aug '14

Hi, Found this in logfiles: Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Error: UIDL: File name lookup failed: Message was expunged Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Disconnected: POP3 UIDLs couldn't be listed top=187/356348, retr=0/0, del=0/1207, size=789346199 Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (fd >= 0) Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0 [0xbdfc1f] -> /usr/lib/dovecot/libdovecot.so.0 [0xbdfc9d] -> /usr/lib/dovecot/libdovecot.so.0 [0xbdf514] -> /usr/lib/dovecot/libdovecot.so.0 [0xbf162d] -> /usr/lib/dovecot/libdovecot.so.0(io_add_istream+0x3f) [0xbf272f] -> dovecot/pop3 [0x804d149] -> /usr/lib/dovecot/libdovecot.so.0 [0xbff17a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x48) [0xbf1d88] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xff) [0xbf375f] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x1e) [0xbf1e2e] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x48) [0xbf1eb8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x2d) [0xb9842d] -> dovecot/pop3(main+0x2a9) [0x804b3b9] -> /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xdc) [0x6f7dec] -> dovecot/pop3 [0x804aeb1] Jul 25 14:12:38 dovecot: pop3(cen(a)so.red): Fatal: master: service(pop3): child 27510 killed with signal 6 (core dumped) (gdb) bt #0 0x00b1a402 in __kernel_vsyscall () #1 0x0070af30 in raise () from /lib/i686/nosegneg/libc.so.6 #2 0x0070c911 in abort () from /lib/i686/nosegneg/libc.so.6 #3 0x00bdfc14 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:193 #4 0x00bdfc9d in i_internal_fatal_handler (ctx=0xbf82cd34, format=0xc0c784 "file %s: line %d (%s): assertion failed: (%s)", args=0xbf82cd54 "\202\232�") at failures.c:657 #5 0x00bdf514 in i_panic (format=0xc0c784 "file %s: line %d (%s): assertion failed: (%s)") at failures.c:267 #6 0x00bf162d in io_add_file (fd=-1, condition=IO_READ, source_linenum=816, callback=0x804d150 <client_input>, context=0x83ba628) at ioloop.c:40 #7 0x00bf272f in io_add_istream (input=0x83ba960, source_linenum=816, callback=0x804d150 <client_input>, context=0x83ba628) at ioloop.c:86 #8 0x0804d149 in client_output (client=0x83ba628) at pop3-client.c:815 #9 0x00bff17a in stream_send_io (fstream=0x83baa48) at ostream-file.c:468 #10 0x00bf1d88 in io_loop_call_io (io=0x83baaf8) at ioloop.c:441 #11 0x00bf375f in io_loop_handler_run_internal (ioloop=0x83a5498) at ioloop-epoll.c:220 #12 0x00bf1e2e in io_loop_handler_run (ioloop=0x83a5498) at ioloop.c:488 #13 0x00bf1eb8 in io_loop_run (ioloop=0x83a5498) at ioloop.c:465 #14 0x00b9842d in master_service_run (service=0x83a53c0, callback=0x804b4e0 <client_connected>) at master-service.c:566 #15 0x0804b3b9 in main (argc=Cannot access memory at address 0x0 ) at main.c:277 (gdb) Thanks, Toni

2 1

[Patch] mail_log plugin supports remote, local ip addresses
by Vitaliy T 20 Aug '14

20 Aug '14

Hello, In attached file you will find patch for mail_log plugin to output local, remote ip addresses. It may be usefull for somebody. To enable support for dovecot 2.2.x versions change the line 'include "net.h"' instead of 'include "network.h"'. Thank you. P.S. Sorry, for previous empty message. -- With Best Regards, Vitaliy T.

2 1

auth_username_translation and LTMP problem
by Arkadiusz Miśkiewicz 20 Aug '14

20 Aug '14

auth_username_translation seems to be appliet on RCPT TO address of LTMP transport. Why dovecot is doing that? And better question - is there a way to disable auth_username_translation for LTMP but leave enabled for the rest (imap, pop3 etc) ? Background: I'm doing auth_username_translation = @= to allow logins like aaa(a)bbb.pl to be internally translated to aaa=bbb.pl. That works fine. Now my exim delivers mail to dovecot using LTMP and it does translation or it own, so it does: RCPT TO: <aaa=bbb.pl(a)mymbox.pl> Unfortunately looks like dovecot it doing translation one more time and looking in user database for aaa=bbb.pl=mymbox.pl where such user doesn't exist. Only aaa=bbb.pl user exists. Thanks, -- Arkadiusz Miśkiewicz, arekm / maven.pl

2 2

Mail Archiving/Lucene Indexes/mbox
by Larry Rosenman 20 Aug '14

20 Aug '14

I have a script (available on demand) that moves all my mail for a given month to a separate NAMESPACE (#ARCHIVE). When I do that, the lucene-indexes folder in the PRIMARY namespace seems to get cleaned out, and NOTHING gets put in the lucene-indexes folder (although it gets created) in the #ARCHIVE namespace. Is this to spec? the archiving is done via doveadm move commands. Thanks! -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

2 3

Segfault with indexer-worker using Tika/Solr
by G H 20 Aug '14

20 Aug '14

Using Dovecot 2.2.13 with Tika 1.5 on Centos 5 64bit Tika is throwing an exception for some reason, which seems to be causing Dovecot's indexer to segfault. I'm using mdbox/gzipped compressed mail and private indexes for the shared folder. I'm importing mail via formail/dovecot-lda: formail -s /usr/libexec/dovecot/dovecot-lda -d admin -m Public/TestSubFolder < /tmp/Inbox.mbox dovecot -n and the GDB backtrace are attached. # find /var/mail/ -maxdepth 2 /var/mail/_public /var/mail/_public/dovecot.mailbox.log /var/mail/_public/dovecot-uidvalidity /var/mail/_public/dovecot-uidvalidity.53ecade1 /var/mail/_public/storage /var/mail/_public/subscriptions /var/mail/_public/mailboxes /var/mail/admin /var/mail/admin/dovecot-uidvalidity.53ecade2 /var/mail/admin/dovecot.mailbox.log /var/mail/admin/public /var/mail/admin/dovecot-uidvalidity /var/mail/admin/storage /var/mail/admin/subscriptions /var/mail/admin/mailboxes

2 1

BUG: Solr FTS engine appends domain name to index records
by G H 20 Aug '14

20 Aug '14

The Solr FTS engine appends domain names to the "user" and "id" fields inside Solr when indexing. However, when an IMAP search is performed, the domain name is not appended, resulting in 0 results, 100% of the time. You can reproduce this by setting up dovecot to have user names like "admin", "james", etc (as opposed to admin(a)example.com) Then import some mail (ie: "dovecot-lda -d admin < /tmp/mymail.txt"). Now attempt to search for that mail and it will not be returned as a result. You can see in the Solr GUI (http://xxxx:8983/solr/) that record is indexed as: { "uid": 1, "box": "7dff0b0d4f84df53f25300003d823887", "user": "admin(a)example.com", "id": "1/7dff0b0d4f84df53f25300003d823887/admin(a)example.com", "_version_": 1475511406809841700 } However, the IMAP search is sent to Solr performed with user:"admin" instead of user:"admin(a)example.com". Either the Solr FTS plugin needs to not append the domain name or the search does need to do so.

2 2

Trouble getting public folders working
by Bjarne Mark Kondrup 20 Aug '14

20 Aug '14

Hi i have trouble getting my public folders working properly. I have a few issues atm with them. First off i'm using version 2.0.9 as it came with CentOS and would greatly prefer to keep this version. Here is my dovecot.conf file # dovecot -n # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-431.23.3.el6.centos.plus.x86_64 x86_64 CentOS release 6.5 (Final) ext4 auth_debug = yes auth_verbose = yes disable_plaintext_auth = no first_valid_uid = 101 last_valid_uid = 101 login_greeting = Dovecot ready. CEGO webmail server mail_debug = yes mail_gid = vmail mail_location = maildir:/home/vmail/domains/%d/%n mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mbox_write_locks = fcntl namespace { inbox = yes location = prefix = separator = / type = private } namespace { location = maildir:/home/vmail/shared:LAYOUT=fs:INDEX=/home/vmail/shared/ prefix = shared/ separator = / subscriptions = no type = public } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { sieve = /home/vmail/domains/%d/%n/dovecot.sieve sieve_dir = /home/vmail/domains/%d/%n/sieve_scripts sieve_global_dir = /etc/dovecot/sieve/global/ } protocols = imap lmtp sieve service auth { unix_listener auth-client { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } ssl = no ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } valid_chroot_dirs = /home/vmail/domains verbose_ssl = yes protocol lmtp { log_path = /var/log/dovecot-deliver.log mail_plugins = quota sieve sieve acl postmaster_address = postmaster(a)cegotest.dk } protocol sieve { info_log_path = /var/log/dovecot-sieve.log log_path = /var/log/dovecot-sieve-errors.log managesieve_implementation_string = dovecot managesieve_max_line_length = 65536 } And my dovecot-ldap file hosts = localhost dn = cn=dovecot,dc=cegotest,dc=dk dnpass = password auth_bind = yes ldap_version = 3 base = o=hosting,dc=cegotest,dc=dk deref = never scope = subtree user_filter = (&(objectClass=JammMailAccount)(mail=%u)(accountActive=TRUE)(delete=FALSE)) pass_attrs = mail,userPassword pass_filter = (&(objectClass=JammMailAccount)(mail=%u)(accountActive=TRUE)(delete=FALSE)) default_pass_scheme = CRYPT Here is my current issues. 1. I cannot get shared folders to work with pr. user settings. What i want is a public folder that all my users can see but each user marks mail as read individually. Atm no matter how i set it up if user1 marks a mail as read, then user2 will also see that mail as read :( 2. ACL to control the permissions. Right now all users can move mail and remove mail from the shared folders. I want people (except a few specific users) to be read only on this public folders. 3. Need to know if i'm creating the folders correctly. Right now i'm under my shared folder directory creating shared_folder and 3 subfolders beneath it (cur, new and tmp). After this it shows up in eg. roundcube and i can subscribe to it. I have read all the documentation and tried all the solutions i can. Mostly moving the index and/or config files to seperate dirs and from my understanding that should give each user their own index file marking what mails have been read ect. But this doesn't happen. Hope you guys are able to help me out a little here. -Bjarne

2 1

Huge difference between the lucene index size created by v2.1 and v2.2
by Akash 20 Aug '14

20 Aug '14

Hi everyone, While examining dovecot versions v2.1 and v2.2 for their lucene search performances, I have noticed a huge difference in the index sizes created by them. Both versions were compiled on same system, against same libclucene, same configure options and were used with same dovecot.conf during run-time. I deleted the lucene-indexes folder and dovecot* files in the Maildir prior to indexing with both versions. The tests were performed on a untouched mail archive folder containing 300000 mails without any dovecot* files in it: root@server:/home/admin/mails/.Archive# ls -l total 78640 drwx------ 2 2500 2500 14098432 Aug 17 22:16 cur drwx------ 2 2500 2500 12435456 Jul 30 09:46 new drwx------ 2 2500 2500 4096 Aug 2 13:02 tmp The command used was: doveadm -v index -u admin Archive After search indexing on v2.1 resulted in: root@server:/home/admin/mails# ls -lh lucene-indexes total 390M -rw------- 1 2500 2500 390M Aug 18 07:03 _25.cfs -rw------- 1 2500 2500 20 Aug 18 07:03 segments.gen -rw------- 1 2500 2500 46 Aug 18 07:03 segments_4d Whereas dovecot v2.2 resulted in: root@server:/home/admin/mails# ls -lh lucene-indexes total 1.5G -rw------- 1 2500 2500 1.5G Aug 18 06:41 _5g.cfs -rw------- 1 2500 2500 20 Aug 18 06:41 segments.gen -rw------- 1 2500 2500 46 Aug 18 06:41 segments_az 390M vs 1.5G. That is a huge difference in size. Why is that? Thanks in advance. -Regards, Akash

2 5