February 2013 - dovecot

[Dovecot] 2.1.12: Panic: file solr-connection.c: line 547 (solr_connection_post_more)
by Marc Schiffbauer 26 Feb '13

26 Feb '13

Hi all, I am having a problem indexing one of my mailboxes using the solr fts backend in dovecot 2.1.12 For many mailboxes it works just fine, but on one mailbox I currently always get a panic. solr setup: Java: icedtea 6.1.11.5 Solr: 3.6.2 running in tomcat 7.0.32 Command to reproduce error: doveadm index -u my@user badmailbox I already noticed that there have been some solr backend fixes just before 2.1.21 release... Is this another bug in the backend? Any idea? IMO relevant logs line parts: java.io.CharConversionException: Invalid UTF-8 start byte 0xf8 (at char #483989, byte #532382) complete dovecot log: doveadm(my@user): Error: fts_solr: Invalid XML input at 85:198: mismatched tag (near: <html><head><title>Apache Tomcat/7.0.32-gentoo - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:wh) doveadm(my@user): Panic: file solr-connection.c: line 547 (solr_connection_post_more): assertion failed: (maxfd >= 0) doveadm(my@user): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x4bcdb) [0x34fbd53ecdb] -> /usr/lib64/dovecot/libdovecot.so.0(default_fatal_handler+0x3a) [0x34fbd53edb8] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x34fbd50eeac] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(solr_connection_post_more+0x2a4) [0x34fbba81072] -> /usr/lib64/dovecot/lib21_fts_solr_plugin.so(+0x40a5) [0x34fbba7d0a5] -> /usr/lib64/dovecot/lib20_fts_plugin.so(fts_build_mail+0x462) [0x34fbbc8d6dc] -> /usr/lib64/dovecot/lib20_fts_plugin.so(+0xd8c2) [0x34fbbc928c2] -> doveadm(+0x17a30) [0x264ecb7a30] -> doveadm(+0x13a45) [0x264ecb3a45] -> doveadm(+0x14762) [0x264ecb4762] -> doveadm(doveadm_mail_try_run+0x173) [0x264ecb4cfd] -> doveadm(main+0x3e9) [0x264ecbd0d5] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x34fbd16a48d] -> doveadm(+0x13809) [0x264ecb3809] Aborted complete java exception: Dec 31, 2012 4:12:58 AM org.apache.solr.core.SolrCore execute INFO: [] webapp=/solr-web path=/update params={} status=500 QTime=118 Dec 31, 2012 4:12:58 AM org.apache.solr.common.SolrException log SEVERE: java.lang.RuntimeException: [was class java.io.CharConversionException] Invalid UTF-8 start byte 0xf8 (at char #483989, byte #532382) at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18) at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731) at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657) at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809) at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:315) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:156) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown Source) at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown Source) at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown Source) at org.apache.catalina.core.StandardContextValve.invoke(Unknown Source) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Unknown Source) at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source) at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source) at org.apache.catalina.valves.AccessLogValve.invoke(Unknown Source) at org.apache.catalina.core.StandardEngineValve.invoke(Unknown Source) at org.apache.catalina.connector.CoyoteAdapter.service(Unknown Source) at org.apache.coyote.http11.AbstractHttp11Processor.process(Unknown Source) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Unknown Source) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0xf8 (at char #483989, byte #532382) at com.ctc.wstx.io.UTF8Reader.reportInvalidInitial(UTF8Reader.java:302) at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:188) at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649) ... 25 more TIA -Marc -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich

4 9

[Dovecot] Dovecot auth works when tested with doveadm, but fails with Postfix
by Darren Pilgrim 26 Feb '13

26 Feb '13

I have three postfix 2.9.5 servers: chombo, rush, yoshi. Chombo relays to rush and yoshi for outbound email. Outbound relay requires SASL authentication. Rush and yoshi run Dovecot 2.1.12 servers with simple passwd-file backends. If I create a new password hash for chombo's user, houseloki, on either rush or yoshi: # doveadm pw -u houseloki -p <password> {CRAM-MD5}... Then I add that to rush and yoshi's passwd file: houseloki@_auth.bluerosetech.com:{CRAM-MD5}... Then `doveadm reload`, it works fine: # doveadm auth houseloki <password> passdb: houseloki auth succeeded extra fields: user=houseloki(a)_auth.bluerosetech.com So I add that username and password to the smtp_sasl_password_maps hash file on chombo, reload postfix, and then try to relay something from chombo, it fails with rush and yoshi logging warnings like: Feb 19 03:32:33 yoshi postfix/smtpd[75783]: warning: chombo.example.com[2001:db8::2]: SASL DIGEST-MD5 authentication failed: <really long string redacted> Rush and yoshi have other hashes in their passwd files, and if I configure chombo to use one of those, it works fine. Those hashes use digest-md5 and are at least a few years old. I tried that scheme instead of the default cram-md5, as well as several others, but none work. I've poured over the wiki and man pages, but can't find the problem. What am I missing? Why can I not generate new hashes correctly?

2 3

Re: [Dovecot] v2.2.rc1 released
by Michael Grimm 25 Feb '13

25 Feb '13

> Please test and report any bugs found. New in rc1: | doveconf: Error: dlopen(/.../lib/dovecot/settings/libmanagesieve_login_settings.so) failed: /.../lib/dovecot/settings/libmanagesieve_login_settings.so: Undefined symbol "hook_config_parser_begin" | doveconf: Error: dlopen(/.../lib/dovecot/settings/libmanagesieve_settings.so) failed: /.../lib/dovecot/settings/libmanagesieve_settings.so: Undefined symbol "mail_user_setting_parser_info" | doveconf: Error: protocols: Unknown protocol: sieve | doveconf: Fatal: Error in configuration file /.../etc/dovecot/dovecot.conf: protocols: Unknown protocol: sieve Regards, Michael

3 9

[Dovecot] 2.2beta2: LSUB broken
by Lutz Preßler 25 Feb '13

25 Feb '13

Hello, I have some problems with (listing) subscriptions on my test system running 2.2beta2 (compared with 2.1.15, configuration and Maildir content unchanged): With 2.1.15: 0 lsub "" "*" * LSUB () "." "INBOX.Trash" * LSUB () "." "INBOX.in2009" * LSUB () "." "INBOX.test" * LSUB () "." "INBOX.test.test2" * LSUB () "." "INBOX.in2010" * LSUB () "." "INBOX.in" * LSUB () "." "INBOX.virtual.small" * LSUB () "." "INBOX.virtual.sent" * LSUB () "." "INBOX.virtual.allin" * LSUB () "." "INBOX.virtual.week" * LSUB () "." "INBOX.virtual.gmail" 0 OK Lsub completed. With 2.2beta2 (output of doveconf -n attached): 0 lsub "" "*" 0 OK Lsub completed. ... empty list. Same with "list (subscribed)". After disableing the "virtual" namespace in the config, it gets: 0 lsub "" "*" * LSUB () "." Trash * LSUB () "." in2009 * LSUB () "." test * LSUB () "." test.test2 * LSUB () "." in2010 * LSUB () "." in * LSUB () "." INBOX.Trash * LSUB () "." INBOX.in2009 * LSUB () "." INBOX.test * LSUB () "." INBOX.test.test2 * LSUB () "." INBOX.in2010 * LSUB () "." INBOX.in 0 OK Lsub completed. Note mailboxes listed with and without INBOX. prefix. Thanks, Lutz

2 3

[Dovecot] v2.2.rc1 released
by Timo Sirainen 25 Feb '13

25 Feb '13

http://dovecot.org/releases/2.2/rc/dovecot-2.2.rc1.tar.gz http://dovecot.org/releases/2.2/rc/dovecot-2.2.rc1.tar.gz.sig A few fixes since beta2. Also added tcp and tcps (=tcp+ssl) targets for dsync, where the destination needs to be a doveadm-server listener. I'm planning on releasing v2.2.0 in 1-2 weeks if no major problems are found. Please test and report any bugs found. BTW. I've also gone through basically all of my unread emails. If you haven't gotten a reply to something, ask again. Here's again all the biggest changes since v2.1: * When creating home directories, the permissions are copied from the parent directory if it has setgid-bit set. For full details, see http://wiki2.dovecot.org/SharedMailboxes/Permissions * "doveadm auth" command was renamed to "doveadm auth test" * IMAP: ID command now advertises server name as Dovecot by default. It was already trivial to guess this from command replies. * dovecot.index.cache files can be safely accessed only by v2.1.11+. Older versions may think they're corrupted and delete them. * LDA/LMTP: If saving a mail brings user from under quota to over quota, allow it based on quota_grace setting (default: 10% above quota limit). * pop3_lock_session=yes now uses a POP3-only dovecot-pop3-session.lock file instead of actually locking the mailbox (and causing IMAP/LDA/LMTP to wait for the POP3 session to close). * mail_shared_explicit_inbox setting's default switched to "no". * ssl_client_ca_dir setting replaced imapc_ssl_ca_dir and pop3c_ssl_ca_dir settings. + Implemented IMAP MOVE and BINARY extensions + Implemented IMAP CATENATE, URLAUTH and URLAUTH=BINARY extensions (by Stephan Bosch). + Implemented IMAP NOTIFY extension. Requires mailbox_list_index=yes to be enabled. + Redesigned and rewritten dsync. The new design makes the syncing faster, more reliable and more featureful. The new dsync protocol isn't backwards compatible with old dsync versions (but is designed to be forwards compatible with future versions). + All mailbox formats now support per-user message flags for shared mailboxes by using a private index. It can be enabled by adding :INDEXPVT=<path> to mail location. This should be used instead of :INDEX also for Maildir/mbox to improve performance. + Improved mailbox list indexes. They should be usable now, although still disabled by default. + Added LAYOUT=index. The mailbox directories are created using their GUIDs in the filesystem, while the actual GUID <-> name mapping exists only in the index. + LMTP proxy: Implemented XCLIENT extension for passing remote IP address through proxy.

1 0

[Dovecot] Bug when a duplicated file is found
by Angel L. Mateo 25 Feb '13

25 Feb '13

Hello, Last days we had to restore some mailboxes from backup. Our mailboxes are in maildir format and during restore, some duplicated files appeared. The situation is: * A user has messages in his new directory when the backup is made * In some moment after the backup is done, the user accesses to his mailbox, so the message is move to the cur directory. * We restore his mailbox directly to his maildir path, so the message is now at new directory too. When this happens, dovecot detect it and it renames one of the files. This is the reported log: Feb 13 23:08:05 myotis31 dovecot: imap(<user>): Warning: Fixed a duplicate: /home/alumnos/59/122559/Maildir/cur/1360169810.M670208P13643.myotis31,S=6282,W=6409:2, -> 1360793285.M721935P29136.myotis31 The problem with this is that we are using zlib plugin so messages are compressed. When the rename is done, the rename is to a name without the S=XXXX field (neither the W=XXX, although this field is less sensible), so the result is that user's can't access to any message after this one. I think than in this situation, rename one of the files is a good solution (although it provokes duplicate mails, but this an expected/acceptable situation after the restore), but the rename should maintain the S= and W= fields (and even other maildir flags). -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337

3 3

[Dovecot] SSL errors for just one client after updaing both dovecot and openssl
by Charles Marcus 25 Feb '13

25 Feb '13

Hi all, Ok, I have a strange problem after updating both dovecot and openssl... OpenSSL was 1.0.0j, now updated to 1.0.1c Dovecot was 2.1.13, now updated to 2.1.15 I'm getting a bunch of lines like the following: Feb 23 10:48:01 myhost dovecot: imap-login: Disconnected (no auth attempts in 29 secs): user=<>, rip=#.#.#.#, lport=993, TLS handshaking: SSL_accept() syscall failed: Connection reset by peer, session=<In+cO2bWngCthJz2> where only the session id (and number of seconds for no auth attempts) is different... This is happening for only the one client. All other clients - I've counted about 25 so far - are working fine. Anyone have any ideas? I can't believe it is a generic openssl problem, since it is only affecting the one client. I've contacted him and asked him to reboot any/all devices that connect to our mail to see if that helps... -- Best regards, */Charles/*

3 3

[Dovecot] understanding namespace configuration
by btb＠bitrate.net 25 Feb '13

25 Feb '13

i'm a little confused about namespace configuration. reading http://wiki2.dovecot.org/Namespaces , and looking at the examples, it seems that when a namespace is declared, you can either do "namespace { ..." or "namespace foo { ..." - i'm having trouble finding clarity on the significance of "foo" though. it would appear it's not required, but then what is its purpose? what does it mean? does it imply something? it's also unclear to me how to create the "default private namespace". i see that there are three types of namespaces, but the list of namespace settings doesn't seem to include a parameter to set the namespace type. what am i missing? it's also not quite clear how to denote a namespace as "default". from coming across this thread [ http://mailman2.u.washington.edu/pipermail/imap-protocol/2010-May/001077.ht… ] it would indicate in theory that a namespace is denoted as default by using an empty prefix - but i wasn't able to find corroboration of that in dovecot's documentation. thanks -ben

2 3

[Dovecot] dovecot-2.2: broken domain quota calculation
by Evgeny Basov 25 Feb '13

25 Feb '13

Hello. Used next version of dovecot: # dovecot --version 20130205 (03a0af22100d+) Quota configuration: plugin { quota = dict:Domain quota:%d:redis:host=127.0.0.1:prefix=domain/ quota_rule = *:storage=0 quota_rule2 = Trash:storage=+5%% quota2 = dict:User quota::redis:host=127.0.0.1:prefix=user/ quota2_rule = *:storage=0 quota2_rule2 = Trash:storage=+5%% } Problem: incorrect quota calculations. Dovecot gets size and messages count of all domain from last user in list. In 2.1 works fine. Steps to reproduce problem: 1. install dovecot 2. create over 50 users with e-mails 3. execute: doveadm quota recalc -A 4. execute: doveadm quota get -A 5. get count of all users message: sum in some application result of execution doveadm quota get -A | grep MESSAGE | grep 'User quota' | awk '{print $5}' (139443 messages) 6. get size of all users message: sum in some application result of execution doveadm quota get -A | grep STORAGE | grep 'User quota' | awk '{print $5}' (2304399 Kb) 7. look at some user user(a)example.org Domain quota STORAGE 82408 51200000 0 user(a)example.org Domain quota MESSAGE 1078 - 0 user(a)example.org User quota STORAGE 0 209715200 0 user(a)example.org User quota MESSAGE 0 - 0 and see 1078 messages and 82408 Kb in domain quota 8. look at last user in list z(a)example.org Domain quota STORAGE 82408 51200000 0 z(a)example.org Domain quota MESSAGE 1078 - 0 z(a)example.org User quota STORAGE 82408 209715200 0 z(a)example.org User quota MESSAGE 1078 - 0 and see matching values.

2 2

[Dovecot] Possible sort optimization (?)
by Michael M Slusarz 25 Feb '13

25 Feb '13

Maybe this is just noise... but I can reproduce this fairly reliably. Mailbox with 21,000+ messages This query: a UID SORT RETURN (ALL COUNT) (DATE) UTF-8 SUBJECT "foo" is always about 10 percent slower than this split query (I've done this 4-5 times, and the numbers are similar): a UID SEARCH RETURN (SAVE) CHARSET UTF-8 SUBJECT "foo" b UID SORT RETURN (ALL COUNT) (DATE) UTF-8 UID $ (The particular query I used matched 5 messages out of the 21,000+) My not-very-scientific benchmarking process: 1.) Stop dovecot process 2.) Delete all dovecot index files for that mailbox 3.) Flush linux paging cache (sync && echo 3 > /proc/sys/vm/drop_caches) 4.) Restart dovecot 5.) Access dovecot via command-line (PREAUTH) 6.) SELECT mailbox 7.) Issue command(s) Could be a potential area for performance improvement or could simply be lazy benchmarking. michael

2 2