- dovecot - dovecot.org

[Dovecot] panic fts_solr for bad attachment
by Robert Strötgen 27 Nov '12

27 Nov '12

Hi! I use dovecot 2.1.7 on Ubuntu 12.10 with fts_solr und decode2text.sh for indexing attachments. This works great in general. Just for one user there is a problem with an unknown bad attachment. I run "doveadm index -A '*'". After a while I receive: doveadm(xyz): Error: fts_solr: Invalid XML input at line 1: mismatched tag doveadm(xyz): Panic: file solr-connection.c: line 545 (solr_connection_post_more): assertion failed: (maxfd >= 0) doveadm(xyz): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3c14a) [0x7f7ce2c1714a] -> /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f7ce2c1720a] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7ce2bee81a] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(solr_connection_post_more+0x249) [0x7f7ce11913a9] -> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so(+0x4597) [0x7f7ce118e597] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x6f57) [0x7f7ce159df57] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0xf5) [0x7f7ce159e085] -> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xba70) [0x7f7ce15a2a70] -> doveadm(+0x15309) [0x7f7ce35cc309] -> doveadm(+0x11f36) [0x7f7ce35c8f36] -> doveadm(+0x12bf1) [0x7f7ce35c9bf1] -> doveadm(doveadm_mail_try_run+0x161) [0x7f7ce35c9ed1] -> doveadm(main+0x3d1) [0x7f7ce35c8ae1] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f7ce283d76d] -> doveadm(+0x11d15) [0x7f7ce35c8d15] In catalina out I find: Nov 18, 2012 2:59:09 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Invalid UTF-8 start byte 0xfc (at char #25214836, byte #26687495) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:81) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:58) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:316) at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:722) Caused by: com.ctc.wstx.exc.WstxIOException: Invalid UTF-8 start byte 0xfc (at char #25214836, byte #26687495) at com.ctc.wstx.sr.StreamScanner.constructFromIOE(StreamScanner.java:625) at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:994) at com.ctc.wstx.sr.StreamScanner.getNext(StreamScanner.java:754) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2691) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1065) at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:309) at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:156) at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79) ... 19 more Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0xfc (at char #25214836, byte #26687495) at com.ctc.wstx.io.UTF8Reader.reportInvalidInitial(UTF8Reader.java:303) at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:189) at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:87) at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:988) ... 25 more doveadm index stops after this error. How can I make doveadm just to skip the error and to continue indexing? Thanks Robert -- Robert Strötgen Abteilungsleiter Informationsmanagement und Publikationen Georg-Eckert-Institut für internationale Schulbuchforschung Celler Str. 3 38114 Braunschweig Tel. +49 (0)531 59099-47 & +49 (0)531 123103-205 Fax +49 (0)531 59099-99 http://www.gei.de/

3 6

[Dovecot] IMAP over SSL
by petsy12＠lavabit.com 27 Nov '12

27 Nov '12

Hello. I've never tried Dovecot. Here is my attempt to enable IMAP over SSL on port 993. (BTW, I don't want to use port 143 at all.) # dovecot -n log_timestamp: %Y-%m-%d %H:%M:%S protocols: imaps listen: *:143,[::]:143 ssl_listen: *:993,[::]:993 ssl: required ssl_cert_file: /etc/dovecot/keycert.pem ssl_key_file: /etc/dovecot/keycert.pem ssl_cipher_list: TLSv1+HIGH:!SSLv2:!aNULL:!eNULL:!NULL:@STRENGTH login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login mail_privileged_group: mail mail_location: maildir:~/Maildir mbox_write_locks: fcntl dotlock auth default: passdb: driver: pam userdb: driver: passwd 1. Here is a snippet from dovecot.conf. Is it correct? Should I change something? (Note that I don't want to enable IMAP on port 143.) protocols = imaps protocol imap { listen = *:143,[::]:143 ssl_listen = *:993,[::]:993 } disable_plaintext_auth = yes ssl_listen = *:993,[::]:933 ssl = required ssl_cert_file = /etc/dovecot/keycert.pem ssl_key_file = /etc/dovecot/keycert.pem ssl_cipher_list = TLSv1+HIGH:!SSLv2:!aNULL:!eNULL:!NULL:@STRENGTH 2. I don't understand the syntax connected with auth. What auth options are enabled by default? dovecot.conf: No sections (e.g. namespace {}) or plugin settings are added by default, they're listed only as examples. Does it mean that passdb pam will use defaults (e.g. session=yes, setrcred=yes)? passdb pam { # [session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=<n>] # [cache_key=<key>] [<service name>] # # session=yes makes Dovecot open and immediately close PAM session. Some # PAM plugins need this to work, such as pam_mkhomedir. # # setcred=yes makes Dovecot establish PAM credentials if some PAM plugins # need that. They aren't ever deleted though, so this isn't enabled by # default. # # max_requests specifies how many PAM lookups to do in one process before # recreating the process. The default is 100, because many PAM plugins # leak memory. # # cache_key can be used to enable authentication caching for PAM # (auth_cache_size also needs to be set). It isn't enabled by default # because PAM modules can do all kinds of checks besides checking password, # such as checking IP address. Dovecot can't know about these checks # without some help. cache_key is simply a list of variables (see # /usr/share/doc/dovecot-common/wiki/Variables.txt) which must match # for the cached data to be used. # Here are some examples: # %u - Username must match. Probably sufficient for most uses. # %u%r - Username and remote IP address must match. # %u%s - Username and service (ie. IMAP, POP3) must match. # # The service name can contain variables, for example %Ls expands to # pop3 or imap. # # Some examples: # args = session=yes %Ls # args = cache_key=%u dovecot #args = dovecot } 3. Here is the output of `openssl s_client -tls1 -connect mail.example.com:993`. Is it OK? [snip] New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: zlib compression Expansion: zlib compression [snip] Verify return code: 18 (self signed certificate) --- * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready. Also, where can I read about these options? Any comments are appreciated.

2 1

[Dovecot] Default fallback behaviour
by Nikita Koshikov 27 Nov '12

27 Nov '12

Hello list, Here is the problem: I have few: passdb { #1 } passdb { #2 } And relative userdb sections. If user not found in 1) section it fallbacks to next one - it's expected and right, IMHO. But when the user exists in both section and password verification fails on 1) database it successfully authenticated on next one. I think this behaviour should be configured. The main goal of 1) section for this server is to overwrite users in main (section2) database. Maybe I missed something and this option is already in dovecot code and I can't find it ? Or if not - will it be added in the future ? Dovecot version 2.1.10.

3 6

[Dovecot] lmtp_rcpt_check_quota working somewhere?
by Steffen Kaiser 27 Nov '12

27 Nov '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I'm running Dovecot v2.1 and enabled the lmtp_rcpt_check_quota option in conf.d/20-lmtp.conf, because I just found it in the sample config. Is this option working somewhere? When I have the option enabled, I get _no_ response in the RCPT TO phase, if the user is under quota. If I go on and enter the DATA keyword eventually, I get "554 5.5.1 No valid recipients", so it looks like, the user is really ignored at all. For an user over quota, I get: 552 5.2.2 <user> Quota exceeded (mailbox for user is full) Without lmtp_rcpt_check_quota LMTP works fine. For the user over quota I get the error after DATA phase, the user under quota gets the message delivered. Regards, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBULOTyWoxLS8a3A9mAQITfggAoC0ujUXE53ePUEsTUHXlL2S0/hQCdAgA ppZnUPv9Gjh2oZjjrOmPa+zYVnd+6xK9m58bV4b7FPC+uz1otaS205WMIeZCuy5l kTtCpNpxxNfsUDzMM8zrRf+AZODAwUzwx7yknGdpXtezIGYg7kTKIXxs3KShgI3i ML/kbj4j+YkpdR0wUnZCIVM7MBukAJ3rhSqvfcY0ekZo8+ownqVjCE1PLhFy6oGE OsN8qY272nuvY9dLjflUjOKsvPVnviIu0kgRAZEsqPPp27Jp4GMsSpqqdAhjgbzH D8n9AnxVc6KYhq8qVpsWBCNy3/CwERsadZcRnl/4zcZmq19VsLLZMQ== =Nyqj -----END PGP SIGNATURE-----

2 2

[Dovecot] Feature request: add information to error message: client doesn't have lookup permissions for this user: userdb reply doesn't contain uid (change userdb socket permissions)
by Steffen Kaiser 27 Nov '12

27 Nov '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Please add the information to this error, which socket has the problem and which uid access is and what is expected. For instance, when the quota dict request fails, because of permission problems, you get a very detailed info about the current problem. And, further more, hints to solve it. BTW: I'm trying to configure shared mailboxes for a virtual user system, when one user with ACLs on other mailboxes requests a LIST, I get this error. In my case, I solved the problem by chmod u+x auth-userdb . Kind regards, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBUJPs/2oxLS8a3A9mAQLQxwf/ZkhmER1q/+rlZCIdQ70x6TDwS1nwJI6J FHyr8UFgV0xXBeX8OW+UDl8N4OzXOiReygHnrcpYE4BCEQmeQC9YUU7mCcC6tpqG cg+uYMAPap3Zx8WkAqsmDnnqHe4xX7GUscoiVh1vxYdiRkHwikiKEfuve+SwTpyj wmAK5CMGLBifWPk6bmFP7YR+sJnG/E1vKxEUMEKEKjMfubsCCqc95NNCOiVTwCTG Y15eIhyJTBBjrGuLSeiHb40sPnOE/OgO9r00+OXrZOtuouYZlsMENsDXSFFLoC7v MpABu57w3GRC0p33bjSuX8ZGinEd8fFz9mbUCSvBmLLL7rOZVR6FDw== =Rf46 -----END PGP SIGNATURE-----

2 2

[Dovecot] Rebuilding indexes fails on inconsistent mdbox
by Milan Holzäpfel 27 Nov '12

27 Nov '12

Hello all, I have a problem with an incosistent mdbox: Oct 24 10:43:23 two dovecot: imap-login: Login: user=<...>, method=PLAIN, rip=..., lip=..., mpid=4977, TLS Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Error: mdbox map .../mdbox/storage/dovecot.map.index corrupted: Unexpectedly lost INBOX uid=638 map_uid=809891 Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Error: mdbox map .../mdbox/storage/dovecot.map.index corrupted: Unexpectedly lost INBOX uid=638 map_uid=809891 Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Disconnected: Internal error occurred. Refer to server log for more information. [2012-10-24 10:43:23] bytes=115/53726 Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Warning: mdbox .../mdbox/storage: Inconsistency in map index (467,31960 != 467,36768) Oct 24 10:43:23 two dovecot: imap(listen(a)mjh.name): Warning: mdbox .../mdbox/storage: rebuilding indexes Oct 24 10:45:19 two dovecot: imap(listen(a)mjh.name): Panic: file mdbox-storage-rebuild.c: line 773 (rebuild_update_refcounts): assertion failed: (map_uid < msgs[i]->map_uid) Oct 24 10:45:19 two dovecot: imap(listen(a)mjh.name): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x380ca) [0x7f99cf35b0ca] -> /usr/lib/dovecot/libdovecot.so.0(+0x3810e) [0x7f99cf35b10e] -> /usr/li b/dovecot/libdovecot.so.0(i_fatal+0) [0x7f99cf334a67] -> /usr/lib/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10a5) [0x7f99cf5f42d5] -> /usr/lib/dovecot/libdovecot-storage.so.0(mdbox_s torage_rebuild+0x24) [0x7f99cf5f4414] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x53565) [0x7f99cf5f4565] -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_close+0x1a) [0x7f99cf5c8caa] -> /usr/lib/dovec ot/libdovecot-storage.so.0(mailbox_free+0x13) [0x7f99cf5c8cf3] -> dovecot/imap(client_destroy+0x109) [0x7f99cfaa69e9] -> dovecot/imap(client_input+0xaa) [0x7f99cfaa6dba] -> /usr/lib/dovecot/libdovecot.so.0(i o_loop_call_io+0x48) [0x7f99cf366c98] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xa7) [0x7f99cf367d27] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x28) [0x7f99cf366c28] -> /usr/lib/dovecot /libdovecot.so.0(master_service_run+0x13) [0x7f99cf354e33] -> dovecot/imap(main+0x304) [0x7f99cfa9e554] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f99cef8576d] -> dovecot/imap(+0x95e5) [0 x7f99cfa9e5e5] Oct 24 10:45:19 two dovecot: master: Error: service(imap): child 4977 killed with signal 6 (core dumps disabled) I use: Dovecot 2.0.19-0ubuntu1 Ubuntu 12.04, x86-64, Kernel 3.2.0-32-generic local XFS filesystem for the mdbox The problem appeared out of nowhere. Many messages been continously delivered to this mailbox on this installation since May 2012, and the mdbox was only accessed with deliver and imap/pop3 from dovecot. About four hours after the problem initially appeared, I did a hard reset of the system because it was unresponsive. The error message is exactly the same before and after the hard reset. The problem is triggered by both IMAP access and dovecot deliver access. The whole mdbox is 6.6 GiB large and I guess that it contains about 300k-600k messages. It's an archive of public mailing lists, so I could give access to the files. Can anybody say something about this? May the mdbox be repaired? Regards, Milan Holzäpfel -- Milan Holzäpfel <listen(a)mjh.name>

5 15

[Dovecot] Dovecot IMAP/POP3 auto creating maildir
by Dominic Malolepszy 27 Nov '12

27 Nov '12

Hi I have observed that after deleting a mailbox and removing the user from the userdb, immediately accessing the mail account via POP3/IMAP causes Dovecot to auto create an empty mailbox, because the userdb/passdb details are still cached for 10minutes. Is there any option to tell Dovecot POP3/IMAP not to auto create the mailbox if it does not exist? Alternatively and less desirably is there a mechanism for telling Dovecot to expire a userdb/passdb cached entry? Running Dovecot 2.1.9 Dominic

2 1

[Dovecot] memory allocation issues
by Kelsey Cummings 27 Nov '12

27 Nov '12

One of our dovecot backend servers ran into a problem with it's auth process a few days ago. This doesn't appear to be the error logged when dovecot hits its internal limit so I'm not sure what is going on here. auth: Error: malloc: 58012: Cannot allocate memory auth: Error: Unable to allocate memory for mutexes from the region auth: Error: PANIC: Cannot allocate memory auth: passwd(test,1.1.1.1,<8HTlNHzNIQBAjhKC>): unknown user pop3: Error: Authenticated user not found from userdb, auth lookup +id=2509111297 (client-pid=4781 client-id=1) pop3-login: Internal login failure (pid=4781 id=1) (internal failure, 1 +succesful auths): user=<test>... There was at least 10+GB free RAM on the server and no indication of a system level issue at the same time. The server is running 2.1.9. There were about 3,200 active sessions, with something like 12 new sessions/sec. The other identical servers are/were handling virtually identical load with the same service uptime and haven't had any issues so far. (Crash happened 7 days ago.) -- Kelsey Cummings - kgc(a)corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407

2 4

[Dovecot] fs_flush_file_handle_cache_dir: rmdir(/var/mail) AGAIN
by Ken Mandelberg 27 Nov '12

27 Nov '12

We are running Dovecot 2.0.12. It runs on only one server and /var/mail is local where all the INBOX's are stored. Users other IMAP folders are NFS mounted on the server. We see frequent Error: nfs_flush_file_handle_cache_dir: rmdir(/var/mail) failed: Device busy I know this error message has been around a long time. Should we still be seeing it on 2.0.12 with a local /var/mail. Perhaps the message is misleading and really complaining about folders other than INBOX that are NFS mounted. Our config has dovecot.conf:mail_nfs_index = no dovecot.conf:mail_nfs_storage = yes is that correct given the above? Also, does the flush error messages have any real consequences? My understanding is that the rmdir is meant to fail, just to force a flush.

2 1

[Dovecot] Filesystem quotas
by Andreas Kasenides 27 Nov '12

27 Nov '12

I could not determine if when using filesystem quotas (http://wiki2.dovecot.org/Quota/FS) Dovecot will use the reported quota and limits without the need of configuring them via "quota_rules". Is this possible somehow? Such a feature would determine the quota limits automatically from the filesystem quotas thereby allowing to make use of them in quota warnings. Dovecot already knows the quota settings and limits. Here is evidence from the logs > Nov 21 19:24:22 iolaos dovecot: imap(user): Debug: box=/Mail/ank > mount=/newmail match=yes > Nov 21 19:24:22 iolaos dovecot: imap(user): Debug: quota-fs: > host=xxxx.xxxx.ucy.xx.cy, path=/mail, uid=211, bytes > Nov 21 19:24:22 iolaos dovecot: imap(user): Debug: quota-fs: uid=211, > value=2779115520, limit=8388608000 I am using (testing) version 2.2alpha1. Thankx. -- Andreas Kasenides Senior IT Officer University of Cyprus, Dept. of Computer Science Tel: 22892714, Fax: 22892701

2 1