- dovecot - dovecot.org

[Dovecot] dovecot-lda (2.1.12) segfaults
by Joseph Tam 18 Dec '12

18 Dec '12

We uograded our dovecot from version 2.1.10 -> 2.10.12, but within the first hour of use, dovecot-lda would segfault during delivery. This left a lock file lying around causing the user's mail readers to hang; much hilarity ensues. All further deliveries to the same user will result in a crash. The only thing that fixes this condition is to blow away the user's INBOX cache index. Reindexing with "doveadm index" does not fix anything. I've installed a test version with one user's problematic INBOX cache indices and ran gdb dovecot-lda (gdb) run -t testuser </dev/null and it crashes with the following backtrace Program received signal SIGSEGV, Segmentation fault. 0xff2a0440 in mail_cache_header_fields_read (cache=0x5c120) at mail-cache-fields.c:323 323 if (field_hdr->size < sizeof(*field_hdr) + (gdb) bt #0 0xff2a0440 in mail_cache_header_fields_read (cache=0x5c120) at mail-cache-fields.c:323 #1 0xff29e8bc in mail_cache_compress_locked (cache=0x5c120, trans=0x63ce8, unlock=0xffbfeeef) at mail-cache-compress.c:361 #2 0xff29efd0 in mail_cache_compress (cache=0x5c120, trans=0x63ce8) at mail-cache-compress.c:489 #3 0xff2a3df4 in mail_cache_transaction_compress (ctx=0x5dac0) at mail-cache-transaction.c:180 #4 0xff2a4084 in mail_cache_transaction_open_if_needed (ctx=0x5dac0) at mail-cache-transaction.c:241 #5 0xff2a6e60 in mail_cache_field_want_add (ctx=0x5dac0, seq=1, field_idx=12) at mail-cache-transaction.c:1048 #6 0xff27e8e8 in index_mail_parse_header_register_all_wanted (mail=0x5e6b0) at index-mail-headers.c:175 #7 0xff27ec90 in index_mail_parse_header_init (mail=0x5e6b0, headers=0x0) at index-mail-headers.c:230 #8 0xff27f5c8 in index_mail_cache_parse_init (_mail=0x5e6b0, input=0x63760) at index-mail-headers.c:376 #9 0xff2299cc in mbox_save_get_input_stream (ctx=0x5dde8, input=0x62ed0) at mbox-save.c:411 #10 0xff22a084 in mbox_save_begin (_ctx=0x5dde8, input=0x62ed0) at mbox-save.c:520 #11 0xff24e9c0 in mailbox_save_begin (ctx=0xffbff514, input=0x62ed0) at mail-storage.c:1652 #12 0xff23f138 in mail_storage_try_copy (_ctx=0xffbff514, mail=0x54cd8) at mail-copy.c:68 #13 0xff23f23c in mail_storage_copy (ctx=0x5dde8, mail=0x54cd8) at mail-copy.c:93 #14 0xff24ec28 in mailbox_copy (_ctx=0xffbff670, mail=0x54cd8) at mail-storage.c:1721 #15 0xff24ec98 in mailbox_save_using_mail (ctx=0xffbff670, mail=0x54cd8) at mail-storage.c:1730 #16 0xff388070 in mail_deliver_save (ctx=0xffbff8a8, mailbox=0x13fe8 "INBOX", flags=0, keywords=0x0, storage_r=0xffbff83c) at mail-deliver.c:317 #17 0xff38869c in mail_deliver (ctx=0xffbff8a8, storage_r=0xffbff83c) at mail-deliver.c:403 #18 0x00012d08 in main (argc=3, argv=0xffbff964) at main.c:434 I dumped the value of field_hdr, and it has some bogus value (gdb) p field_hdr $1 = (const struct mail_cache_header_fields *) 0x20 Other observations - blowing away the indices does not fix it permanantly. The the LDA will eventually start crashing again. - the problematic indices do not cause problems with 2.1.10. Reverting to 2.1.10 restores proper mail delivery. Is this enough debug info to work your magic, Timo? I can supply the cache data if you neded it. It's compiled on a Solaris10. Joseph Tam <jtam.home(a)gmail.com>

2 1

[Dovecot] NFS, deleting folder issue
by Catalin Petrescu 18 Dec '12

18 Dec '12

Hi there, I use dovecot on a server with some Nightmare File Systerm (aka NFS) storage. When I delete a folder, its content gets deleted, but the folder itself does not. Here's what I found: #### Step 1: From a shell, telnet to imap, then create a folder cpetrescu@cpetrescu-mobile:~$ telnet server1.localdomain 143 Trying 10.200.28.36... Connected to server1.localdomain. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready. 01 login testuser1 Password 01 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in 02 create INBOX/F1 02 OK Create completed. #### Step2: On the server, everything is OK, the folder is there [server1: testuser1]# ls -lrtA Maildir/INBOX/ total 4 drwxr-xr-x 5 root root 4096 Dec 5 10:28 F1 [server1: testuser1]# ls -lrtA Maildir/INBOX/F1/ total 12 drwxr-xr-x 2 root root 4096 Dec 5 10:28 tmp drwxr-xr-x 2 root root 4096 Dec 5 10:28 new drwxr-xr-x 2 root root 4096 Dec 5 10:28 cur #### Step3: in the telnet session, delete the folder 03 delete INBOX/F1 03 OK Delete completed. #### Step4: On the server, the content of the folder is gone, but the folder is still there [server1: testuser1]# ls -lrtA Maildir/INBOX/ total 4 drwxr-xr-x 2 root root 4096 Dec 5 10:29 F1 [server1: testuser1]# ls -lrtA Maildir/INBOX/F1/ total 0 #### Debugging: I defined my own my_delete_dir and added it to the delete_dir list in mailbox_list_vfuncs In my_delete_dir I just call readdir on that folder, and print the files via syslog. Here's the content of F1 before calling delete_dir: Dec 05 10:29:14 imap(testuser1): my_delete_dir: d_name: "new" inode: 843617 Dec 05 10:29:14 imap(testuser1): my_delete_dir: d_name: "tmp" inode: 843623 Dec 05 10:29:14 imap(testuser1): my_delete_dir: d_name: "cur" inode: 843616 Dec 05 10:29:14 imap(testuser1): my_delete_dir: d_name: "dovecot.index.log" inode: 2363059 Dec 05 10:29:14 imap(testuser1): my_delete_dir: d_name: "." inode: 843612 Dec 05 10:29:14 imap(testuser1): my_delete_dir: d_name: ".." inode: 807717 And here's its content after calling delete_dir Dec 05 10:29:15 imap(testuser1): my_delete_dir: name: INBOX/F1 Dec 05 10:29:15 imap(testuser1): my_delete_dir: path: /data/testuser1/Maildir/INBOX/F1 Dec 05 10:29:15 imap(testuser1): my_delete_dir: ret: -1 Dec 05 10:29:15 imap(testuser1): my_delete_dir: error: "Directory not empty" trying again Dec 05 10:29:15 imap(testuser1): my_delete_dir: d_name: "." inode: 843612 Dec 05 10:29:15 imap(testuser1): my_delete_dir: d_name: ".." inode: 807717 Dec 05 10:29:15 imap(testuser1): my_delete_dir: d_name: ".nfs0000000000240eb3000000e0" inode: 2363059 Dec 05 10:29:15 imap(testuser1): my_delete_dir: ret: -1 So, the dovecot.index.log was deleted, but it's still open, and the folder can't be deleted because of the pesky .nfs* file. Now, the question: Is there a *magic* function that closes "dovecot.index.log" that I can call before delete_dir()? I tried mailbox_log_close() but it didn't help. Thanks a lot. p.s. Please don't suggest that I should not use NFS! I know that, and I don't really like NFS, but I have no other choice. Catalin Petrescu <catalin.petrescu gmail com>

2 2

[Dovecot] SPECIAL-USE and mailbox names in various IMAP clients
by Timo Sirainen 18 Dec '12

18 Dec '12

http://imapwiki.org/SpecialUse Please fill out for all clients with all languages you know of. Knowing the mailbox names can be helpful with Dovecot's mailbox_alias plugin.

2 2

[Dovecot] Manually creating dovecot- files after isync/mbsync mirror
by Dale Gallagher 18 Dec '12

18 Dec '12

Hi everyone I'm using isync 1.0.5 patched to do recursive folder listing and mirroring mail from an external party's IMAP server to a local Maildir, which will be accessed by Dovecot 2.0.9. In order to prevent IMAP clients from re-downloading their mail, when connecting to Dovecot, I'm busy writing a script to generate appropriate dovecot-uidlist and dovecot-uidvalidity files, per Maildir. isync stores the uid in the filename, so that's extractable, per message, and the uidvalidity and current max uid are stored in a file .uidvalidity, in each Maildir. I've read the docs here: http://wiki2.dovecot.org/MailboxFormat/Maildir Regarding the dovecot-uidlist header (first line) - I've tried to search for and figure out what exactly is meant by the "mailbox global uid" and once I do, how to manually generate the 128 bit hex variation of it? Could someone elaborate on this, please. How would omitting the global uid field, beginning with G, affect remote clients' syncing of mail? Is this field mandatory? I also see that Dovecot's uidvalidity is of the form 4df8a4e2, whereas isync uses an integer. I'd appreciate it if someone could elaborate on what Dovecot uses and whether I can convert one to the other? I'll post the full solution, once complete, and working. Thanks. Dale

2 1

Re: [Dovecot] Automatically Cleaning Kerberos Credential Cache Files
by Joseph Tam 18 Dec '12

18 Dec '12

Ben Morrow writes: > In any case, it's likely that the delivery process looks in the user's > home directory even if delivery is to a separate mail spool, unless > you've taken steps to prevent this. I can confirm this. I think the reason is that dovecot changes to the user's home directory so that it has a place to dump core. This causes problem for my setup since I user per-user NFS mounts, so a delivery to an internal mailing list caused hundreds of NFS automounts. I patched this behaviour out, but there is probably a more elegant solution (unset MAIL_STORAGE_SERVICE_FLAG_ENABLE_CORE_DUMPS? How?). Joseph Tam <jtam.home(a)gmail.com> --- ./src/lib-storage/mail-storage-service.c~ Sat May 19 08:42:12 2012 +++ ./src/lib-storage/mail-storage-service.c Mon Jun 25 22:00:57 2012 @@ -615,21 +615,21 @@ - if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP) != 0 && - (user->flags & MAIL_STORAGE_SERVICE_FLAG_ENABLE_CORE_DUMPS) == 0) { - /* we don't want to write core files to any users' home - directories since they could contain information about other - users' mails as well. so do no chdiring to home. */ - } else if (*home != '\0' && - (user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_CHDIR) == 0) { - /* If possible chdir to home directory, so that core file - could be written in case we crash. */ - if (chdir(home) < 0) { - if (errno == EACCES) { - i_error("%s", eacces_error_get("chdir", - t_strconcat(home, "/", NULL))); - } if (errno != ENOENT) - i_error("chdir(%s) failed: %m", home); - else if (mail_set->mail_debug) - i_debug("Home dir not found: %s", home); - } - } +// if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP) != 0 && +// (user->flags & MAIL_STORAGE_SERVICE_FLAG_ENABLE_CORE_DUMPS) == 0) { +// /* we don't want to write core files to any users' home +// directories since they could contain information about other +// users' mails as well. so do no chdiring to home. */ +// } else if (*home != '\0' && +// (user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_CHDIR) == 0) { +// /* If possible chdir to home directory, so that core file +// could be written in case we crash. */ +// if (chdir(home) < 0) { +// if (errno == EACCES) { +// i_error("%s", eacces_error_get("chdir", +// t_strconcat(home, "/", NULL))); +// } if (errno != ENOENT) +// i_error("chdir(%s) failed: %m", home); +// else if (mail_set->mail_debug) +// i_debug("Home dir not found: %s", home); +// } +// }

2 1

[Dovecot] Removing configured SSL certificate file from disk while Dovecot is running spams syslog
by Nils 18 Dec '12

18 Dec '12

Hi! [root@mikan ~]# dovecot --version 2.1.12 When I remove the SSL certificate from disk while Dovecot is running, it spams syslog with the following log message over and over: Dec 6 17:41:41 mikan dovecot: config: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert: Can't open file /etc/ssl/private/dovecot.pem: No such file or directory Dec 6 17:41:41 mikan dovecot: config: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert: Can't open file /etc/ssl/private/dovecot.pem: No such file or directory Dec 6 17:41:41 mikan dovecot: config: Fatal: Error in configuration file /etc/dovecot/conf.d/10-ssl.conf line 12: ssl_cert: Can't open file /etc/ssl/private/dovecot.pem: No such file or directory [...] Reporting that the certificate went away is fine and useful, but spamming the syslog over and over with hundreds of entries per second(!) is annoying. :( I know what I'm doing, so reporting it once per connection attempt or something is enough. Even if Dovecot just decides to terminate, that would be fine. But a denial of service attack on syslog is not nice. Configuration output follows below. All the best, Nils [root@mikan ~]# dovecot -n # 2.1.12: /etc/dovecot/dovecot.conf # OS: Linux 3.5.4-1-ARCH x86_64 auth_failure_delay = 5 secs auth_mechanisms = plain login auth_verbose = yes listen = *,:: login_greeting = mail_location = maildir:~/.maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } protocols = pop3 service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service pop3 { process_limit = 10 } ssl_cert = </etc/ssl/private/dovecot.pem ssl_key = </etc/ssl/private/dovecot.key userdb { driver = passwd } protocol imap { imap_idle_notify_interval = 2 mins mail_max_userip_connections = 5 } protocol pop3 { mail_max_userip_connections = 5 pop3_logout_format = pop3_save_uidl = no pop3_uidl_format = %08Xu%08Xv }

2 1

[Dovecot] During FTS reindexing, search commands unceremoniously exit
by Dave Abrahams 18 Dec '12

18 Dec '12

Is this intentional? Closing the connection without warning doesn't seem like the best response... --8<---------------cut here---------------start------------->8--- % printf "1 select \"INBOX\"\n2 search body quack\n3 logout\n" | /opt/local/libexec/dovecot/imap * PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE SEARCH=FUZZY] Logged in as dave * FLAGS (\Answered \Flagged \Deleted \Seen \Draft gnus-forward gnus-expire) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft gnus-forward gnus-expire \*)] Flags permitted. * 31 EXISTS * 0 RECENT * OK [UIDVALIDITY 1349969399] UIDs valid * OK [UIDNEXT 1376] Predicted next UID * OK [HIGHESTMODSEQ 2908] Highest 1 OK [READ-WRITE] Select completed. imap(dave): Info: Connection closed in=45 out=2027404 % --8<---------------cut here---------------end--------------->8--- -- Dave Abrahams BoostPro Computing Software Development Training http://www.boostpro.com Clang/LLVM/EDG Compilers C++ Boost

2 1

[Dovecot] Documentation of Redis and Memcache Backends
by Urban Loesch 18 Dec '12

18 Dec '12

Hi, in the release notes of 2.1.9 I read that dovecot supports memcache and redis backends for userdb/passdb authentication. This is very interesting for me and should reduce queries and database load of our mysql servers. My idea is to use memcache or redis in our IMAP/POP3 proxies in front of our backend servers. So I would like to try out if it's possible to store proxy information of our backends. For example the backend ip adress. But in the wiki I found only a few configuration settings for redis backend. http://master.wiki2.dovecot.org/AuthDatabase/Dict Also the mentioned example config file "dovecot-dict-auth.conf.ext" with full list of configuration options does not exits in the source of 2.1.11. Have you any idea where I can find the full info or any howtos? Many thanks and regards Urban

2 1

[Dovecot] Error: unlink_directory(.../lucene-indexes) with doveadm index
by Fred Kilbourn 18 Dec '12

18 Dec '12

Hi, Per discussion on this previous thread: http://dovecot.org/pipermail/dovecot/2012-October/068849.html I am updating search indexes on all my mailboxes via a nightly cronjob that runs: doveadm index -A '*' The problem we're running into is that random mailboxes will throw the following error during the update (sanitized to conceal actual domain name): doveadm(user(a)domain.com): Error: unlink_directory(/var/spool/mail/domain.com/user/sdbox/lucene-indexes) failed: Permission denied We have around 250 domains on the server, with a handful of mailboxes on average in each domain. We get this for a varying number of total mailboxes (1 to 6 every night). It is common for the same mailbox to show up a few days in a row and then stop throwing the error. The script is running as root, so there shouldn't be any file level permissions. Any thoughts on the root cause of these errors? What can I provide to help further diagnose? Thanks, Fred Kilbourn

2 2

[Dovecot] indexer-worker Panic
by Dave Abrahams 18 Dec '12

18 Dec '12

I just found this in my logs. Any idea? --8<---------------cut here---------------start------------->8--- 12/4/12 12:33:43.831 PM dovecot[2660]: indexer-worker(dave): Panic: file mailbox-list-fs.c: line 156 (fs_list_get_path): assertion failed: (mailbox_list_is_valid_pattern(_list, name)) 12/4/12 12:33:43.831 PM dovecot[2660]: indexer-worker(dave): Error: Raw backtrace: 2 libdovecot.0.dylib 0x000000010840e6c4 default_fatal_finish + 36 -> 3 libdovecot.0.dylib 0x000000010840f3e1 i_internal_fatal_handler + 49 -> 4 libdovecot.0.dylib 0x000000010840e98e i_panic + 158 -> 5 libdovecot-storage.0.dylib 0x0000000108340693 fs_list_get_path + 467 -> 6 indexer-worker 0x00000001082e7341 master_connection_input + 513 -> 7 libdovecot.0.dylib 0x000000010841b5 12/4/12 12:33:43.831 PM dovecot[2660]: indexer-worker(dave): Error: ce io_loop_call_io + 46 -> 8 libdovecot.0.dylib 0x000000010841c737 io_loop_handler_run + 215 -> 9 libdovecot.0.dylib 0x000000010841b7e8 io_loop_run + 88 -> 10 libdovecot.0.dylib 0x00000001084079c8 master_service_run + 24 -> 11 indexer-worker 0x00000001082e7001 main + 321 -> 12 libdyld.dylib 0x00007fff93d117e1 start + 0 -> 13 ??? 0x0000000000000001 0x0 + 1 12/4/12 12:33:44.035 PM dovecot[2660]: indexer: Error: Indexer worker disconnected, discarding 5 requests for dave 12/4/12 12:33:44.036 PM dovecot[2660]: indexer-worker(dave): Fatal: master: service(indexer-worker): child 2665 killed with signal 6 (core dumps disabled) 12/4/12 12:34:17.094 PM dovecot[2660]: indexer-worker(dave): Panic: file mailbox-list-fs.c: line 156 (fs_list_get_path): assertion failed: (mailbox_list_is_valid_pattern(_list, name)) 12/4/12 12:34:17.094 PM dovecot[2660]: indexer-worker(dave): Error: Raw backtrace: 2 libdovecot.0.dylib 0x000000010d1dd6c4 default_fatal_finish + 36 -> 3 libdovecot.0.dylib 0x000000010d1de3e1 i_internal_fatal_handler + 49 -> 4 libdovecot.0.dylib 0x000000010d1dd98e i_panic + 158 -> 5 libdovecot-storage.0.dylib 0x000000010d10d693 fs_list_get_path + 467 -> 6 indexer-worker 0x000000010d0b7341 master_connection_input + 513 -> 7 libdovecot.0.dylib 0x000000010d1ea5 12/4/12 12:34:17.094 PM dovecot[2660]: indexer-worker(dave): Error: ce io_loop_call_io + 46 -> 8 libdovecot.0.dylib 0x000000010d1eb737 io_loop_handler_run + 215 -> 9 libdovecot.0.dylib 0x000000010d1ea7e8 io_loop_run + 88 -> 10 libdovecot.0.dylib 0x000000010d1d69c8 master_service_run + 24 -> 11 indexer-worker 0x000000010d0b7001 main + 321 -> 12 libdyld.dylib 0x00007fff93d117e1 start + 0 -> 13 ??? 0x0000000000000001 0x0 + 1 12/4/12 12:34:17.286 PM dovecot[2660]: indexer: Error: Indexer worker disconnected, discarding 5 requests for dave 12/4/12 12:34:17.286 PM dovecot[2660]: indexer-worker(dave): Fatal: master: service(indexer-worker): child 2677 killed with signal 6 (core dumps disabled) 12/4/12 12:35:48.633 PM dovecot[2660]: indexer-worker(dave): Panic: file mailbox-list-fs.c: line 156 (fs_list_get_path): assertion failed: (mailbox_list_is_valid_pattern(_list, name)) 12/4/12 12:35:48.633 PM dovecot[2660]: indexer-worker(dave): Error: Raw backtrace: 2 libdovecot.0.dylib 0x0000000107b606c4 default_fatal_finish + 36 -> 3 libdovecot.0.dylib 0x0000000107b613e1 i_internal_fatal_handler + 49 -> 4 libdovecot.0.dylib 0x0000000107b6098e i_panic + 158 -> 5 libdovecot-storage.0.dylib 0x0000000107a95693 fs_list_get_path + 467 -> 6 indexer-worker 0x0000000107a3d341 master_connection_input + 513 -> 7 libdovecot.0.dylib 0x0000000107b6d5 12/4/12 12:35:48.634 PM dovecot[2660]: indexer-worker(dave): Error: ce io_loop_call_io + 46 -> 8 libdovecot.0.dylib 0x0000000107b6e737 io_loop_handler_run + 215 -> 9 libdovecot.0.dylib 0x0000000107b6d7e8 io_loop_run + 88 -> 10 libdovecot.0.dylib 0x0000000107b599c8 master_service_run + 24 -> 11 indexer-worker 0x0000000107a3d001 main + 321 -> 12 libdyld.dylib 0x00007fff93d117e1 start + 0 -> 13 ??? 0x0000000000000001 0x0 + 1 12/4/12 12:35:48.838 PM dovecot[2660]: indexer: Error: Indexer worker disconnected, discarding 5 requests for dave 12/4/12 12:35:48.838 PM dovecot[2660]: indexer-worker(dave): Fatal: master: service(indexer-worker): child 2694 killed with signal 6 (core dumps disabled) --8<---------------cut here---------------end--------------->8--- -- Dave Abrahams BoostPro Computing Software Development Training http://www.boostpro.com Clang/LLVM/EDG Compilers C++ Boost

2 1