May 2007 - dovecot - dovecot.org

[Dovecot] assertion failed
by Adrian Stoica 15 May '07

15 May '07

What is this ? dovecot: Apr 24 13:03:29 Error: IMAP(tina.burdujan): file maildir-sync.c: line 1075 (maildir_sync_index): assertion failed: (uid >prev_uid) dovecot: Apr 24 13:03:29 Error: IMAP(tina.burdujan): Raw backtrace: /usr/libexec/dovecot/imap [0x80b8741] -> /usr/libexec/dovecot/imap [0x80b819c] -> /usr/libexec/dovecot/imap(maildir_sync_index+0x898) [0x80695f8] -> /usr/libexec/dovecot/imap [0x80698b3] -> /usr/libexec/dovecot/imap(maildir_ storage_sync_init+0x49) [0x8069aa9] -> /usr/libexec/dovecot/imap(imap_sync_init+0x40) [0x8062620] -> /usr/libexec/dovecot/imap(cmd_sync+0x71) [0x8 062b81] -> /usr/libexec/dovecot/imap(cmd_noop+0x26) [0x8059c76] -> /usr/libexec/dovecot/imap [0x805be7f] -> /usr/libexec/dovecot/imap [0x805bf20] -> /usr/libexec/dovecot/imap(_client_input+0x6c) [0x805c0fc] -> /usr/libexec/dovecot/imap(io_loop_handler_run+0xff) [0x80bef6f] -> /usr/libexec/do vecot/imap(io_loop_run+0x36) [0x80be366] -> /usr/libexec/dovecot/imap(main+0x46c) [0x806448c] -> /lib/libc.so.6(__libc_start_main+0xb9) [0xb7eb4e1 9] -> /usr/libexec/dovecot/imap [0x80568b1] dovecot: Apr 24 13:03:29 Error: child 25748 (imap) killed with signal 6 My conf is: exim + dovecot + mysql exim deliver mail to maildir , because i don't know how to use LDA from dovecot. the maildirs are very large, for one account could be 30000 mails. dovecot is version 1.0.0 dovecot -n shows: # /etc/dovecot.conf base_dir: /var/run/dovecot/login log_path: /var/log/mail.log login_dir: /var/run/dovecot login_executable: /usr/libexec/dovecot/imap-login login_process_per_connection: no login_process_size: 80 login_processes_count: 10 login_max_processes_count: 50 login_max_connections: 80 first_valid_uid: 1001 last_valid_uid: 1001 mail_extra_groups: exim,vmail mail_location: maildir:%Lh/Maildir mail_cache_min_mail_count: 1 dotlock_use_excl: yes fsync_disable: yes maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_drop_priv_before_exec: yes mail_executable: /bin/imap.sh mail_plugins: quota imap_quota auth default: mechanisms: PLAIN CRAM-MD5 default_realm: testing.net cache_size: 256 worker_max_count: 100 passdb: driver: sql args: /etc/dovecot-crammd5.conf userdb: driver: prefetch plugin: quota: maildir

3 12

[Dovecot] Problems with Shared Mailbox
by Gerhard Schmidt 15 May '07

15 May '07

Hi, i've installed dovecot on a new mailserverand try to set up some shared mailboxes. I have everiting up an runnig so far. I have 3 mboxes to be shared. I have use the following setup namespace private { separator = prefix = location = inbox = yes hidden = yes } namespace public { separator = / prefix = zentrale/ location = mbox:/var/mail/shared:CONTROL=/var/mail/shared/.control:INDEX=/var/mail/shared/.index } the problem is with the dovecot.index files [root@mailserv /var/mail/shared/.index]# ls -alR total 6 drwxrwx--- 3 dovecot vhsal 512 May 7 13:49 . drwxrwx--- 4 mailnull vhsal 512 May 7 13:52 .. drwxrwx--- 5 dovecot vhsal 512 May 7 13:49 .imap -rw-rw---- 1 dovecot vhsal 0 May 7 13:49 dovecot-shared ./.imap: total 10 drwxrwx--- 5 dovecot vhsal 512 May 7 13:49 . drwxrwx--- 3 dovecot vhsal 512 May 7 13:49 .. drwxrwx--- 2 dovecot vhsal 512 May 7 13:50 Eingang drwxrwx--- 2 dovecot vhsal 512 May 7 13:49 Erledigt drwxrwx--- 2 dovecot vhsal 512 May 7 13:50 Spam -rw-rw---- 1 dovecot vhsal 0 May 7 13:49 dovecot-shared ./.imap/Eingang: total 38 drwxrwx--- 2 dovecot vhsal 512 May 7 13:50 . drwxrwx--- 5 dovecot vhsal 512 May 7 13:49 .. -rw-rw---- 1 dovecot vhsal 0 May 7 13:50 dovecot-shared -rw-rw---- 1 dovecot vhsal 896 May 7 13:51 dovecot.index -rw-rw---- 1 dovecot vhsal 23552 May 7 13:51 dovecot.index.cache -rw-rw---- 1 dovecot vhsal 6824 May 7 13:51 dovecot.index.log ./.imap/Erledigt: total 30 drwxrwx--- 2 dovecot vhsal 512 May 7 13:49 . drwxrwx--- 5 dovecot vhsal 512 May 7 13:49 .. -rw-rw---- 1 dovecot vhsal 0 May 7 13:49 dovecot-shared -rw-rw---- 1 dovecot vhsal 456 May 7 13:52 dovecot.index -rw-rw---- 1 dovecot vhsal 20480 May 7 13:45 dovecot.index.cache -rw-rw---- 1 dovecot vhsal 3364 May 7 13:52 dovecot.index.log ./.imap/Spam: total 26 drwxrwx--- 2 dovecot vhsal 512 May 7 13:50 . drwxrwx--- 5 dovecot vhsal 512 May 7 13:49 .. -rw-rw---- 1 dovecot vhsal 0 May 7 13:50 dovecot-shared -rw-rw---- 1 dovecot vhsal 240 May 7 13:51 dovecot.index -rw-rw---- 1 dovecot vhsal 17408 May 7 13:45 dovecot.index.cache -rw-rw---- 1 dovecot vhsal 648 May 7 13:51 dovecot.index.log When the directory is setup like this everthing works as planed. But sometimes (havn't quite figured when exactly the dovecot.index file of one of the Mailboxes is regenated with the false permissions set ./.imap/Eingang: total 38 drwxrwx--- 2 dovecot vhsal 512 May 7 13:53 . drwxrwx--- 5 dovecot vhsal 512 May 7 13:49 .. -rw-rw---- 1 dovecot vhsal 0 May 7 13:50 dovecot-shared -rw------- 1 estartu vhsal 376 May 7 13:53 dovecot.index -rw-rw---- 1 dovecot vhsal 23552 May 7 13:53 dovecot.index.cache -rw-rw---- 1 dovecot vhsal 7116 May 7 13:53 dovecot.index.log How can I gate dovecot to set the permissions of this files to rw-rw---- I am using Dovecot v1.0.rc29 Regards Gerhard Schmidt -- ---------------------------------------------------------------------------- Gerhard Schmidt | Nick : estartu IRC : Estartu | Fischbachweg 3 | | PGP Public Key 86856 Hiltenfingen | EMail: estartu(a)augusta.de | on request Germany | |

2 3

[Dovecot] Thunderbird or Dovecot bug? (multiple keywords)
by Troy Engel 15 May '07

15 May '07

Here's a strange one. I have a user who's using Thunderbird 2.0.0 and assigning multiple tags (keywords) to emails, which are getting "lost" (I'll explain) when he leaves the folder and comes back. I can reproduce this on *his* account myself, but cannot reproduce it on *my* account (separate new TBird 2.0.0 profiles on Windows XP). Use the two TBird default tags 'Important' and 'Work' for all these tests, which it refers to as $Label1 and $Label2 in IMAP-keywords land. If I go into my account, label an email with both tags, it sticks -- I can click on Drafts then back to Inbox, and both tags/keywords still show in the client. If I go into his account, add the two tags, then click on Drafts and back on Inbox, only the *second* tag (Work, $Label2) shows. If I remove that tag, then the first tag (Important, $Label1) immediately pops up! On the backend the actual Maildir files match the dovecot-keywords and have the correct letter appended in both cases - so, in his account when the email only shows one tag, the physical mail file has the letters for two tags. So Thunderbird should be showing both tags, like it does in my account where everything is working correctly. This leads me to believe that the problem has something to do with the difference of dovecot-keywords content, and the interaction TBird has with Dovecot. My account, working correctly, looks like: dovecot-keywords: 0 $Forwarded 1 $MDNSent 2 $Label2 3 $Label1 file: Maildir/cur/1178712608.H529732P11686.yak.fluid.com:2,RSdc ...which matches, d=$Label1 (Important), c=$Label2 (Work) and TBird is displaying the email as having both tags - all is well and good. Now, his account looks like: dovecot-keywords: 0 Junk 1 NonJunk 2 $Label4 3 $Label1 4 $Label2 5 $Label3 6 $Label5 file: Maildir/cur/1178728930.P28392Q0M960111.yak.fluid.com:2,RSbde ...again, everything matches on the server; yet, it Thunderbird is only displaying $Label2 (d,3). I also noticed something interesting, even if I have the TB 'Junk Controls' completely disabled on his account, and remove all tags using the menu option, that file still retains the 'b' (NonJunk) tag in the filename: Maildir/cur/1178728930.P28392Q0M960111.yak.fluid.com:2,RSb All this long-winded explanation leads me to believe that somehow the difference in dovecot-keywords is causing this problem, but what I can't tell is who's problem it is -- Dovecot, or Thunderbird? Thoughts welcome, -te PS: all the normal things tried like deleting the dovecot index files, Thunderbird cache files and profiles, etc. -- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com

2 3

[Dovecot] Why is direct invocation of dovecot preferred over inet inclusion?
by Stewart Dean 15 May '07

15 May '07

I'm used to putting things in inetd and, 1) should dovecot crash, I'd think it would get restarted if it was invoked from inet's imap-login 2) it would get started well after the boot ...wouldn't that be preferable. Enlighten me, please -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

2 1

[Dovecot] Refreshing Dovecot
by Stewart Dean 15 May '07

15 May '07

If I've made configuration changes, is there a way to refresh...without killing and restarting....Dovecot? -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

5 5

[Dovecot] maildir_file_do(/var/admhome/user/Maildir) racing
by Ralf Hildebrandt 15 May '07

15 May '07

dovecot version 1.0.0, compile from the sources. --------- snip -------- # /usr/local/etc/dovecot.conf listen: *:10143 ssl_listen: *:10943 ssl_cert_file: /etc/ssl/certs/postamt.pem ssl_key_file: /etc/ssl/private/postamt.key disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_location: maildir:~/Maildir mail_plugins: quota imap_quota namespace: type: private separator: . prefix: INBOX. inbox: yes auth default: passdb: driver: pam userdb: driver: passwd plugin: quota: maildir quota: maildir:ignore=Trash --------- snip -------- We're using these odd ports, because there's still courier-imap on the original ports. One, just one (!) of our users gets: May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: IMAP(username): maildir_file_do(/var/admhome/username/Maildir) racing May 14 16:47:17 postamt dovecot: IMAP(username): Disconnected: Internal error occurred. Refer to server log for more information. [2007-05-14 16:47:17] Now it's a bit hard for me to see what's going on. /var/admhome/username/Maildir is on /, thus: $ mount /dev/cciss/c0d0p6 on / type ext3 (rw,errors=panic) $ dmesg does not complain about any corruption $ uname -a Linux postamt.charite.de 2.6.21.1 #1 SMP Tue May 1 15:50:11 CEST 2007 i686 GNU/Linux OS: Debian testing -- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt(a)charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk(a)charite.de

3 6

[Dovecot] Question: contention with blackberry
by Stewart Dean 15 May '07

15 May '07

My production imap is currently UWIMAP with mbox.. One of the problems commonly observed is mailbox lock loss, which happens when: a) a VIP has multiple secretaries accessing a single mailbox (actually, they are professional enough to have figured the realities of conflicting access and rarely have a problem), b) somebody leaving their machine on at home and coming in and firing up their work computer c) important people with Blackberries....and the BB service polls the mailbox every so often and breaks the lock. In any case, the maillog ends up with a message like this: > May 5 19:45:04 mercury mail:info imapd[2035774]: Logout user=xxxxx host=bda056.bis.na.blackberry.com [216.9.249.56] > May 5 19:46:52 mercury mail:info imapd[2502902]: Killed (lost mailbox lock) user=xxxxx host=cpe-24-161-103-11.hvc.res.rr.com [24.161.103.11] How will this work with Dovecot? I regard what is happening now as OK, working as designed. Although we may eventually move to a different mailbox format, I am only switching from UW to DC in the first step. -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

3 2

[Dovecot] global vs user script
by M1 15 May '07

15 May '07

What is the order of execution if both user and global sieve script exists? Thanks.

4 10

[Dovecot] Mails, work and so on
by Timo Sirainen 14 May '07

14 May '07

You can probably guess from my flood of mails that I'm no longer busy. I still have a couple of exams but since I haven't spent any time at all studying for them or attending the lectures, I can just as easily leave them for next year. I've read and answered most of the "easy" mails in this list and in my INBOX. I'll try to answer the rest of them this weekend, so no need to forward them to me again, at least not yet. :) Now that I'm finally non-busy, it also appears that I haven't worked for a long time and I'm almost broke. :) So now would be a great time for your corporation to buy/sponsor some Dovecot development. Otherwise I'm probably soon busy with non-Dovecot related work. My own high priority TODO contains: - Get the rewritten Squat full text search index working. It probably needs a name change, since it's not all that close to Cyrus Squat anymore. Suggestions? Could of course be just "fts_dovecot". - Shared mailboxes / ACLs - dbox cleanups and possibly partial redesign. I'll write a separate mail about this later. - Index file code cleanups, simplifications and optimizations. Make it work with NFS with attribute cache enabled.

5 7

[Dovecot] How to get rid of locks
by Timo Sirainen 14 May '07

14 May '07

Although Dovecot is already read-lockless and it uses only short- lived write locks, it's be really nice to just get rid of the locking completely. :) I just figured out that O_APPEND is pretty great. If the operating system updates seek position after writing to a file opened with O_APPEND, writes to Dovecot's transaction log file can be made lockless. I see that this works with Linux and Solaris, but not with OS X. Could you BSD people try if it works there? http://dovecot.org/ tmp/append.c and see if it says "offset = 0" (bad) or non-zero (yay). The O_APPEND at least doesn't work with NFS, so it'll have to be optional anyway. Currently Dovecot always updates dovecot.index file after it has done any changes. This isn't really necessary, because the changes are already in transaction log, so the dovecot.index file can be read to memory and the new changes applied on top of it from transaction log (this is pretty much how mmap_disable=yes works). So I'm going to change this to work so that the dovecot.index is updated only if a) there are enough changes in transaction log (eg. 8kB or so) and b) it can be write-locked without waiting. Maildir then. It has this annoying problem that readdir() can skip files if another process is rename()ing them, causing Dovecot to think that the message was expunged. The only way I could avoid this by locking the maildir while synchronizing it. Today I noticed that this doesn't happen with OS X. I'm not sure if I was just lucky or if there really is something special implemented in it, because it doesn't work anywhere else. I'm not sure if this is tied to HFS+, or if it will work with zfs also (Solaris+zfs didn't work). So perhaps the locking could be disabled while running with OS X. More importantly I figured out that it can also be avoided with Linux +inotify. As long as the inotify event buffer doesn't overflow, the full list of files can be read by combining the readdir() output and files listed by inotify events. If the inotify buffer overflows (highly unlikely), the operation can just be retried and it most likely works the next time. So with these changes in place, changing a message flag or expunging a message would usually result in: - lockless write() call to dovecot.index.log - lockless read()ing (or looking into mmaped) dovecot.index.log to see if there's some new data besides what we just wrote that needs to be synchronized to maildir - rename() or unlink() calls to maildir. If a call return ENOENT, the maildir needs to be readdir()ed with inotify enabled to find the new filename. Not a single lock in the operation, assuming that dovecot.index file wasn't updated. Assigning UIDs to newly delivered mails would require locking though. dovecot-uidlist needs to be locked, and the UIDs need to be written to dovecot.index.log file in the correct order, which can also be done with dovecot-uidlist locking. Actually a single write() to dovecot.index.log isn't enough. I think there needs to be some kind of a flag written to the beginning of the transaction which marks the transaction as truly finished. If the flag isn't there, any reader knows to stop and wait until the flag is set. So this means that the writer needs to: 1. Do a single O_APPENDed write() call writing the whole transaction 2. Get the current offset with lseek(fd, 0, SEEK_CUR) (this is what the append.c tester checks) 3. pwrite() the finished-flag to beginning of the transaction Except at least with Linux pwrite() doesn't work if O_APPEND is enabled. There are two ways to work around this: a) fcntl(disable O_APPEND) + pwrite() + fcntl(enable O_APPEND) b) Keep two file descriptors open for the transaction log. First with O_APPEND flag and second without. pwrite() to the second one. a) is probably better because it doesn't waste file descriptors.

8 10