- dovecot - dovecot.org

[Dovecot] Thunderbird or Dovecot bug? (multiple keywords)
by Troy Engel 15 May '07

15 May '07

Here's a strange one. I have a user who's using Thunderbird 2.0.0 and assigning multiple tags (keywords) to emails, which are getting "lost" (I'll explain) when he leaves the folder and comes back. I can reproduce this on *his* account myself, but cannot reproduce it on *my* account (separate new TBird 2.0.0 profiles on Windows XP). Use the two TBird default tags 'Important' and 'Work' for all these tests, which it refers to as $Label1 and $Label2 in IMAP-keywords land. If I go into my account, label an email with both tags, it sticks -- I can click on Drafts then back to Inbox, and both tags/keywords still show in the client. If I go into his account, add the two tags, then click on Drafts and back on Inbox, only the *second* tag (Work, $Label2) shows. If I remove that tag, then the first tag (Important, $Label1) immediately pops up! On the backend the actual Maildir files match the dovecot-keywords and have the correct letter appended in both cases - so, in his account when the email only shows one tag, the physical mail file has the letters for two tags. So Thunderbird should be showing both tags, like it does in my account where everything is working correctly. This leads me to believe that the problem has something to do with the difference of dovecot-keywords content, and the interaction TBird has with Dovecot. My account, working correctly, looks like: dovecot-keywords: 0 $Forwarded 1 $MDNSent 2 $Label2 3 $Label1 file: Maildir/cur/1178712608.H529732P11686.yak.fluid.com:2,RSdc ...which matches, d=$Label1 (Important), c=$Label2 (Work) and TBird is displaying the email as having both tags - all is well and good. Now, his account looks like: dovecot-keywords: 0 Junk 1 NonJunk 2 $Label4 3 $Label1 4 $Label2 5 $Label3 6 $Label5 file: Maildir/cur/1178728930.P28392Q0M960111.yak.fluid.com:2,RSbde ...again, everything matches on the server; yet, it Thunderbird is only displaying $Label2 (d,3). I also noticed something interesting, even if I have the TB 'Junk Controls' completely disabled on his account, and remove all tags using the menu option, that file still retains the 'b' (NonJunk) tag in the filename: Maildir/cur/1178728930.P28392Q0M960111.yak.fluid.com:2,RSb All this long-winded explanation leads me to believe that somehow the difference in dovecot-keywords is causing this problem, but what I can't tell is who's problem it is -- Dovecot, or Thunderbird? Thoughts welcome, -te PS: all the normal things tried like deleting the dovecot index files, Thunderbird cache files and profiles, etc. -- Troy Engel | Systems Engineer Fluid, Inc | http://www.fluid.com

2 3

[Dovecot] Why is direct invocation of dovecot preferred over inet inclusion?
by Stewart Dean 15 May '07

15 May '07

I'm used to putting things in inetd and, 1) should dovecot crash, I'd think it would get restarted if it was invoked from inet's imap-login 2) it would get started well after the boot ...wouldn't that be preferable. Enlighten me, please -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

2 1

[Dovecot] Refreshing Dovecot
by Stewart Dean 15 May '07

15 May '07

If I've made configuration changes, is there a way to refresh...without killing and restarting....Dovecot? -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

5 5

[Dovecot] maildir_file_do(/var/admhome/user/Maildir) racing
by Ralf Hildebrandt 15 May '07

15 May '07

dovecot version 1.0.0, compile from the sources. --------- snip -------- # /usr/local/etc/dovecot.conf listen: *:10143 ssl_listen: *:10943 ssl_cert_file: /etc/ssl/certs/postamt.pem ssl_key_file: /etc/ssl/private/postamt.key disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_location: maildir:~/Maildir mail_plugins: quota imap_quota namespace: type: private separator: . prefix: INBOX. inbox: yes auth default: passdb: driver: pam userdb: driver: passwd plugin: quota: maildir quota: maildir:ignore=Trash --------- snip -------- We're using these odd ports, because there's still courier-imap on the original ports. One, just one (!) of our users gets: May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: IMAP(username): maildir_file_do(/var/admhome/username/Maildir) racing May 14 16:47:17 postamt dovecot: IMAP(username): Disconnected: Internal error occurred. Refer to server log for more information. [2007-05-14 16:47:17] Now it's a bit hard for me to see what's going on. /var/admhome/username/Maildir is on /, thus: $ mount /dev/cciss/c0d0p6 on / type ext3 (rw,errors=panic) $ dmesg does not complain about any corruption $ uname -a Linux postamt.charite.de 2.6.21.1 #1 SMP Tue May 1 15:50:11 CEST 2007 i686 GNU/Linux OS: Debian testing -- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt(a)charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk(a)charite.de

3 6

[Dovecot] Question: contention with blackberry
by Stewart Dean 15 May '07

15 May '07

My production imap is currently UWIMAP with mbox.. One of the problems commonly observed is mailbox lock loss, which happens when: a) a VIP has multiple secretaries accessing a single mailbox (actually, they are professional enough to have figured the realities of conflicting access and rarely have a problem), b) somebody leaving their machine on at home and coming in and firing up their work computer c) important people with Blackberries....and the BB service polls the mailbox every so often and breaks the lock. In any case, the maillog ends up with a message like this: > May 5 19:45:04 mercury mail:info imapd[2035774]: Logout user=xxxxx host=bda056.bis.na.blackberry.com [216.9.249.56] > May 5 19:46:52 mercury mail:info imapd[2502902]: Killed (lost mailbox lock) user=xxxxx host=cpe-24-161-103-11.hvc.res.rr.com [24.161.103.11] How will this work with Dovecot? I regard what is happening now as OK, working as designed. Although we may eventually move to a different mailbox format, I am only switching from UW to DC in the first step. -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

3 2

[Dovecot] global vs user script
by M1 15 May '07

15 May '07

What is the order of execution if both user and global sieve script exists? Thanks.

4 10

[Dovecot] Mails, work and so on
by Timo Sirainen 14 May '07

14 May '07

You can probably guess from my flood of mails that I'm no longer busy. I still have a couple of exams but since I haven't spent any time at all studying for them or attending the lectures, I can just as easily leave them for next year. I've read and answered most of the "easy" mails in this list and in my INBOX. I'll try to answer the rest of them this weekend, so no need to forward them to me again, at least not yet. :) Now that I'm finally non-busy, it also appears that I haven't worked for a long time and I'm almost broke. :) So now would be a great time for your corporation to buy/sponsor some Dovecot development. Otherwise I'm probably soon busy with non-Dovecot related work. My own high priority TODO contains: - Get the rewritten Squat full text search index working. It probably needs a name change, since it's not all that close to Cyrus Squat anymore. Suggestions? Could of course be just "fts_dovecot". - Shared mailboxes / ACLs - dbox cleanups and possibly partial redesign. I'll write a separate mail about this later. - Index file code cleanups, simplifications and optimizations. Make it work with NFS with attribute cache enabled.

5 7

[Dovecot] How to get rid of locks
by Timo Sirainen 14 May '07

14 May '07

Although Dovecot is already read-lockless and it uses only short- lived write locks, it's be really nice to just get rid of the locking completely. :) I just figured out that O_APPEND is pretty great. If the operating system updates seek position after writing to a file opened with O_APPEND, writes to Dovecot's transaction log file can be made lockless. I see that this works with Linux and Solaris, but not with OS X. Could you BSD people try if it works there? http://dovecot.org/ tmp/append.c and see if it says "offset = 0" (bad) or non-zero (yay). The O_APPEND at least doesn't work with NFS, so it'll have to be optional anyway. Currently Dovecot always updates dovecot.index file after it has done any changes. This isn't really necessary, because the changes are already in transaction log, so the dovecot.index file can be read to memory and the new changes applied on top of it from transaction log (this is pretty much how mmap_disable=yes works). So I'm going to change this to work so that the dovecot.index is updated only if a) there are enough changes in transaction log (eg. 8kB or so) and b) it can be write-locked without waiting. Maildir then. It has this annoying problem that readdir() can skip files if another process is rename()ing them, causing Dovecot to think that the message was expunged. The only way I could avoid this by locking the maildir while synchronizing it. Today I noticed that this doesn't happen with OS X. I'm not sure if I was just lucky or if there really is something special implemented in it, because it doesn't work anywhere else. I'm not sure if this is tied to HFS+, or if it will work with zfs also (Solaris+zfs didn't work). So perhaps the locking could be disabled while running with OS X. More importantly I figured out that it can also be avoided with Linux +inotify. As long as the inotify event buffer doesn't overflow, the full list of files can be read by combining the readdir() output and files listed by inotify events. If the inotify buffer overflows (highly unlikely), the operation can just be retried and it most likely works the next time. So with these changes in place, changing a message flag or expunging a message would usually result in: - lockless write() call to dovecot.index.log - lockless read()ing (or looking into mmaped) dovecot.index.log to see if there's some new data besides what we just wrote that needs to be synchronized to maildir - rename() or unlink() calls to maildir. If a call return ENOENT, the maildir needs to be readdir()ed with inotify enabled to find the new filename. Not a single lock in the operation, assuming that dovecot.index file wasn't updated. Assigning UIDs to newly delivered mails would require locking though. dovecot-uidlist needs to be locked, and the UIDs need to be written to dovecot.index.log file in the correct order, which can also be done with dovecot-uidlist locking. Actually a single write() to dovecot.index.log isn't enough. I think there needs to be some kind of a flag written to the beginning of the transaction which marks the transaction as truly finished. If the flag isn't there, any reader knows to stop and wait until the flag is set. So this means that the writer needs to: 1. Do a single O_APPENDed write() call writing the whole transaction 2. Get the current offset with lseek(fd, 0, SEEK_CUR) (this is what the append.c tester checks) 3. pwrite() the finished-flag to beginning of the transaction Except at least with Linux pwrite() doesn't work if O_APPEND is enabled. There are two ways to work around this: a) fcntl(disable O_APPEND) + pwrite() + fcntl(enable O_APPEND) b) Keep two file descriptors open for the transaction log. First with O_APPEND flag and second without. pwrite() to the second one. a) is probably better because it doesn't waste file descriptors.

8 10

[Dovecot] PATCH: sendmail-like DSNs in Dovecot deliver (EX_TEMPFAIL always)
by Steffen Kaiser 14 May '07

14 May '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I've put together an always tempfail & sendmail-like DSN report back patch. Because I don't know how to transform the rejection reason into sysexit.h codes, it always tempfails. The hooks are so that: a) config file setting: lda_mode = mode b) command line: deliver --mode mode where mode is default or sendmail. When no mode is specified, it defaults to "default", which is the current behaviour. In sendmail mode, mail_send_rejection() simply prints the rejection reason to stdout, logs it and returns EX_TEMPFAIL. Ideally, the return value would be specific to the reason. I haven't poked into this deeply, but it doesn't look straight forward. Bye, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRkHNQS9SORjhbDpvAQIhdggAq79RHKa1gIUTFwAu7Gd2maIUkDTgbNu6 8+MEQHujOPkV6wcF+5ih/hkht5BnuBUiIbnp2dvKGaUcHY8avzYD6BEc6gIAyJjM YIKDANI5+BVnQWUdiQMlQrSrRIZBqFziL8v/LvICJUsiOqCKT7se37KuUOK14gXO FSghuPta0rorogJ8kh+VGdmnqNVzKb6eKrnFQUggiezTmTcHY6/DRsLbsKV//fyW cybualj+THF9cj+EajwCMsYypc9KOdVg6lhcCD87oskDmMqcG5lwF2HU9nclnMxE 6pOnaLGNfKmWyeAKOLgvMimjhkU74+NBwQ+b4h61JzZ6lbiZKWX7wA== =gE8Z -----END PGP SIGNATURE-----

3 5

[Dovecot] PATCH Deliver forgets sendmail_path if DEBUG is set
by Steffen Kaiser 14 May '07

14 May '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, Dovecot v1.0.0 deliver erases the environment if DEBUG is set, hence, also (probably) erases SENDMAIL_PATH if it has been read from configuration file. This happens with: Debian Etch, libc-2.3.6.so, gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21). Problem in log: May 10 12:33:07 ux-2s11-9 deliver(dvtest): execv() failed: No such file or directory May 10 12:33:07 ux-2s11-9 deliver(dvtest): Sendmail process terminated abnormally, exit status 75 Note the empty string in the function call execv(). Attached patch just deletes the DEBUG variable. Bye, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRkL45y9SORjhbDpvAQKeTAgAr6i6XrICqZsTaezUFqXO69cuwOt4iL5u K3PAF0ywAtrRkpWG+G8TUtypUkLp+CGEK3Ery1mWVp2/TeNPv6LxeUWWGxmdlSV3 XFQSV/VRuPtJ/Uc8wFWOBcpAn0H9rnLNkF5vv512t4zs99eF10++US5fZwFOXK8M azgaCEYwOrmZ4U2sCQwVQChXcZQq6MI5IcE3ChpzlUP45bq2dVZrXqAZ/xNo1iuG lQd/43XJu1++LxB8N1Fkbaeme+Ojt/c5fOKApjhnCJuQwi46jnSEeHbO2VUCz+ST mU34QR3zx+h4YW91oKQhFYkbSWEYmuXk2OoZaFUd1Q+E+oXHHuDBng== =CiqO -----END PGP SIGNATURE-----

2 2