- dovecot - dovecot.org

[Dovecot] maildir_file_do(/var/admhome/user/Maildir) racing
by Ralf Hildebrandt 15 May '07

15 May '07

dovecot version 1.0.0, compile from the sources. --------- snip -------- # /usr/local/etc/dovecot.conf listen: *:10143 ssl_listen: *:10943 ssl_cert_file: /etc/ssl/certs/postamt.pem ssl_key_file: /etc/ssl/private/postamt.key disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_location: maildir:~/Maildir mail_plugins: quota imap_quota namespace: type: private separator: . prefix: INBOX. inbox: yes auth default: passdb: driver: pam userdb: driver: passwd plugin: quota: maildir quota: maildir:ignore=Trash --------- snip -------- We're using these odd ports, because there's still courier-imap on the original ports. One, just one (!) of our users gets: May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: imap-login: Login: user=<username>, method=PLAIN, rip=141.42.x.y, lip=ip.of.the.server May 14 16:47:17 postamt dovecot: IMAP(username): maildir_file_do(/var/admhome/username/Maildir) racing May 14 16:47:17 postamt dovecot: IMAP(username): Disconnected: Internal error occurred. Refer to server log for more information. [2007-05-14 16:47:17] Now it's a bit hard for me to see what's going on. /var/admhome/username/Maildir is on /, thus: $ mount /dev/cciss/c0d0p6 on / type ext3 (rw,errors=panic) $ dmesg does not complain about any corruption $ uname -a Linux postamt.charite.de 2.6.21.1 #1 SMP Tue May 1 15:50:11 CEST 2007 i686 GNU/Linux OS: Debian testing -- Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt(a)charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to plonk(a)charite.de

3 6

[Dovecot] Question: contention with blackberry
by Stewart Dean 15 May '07

15 May '07

My production imap is currently UWIMAP with mbox.. One of the problems commonly observed is mailbox lock loss, which happens when: a) a VIP has multiple secretaries accessing a single mailbox (actually, they are professional enough to have figured the realities of conflicting access and rarely have a problem), b) somebody leaving their machine on at home and coming in and firing up their work computer c) important people with Blackberries....and the BB service polls the mailbox every so often and breaks the lock. In any case, the maillog ends up with a message like this: > May 5 19:45:04 mercury mail:info imapd[2035774]: Logout user=xxxxx host=bda056.bis.na.blackberry.com [216.9.249.56] > May 5 19:46:52 mercury mail:info imapd[2502902]: Killed (lost mailbox lock) user=xxxxx host=cpe-24-161-103-11.hvc.res.rr.com [24.161.103.11] How will this work with Dovecot? I regard what is happening now as OK, working as designed. Although we may eventually move to a different mailbox format, I am only switching from UW to DC in the first step. -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

3 2

[Dovecot] global vs user script
by M1 15 May '07

15 May '07

What is the order of execution if both user and global sieve script exists? Thanks.

4 10

[Dovecot] Mails, work and so on
by Timo Sirainen 14 May '07

14 May '07

You can probably guess from my flood of mails that I'm no longer busy. I still have a couple of exams but since I haven't spent any time at all studying for them or attending the lectures, I can just as easily leave them for next year. I've read and answered most of the "easy" mails in this list and in my INBOX. I'll try to answer the rest of them this weekend, so no need to forward them to me again, at least not yet. :) Now that I'm finally non-busy, it also appears that I haven't worked for a long time and I'm almost broke. :) So now would be a great time for your corporation to buy/sponsor some Dovecot development. Otherwise I'm probably soon busy with non-Dovecot related work. My own high priority TODO contains: - Get the rewritten Squat full text search index working. It probably needs a name change, since it's not all that close to Cyrus Squat anymore. Suggestions? Could of course be just "fts_dovecot". - Shared mailboxes / ACLs - dbox cleanups and possibly partial redesign. I'll write a separate mail about this later. - Index file code cleanups, simplifications and optimizations. Make it work with NFS with attribute cache enabled.

5 7

[Dovecot] How to get rid of locks
by Timo Sirainen 14 May '07

14 May '07

Although Dovecot is already read-lockless and it uses only short- lived write locks, it's be really nice to just get rid of the locking completely. :) I just figured out that O_APPEND is pretty great. If the operating system updates seek position after writing to a file opened with O_APPEND, writes to Dovecot's transaction log file can be made lockless. I see that this works with Linux and Solaris, but not with OS X. Could you BSD people try if it works there? http://dovecot.org/ tmp/append.c and see if it says "offset = 0" (bad) or non-zero (yay). The O_APPEND at least doesn't work with NFS, so it'll have to be optional anyway. Currently Dovecot always updates dovecot.index file after it has done any changes. This isn't really necessary, because the changes are already in transaction log, so the dovecot.index file can be read to memory and the new changes applied on top of it from transaction log (this is pretty much how mmap_disable=yes works). So I'm going to change this to work so that the dovecot.index is updated only if a) there are enough changes in transaction log (eg. 8kB or so) and b) it can be write-locked without waiting. Maildir then. It has this annoying problem that readdir() can skip files if another process is rename()ing them, causing Dovecot to think that the message was expunged. The only way I could avoid this by locking the maildir while synchronizing it. Today I noticed that this doesn't happen with OS X. I'm not sure if I was just lucky or if there really is something special implemented in it, because it doesn't work anywhere else. I'm not sure if this is tied to HFS+, or if it will work with zfs also (Solaris+zfs didn't work). So perhaps the locking could be disabled while running with OS X. More importantly I figured out that it can also be avoided with Linux +inotify. As long as the inotify event buffer doesn't overflow, the full list of files can be read by combining the readdir() output and files listed by inotify events. If the inotify buffer overflows (highly unlikely), the operation can just be retried and it most likely works the next time. So with these changes in place, changing a message flag or expunging a message would usually result in: - lockless write() call to dovecot.index.log - lockless read()ing (or looking into mmaped) dovecot.index.log to see if there's some new data besides what we just wrote that needs to be synchronized to maildir - rename() or unlink() calls to maildir. If a call return ENOENT, the maildir needs to be readdir()ed with inotify enabled to find the new filename. Not a single lock in the operation, assuming that dovecot.index file wasn't updated. Assigning UIDs to newly delivered mails would require locking though. dovecot-uidlist needs to be locked, and the UIDs need to be written to dovecot.index.log file in the correct order, which can also be done with dovecot-uidlist locking. Actually a single write() to dovecot.index.log isn't enough. I think there needs to be some kind of a flag written to the beginning of the transaction which marks the transaction as truly finished. If the flag isn't there, any reader knows to stop and wait until the flag is set. So this means that the writer needs to: 1. Do a single O_APPENDed write() call writing the whole transaction 2. Get the current offset with lseek(fd, 0, SEEK_CUR) (this is what the append.c tester checks) 3. pwrite() the finished-flag to beginning of the transaction Except at least with Linux pwrite() doesn't work if O_APPEND is enabled. There are two ways to work around this: a) fcntl(disable O_APPEND) + pwrite() + fcntl(enable O_APPEND) b) Keep two file descriptors open for the transaction log. First with O_APPEND flag and second without. pwrite() to the second one. a) is probably better because it doesn't waste file descriptors.

8 10

[Dovecot] PATCH: sendmail-like DSNs in Dovecot deliver (EX_TEMPFAIL always)
by Steffen Kaiser 14 May '07

14 May '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I've put together an always tempfail & sendmail-like DSN report back patch. Because I don't know how to transform the rejection reason into sysexit.h codes, it always tempfails. The hooks are so that: a) config file setting: lda_mode = mode b) command line: deliver --mode mode where mode is default or sendmail. When no mode is specified, it defaults to "default", which is the current behaviour. In sendmail mode, mail_send_rejection() simply prints the rejection reason to stdout, logs it and returns EX_TEMPFAIL. Ideally, the return value would be specific to the reason. I haven't poked into this deeply, but it doesn't look straight forward. Bye, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRkHNQS9SORjhbDpvAQIhdggAq79RHKa1gIUTFwAu7Gd2maIUkDTgbNu6 8+MEQHujOPkV6wcF+5ih/hkht5BnuBUiIbnp2dvKGaUcHY8avzYD6BEc6gIAyJjM YIKDANI5+BVnQWUdiQMlQrSrRIZBqFziL8v/LvICJUsiOqCKT7se37KuUOK14gXO FSghuPta0rorogJ8kh+VGdmnqNVzKb6eKrnFQUggiezTmTcHY6/DRsLbsKV//fyW cybualj+THF9cj+EajwCMsYypc9KOdVg6lhcCD87oskDmMqcG5lwF2HU9nclnMxE 6pOnaLGNfKmWyeAKOLgvMimjhkU74+NBwQ+b4h61JzZ6lbiZKWX7wA== =gE8Z -----END PGP SIGNATURE-----

3 5

[Dovecot] PATCH Deliver forgets sendmail_path if DEBUG is set
by Steffen Kaiser 14 May '07

14 May '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, Dovecot v1.0.0 deliver erases the environment if DEBUG is set, hence, also (probably) erases SENDMAIL_PATH if it has been read from configuration file. This happens with: Debian Etch, libc-2.3.6.so, gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21). Problem in log: May 10 12:33:07 ux-2s11-9 deliver(dvtest): execv() failed: No such file or directory May 10 12:33:07 ux-2s11-9 deliver(dvtest): Sendmail process terminated abnormally, exit status 75 Note the empty string in the function call execv(). Attached patch just deletes the DEBUG variable. Bye, - -- Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBRkL45y9SORjhbDpvAQKeTAgAr6i6XrICqZsTaezUFqXO69cuwOt4iL5u K3PAF0ywAtrRkpWG+G8TUtypUkLp+CGEK3Ery1mWVp2/TeNPv6LxeUWWGxmdlSV3 XFQSV/VRuPtJ/Uc8wFWOBcpAn0H9rnLNkF5vv512t4zs99eF10++US5fZwFOXK8M azgaCEYwOrmZ4U2sCQwVQChXcZQq6MI5IcE3ChpzlUP45bq2dVZrXqAZ/xNo1iuG lQd/43XJu1++LxB8N1Fkbaeme+Ojt/c5fOKApjhnCJuQwi46jnSEeHbO2VUCz+ST mU34QR3zx+h4YW91oKQhFYkbSWEYmuXk2OoZaFUd1Q+E+oXHHuDBng== =CiqO -----END PGP SIGNATURE-----

2 2

[Dovecot] header-lines of subfolders have to be rescanned after each client startup
by Rainer Sigl 14 May '07

14 May '07

Hi all, after each startup of my different mail clients (thunderbird, outlook, ...) the contents of subfolders if I click in has to be renewed. This takes some time for large folders. I just migrated from courier. And there all subfolders were available immediately. Is there something to configure? Another effect regarding subfolders is that the client forgets, which subfolders were opened after shutdown and shows the folders closed. This means, that necessary subfolders have to be reopened after each startup. This was managed as well by my previous courier server. Many thanks for answers Rainer Sigl -- ----------------- Rainer Sigl, | email: sigl(a)mpe.mpg.de MPI f. extraterrestrische Physik, | phone: +49 (89) 30000-3557 Postfach 1312, | fax: +49 (89) 30000-3569 D-85741 Garching, Germany | web: http://www.mpe.mpg.de/~sigl/

2 4

[Dovecot] v1.1 plans
by Timo Sirainen 14 May '07

14 May '07

I think I won't do any actual releases until it's mostly feature complete. Then maybe v1.1.alpha1 or v1.1.beta1. So if you want to test it before then, use CVS or the nightly snapshots. I'm planning on keeping v1.1 almost completely compatible with v1.0. There could be some minor configuration file changes, but for most people v1.0's dovecot.conf should work with v1.1. I want to update index file handling to use less locks and update dovecot.index file less often. The file formats would still stay backwards compatible with v1.0. After this I'm hoping to get index files working well with NFS even with attribute cache enabled. Dovecot v2.0 then brings incompatible index files and more or less rewritten configuration file handling. It could still support old config file format and settings, but the default will be somewhat different. I'm hoping to release the first alphas/betas in 2-3 months, with v1.1.0 maybe even as early as next summer. The good thing here is that index file code hasn't changed much since v1.0, so it should be stable. The bad thing is that I've added a couple of completely new indexes. I'm also quite busy for the next month, but I'm not sure how much it affects my coding. :) The largest already implemented new features are: + Added support for IMAP UIDPLUS extension + IMAP SORT: Sort keys are indexed, which makes SORT commands faster. + IMAP THREAD: Threads are indexed into a dovecot.index.thread file. It can usually be updated incrementally, so this makes THREAD fast. + Added Dovecot-specific X-REFERENCES2 threading algorithm. It's similar to REFERENCES, but it doesn't do base subject merging and it sorts the threads by their newest message. + When saving messages, update cache file immediately with the data that we expect client to fetch later. Maildir-only currently. + Mailbox list indexes. Mailbox metadata is stored there, so STATUS commands can return synchronized mailboxes' metadata simply by reading a single mailbox list index file. - FIXME: Currently works only with Maildir++ layout. + Expire plugin can be used to keep track of oldest messages in specific mailboxes. A nightly run can then quickly expunge old messages from the mailboxes that have them. The tracking is done using lib-dict, so you can use either Berkeley DB or SQL database. + Namespaces are supported everywhere now. - FIXME: except by convert plugin + Full text search indexing support with Lucene and Squat backends. + OTP and S/KEY authentication mechanisms (by Andrey Panin). + mbox and Maildir works with both Maildir++ and FS layouts. You can change these by appending :LAYOUT=maildir++ or :LAYOUT=fs to mail_location. Features that I'm planning on implementing: - Fully supported shared mailboxes and IMAP ACL extension - Replace Squat FTS indexes with my new design - Case-insensitive searches with non-ASCII text as well - Maybe add support for all kinds of IMAP extensions that can be easily supported. LEMONADE extensions especially: CONDSTORE, CATENATE and maybe even URLAUTH if I can figure out how it should work. If a feature can be implemented completely as a plugin, it could still come in v1.1.beta stage, because it won't destabilize the rest of the Dovecot anyway.

17 36

[Dovecot] Feature Request: content-filter / MOVE scripts
by Daniel L. Miller 14 May '07

14 May '07

Since your "1.1 plans" announcement somehow turned into everybody handing in a wish list . . . thought I'd start one by itself - since I expect a few people to have opinions on this matter. I'm aware of a plug-in functionality in Dovecot, and that a plug-in exists for dspam processing - but my review of the plug-ins indicate they require compiling your own Dovecot - the plug-in setup isn't really suitable for packaged distros. As a packaged software user - I really really really don't like compiling my own software, either for features or bugfixes - I want leave that to the experts. On learning of 1.0's official release, I nudged the Debian package maintainers to check - and they were already on it and it updated. Thanx Timo, Jaldhar, and Fabio. I have some thoughts on an interface that would eliminate the need for compile-time plug-ins, but provide some major power to sysadmins. Basically, I want a way of telling an external program the following: 1. A message was moved. 2. The folder is moved from was XXX. 3. The folder it moved to was YYY. 4. The message moved was ZZZ. Armed with this information, just about anybody's spam fighting learn scripts can be called to do their thing. A few different ways have occurred to me: 1. Have a dedicated "content-filter" functionality built-in to Dovecot. Under this model, Dovecot.conf would include at least three new parameters: FILTER-FOLDER-SUFFIX, FILTER-FOLDER-IN-SCRIPT, and FILTER-FOLDER-OUT-SCRIPT. Whenever a message is moved into or out of a folder ending in SUFFIX the appropriate SCRIPT is called, with the arguments above being passed. 2. A bit more generic, less processing for Dovecot. Dovecot.conf parameter ACTION-SCRIPT (hey, feel free to use other names - I'm just brainstorming here). Any time any message is moved anywhere, call SCRIPT with the arguments. Having never looked at Dovecot's code, I'm just guessing, but this probably wouldn't take much to implement - but might be the most performance affecting. 3. Have a definable named-pipe to send the same information to. 4. Have a definable TCP socket to send the same information to. In any case, I figure it's best to prevent any message processing by Dovecot itself - so while it might be nice to define a list of headers to send to the scripts, instead of the whole message, that's probably not appropriate for Dovecot to focus on. On the other hand, any of the above options would probably benefit from a MAXIMUM_FILTER_SIZE parameter to limit the maximum message size. So what's the world got to say about this? (More importantly, how does TIMO feel about this?) ;). -- Daniel

3 2