April 2007 - dovecot - dovecot.org

[Dovecot] Deliver sending bounces from 'MAILER-DAEMON@'
by Erland Nylend 16 May '07

16 May '07

Hello, I'm having a problem with postfix and/or dovecots "deliver". I'm getting these lines in the mail log: | Apr 24 11:42:08 smtp2 postfix/qmgr[6176]: 05BAE3B67E: from=<>, size=3055, nrcpt=1 (queue active) | Apr 24 11:42:08 smtp2 postfix/qmgr[6176]: 05BAE3B67E: to=<MAILER-DAEMON@>, relay=none, delay=0.01, delays=0.01/0.01/0/0, dsn=5.1.3, status=bounced (bad address syntax) After some debugging, it seems that those lines in the log appear if I send mail to an account which is over quota. (I'm using the Maildir++ quota) This is from my dovecot.conf: | protocol lda { | postmaster_address = MAILER-DAEMON(a)my.domain.tld | mail_plugins = quota | auth_socket_path = /var/run/dovecot/auth-master | hostname = my.mail.server | sendmail_path = /usr/sbin/sendmail | } It seems to me that dovecot is sending bounce messages from 'MAILER-DAEMON@' .. Does anyone know why this is happening, or how I can specify the address to be used? I'm using dovecot 1.0.0. -- Erland Nylend

2 5

[Dovecot] assertion failed
by Adrian Stoica 15 May '07

15 May '07

What is this ? dovecot: Apr 24 13:03:29 Error: IMAP(tina.burdujan): file maildir-sync.c: line 1075 (maildir_sync_index): assertion failed: (uid >prev_uid) dovecot: Apr 24 13:03:29 Error: IMAP(tina.burdujan): Raw backtrace: /usr/libexec/dovecot/imap [0x80b8741] -> /usr/libexec/dovecot/imap [0x80b819c] -> /usr/libexec/dovecot/imap(maildir_sync_index+0x898) [0x80695f8] -> /usr/libexec/dovecot/imap [0x80698b3] -> /usr/libexec/dovecot/imap(maildir_ storage_sync_init+0x49) [0x8069aa9] -> /usr/libexec/dovecot/imap(imap_sync_init+0x40) [0x8062620] -> /usr/libexec/dovecot/imap(cmd_sync+0x71) [0x8 062b81] -> /usr/libexec/dovecot/imap(cmd_noop+0x26) [0x8059c76] -> /usr/libexec/dovecot/imap [0x805be7f] -> /usr/libexec/dovecot/imap [0x805bf20] -> /usr/libexec/dovecot/imap(_client_input+0x6c) [0x805c0fc] -> /usr/libexec/dovecot/imap(io_loop_handler_run+0xff) [0x80bef6f] -> /usr/libexec/do vecot/imap(io_loop_run+0x36) [0x80be366] -> /usr/libexec/dovecot/imap(main+0x46c) [0x806448c] -> /lib/libc.so.6(__libc_start_main+0xb9) [0xb7eb4e1 9] -> /usr/libexec/dovecot/imap [0x80568b1] dovecot: Apr 24 13:03:29 Error: child 25748 (imap) killed with signal 6 My conf is: exim + dovecot + mysql exim deliver mail to maildir , because i don't know how to use LDA from dovecot. the maildirs are very large, for one account could be 30000 mails. dovecot is version 1.0.0 dovecot -n shows: # /etc/dovecot.conf base_dir: /var/run/dovecot/login log_path: /var/log/mail.log login_dir: /var/run/dovecot login_executable: /usr/libexec/dovecot/imap-login login_process_per_connection: no login_process_size: 80 login_processes_count: 10 login_max_processes_count: 50 login_max_connections: 80 first_valid_uid: 1001 last_valid_uid: 1001 mail_extra_groups: exim,vmail mail_location: maildir:%Lh/Maildir mail_cache_min_mail_count: 1 dotlock_use_excl: yes fsync_disable: yes maildir_copy_with_hardlinks: yes maildir_copy_preserve_filename: yes mail_drop_priv_before_exec: yes mail_executable: /bin/imap.sh mail_plugins: quota imap_quota auth default: mechanisms: PLAIN CRAM-MD5 default_realm: testing.net cache_size: 256 worker_max_count: 100 passdb: driver: sql args: /etc/dovecot-crammd5.conf userdb: driver: prefetch plugin: quota: maildir

3 12

[Dovecot] How to get rid of locks
by Timo Sirainen 14 May '07

14 May '07

Although Dovecot is already read-lockless and it uses only short- lived write locks, it's be really nice to just get rid of the locking completely. :) I just figured out that O_APPEND is pretty great. If the operating system updates seek position after writing to a file opened with O_APPEND, writes to Dovecot's transaction log file can be made lockless. I see that this works with Linux and Solaris, but not with OS X. Could you BSD people try if it works there? http://dovecot.org/ tmp/append.c and see if it says "offset = 0" (bad) or non-zero (yay). The O_APPEND at least doesn't work with NFS, so it'll have to be optional anyway. Currently Dovecot always updates dovecot.index file after it has done any changes. This isn't really necessary, because the changes are already in transaction log, so the dovecot.index file can be read to memory and the new changes applied on top of it from transaction log (this is pretty much how mmap_disable=yes works). So I'm going to change this to work so that the dovecot.index is updated only if a) there are enough changes in transaction log (eg. 8kB or so) and b) it can be write-locked without waiting. Maildir then. It has this annoying problem that readdir() can skip files if another process is rename()ing them, causing Dovecot to think that the message was expunged. The only way I could avoid this by locking the maildir while synchronizing it. Today I noticed that this doesn't happen with OS X. I'm not sure if I was just lucky or if there really is something special implemented in it, because it doesn't work anywhere else. I'm not sure if this is tied to HFS+, or if it will work with zfs also (Solaris+zfs didn't work). So perhaps the locking could be disabled while running with OS X. More importantly I figured out that it can also be avoided with Linux +inotify. As long as the inotify event buffer doesn't overflow, the full list of files can be read by combining the readdir() output and files listed by inotify events. If the inotify buffer overflows (highly unlikely), the operation can just be retried and it most likely works the next time. So with these changes in place, changing a message flag or expunging a message would usually result in: - lockless write() call to dovecot.index.log - lockless read()ing (or looking into mmaped) dovecot.index.log to see if there's some new data besides what we just wrote that needs to be synchronized to maildir - rename() or unlink() calls to maildir. If a call return ENOENT, the maildir needs to be readdir()ed with inotify enabled to find the new filename. Not a single lock in the operation, assuming that dovecot.index file wasn't updated. Assigning UIDs to newly delivered mails would require locking though. dovecot-uidlist needs to be locked, and the UIDs need to be written to dovecot.index.log file in the correct order, which can also be done with dovecot-uidlist locking. Actually a single write() to dovecot.index.log isn't enough. I think there needs to be some kind of a flag written to the beginning of the transaction which marks the transaction as truly finished. If the flag isn't there, any reader knows to stop and wait until the flag is set. So this means that the writer needs to: 1. Do a single O_APPENDed write() call writing the whole transaction 2. Get the current offset with lseek(fd, 0, SEEK_CUR) (this is what the append.c tester checks) 3. pwrite() the finished-flag to beginning of the transaction Except at least with Linux pwrite() doesn't work if O_APPEND is enabled. There are two ways to work around this: a) fcntl(disable O_APPEND) + pwrite() + fcntl(enable O_APPEND) b) Keep two file descriptors open for the transaction log. First with O_APPEND flag and second without. pwrite() to the second one. a) is probably better because it doesn't waste file descriptors.

8 10

[Dovecot] v1.1 plans
by Timo Sirainen 14 May '07

14 May '07

I think I won't do any actual releases until it's mostly feature complete. Then maybe v1.1.alpha1 or v1.1.beta1. So if you want to test it before then, use CVS or the nightly snapshots. I'm planning on keeping v1.1 almost completely compatible with v1.0. There could be some minor configuration file changes, but for most people v1.0's dovecot.conf should work with v1.1. I want to update index file handling to use less locks and update dovecot.index file less often. The file formats would still stay backwards compatible with v1.0. After this I'm hoping to get index files working well with NFS even with attribute cache enabled. Dovecot v2.0 then brings incompatible index files and more or less rewritten configuration file handling. It could still support old config file format and settings, but the default will be somewhat different. I'm hoping to release the first alphas/betas in 2-3 months, with v1.1.0 maybe even as early as next summer. The good thing here is that index file code hasn't changed much since v1.0, so it should be stable. The bad thing is that I've added a couple of completely new indexes. I'm also quite busy for the next month, but I'm not sure how much it affects my coding. :) The largest already implemented new features are: + Added support for IMAP UIDPLUS extension + IMAP SORT: Sort keys are indexed, which makes SORT commands faster. + IMAP THREAD: Threads are indexed into a dovecot.index.thread file. It can usually be updated incrementally, so this makes THREAD fast. + Added Dovecot-specific X-REFERENCES2 threading algorithm. It's similar to REFERENCES, but it doesn't do base subject merging and it sorts the threads by their newest message. + When saving messages, update cache file immediately with the data that we expect client to fetch later. Maildir-only currently. + Mailbox list indexes. Mailbox metadata is stored there, so STATUS commands can return synchronized mailboxes' metadata simply by reading a single mailbox list index file. - FIXME: Currently works only with Maildir++ layout. + Expire plugin can be used to keep track of oldest messages in specific mailboxes. A nightly run can then quickly expunge old messages from the mailboxes that have them. The tracking is done using lib-dict, so you can use either Berkeley DB or SQL database. + Namespaces are supported everywhere now. - FIXME: except by convert plugin + Full text search indexing support with Lucene and Squat backends. + OTP and S/KEY authentication mechanisms (by Andrey Panin). + mbox and Maildir works with both Maildir++ and FS layouts. You can change these by appending :LAYOUT=maildir++ or :LAYOUT=fs to mail_location. Features that I'm planning on implementing: - Fully supported shared mailboxes and IMAP ACL extension - Replace Squat FTS indexes with my new design - Case-insensitive searches with non-ASCII text as well - Maybe add support for all kinds of IMAP extensions that can be easily supported. LEMONADE extensions especially: CONDSTORE, CATENATE and maybe even URLAUTH if I can figure out how it should work. If a feature can be implemented completely as a plugin, it could still come in v1.1.beta stage, because it won't destabilize the rest of the Dovecot anyway.

17 36

[Dovecot] Feature Request: content-filter / MOVE scripts
by Daniel L. Miller 13 May '07

13 May '07

Since your "1.1 plans" announcement somehow turned into everybody handing in a wish list . . . thought I'd start one by itself - since I expect a few people to have opinions on this matter. I'm aware of a plug-in functionality in Dovecot, and that a plug-in exists for dspam processing - but my review of the plug-ins indicate they require compiling your own Dovecot - the plug-in setup isn't really suitable for packaged distros. As a packaged software user - I really really really don't like compiling my own software, either for features or bugfixes - I want leave that to the experts. On learning of 1.0's official release, I nudged the Debian package maintainers to check - and they were already on it and it updated. Thanx Timo, Jaldhar, and Fabio. I have some thoughts on an interface that would eliminate the need for compile-time plug-ins, but provide some major power to sysadmins. Basically, I want a way of telling an external program the following: 1. A message was moved. 2. The folder is moved from was XXX. 3. The folder it moved to was YYY. 4. The message moved was ZZZ. Armed with this information, just about anybody's spam fighting learn scripts can be called to do their thing. A few different ways have occurred to me: 1. Have a dedicated "content-filter" functionality built-in to Dovecot. Under this model, Dovecot.conf would include at least three new parameters: FILTER-FOLDER-SUFFIX, FILTER-FOLDER-IN-SCRIPT, and FILTER-FOLDER-OUT-SCRIPT. Whenever a message is moved into or out of a folder ending in SUFFIX the appropriate SCRIPT is called, with the arguments above being passed. 2. A bit more generic, less processing for Dovecot. Dovecot.conf parameter ACTION-SCRIPT (hey, feel free to use other names - I'm just brainstorming here). Any time any message is moved anywhere, call SCRIPT with the arguments. Having never looked at Dovecot's code, I'm just guessing, but this probably wouldn't take much to implement - but might be the most performance affecting. 3. Have a definable named-pipe to send the same information to. 4. Have a definable TCP socket to send the same information to. In any case, I figure it's best to prevent any message processing by Dovecot itself - so while it might be nice to define a list of headers to send to the scripts, instead of the whole message, that's probably not appropriate for Dovecot to focus on. On the other hand, any of the above options would probably benefit from a MAXIMUM_FILTER_SIZE parameter to limit the maximum message size. So what's the world got to say about this? (More importantly, how does TIMO feel about this?) ;). -- Daniel

3 2

[Dovecot] Dovecot, Postfix and SMTP AUTH....
by Bartosz Toczek 13 May '07

13 May '07

Hello. I've notice a problem with SMTP AUTH in postfix with dovecot. My configuration is based on Postgresql. I've created a function in postgresql which returns password and username and attached it to password_query. Query looks like this password_query = SELECT username_out as username, password_out as password FROM get_password(lower('%n'),lower('%d')); Problem is when get_password returns something like this... vmail=# SELECT username_out as username, password_out FROM get_password(lower(''),lower('')); LOG: statement: SELECT username_out as username, password_out FROM get_password(lower(''),lower('')); LOG: duration: 2.342 ms statement: SELECT username_out as username, password_out FROM get_password(lower(''),lower('')); username | password_out ----------+-------------- | (1 row) It returns 1 empty row.... When this is attached to postfix with smtp auth with such configuration # SMTP AUTH broken_sasl_auth_clients = yes smtpd_sasl_auth_enable = yes smtpd_sasl_security_options = noanonymous smtpd_sasl_authenticated_header = yes smtpd_sasl_local_domain = smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth my serwer starts to be open relay... I now that it returns empty string for username and password... so where is the password checked... if I dont send password as a parameter to my function. Definition of the function looks like this: CREATE FUNCTION get_password("login" character varying, "domain" character varying, OUT username_out character varying, OUT password_out character varying) RETURNS record Can you help with that... BT ---------------------------------------------------------------------- Zrob numer kumplom >> http://link.interia.pl/f1a5d

4 4

[Dovecot] Public folders using NAMESPACE
by Joakim Ryden 13 May '07

13 May '07

Howdy folks! I've been trying to research how to (or if possible at all) implement public folders using the namespace feature on a server with multiple virtual domains. After about an hour of looking around I'm a little confused. :) I guess there are two questions: 1) Is it possible / does it work? 2) If yes, does anyone have a sample configuration or write-up of How You Did It[TM] to share? I've been mainly focusing on the thread from here: http://www.dovecot.org/list/dovecot/2006-June/013683.html and the wiki ACL documentation: http://wiki.dovecot.org/ACL but I'm having a hard time piecing it all together... Thanks for any and all tips! --Jo

2 5

[Dovecot] Deferring, instead of rejecting, messages when at quota
by Mike Brudenell 13 May '07

13 May '07

Greetings - I'm having trouble working out where to alter this behaviour (so please forgive me if it turns out to be an Exim thing rather than Dovecot!)... My test service is using Exim as the MTA and Dovecot's "deliver" for the LDA. We are using Maildir mailboxes with filestore quotas. When I send a large test message to myself that cannot be delivered because I don't have sufficient available quota remaining an error message is immediately returned to the sender and the incoming message discarded. The error message says this: Your message was automatically rejected by Dovecot Mail Delivery Agent. The following reason was given: Not enough disk space Instead of Exim rejecting the incoming message and generating the error response because of deliver's failure to, umm, deliver I'd like the message to be retained in Exim's queue so that it can be retried periodically. With our old/live system (Sendmail + UW IMAP's "tmail" LDA) I edited the tmail source code to return a different exit code: one that caused Sendmail to defer (re-queue) the message rather than reject it. How do I achieve this effect with Exim + deliver please? Is it a case of: * Setting something in Dovecot's configuration file (I can't spot what!); * Changing a return code in deliver's source (if so, whereabouts and to what?); * Or is there an Exim directive I need to add to the transport definition in its configuration file? With thanks, Mike B-) -- The Computing Service, University of York, Heslington, York Yo10 5DD, UK Tel:+44-1904-433811 FAX:+44-1904-433740 * Unsolicited commercial e-mail is NOT welcome at this e-mail address. *

3 3

[Dovecot] fs quota plugin and NFS
by Nicolas STRANSKY 13 May '07

13 May '07

Hi, I'm trying to use the Dovecot v1 fs quota plugin. The server uses NFS mounted volumes for INBOX and other maildir folders. The /usr/bin/quota command is working seamlessly but I get errors with the quota plugin, which gives the following logs: Apr 19 17:46:15 rouge dovecot: IMAP(xyxyxyx): quotactl(Q_GETQUOTA, nfs.xxx.yyy.org:/home) failed: No such file or directory Apr 19 17:46:18 rouge dovecot: IMAP(yxyxyxy): quotactl(Q_GETQUOTA, nfs.xxx.yyy.org:/home) failed: No such file or directory Is there a way to get it work ? Thanks, -- Nico Il faut s'entraider, c'est la loi de nature. -+- Jean de La Fontaine (1621-1695), L'Âne et le Chien (Fables VIII.17) -+-

2 3

[Dovecot] Mixing MBOX and Maildir?
by Marc Perkel 13 May '07

13 May '07

Is it possible to have some folders be mbox and some be maildir format? Why you might ask? Sometime for diagnostics I have the Exim MTA archive certain messages in MBOX format and then I want to copy them into an imao folder to look at them with Thunderbird. When I used MBOX all I had to do was copy the file into a folder and it worked. So what I'd like to do is have a folder with a special name that is MBOX and the rest be maildir. Ideally have Dovecot be smart so that if the file exists then it is assumed to be MBOX and if the directory exists then it is assumed to be maildir. Has anyone done anything like this?

6 10