December 2009 - dovecot

[Dovecot] multiple attribute templates?
by Mariano Absatz 29 Dec '09

29 Dec '09

Hi, I've been using an attribute template for some time without any problems. My accounts are on an LDAP server and I have an attribute maildropPath which is the maildir path for the user relative to /home/vmail That is, user "mabsatz(a)example.org" has, in attribute maildropPath the string "example.org/mabsatz", meaning that the maildir is in /home/vmail/example.org/mabsatz This is configured like this (taken from http://wiki.dovecot.org/AuthDatabase/LDAP/Userdb) user_attrs = maildropPath=mail=maildir:/home/vmail/%$ pass_attrs = uid=user,userPassword=password,maildropPath=userdb_mail=maildir:/home/vmail/%$ and it works OK. Now I want to start using sieve and, since all accounts are virtual, I want to use a subdirectory "sieve" within the maildir to hold the scripts, so I need (I think) a userdb home to point to the same path... being a monkey I tried to imitate (or rather, extend) the previous configuration like this: user_attrs = maildropPath=mail=maildir:/home/vmail/%$,maildropPath=home=/home/vmail/%$ pass_attrs = uid=user,userPassword=password,maildropPath=userdb_mail=maildir:/home/vmail/%$,maildropPath=userdb_home=/home/vmail/%$ Alas... this didn't work... that is, apparently, the "home" attribute works OK (I got my scripts on the /home/vmail/example.org/mabsatz/sieve directory), but the "mail" attribute got lost and mail is being delivered to /home/vmail/mabsatz. Is there a way to use more than one attribute template AND that both use the same LDAP attribute? Otherwise, can someone think of a better way to do what I'm trying to do? I'm using dovecot 1.1.11 from the standard ubuntu 9.04 package (dovecot-imapd_1.1.11-0ubuntu4.1). -- Mariano Absatz - "El Baby" el.baby(a)gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Real programmers use: cat > a.out http://xkcd.com/378/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org

2 1

[Dovecot] Deliver EX_TEMPFAIL's without giving any information
by Denis Khromov 29 Dec '09

29 Dec '09

Hi all. I've had a hard time trying to find out why deliver isn't working after I've updated dovecot from v1.11 to v1.2.8. It just gave me EX_TEMPFAIL without any info in the logs. My deliver was setuid-root. Once I've made a simple shell wrapper script for the deliver executable which saves deliver's stdout+stderr, I've found the reason: /usr/local/libexec/dovecot/deliver must not be both world-executable and setuid-root. This allows root exploits. See http://wiki.dovecot.org/LDA#multipleuids Did a 'chmod o-x deliver' and fixed groups/owners and now everything works as it should. I think this error message should go to log files, not just to stdout/stderr. And it's worth to describe this behaviour in the Wiki. Cheers, Denis

2 1

[Dovecot] dovecot -n output of (default) is confusing me
by john.blue＠cox.net 29 Dec '09

29 Dec '09

Hello, I'm new to dovecot and I've been taking my time examining what makes dovecot work. If I have imap and pop3 listed as protocols, dovecot -n outputs: mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 I don't understand why imap is listed twice once as (default), and the other without (default). As a secondary question, why would you want a default of imap as opposed to pop3? Thanks. John

2 1

[Dovecot] Log rotation?
by Beto Moreno 29 Dec '09

29 Dec '09

Hi people. I had read dovecot logging info, they say something about using log rotation to help us rotate our log, with this setup: # dovecot SIGUSR1: Re-opens the log files. /var/log/dovecot*.log { missingok notifempty delaycompress sharedscripts postrotate /bin/kill -USR1 `cat /var/run/dovecot/master.pid 2>/dev/null` 2> /dev/null || true endscript } Well, my question is simple, Do we need to restart the service each time logrotation runs? In my case, is wrong this? var/log/dovecot*.log { weekly rotate 8 notifempty missingok copytruncate create 600 root start 0 } I have 2 dovecot logs: dovecot-info.log dovecot.log dovecot-info.log is the biggest, 5MB by week, is to much? Do I have to change our log rotation to "daily"? Centos 5.4 updated, dovecot 1.0.7-7.el5, thanks!!!

3 3

[Dovecot] Retrieve Physical Message Filename
by Tony Rutherford 29 Dec '09

29 Dec '09

Hi,Using Dovecot 1.2.9 with Maildir and Layout=FS.In a plugin, i'm overriding "update_flags":void my_mail_update_flags(struct mail *_mail, enum modify_type modify_type, enum mail_flags flags)For this email which is having its flags updated, I need to retrieve the physical filename of the message. I see where using:mail_get_special(_mail, MAIL_FETCH_UIDL_FILE_NAME, &fname);...I can retrieve part of the filename, but without the ":2" and the flags part.Is there a method I can use to get the whole filename (without having to reassemble it myself). I've looked around, and I'm guessing it exists, but I don't see it offhand.Thanks,...Tony RutherfordSoftware EngineerBlueTie Inc.arutherford(a)bluetie.com(585) 586-2000 x1040 This message is intended solely for the individual(s) to whom it is addressed. If you are not the intended recipient, any dissemination or copying is strictly prohibited. If you believe you received this message in error, please notify the sender and delete from your system. Thank you.

2 1

[Dovecot] Design: Optimizations for high-latency storage backend
by Timo Sirainen 29 Dec '09

29 Dec '09

Mainly intended for future support of key-value (etc.) databases, but this is probably useful even for regular disk I/O. By using async I/O for disk access, the OS could more intelligently do seeks on disk to read the wanted data with less latency. Problem #1: Accessing message body while searching (e.g. SEARCH BODY). The idea is that searching wouldn't block. Caller would typically use mailbox_search_next_nonblock() function, which can return "try again later" result. Whenever search sees a query that could potentially block (e.g. SEARCH BODY, or searching a header that might not be cached), it goes into a slightly different search mode: - the whole search would typically be run from beginning to end immediately, and the search result state would be stored for each message. - whenever search runs into a situation where the result can't be determined without opening message body, it issues a prefetch command for that message, marks the result for the message as "incomplete", and continues to the next message - once the search is done with all messages, it starts replying to the caller. if the next search result is incomplete at the time, the search returns "try again later". optimizations: - probably there should be prefetch limit (a queue) so that only maybe 10 requests are actually pending at a time. - probably the whole search shouldn't run immediately. maybe it should stop after having 10 requests pending. - prefetch could return only the needed parts of the message, such as "only header" or "only body" (or in future something like "message without attachments") There would be a new call, something like mailbox_search_set_nonblock_callback() that specifies a function to be called when next search result is available. The callback then would continue the search. This is similar to how IMAP SEARCH is already implemented, except currently when mailbox_search_next_nonblock() returns "try again later", Dovecot simply adds a 0 second timeout after which it's called again (that's enough for Dovecot to do some other pending work). Problem #2: Accessing index cache while searching. Cache file is typically in memory or in low-latency disk already. Anyway something similar to #1 could be implemented for it. "Make sure cache x..y bytes in cache file have been read to memory" call or something. Problem #3: Fetching message body (and uncached fields). The nice thing about lib-storage's fetching API is that the caller already specifies what fields it intends to access in mailbox_search_init(). So this code can use a similar technique than searching: - callers only use mailbox_search_next_nonblock() - which keeps returning "try again later" until the message body has been read to memory - trying to access a field that's not specified in wanted_fields in mailbox_search_init() could be (optionally) treated as a programming error and cause a crash instead of a blocking wait. Problem #4: How to drop latencies between IMAP commands? For example a (stupid) client issues commands: a FETCH 1 BODY.PEEK[] a FETCH 2 BODY.PEEK[] ..etc.. You really don't want to a delay between each command. So what Dovecot code needs to happen is to start processing the next command while waiting for previous one to finish reading the body to memory. A lot of this actually already works as intended, but there are some cases that don't, such as FETCH + SELECT (see #5). Problem #5: Opening mailbox index. So if your indexes are also stored in high-latency disk, how do you optimize getting STATUS for each mailbox? Again, a similar idea than before: Create a new mailbox_open_nonblock() function that calls a given callback function when the indexes are read to memory. During the wait it can continue processing more STATUS commands (hopefully client sends them all at once, instead of waiting for each reply before sending next STATUS). .. I think that's all of the problematic cases that could cause blocking. And the solution is always the same: Use prefetching and "try again later" status code with a callback function to be called when it's ready. Luckily there are only two functions that need the above behavior, implementing async callbacks for all functions would have made programming it horrible. The main problem will then be to optimize what and how much to prefetch. For example in the cache file case, should the cache be read to memory when mailbox is opened, or should it be delayed until something is actually wanted to be read from it? Should all of it be read, or should Dovecot attempt to be smarter to reduce memory usage and I/O bandwidth, so e.g. if the cache is wanted to be accessed to 3rd last message, maybe just read the cache file beginning from that message to EOF? Transaction log files have a similar problem. Actually all of this is similar to how Dovecot attempts to cache them to memory when using mmap_disable=yes, so it's not really a new problem.

2 2

[Dovecot] Design: Mailbox list indexes
by Timo Sirainen 29 Dec '09

29 Dec '09

I drove over 2000 miles in recent days, so I had some time to think about future Dovecot things. :) Maybe for v2.1 or v3.0, whichever I happen to release next. Here's the first thought: The main idea behind mailbox list indexes was to make getting STATUS information for mailboxes from a single index file. The code was already written for Dovecot v1.1, but it was disabled because of bugs I never got around to fixing. It also was maybe a bit overdesigned to make it fast with thousands of mailboxes. There were two indexes: Mailbox UID -> name map index, and UID -> status info index. I don't think most people have that many mailboxes. Using two index files just makes the common case slower. So the new mailbox list index could be a single index (read/written to using the same index file API as for mails). Index's header would contain an ID -> string map. So basically an array of {uint32_t id, char nul-terminated-name[]} records. Each index record would contain: - mailbox UID number (32 bit) - ID number (32 bit), pointing to a string in the header. That'll be this mailbox's name. - parent mailbox UID (e.g. if parent's name is "foo" and our name is "bar", the mailbox is "foo/bar") - mailbox global UID (128 bit) - mailbox flags (\Noselect, \Nonexistent) - STATUS information: message/unseen counts, last-non-recent-uid (for recent count), uidvalidity, uidnext, highest-modseq - storage backend specific timestamps So typically the whole index would be read into memory and the mailbox tree would be generated based on the parent UID pointers. If there are any \Noselect or \Nonexistent mailboxes, they just have their GUIDs and STATUS info zeroed out. Each time a mailbox is created or renamed, a new name ID is added to the header with the given name. I don't think it's worth it to try to reuse old IDs. Unused names (after delete/rename) will be dropped sometimes later. Whenever STATUS information changes while accessing mailbox, the STATUS info is updated in mailbox list index. Also unless mailbox_list_index_dirty_syncs=yes (or something similar), the index can't be trusted by itself. So before trusting the index, Dovecot needs to check if the index is up to date by comparing the timestamps to what exists in filesystem. For example with Maildir there would be new/ and cur/ directories' mtimes. Then there's the problem of how to notice externally added/deleted/renamed mailboxes. It'll be immediately noticed if such mailbox is tried to be opened, but what about LIST replies? One possibility would be to refresh the list once when session begins and after that just have LIST return indexed mailboxes. Another would be to keep track of directory mtimes and keep stat()ing the dirs to see if something changes. The latter is probably more trouble than worth.. Besides caching LIST and STATUS replies, there's one additional thing I thought of that the mailbox list indexes will be useful for: atomic mailbox list updates. Even if the backend doesn't support e.g. atomic RENAME (such as Maildir++), this can be made practically atomic using the list index. CREATE, RENAME and DELETE commands begin by writing and committing a record to the mailbox list index about the change. Then they'll start the actual change, and finally write to the index that the change is complete. If it crashed in the middle, the next session would finish it. If another session tries to do a conflicting change before the change is finished, it would wait for the previous one to finish. (Probably practically the index would be write-locked and all changes would wait for previous ones to finish.) And the final functionality of this mailbox list index is for the future support of key-value (etc.) databases: The mailbox list index would be the authoritative list of mailboxes. It wouldn't be stored anywhere else.

1 0

[Dovecot] checking all folders for new mail (IMAP STATUS command)
by Florian Effenberger 29 Dec '09

29 Dec '09

Hello, I use Thunderbird as IMAP client and do a lot of server-side filtering with Sieve scripts. To be notified of new mails, I want to check all folders (a lot of them...) at once for new mails, and not only the inbox. According to http://kb.mozillazine.org/Checking_for_new_messages_in_other_folders_-_Thun… it should be sufficient to enable one setting -- given that the IMAP server returns the STATUS command correctly. I still run Dovecot 1.0 (from Debian 4.0), and it seems this version doesn't support the IMAP STATUS command, as I have to explictly query each folder with the second mentioned setting. What is the recommended way of checking all folders for new mails? Is there any server-side setting I can activate? Do newer versions of Dovecot support the STATUS command in the way Thunderbird expects it? Or does Dovecot even offer an option like the mentioned IMAP_CHECK_ALL_FOLDERS=1 and IMAP_ENHANCEDIDLE=1 in Courier? Sorry for my maybe dumb question... :-) Thanks, Florian

6 13

Re: [Dovecot] Dovecot-deliver sometimes fails when message set to discard via sieve
by Tobias Lang 29 Dec '09

29 Dec '09

Am 28.12.2009 um 21:51 schrieb Steve Wagner: > Sorry it took some time to get in to this failing situation. > > Procmail log--> > > procmail: [29952] Mon Dec 28 21:36:33 2009 > procmail: Match on "< 256000" > procmail: Locking "spamassassin.lock" > procmail: Executing "/usr/bin/spamc,-U,/var/run/spamd.sock" > procmail: [29952] Mon Dec 28 21:36:35 2009 > procmail: Unlocking "spamassassin.lock" > procmail: No match on "^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*" > procmail: No match on "^X-Spam-Status: YES" > procmail: No match on "^List-Id: .*debian-arm.lists.debian.org" > procmail: No match on "^^rom[ ]" > procmail: Executing "/opt/dovecot/libexec/dovecot/deliver" > procmail: Error while writing to "/opt/dovecot/libexec/dovecot/deliver" > procmail: Assigning "LASTFOLDER=/opt/dovecot/libexec/dovecot/deliver" > procmail: Assigning > "LASTFOLDER=/home/lanwin/Maildir/new/1262032593.29952_2.vs5923" > procmail: Notified comsat: > "lanwin@0:/home/lanwin/Maildir/new/1262032593.29952_2.vs5923" > From KeineAntwortAdresse(a)web.de Mon Dec 28 21:36:33 2009 > Subject: discoTEL > =?ISO-8859-1?Q?Tarif=3A_F=FCr_7=2C5_Cent=2FMin=2E_telefonier > Folder: /home/lanwin/Maildir/new/1262032593.29952_2.vs5923 > 41096 > procmail: Unlocking "/home/lanwin/Maildir/.lockmail" > > Dovecot Deliver Log--> > > 2009-12-28 21:36:35 deliver(lanwin): Info: sieve: > msgid=<E1NPMKF-00014k-00(a)mxnl04.web.de>: marked message to be discarded > if not explicitly delivered (discard action) Does it always fail, if you try to do a discard, just sometimes? I am using Dovecot 1.2.9 "dovecot-1.2-sieve-0.1.14" and discarding email seems to work fine. The things I can think of are: 1) Wrong privileges for deleting. What user does procmail / deliver run as? 2) Old Bug in Sieve (http://www.dovecot.org/list/dovecot/2007-July/024290.html) 3) If it fails just sometimes, what is the difference between discarded and nondiscarded mails. If this does not turn up something, I do not have an idea what is going wrong. Only bugfix I then can you provide with, is not to discard but to move those files to a folder 'SPAM' and delete them by hand. Tobi

1 0

[Dovecot] Shared mailboxes do not support subfolders
by Nick Rosier 29 Dec '09

29 Dec '09

Hi, I'm currently running Dovecot 1.2.8 with dbox mailboxes. I've configured the use of shared mailboxes. I'm able to only see the top-level mailboxes: e.g. Inbox LogWatch Trash Sent but not LogWatch/Server1 LogWatch/Server2 Is this a know limitation of a bug? Rgds, N.

2 3