- dovecot - dovecot.org

[Dovecot] Retrieve Physical Message Filename
by Tony Rutherford 29 Dec '09

29 Dec '09

Hi,Using Dovecot 1.2.9 with Maildir and Layout=FS.In a plugin, i'm overriding "update_flags":void my_mail_update_flags(struct mail *_mail, enum modify_type modify_type, enum mail_flags flags)For this email which is having its flags updated, I need to retrieve the physical filename of the message. I see where using:mail_get_special(_mail, MAIL_FETCH_UIDL_FILE_NAME, &fname);...I can retrieve part of the filename, but without the ":2" and the flags part.Is there a method I can use to get the whole filename (without having to reassemble it myself). I've looked around, and I'm guessing it exists, but I don't see it offhand.Thanks,...Tony RutherfordSoftware EngineerBlueTie Inc.arutherford(a)bluetie.com(585) 586-2000 x1040 This message is intended solely for the individual(s) to whom it is addressed. If you are not the intended recipient, any dissemination or copying is strictly prohibited. If you believe you received this message in error, please notify the sender and delete from your system. Thank you.

2 1

[Dovecot] Design: Optimizations for high-latency storage backend
by Timo Sirainen 29 Dec '09

29 Dec '09

Mainly intended for future support of key-value (etc.) databases, but this is probably useful even for regular disk I/O. By using async I/O for disk access, the OS could more intelligently do seeks on disk to read the wanted data with less latency. Problem #1: Accessing message body while searching (e.g. SEARCH BODY). The idea is that searching wouldn't block. Caller would typically use mailbox_search_next_nonblock() function, which can return "try again later" result. Whenever search sees a query that could potentially block (e.g. SEARCH BODY, or searching a header that might not be cached), it goes into a slightly different search mode: - the whole search would typically be run from beginning to end immediately, and the search result state would be stored for each message. - whenever search runs into a situation where the result can't be determined without opening message body, it issues a prefetch command for that message, marks the result for the message as "incomplete", and continues to the next message - once the search is done with all messages, it starts replying to the caller. if the next search result is incomplete at the time, the search returns "try again later". optimizations: - probably there should be prefetch limit (a queue) so that only maybe 10 requests are actually pending at a time. - probably the whole search shouldn't run immediately. maybe it should stop after having 10 requests pending. - prefetch could return only the needed parts of the message, such as "only header" or "only body" (or in future something like "message without attachments") There would be a new call, something like mailbox_search_set_nonblock_callback() that specifies a function to be called when next search result is available. The callback then would continue the search. This is similar to how IMAP SEARCH is already implemented, except currently when mailbox_search_next_nonblock() returns "try again later", Dovecot simply adds a 0 second timeout after which it's called again (that's enough for Dovecot to do some other pending work). Problem #2: Accessing index cache while searching. Cache file is typically in memory or in low-latency disk already. Anyway something similar to #1 could be implemented for it. "Make sure cache x..y bytes in cache file have been read to memory" call or something. Problem #3: Fetching message body (and uncached fields). The nice thing about lib-storage's fetching API is that the caller already specifies what fields it intends to access in mailbox_search_init(). So this code can use a similar technique than searching: - callers only use mailbox_search_next_nonblock() - which keeps returning "try again later" until the message body has been read to memory - trying to access a field that's not specified in wanted_fields in mailbox_search_init() could be (optionally) treated as a programming error and cause a crash instead of a blocking wait. Problem #4: How to drop latencies between IMAP commands? For example a (stupid) client issues commands: a FETCH 1 BODY.PEEK[] a FETCH 2 BODY.PEEK[] ..etc.. You really don't want to a delay between each command. So what Dovecot code needs to happen is to start processing the next command while waiting for previous one to finish reading the body to memory. A lot of this actually already works as intended, but there are some cases that don't, such as FETCH + SELECT (see #5). Problem #5: Opening mailbox index. So if your indexes are also stored in high-latency disk, how do you optimize getting STATUS for each mailbox? Again, a similar idea than before: Create a new mailbox_open_nonblock() function that calls a given callback function when the indexes are read to memory. During the wait it can continue processing more STATUS commands (hopefully client sends them all at once, instead of waiting for each reply before sending next STATUS). .. I think that's all of the problematic cases that could cause blocking. And the solution is always the same: Use prefetching and "try again later" status code with a callback function to be called when it's ready. Luckily there are only two functions that need the above behavior, implementing async callbacks for all functions would have made programming it horrible. The main problem will then be to optimize what and how much to prefetch. For example in the cache file case, should the cache be read to memory when mailbox is opened, or should it be delayed until something is actually wanted to be read from it? Should all of it be read, or should Dovecot attempt to be smarter to reduce memory usage and I/O bandwidth, so e.g. if the cache is wanted to be accessed to 3rd last message, maybe just read the cache file beginning from that message to EOF? Transaction log files have a similar problem. Actually all of this is similar to how Dovecot attempts to cache them to memory when using mmap_disable=yes, so it's not really a new problem.

2 2

[Dovecot] Design: Mailbox list indexes
by Timo Sirainen 29 Dec '09

29 Dec '09

I drove over 2000 miles in recent days, so I had some time to think about future Dovecot things. :) Maybe for v2.1 or v3.0, whichever I happen to release next. Here's the first thought: The main idea behind mailbox list indexes was to make getting STATUS information for mailboxes from a single index file. The code was already written for Dovecot v1.1, but it was disabled because of bugs I never got around to fixing. It also was maybe a bit overdesigned to make it fast with thousands of mailboxes. There were two indexes: Mailbox UID -> name map index, and UID -> status info index. I don't think most people have that many mailboxes. Using two index files just makes the common case slower. So the new mailbox list index could be a single index (read/written to using the same index file API as for mails). Index's header would contain an ID -> string map. So basically an array of {uint32_t id, char nul-terminated-name[]} records. Each index record would contain: - mailbox UID number (32 bit) - ID number (32 bit), pointing to a string in the header. That'll be this mailbox's name. - parent mailbox UID (e.g. if parent's name is "foo" and our name is "bar", the mailbox is "foo/bar") - mailbox global UID (128 bit) - mailbox flags (\Noselect, \Nonexistent) - STATUS information: message/unseen counts, last-non-recent-uid (for recent count), uidvalidity, uidnext, highest-modseq - storage backend specific timestamps So typically the whole index would be read into memory and the mailbox tree would be generated based on the parent UID pointers. If there are any \Noselect or \Nonexistent mailboxes, they just have their GUIDs and STATUS info zeroed out. Each time a mailbox is created or renamed, a new name ID is added to the header with the given name. I don't think it's worth it to try to reuse old IDs. Unused names (after delete/rename) will be dropped sometimes later. Whenever STATUS information changes while accessing mailbox, the STATUS info is updated in mailbox list index. Also unless mailbox_list_index_dirty_syncs=yes (or something similar), the index can't be trusted by itself. So before trusting the index, Dovecot needs to check if the index is up to date by comparing the timestamps to what exists in filesystem. For example with Maildir there would be new/ and cur/ directories' mtimes. Then there's the problem of how to notice externally added/deleted/renamed mailboxes. It'll be immediately noticed if such mailbox is tried to be opened, but what about LIST replies? One possibility would be to refresh the list once when session begins and after that just have LIST return indexed mailboxes. Another would be to keep track of directory mtimes and keep stat()ing the dirs to see if something changes. The latter is probably more trouble than worth.. Besides caching LIST and STATUS replies, there's one additional thing I thought of that the mailbox list indexes will be useful for: atomic mailbox list updates. Even if the backend doesn't support e.g. atomic RENAME (such as Maildir++), this can be made practically atomic using the list index. CREATE, RENAME and DELETE commands begin by writing and committing a record to the mailbox list index about the change. Then they'll start the actual change, and finally write to the index that the change is complete. If it crashed in the middle, the next session would finish it. If another session tries to do a conflicting change before the change is finished, it would wait for the previous one to finish. (Probably practically the index would be write-locked and all changes would wait for previous ones to finish.) And the final functionality of this mailbox list index is for the future support of key-value (etc.) databases: The mailbox list index would be the authoritative list of mailboxes. It wouldn't be stored anywhere else.

1 0

[Dovecot] checking all folders for new mail (IMAP STATUS command)
by Florian Effenberger 29 Dec '09

29 Dec '09

Hello, I use Thunderbird as IMAP client and do a lot of server-side filtering with Sieve scripts. To be notified of new mails, I want to check all folders (a lot of them...) at once for new mails, and not only the inbox. According to http://kb.mozillazine.org/Checking_for_new_messages_in_other_folders_-_Thun… it should be sufficient to enable one setting -- given that the IMAP server returns the STATUS command correctly. I still run Dovecot 1.0 (from Debian 4.0), and it seems this version doesn't support the IMAP STATUS command, as I have to explictly query each folder with the second mentioned setting. What is the recommended way of checking all folders for new mails? Is there any server-side setting I can activate? Do newer versions of Dovecot support the STATUS command in the way Thunderbird expects it? Or does Dovecot even offer an option like the mentioned IMAP_CHECK_ALL_FOLDERS=1 and IMAP_ENHANCEDIDLE=1 in Courier? Sorry for my maybe dumb question... :-) Thanks, Florian

6 13

Re: [Dovecot] Dovecot-deliver sometimes fails when message set to discard via sieve
by Tobias Lang 29 Dec '09

29 Dec '09

Am 28.12.2009 um 21:51 schrieb Steve Wagner: > Sorry it took some time to get in to this failing situation. > > Procmail log--> > > procmail: [29952] Mon Dec 28 21:36:33 2009 > procmail: Match on "< 256000" > procmail: Locking "spamassassin.lock" > procmail: Executing "/usr/bin/spamc,-U,/var/run/spamd.sock" > procmail: [29952] Mon Dec 28 21:36:35 2009 > procmail: Unlocking "spamassassin.lock" > procmail: No match on "^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*" > procmail: No match on "^X-Spam-Status: YES" > procmail: No match on "^List-Id: .*debian-arm.lists.debian.org" > procmail: No match on "^^rom[ ]" > procmail: Executing "/opt/dovecot/libexec/dovecot/deliver" > procmail: Error while writing to "/opt/dovecot/libexec/dovecot/deliver" > procmail: Assigning "LASTFOLDER=/opt/dovecot/libexec/dovecot/deliver" > procmail: Assigning > "LASTFOLDER=/home/lanwin/Maildir/new/1262032593.29952_2.vs5923" > procmail: Notified comsat: > "lanwin@0:/home/lanwin/Maildir/new/1262032593.29952_2.vs5923" > From KeineAntwortAdresse(a)web.de Mon Dec 28 21:36:33 2009 > Subject: discoTEL > =?ISO-8859-1?Q?Tarif=3A_F=FCr_7=2C5_Cent=2FMin=2E_telefonier > Folder: /home/lanwin/Maildir/new/1262032593.29952_2.vs5923 > 41096 > procmail: Unlocking "/home/lanwin/Maildir/.lockmail" > > Dovecot Deliver Log--> > > 2009-12-28 21:36:35 deliver(lanwin): Info: sieve: > msgid=<E1NPMKF-00014k-00(a)mxnl04.web.de>: marked message to be discarded > if not explicitly delivered (discard action) Does it always fail, if you try to do a discard, just sometimes? I am using Dovecot 1.2.9 "dovecot-1.2-sieve-0.1.14" and discarding email seems to work fine. The things I can think of are: 1) Wrong privileges for deleting. What user does procmail / deliver run as? 2) Old Bug in Sieve (http://www.dovecot.org/list/dovecot/2007-July/024290.html) 3) If it fails just sometimes, what is the difference between discarded and nondiscarded mails. If this does not turn up something, I do not have an idea what is going wrong. Only bugfix I then can you provide with, is not to discard but to move those files to a folder 'SPAM' and delete them by hand. Tobi

1 0

[Dovecot] Shared mailboxes do not support subfolders
by Nick Rosier 29 Dec '09

29 Dec '09

Hi, I'm currently running Dovecot 1.2.8 with dbox mailboxes. I've configured the use of shared mailboxes. I'm able to only see the top-level mailboxes: e.g. Inbox LogWatch Trash Sent but not LogWatch/Server1 LogWatch/Server2 Is this a know limitation of a bug? Rgds, N.

2 3

[Dovecot] sieve vs deliver -m
by Frank Cusack 29 Dec '09

29 Dec '09

I'm invoking deliver through postfix with a dovecot transport in master.cf, using the -m ${extension} argument. If I have no sieve script, or a sieve script with an implicit keep, the mail gets correctly filed into the -m mailbox. However if the sieve script has an explicit discard, this gets honored if it doesn't match another sieve action. Shouldn't -m take precedence? I would say that if an extension is present any other sieve action should be ignored, perhaps just sieve_before should be run then the message filed per the extension. -frank

2 7

[Dovecot] Fwd: Dovecot-deliver sometimes fails when message set to discard via sieve
by Tobias Lang 28 Dec '09

28 Dec '09

Anfang der weitergeleiteten E-Mail: > Von: Steve Wagner <lanwin(a)lanwin.de> > Datum: 28. Dezember 2009 21:51:39 MEZ > An: Tobias Lang <tobias.lang(a)evoscout.de> > Betreff: Re: [Dovecot] Dovecot-deliver sometimes fails when message set to discard via sieve > > Sorry it took some time to get in to this failing situation. > > Procmail log--> > > procmail: [29952] Mon Dec 28 21:36:33 2009 > procmail: Match on "< 256000" > procmail: Locking "spamassassin.lock" > procmail: Executing "/usr/bin/spamc,-U,/var/run/spamd.sock" > procmail: [29952] Mon Dec 28 21:36:35 2009 > procmail: Unlocking "spamassassin.lock" > procmail: No match on "^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*" > procmail: No match on "^X-Spam-Status: YES" > procmail: No match on "^List-Id: .*debian-arm.lists.debian.org" > procmail: No match on "^^rom[ ]" > procmail: Executing "/opt/dovecot/libexec/dovecot/deliver" > procmail: Error while writing to "/opt/dovecot/libexec/dovecot/deliver" > procmail: Assigning "LASTFOLDER=/opt/dovecot/libexec/dovecot/deliver" > procmail: Assigning > "LASTFOLDER=/home/lanwin/Maildir/new/1262032593.29952_2.vs5923" > procmail: Notified comsat: > "lanwin@0:/home/lanwin/Maildir/new/1262032593.29952_2.vs5923" > From KeineAntwortAdresse(a)web.de Mon Dec 28 21:36:33 2009 > Subject: discoTEL > =?ISO-8859-1?Q?Tarif=3A_F=FCr_7=2C5_Cent=2FMin=2E_telefonier > Folder: /home/lanwin/Maildir/new/1262032593.29952_2.vs5923 > 41096 > procmail: Unlocking "/home/lanwin/Maildir/.lockmail" > > Dovecot Deliver Log--> > > 2009-12-28 21:36:35 deliver(lanwin): Info: sieve: > msgid=<E1NPMKF-00014k-00(a)mxnl04.web.de>: marked message to be discarded > if not explicitly delivered (discard action) > > Dose not look better to me. > > Steve > > Am 28.12.2009 18:06, schrieb Tobias Lang: >> >> Am 28.12.2009 um 15:09 schrieb Steve Wagner: >> >>> [snip] >>> >>> :0 w >>> | /opt/dovecot/libexec/dovecot/deliver >> >> Looks uncritical to me. Although there is no default rule at the end of >> .procmailrc, Dovecot deliver and Sieve should reroute everything not >> catched by a rule to INBOX. >> >> Did Procmails "VERBOSE=on" reveal something? >> >> Tobi >>

1 0

[Dovecot] managesieve error
by Lampa 26 Dec '09

26 Dec '09

Hello, when using sieve-php and using LOGIN auth, getting: 2009-11-21T15:13:48+0100 managesieve-login: Panic: file client.c: line 449 (client_set_auth_waiting): assertion failed: (client->to_auth_waiting == NULL) 2009-11-21T15:13:48+0100 dovecot: Error: child 8474 (login) killed with signal 6 (core dumps disabled) (latest ip=127.0.0.1) I'm using dovecot 1.2.6 from package debian testing (1:1.2.6-1) -- Lampa

2 1

[Dovecot] basic understanding of imap in dovecot
by dovecot＠paz.bz 24 Dec '09

24 Dec '09

I have always used a pop3 daemon, and switching to dovecot has been my introduction to imap. Imap is foreign to me. I am wondering if I am a little off-track here, and would like some clarification. My MTA (exim) delivers email to: /mail/user dovecot, sucks that mail from /mail/user TO /home/user/mail/<filenamesoffolders> dovecot IMAP further uses /home/user/mail/.imap/folders/{.index / .index.cache / .index.log} the mail folder which exim uses doesn't seem to get emptied by dovecot when it sucks the email into /home/user/mail/<filenamesoffolders> when using IMAP. If POP3 is used via dovecot, then it appears that the interaction is directly to the original exim folder and that it does indeed empty (or at least it has the "FOLDER INTERNAL DATA" entry). I suppose my question is, is there a way to have dovecot empty/parse the original exim mailbox so that the customer email isn't sitting in two spots, taking up double the space. This would be in the IMAP environment only, as POP3 seems to be fine. Any online references available to help a newbee understand IMAP? Thanks!

3 2