[Dovecot] outstanding issues with 0.99.13
hi list,
i'm in the process of testing dovecot before a hopeful switchover from uw-imap/qpopper to dovecot, primarily to get off of the uw-imap/mbox combo and moved to Maildir. i think we've worked out the logistics to avoid any service downtime, with only a short window of a few minutes per user for when we convert from mbox to maildir format.
however, during testing a few issues have come up, some of which i've been able to work around, others which i haven't. for those that i have been able to work around, i'll include my solution so that fellow googlers may profit, and for those that i haven't i'd really appreciate your feedback.
logistical problems:
1 - automagic detection and uw-imap inbox in ~/mbox. i'm not sure if this is a default characteristic or if it's specific to us, but our spool is in /var/mail/foo and inbox in ~/mbox. the automagic Maildir/mbox detection doesn't find the "mbox", and i can't tell it where it is because then it doesn't look for the Maildir. solution: i wrote a small patch to dovecot's autodetect code to help it find the location of our mailboxes. if anyone would like a copy of said patch i can send it off list.
2 - uw-imap's spool/inbox message moving. i read from the wiki that this isn't a feature currently being worked on. solution: deliver all mail to the same place. since we use procmail as our MDA, this isn't too hard of a problem. basically we redirect mail to the inbox and redeliver everything in the spool.
3 - supporting multi-homed services + ssl. for historical reasons we provide services from multiple hostnames (mail.foo, imap.foo, etc). with ssl, that means a different cert for each, otherwise some clients will break. solution: we're spawning off three seperate dovecot instances, each using a different config file. not the most graceful, but it works...
application problems:
4 - dovecot imap + ms lookout 2002 + deleting messages. i'm not sure if this is an app or server side problem. basically, a user will delete a message, only to have it reappear in the mailbox a short time (few minutes) later. if i monitor the state of the mbox file, no changes are made to it until the user selects "empty trash", after which point they are expunged. looking at a protocol dump, i see the client setting the deleted flag on the message when they originally delete it. here's an excerpt:
<snip> tg7i OK Idle completed.
6gzd UID STORE 30021 +FLAGS (\Deleted \Seen)
- 51 FETCH (FLAGS (\Deleted \Seen \Recent) UID 30021) 6gzd OK Store completed.
stpw IDLE <snip>
i'm not too familiar with imap, but it looks like the server is
caching the deletion request and then forgetting later forgetting
about it? i have all the "client workarounds" enabled, fwiw.solution: ???
5 - mac os x + eudora + deleting mail. a different problem alltogether. in eudora deleting a mail seems to immediately move it to the "trash" mailbox, so the above problem does not occur. what *does* happen is that if a user rapidly deletes messages one-by-one, an error message pops up about UID's not existing. i haven't gotten a full description of this problem, but here's an excerpt from what i think is the symptom:
<snip> A00109 UID COPY 30002 Trash
A00109 OK Copy completed.
A00110 UID STORE 30002 +Flags.silent (\Deleted)
A00110 OK Store completed.
A00111 EXPUNGE
- 31 EXPUNGE A00111 OK Expunge completed. <snip> and later: <snip> A00040 UID FETCH 30002 (UID BODYSTRUCTURE)
A00040 NO Some of the requested messages no longer exist.
A00041 UID FETCH 30002 RFC822.SIZE
A00041 NO Some of the requested messages no longer exist.
A00042 UID FETCH 30002 (BODY.PEEK[])
A00042 NO Some of the requested messages no longer exist. <snip> which i suppose is correct that it doesn't exist (since it was moved out of the inbox). however, eudora does not have this problem with uw-imap, and i'm wondering what might be causing it? solution: ???
okay, i think that's everything that i've ran into. we're still pushing ahead with this because we need to get the hell off of uw-imap/mbox, but i don't think we'll be able to stick with dovecot if these are unresolvable issues. in that case, we'll probably use dovecot with these minor nuisances while we transition, and then perhaps use courier in the long run. hopefully that won't have to be the case though :)
if anyone has any experience specifically with "5" and "5", i'd really, really like to hear about it. in the meantime, i'll continue to test other things, and also see if the problem also persists when mail is stored in Maildir format.
anyway, thanks for your time! sean
On Wed, 2005-02-09 at 11:48 -0500, sean finney wrote:
3 - supporting multi-homed services + ssl. for historical reasons we provide services from multiple hostnames (mail.foo, imap.foo, etc). with ssl, that means a different cert for each, otherwise some clients will break. solution: we're spawning off three seperate dovecot instances, each using a different config file. not the most graceful, but it works...
With 1.0-tests you can create server { .. } blocks so there's only one dovecot master process, but still multiple login processes. I'm thinking about removing this feature and rather create a separate configuration process which allows dynamic configuration based on different rules. For example in your case you'd set cert file based on target IP address.
I'm not really sure how that dynamic configuration lookup would work. At least with SQL it would be pretty simple because you can put the rules into the SQL queries, but I'm not sure if I should support anything else..
I was also thinking about modifying dovecot-auth so that you could include whatever configuration settings there to make user-specific settings. Especially the mail-setting wouldn't be a special case anymore then.
4 - dovecot imap + ms lookout 2002 + deleting messages. i'm not sure if this is an app or server side problem. basically, a user will delete a message, only to have it reappear in the mailbox a short time (few minutes) later. if i monitor the state of the mbox file, no changes are made to it until the user selects "empty trash", after which point they are expunged. looking at a protocol dump, i see the client setting the deleted flag on the message when they originally delete it. here's an excerpt:
<snip> tg7i OK Idle completed.
6gzd UID STORE 30021 +FLAGS (\Deleted \Seen)
- 51 FETCH (FLAGS (\Deleted \Seen \Recent) UID 30021) 6gzd OK Store completed.
stpw IDLE <snip>
i'm not too familiar with imap, but it looks like the server is caching the deletion request and then forgetting later forgetting about it? i have all the "client workarounds" enabled, fwiw.solution: ???
So later the \Deleted flag is lost? And this is with maildir? Is the maildir filename updated by adding 'T' flag to it?
A00040 UID FETCH 30002 (UID BODYSTRUCTURE) A00040 NO Some of the requested messages no longer exist.
This is actually a bug, UID FETCH should never fail.
1.0-tests also has the same problem, except instead of replying with "NO" it disconnects the client. This only happens when one connection expunges the mail and another tries to fetch it but notices that it's gone.
I'll try to get this fixed for 0.99.14 and next 1.0-test.
On Wed, 9 Feb 2005, Timo Sirainen wrote:
With 1.0-tests you can create server { .. } blocks so there's only one dovecot master process, but still multiple login processes. I'm thinking about removing this feature and rather create a separate configuration process which allows dynamic configuration based on different rules. For example in your case you'd set cert file based on target IP address.
I'm not really sure how that dynamic configuration lookup would work. At least with SQL it would be pretty simple because you can put the rules into the SQL queries, but I'm not sure if I should support anything else..
you could include config file fragements from the fs...
specifically suppose %a expands to the local ip address, %p the local port, and %d the reverse dns result... then you could have a vhosts config element like this:
vhosts = /etc/dovecot/vhosts/%a.%p:/etc/dovecot/vhosts/%a
which would look for .../a.b.c.d.port first, if that fails then look for .../a.b.c.d ...
-dean
sean finney wrote:
1 - automagic detection and uw-imap inbox in ~/mbox. i'm not sure if this is a default characteristic or if it's specific to us, but our spool is in /var/mail/foo and inbox in ~/mbox. the automagic Maildir/mbox detection doesn't find the "mbox", and i can't tell it where it is because then it doesn't look for the Maildir. solution: i wrote a small patch to dovecot's autodetect code to help it find the location of our mailboxes. if anyone would like a copy of said patch i can send it off list.
I tried a combined mbox-for-inbox/Maildir-for-other setup once, only to find it's not an option in the 0.99 series. You can do it in 1.0, apparently, but I'm yet to take the plunge :)
2 - uw-imap's spool/inbox message moving. i read from the wiki that this isn't a feature currently being worked on. solution: deliver all mail to the same place. since we use procmail as our MDA, this isn't too hard of a problem. basically we redirect mail to the inbox and redeliver everything in the spool.
Umm... not being familiar with uw-imap's features, I'm not clear on what you're asking here. My understanding of the flow of things is your MDA (at least) delivers to your INBOX. More advanced MDAs will deliver to multiple folders, but your MUA needs to know to check these places for new mail. I know Thunderbird, by default, only checks INBOX, but can be told to check other folders, or all, for new mail.
-- Curtis Maloney cmaloney@cardgate.net
hi guys,
thanks for all the responses...
On Wed, Feb 09, 2005 at 11:48:30PM +0200, Timo Sirainen wrote:
With 1.0-tests you can create server { .. } blocks so there's only one dovecot master process, but still multiple login processes. I'm thinking about removing this feature and rather create a separate configuration process which allows dynamic configuration based on different rules. For example in your case you'd set cert file based on target IP address.
a few ideas that bounce off the top of my head:
- add a couple of expandable macros for hostname/ip-address
- add an "include" command in the config file syntax (all instances could then include a common config with all the identical stuff).
- add some default locations for cert files if they're not specified that is based on the ip address (uw-imap does this).
6gzd UID STORE 30021 +FLAGS (\Deleted \Seen)
So later the \Deleted flag is lost? And this is with maildir? Is the maildir filename updated by adding 'T' flag to it?
this is with mbox, actually. on friday i'll convert my outlook tester to Maildir and verify whether this happens then as well.
A00040 UID FETCH 30002 (UID BODYSTRUCTURE) A00040 NO Some of the requested messages no longer exist.
This is actually a bug, UID FETCH should never fail.
1.0-tests also has the same problem, except instead of replying with "NO" it disconnects the client. This only happens when one connection expunges the mail and another tries to fetch it but notices that it's gone.
looking at the packet dump, this was in fact two seperate connections from the same client program. if he takes his time (click, pause, click, pause) it doesn't seem to happen, it only occurs when he deletes messages in rapid succession (click, click, click, click)
I'll try to get this fixed for 0.99.14 and next 1.0-test.
awesome. in the meantime i'll also give the dovecot-stable release a try. do i need to nuke off all the .imap cache files before i change?
On Thu, Feb 10, 2005 at 10:19:29AM +1100, Curtis Maloney wrote:
I tried a combined mbox-for-inbox/Maildir-for-other setup once, only to find it's not an option in the 0.99 series. You can do it in 1.0, apparently, but I'm yet to take the plunge :)
our plan is all-in-one format, but with a userbase of ~2k, we don't have the liberty of doing it all at once. so, i will go user by user, converting everything from mbox to maildir. with the exception of the 2-3 minute window as their mail is being converted (and their account is locked), no-one should notice a change... except for the pop users who will have to download all their mail all over again.
Umm... not being familiar with uw-imap's features, I'm not clear on what you're asking here. My understanding of the flow of things is your MDA (at least) delivers to your INBOX. More advanced MDAs will deliver to multiple folders, but your MUA needs to know to check these places for new mail. I know Thunderbird, by default, only checks INBOX, but can be told to check other folders, or all, for new mail.
uw-imap differentiates between an "unread mail spool" (/var/mail/$user or $home/.mail) and an "inbox" ($home/INBOX). if it finds any mail in the former when a user logs in, it moves it to the latter. kind of an annoying "feature", really.
anyway, thanks for all the input! sean
participants (4)
-
Curtis Maloney
-
dean gaudet
-
sean finney
-
Timo Sirainen