[Dovecot] transitional weirdnesses
not reproducible but when it occurred, its stuck: in the early transition to dovecot I would notice occasionally that messages would come up, I would delete them and then something would hang causing a major delay in the (Mozilla) e-mail client's response. When the system responded, I would notice that occasionally there would be an off by one offset between the index of messages and the message displayed. This error would persist until I killed off the index and the dovecot process.
very reproducible: after fetching messages from my inbox, there will be a long (30 sec.) delay. Dovecot demon is clearly busy doing something. I do have a large set of filter rules (approximately 150) sorting messages into different boxes. Is this a reason to switch to maildir?
just ended up with a message that displayed the entire header. Will look into this more but it's very strange.
---eric
On Tue, 2003-08-19 at 00:24, Eric S. Johansson wrote:
not reproducible but when it occurred, its stuck: in the early transition to dovecot I would notice occasionally that messages would come up, I would delete them and then something would hang causing a major delay in the (Mozilla) e-mail client's response. When the system responded, I would notice that occasionally there would be an off by one offset between the index of messages and the message displayed. This error would persist until I killed off the index and the dovecot process.
just ended up with a message that displayed the entire header. Will look into this more but it's very strange.
Corrupted messages with Mozilla sounds a bit like the bug I fixed in 0.99.10.. You don't happen to be using CR+LF linefeeds? Or alternatively it's just some mbox weirdness. I haven't noticed that myself though..
- very reproducible: after fetching messages from my inbox, there will be a long (30 sec.) delay. Dovecot demon is clearly busy doing something. I do have a large set of filter rules (approximately 150) sorting messages into different boxes. Is this a reason to switch to maildir?
You could check with strace -p pid what imap process is doing. Or if those filter rules are done by client and one message at a time doing COPY + EXPUNGE, it's really no wonder.
I'd suggest maildir anyway, it probably fixes your other problems too :)
Timo Sirainen explained:
On Tue, 2003-08-19 at 00:24, Eric S. Johansson wrote:
Corrupted messages with Mozilla sounds a bit like the bug I fixed in 0.99.10.. You don't happen to be using CR+LF linefeeds? Or alternatively it's just some mbox weirdness. I haven't noticed that myself though..
no, I don't think I am. I will doublecheck however in any case, it only happened once so I'm not too worried
You could check with strace -p pid what imap process is doing. Or if those filter rules are done by client and one message at a time doing COPY + EXPUNGE, it's really no wonder.
you know, I would not be surprised if Mozilla was doing that. It's not a very smart imap client but it sucks far less than the other ones out there. Now all I need is server side filtering that is manageable (unlike procmail) and I will be very happy (until the next time I stub my toe ;-)
I'd suggest maildir anyway, it probably fixes your other problems too :)
I figured you'd say that. It's where I was going anyway... now I need to figure out how to make the delivery agent (procmail) deliver to my inbox using maildir and everybody else's mailbox in mbox.... time to STFW
---eric
Eric S. Johansson wrote:
I figured you'd say that. It's where I was going anyway... now I need to figure out how to make the delivery agent (procmail) deliver to my inbox using maildir and everybody else's mailbox in mbox.... time to STFW
Just follow your destination folder with a / and it will create a Maildir.
--Ian.
On Tuesday, Aug 19, 2003, at 16:44 Europe/Helsinki, Eric S. Johansson wrote:
Now all I need is server side filtering that is manageable (unlike procmail) and I will be very happy (until the next time I stub my toe ;-)
That would probably be Sieve then. I think there's some GUIs for it as well. I'd just have to get around plugging libSieve into Dovecot. Last I tried it wasn't as simple as I thought.
Timo Sirainen explained:
On Tuesday, Aug 19, 2003, at 16:44 Europe/Helsinki, Eric S. Johansson wrote:
That would probably be Sieve then. I think there's some GUIs for it as well. I'd just have to get around plugging libSieve into Dovecot. Last I tried it wasn't as simple as I thought.
why is none of this stuff as easy as it should be? We should know enough now to be able to abstract appropriate bits functionality and communicate that abstraction clearly. Instead we still have a rather poor mishmash of knowledge compounded by a cluster of point simplifications when used together increase overall complexity.
bah, humbug.. should have stayed in astronomy and physics.
---eric
On Tue, Aug 19, 2003 at 06:49:03PM +0300, Timo Sirainen wrote:
On Tuesday, Aug 19, 2003, at 16:44 Europe/Helsinki, Eric S. Johansson wrote:
Now all I need is server side filtering that is manageable (unlike procmail) and I will be very happy (until the next time I stub my toe ;-)
That would probably be Sieve then. I think there's some GUIs for it as well. I'd just have to get around plugging libSieve into Dovecot. Last I tried it wasn't as simple as I thought.
I thought he was just musing about finding a delivery agent that included a filtering language. Not sure what good having such a thing in the IMAP server does, although I know some support it. Seems to me that a facility to re-deliver mail or other general ways to give a message to a standalone tool (as briefly discussed a while ago) would be superior.
-mm- (currently using a sieve-ish filter on my inbox..)
Mark E. Mallett explained:
On Tue, Aug 19, 2003 at 06:49:03PM +0300, Timo Sirainen wrote:
I thought he was just musing about finding a delivery agent that
Mark, I know you're just being polite. It's OK to say "grousing" or "whingeing" because that's what I was doing. ;-)
included a filtering language. Not sure what good having such a thing in the IMAP server does, although I know some support it. Seems to me that a facility to re-deliver mail or other general ways to give a message to a standalone tool (as briefly discussed a while ago) would be superior.
in all seriousness, what you are describing is the right place to put a filter as long as the user agent has a channel for notifications of mail delivery to different mailboxes. Remember my grousing about complexity? This kind of simplification is necessary.
So, to go to the next step, there are 4 components on the receive side of a mail system, MTA, LDA, MPA, MUA; mail transport agent, local delivery agent, mailbox presentation agent, and mail user agent. Sendmail and postfix are examples of mail transport agents. Procmail is an example of a local delivery agent. dovecot is an example of a mailbox presentation agent. And Eudora, Mozilla, outlook are all examples of mail user agents.
if I understand you correctly Mark, you think that putting filter and plug-in facilities into the local delivery agent would be the appropriate place. Please correct me if I'm wrong.
Architecturally, I consider putting the filter in the LDA appropriate because the filter operations happen at delivery time when it's appropriate not as you are trying to read your inbox. It's also the appropriate place to put other types of processing such as antispam and antivirus checking.
One interesting thing about this model is that it implies if you can push mail around from the MPA, there is a question of whether or not you want to replicate LDA functionality in the MPA and if so, how much? or how much common code can a LDA and MPA share? Is there a different architectural structure where all inbound mail is injected through common path.
I don't expect anyone to do anything about this. I'm just treading the ground to see what the terrain feels like.
---eric
On Tue, Aug 19, 2003 at 06:55:38PM -0400, Eric S. Johansson wrote:
Mark, I know you're just being polite.
I guess there's always a first time :-)
So, to go to the next step, there are 4 components on the receive side of a mail system, MTA, LDA, MPA, MUA; mail transport agent, local delivery agent, mailbox presentation agent, and mail user agent. Sendmail and postfix are examples of mail transport agents. Procmail is an example of a local delivery agent. dovecot is an example of a mailbox presentation agent. And Eudora, Mozilla, outlook are all examples of mail user agents.
In my extreme ignorance, I have not heard of the term "MPA". However it does seem to fit.
if I understand you correctly Mark, you think that putting filter and plug-in facilities into the local delivery agent would be the appropriate place. Please correct me if I'm wrong.
I dunno. Maybe I was just spouting off too much. I do think that filters that are designed to accomplish mail delivery are most appropriate in the delivery agent. But I am also in favor of having controls at various stages. The filtering controls at the MTA level are not likely to be the same as those at the delivery agent level-- here, I use both. And I suppose I can see where there might be a demand for filters that are active when the mail is fetched from the mailbox server (e.g. between IMAP server and mail user agent). But is something like Sieve the right fit there? Sieve is designed to control what happens to mail on delivery. (One might have such filters at the user agent side inasmuch as there is delivery to local folders going on there.) All I really said was that if you want to have filters that control how mail is filed on the mailbox server, have a mechanism to re-deliver the mail so that new delivery filters can be applied. This would isolate the delivery logic to one place. I would think there would be a different set of requirements for filters (or selection criteria) that are engaged when the mail client talks to the mailbox server. And a lot of that is already present and available by using the IMAP or POP3 protocol.
Architecturally, I consider putting the filter in the LDA appropriate because the filter operations happen at delivery time when it's appropriate not as you are trying to read your inbox. It's also the appropriate place to put other types of processing such as antispam and antivirus checking.
Well- there are conflicts. One wants to conserve resources, but the resources available are sometimes in opposition (horsepower on the server, bandwidth between the server and the client, time and effort involved to set up each client, the are all resources that have to be juggled). One puts filtering on the server in order to conserve bandwidth between the server and client. One puts heavy duty work on the client system because doing so on the server can't scale. (So whereas I can't run a CRM114 analysis on every message that hits the server, each client could do so on their own machine.) One may relegate virus checking to the client machine because the corrective stuff has to be done there anyway if a virus gets through. Etc.
One interesting thing about this model is that it implies if you can push mail around from the MPA, there is a question of whether or not you want to replicate LDA functionality in the MPA and if so, how much? or how much common code can a LDA and MPA share? Is there a different architectural structure where all inbound mail is injected through common path.
I don't expect anyone to do anything about this. I'm just treading the ground to see what the terrain feels like.
(no comment, other than just nodding)
-mm-
On Wed, 2003-08-20 at 01:55, Eric S. Johansson wrote:
Architecturally, I consider putting the filter in the LDA appropriate because the filter operations happen at delivery time when it's appropriate not as you are trying to read your inbox. It's also the appropriate place to put other types of processing such as antispam and antivirus checking.
It's also where I was going to put the Sieve scripting. I was going to create a separate "deliver" binary which would read the mail from stdin and place it into proper mailbox. This could be also used to update indexes immediately while mail is being delivered.
On Wed, Aug 20, 2003 at 06:14:23AM +0300, Timo Sirainen wrote:
On Wed, 2003-08-20 at 01:55, Eric S. Johansson wrote:
Architecturally, I consider putting the filter in the LDA appropriate because the filter operations happen at delivery time when it's appropriate not as you are trying to read your inbox. It's also the appropriate place to put other types of processing such as antispam and antivirus checking.
It's also where I was going to put the Sieve scripting. I was going to create a separate "deliver" binary which would read the mail from stdin and place it into proper mailbox. This could be also used to update indexes immediately while mail is being delivered.
How important do you think it is for the local delivery agent to update the indexes upon delivery? I assume that would be the primary purpose of your including a delivery agent in the package..
If it's important to update the indexes upon delivery, perhaps it would be worthwhile to provide some kind of toolkit that could be included into other delivery agents. I'm already using a sieve-ish filtering delivery agent, and I imagine I would keep on using it. But I wouldn't be averse to adding logic to update indexes if it meant some kind of performance gain.
mm
On Wednesday, Aug 20, 2003, at 19:30 Europe/Helsinki, Mark E. Mallett wrote:
It's also where I was going to put the Sieve scripting. I was going to create a separate "deliver" binary which would read the mail from stdin and place it into proper mailbox. This could be also used to update indexes immediately while mail is being delivered.
How important do you think it is for the local delivery agent to update the indexes upon delivery? I assume that would be the primary purpose of your including a delivery agent in the package..
No, the primary purpose would be to have Sieve scripting.
The index updating then .. well, there's two reasons why it's useful at that point:
First it reduces the overall server load. We have to read the mail into memory and write it to disk in any case, so we might as well parse it's contents and update indexes. If we do it later, we might have to fully read the mail from disk just for some client request. Although I don't know how smart the SMTP servers are with maildir - do they just link() their temporary file into the new/ dir rather than copy it?
Second it distributes the load. If user hasn't opened some mailbox for a long time (eg. spambox) and it contains thousands of messages, the opening could take quite a long time if the wanted data isn't indexed.
If it's important to update the indexes upon delivery, perhaps it would be worthwhile to provide some kind of toolkit that could be included into other delivery agents. I'm already using a sieve-ish filtering delivery agent, and I imagine I would keep on using it. But I wouldn't be averse to adding logic to update indexes if it meant some kind of performance gain.
There already is, kind of. lib-storage/mail-storage.h is the main API that can be used to access mailboxes.
On Wed, Aug 20, 2003 at 08:15:14PM +0300, Timo Sirainen wrote:
On Wednesday, Aug 20, 2003, at 19:30 Europe/Helsinki, Mark E. Mallett wrote:
It's also where I was going to put the Sieve scripting. I was going to create a separate "deliver" binary which would read the mail from stdin and place it into proper mailbox. This could be also used to update indexes immediately while mail is being delivered.
How important do you think it is for the local delivery agent to update the indexes upon delivery? I assume that would be the primary purpose of your including a delivery agent in the package..
No, the primary purpose would be to have Sieve scripting.
Ah. I already have that, and I suspect that anybody who wants it can have it, too.
If it's important to update the indexes upon delivery, perhaps it would be worthwhile to provide some kind of toolkit that could be included into other delivery agents. I'm already using a sieve-ish filtering delivery agent, and I imagine I would keep on using it. But I wouldn't be averse to adding logic to update indexes if it meant some kind of performance gain.
There already is, kind of. lib-storage/mail-storage.h is the main API that can be used to access mailboxes.
I don't think that's an exact match. A delivery agent already has its own method of internalizing information about the mail message, (and already has a mail message open and parsed and in memory), and to have a hook into updating the indexes all that it really needs is some way of passing the indexable information to the indexer. If indeed it's worth doing.
mm
On Wednesday, Aug 20, 2003, at 20:20 Europe/Helsinki, Mark E. Mallett wrote:
No, the primary purpose would be to have Sieve scripting.
Ah. I already have that, and I suspect that anybody who wants it can have it, too.
What program do you use for it? I haven't seen any yet.
There already is, kind of. lib-storage/mail-storage.h is the main API that can be used to access mailboxes.
I don't think that's an exact match. A delivery agent already has its own method of internalizing information about the mail message, (and already has a mail message open and parsed and in memory), and to have a hook into updating the indexes all that it really needs is some way of passing the indexable information to the indexer. If indeed it's worth doing.
Well, there's lib-index/ too .. :) But it requires some more work from the caller.
Anyway I would like to improve the APIs so that it's all pretty and easy to use even by external projects. I don't think they're too bad now, but they're not really pretty yet either.
On Wed, Aug 20, 2003 at 08:45:19PM +0300, Timo Sirainen wrote:
On Wednesday, Aug 20, 2003, at 20:20 Europe/Helsinki, Mark E. Mallett wrote:
No, the primary purpose would be to have Sieve scripting.
Ah. I already have that, and I suspect that anybody who wants it can have it, too.
What program do you use for it? I haven't seen any yet.
homebrew (see http://www.mv.com/tools/mvmf/ if you care)
but my claim was based on a number of sieve-related articles, and on putting "sieve" and "spam" into google. (and looking at various comments on the ietf-mta-filters mailing list.)
Anyway I would like to improve the APIs so that it's all pretty and easy to use even by external projects. I don't think they're too bad now, but they're not really pretty yet either.
I suppose it might be nice to have a standalone utility that one could fork and give information to add to an index.
mm
participants (4)
-
Eric S. Johansson
-
Ian R. Justman
-
Mark E. Mallett
-
Timo Sirainen