[Dovecot] what is the current state
hi, what is the current state of dovecot? is it worth to upgrade now or wait for some release in the near future?
-- Levente "Si vis pacem para bellum!"
On Mon, Sep 08, 2003 at 05:59:20PM +0200, Farkas Levente wrote:
what is the current state of dovecot? is it worth to upgrade now or wait for some release in the near future?
It's probably not a good idea to use CVS yet. There's a few small bugs (which I'll fix today) and it doesn't yet do any smart decisions on what it should save into cache.
It might work better than 0.99.10, but I'll probably break the index format a few times more before 0.99.11.
I wasted yesterday trying to figure out why Cyrus was faster in simple fetches. Such as fetching UID for 360k mails took a bit over second in Cyrus while Dovecot used two seconds. Annoying :) Probably has something to do with Cyrus printing things directly into stdout while Dovecot copies the data through a couple of buffers.
Anyway, I would really like to know if the new indexing code helps with your load problems. It should use much less disk I/O.
On Mon, Sep 08, 2003 at 12:54:26PM -0400, Eric S. Johansson wrote:
Anyway, I would really like to know if the new indexing code helps with your load problems. It should use much less disk I/O.
sounds like this would also help with some of the problems I was having with long delays. Think it would be worth trying?
I think your problems were mostly due to general mbox slowness :) mbox support also isn't really working in CVS now. I'd have to figure out some nice algorithm how to quickly write changes to mbox.
Timo Sirainen wrote:
On Mon, Sep 08, 2003 at 12:54:26PM -0400, Eric S. Johansson wrote:
I think your problems were mostly due to general mbox slowness :) mbox support also isn't really working in CVS now. I'd have to figure out some nice algorithm how to quickly write changes to mbox.
:-) crush dreams why don't you?
seriously, on mbox, the best thing we can do is to provide minimal support (i.e. reliable and it works no worse than uw) and make maildir transition as trivial as possible (i.e. switch mbox to maildir on first write).
the automatic switching code should be relatively trivial and the process would be something like:
Move mbox to mbox-xyxzzy make directory mbox iterate over mbox-xyzzy copying each message into maildir entry. move mbox-xyzzy safe-dir
I'm willing to bet that 90 percent of this code exists in dovecot. there are probably a few other guards when they want to put around this code to make it truly safe but I believe it would be really good idea.
---eric
On Mon, Sep 08, 2003 at 01:35:46PM -0400, Eric S. Johansson wrote:
Timo Sirainen wrote:
On Mon, Sep 08, 2003 at 12:54:26PM -0400, Eric S. Johansson wrote:
I think your problems were mostly due to general mbox slowness :) mbox support also isn't really working in CVS now. I'd have to figure out some nice algorithm how to quickly write changes to mbox.
:-) crush dreams why don't you?
seriously, on mbox, the best thing we can do is to provide minimal support (i.e. reliable and it works no worse than uw) and make maildir transition as trivial as possible (i.e. switch mbox to maildir on first write).
I would not consider that to be the best thing. I'm not interested in having our mboxes converted automatically to maildirs. We have systems where each is used, and enviromnents where both are used, by choice.
-mm-
Mark E. Mallett wrote:
On Mon, Sep 08, 2003 at 01:35:46PM -0400, Eric S. Johansson wrote:
seriously, on mbox, the best thing we can do is to provide minimal support (i.e. reliable and it works no worse than uw) and make maildir transition as trivial as possible (i.e. switch mbox to maildir on first write).
I would not consider that to be the best thing. I'm not interested in having our mboxes converted automatically to maildirs. We have systems where each is used, and enviromnents where both are used, by choice.
I think we're in more agreement than not. Like you, I would not want them to be converted automatically. That is *unless* I set configuration to convert them automatically. mbox accesse is never going to be fast or efficient. If it was possible to make mbox access faster, it would be because enough bright people have thought about the problem to find a solution if there was one. I'm not saying there isn't an undiscovered magic option like marking a message as replaced, writing an updated copy to the end of the file, and sorting out the order at read time. Only that it's highly unlikely.
the reason I advocate automatic conversion is that the process is usually painful and error prone when it does not need to be. Someone should be able to turn on dovecot and have the conversion happen automatically and feel confident that they will be able to get their e-mail quickly and reliably.
---eric
On Monday, Sep 8, 2003, at 21:15 Europe/Helsinki, Eric S. Johansson wrote:
mbox accesse is never going to be fast or efficient. If it was possible to make mbox access faster, it would be because enough bright people have thought about the problem to find a solution if there was one. I'm not saying there isn't an undiscovered magic option like marking a message as replaced, writing an updated copy to the end of the file, and sorting out the order at read time. Only that it's highly unlikely.
Well .. mbox performance does depend on what the client does. Synchronizing can do all kinds of tricks to speed it up when Dovecot is the only one accessing the mbox. I'll probably do these once I get mbox working in general.
Sync optimization would work so that whenever Dovecot modifies mbox or synchronizes it, it keeps the mbox read-locked for at least two seconds. This guarantees that if someone else modifies the mbox, the timestamp changes and we know it. If it doesn't change, our index is up to date and we don't need any syncing.
Only problem is that if there's lots of unsynced mailboxes, we don't really want to just keep waiting for those 2 seconds. We'd rather just want to keep the lock and remove it two seconds later. But we don't necessarily want to keep all the mboxes open and eat all available file decriptors.. Maybe it should keep max. 10 mboxes open before waiting for one of them to finish the wait.
Expunging can't do much optimizations or it wouldn't be mbox-compatible anymore. But luckily most expunges are done for new messages and then it doesn't have to move that much data.
That sync optimization would actually help with your slowdown, if Dovecot also indexed mails at the time they were copied/appended. That also makes UIDPLUS extension possible with little effort..
Timo Sirainen wrote:
On Mon, Sep 08, 2003 at 05:59:20PM +0200, Farkas Levente wrote:
what is the current state of dovecot? is it worth to upgrade now or wait for some release in the near future?
It's probably not a good idea to use CVS yet. There's a few small bugs (which I'll fix today) and it doesn't yet do any smart decisions on what it should save into cache.
It might work better than 0.99.10, but I'll probably break the index format a few times more before 0.99.11.
I wasted yesterday trying to figure out why Cyrus was faster in simple fetches. Such as fetching UID for 360k mails took a bit over second in Cyrus while Dovecot used two seconds. Annoying :) Probably has something to do with Cyrus printing things directly into stdout while Dovecot copies the data through a couple of buffers.
Anyway, I would really like to know if the new indexing code helps with your load problems. It should use much less disk I/O.
as you know we'd a lot of problem with dovecot at the begining (with stability and the load). now there is one version of dovecot (a cvs around 99.10) which still has a high load, but work both with mozilla and OE6. this's a production system about 300 user so we can't play with it too much. that was the reason why I ask it. try now or wait a bit more? but as I see it's better to wait a bit more... thanks.
-- Levente "Si vis pacem para bellum!"
participants (4)
-
Eric S. Johansson
-
Farkas Levente
-
Mark E. Mallett
-
Timo Sirainen