[Dovecot] considering dovecot
Hi all.
I'm making a case to use Dovecot over Courier and would like some
feedback. I've searched the web extensively and have been unable to
find any hard information regarding Dovecot's IMAP performance. I've
got some pressure to use Courier but my gut tells me that Dovecot
should be much faster. In particular, I will have typical users with
mailboxes exceeding 10,000 emails and in some cases up to 100,000.
Courier might be the "safe" choice but I'm not confident it can
handle inboxes with that many messages. Dovecot, with its indexes,
should be much, much faster. Can anyone point me to some benchmarks
or production uses of dovecot that bear this out?
From my research, Courier maildrop appears to be what I need for
local delivery. Is it possible to use Maildrop for local delivery
with Dovecot? Or is the Dovecot LDA needed to update the maildir
indexes?
I also need the capability for users to set their own vacation
messages via a web page. Maildrop looks to handle this easily (by
allowing me to link the maildrop filter to a file with the vacation
message) but I don't see similar capability with the Sieve plugin. I
do see where I can create a vacation message with Sieve but the text
of the message seems to be required to reside within the Sieve
script. Perhaps I'm missing something?
Any feedback would be greatly appreciated.
On Tue, 2006-11-28 at 15:01 -0600, Rick Hazey wrote:
From my research, Courier maildrop appears to be what I need for
local delivery. Is it possible to use Maildrop for local delivery
with Dovecot?
Yes, I use dovecot imap along with maildrop.
Or is the Dovecot LDA needed to update the maildir
indexes?
There's just a slight performance increase if you use dovecot's lda, if you don't dovecot indexes the mails on the next access.
Can't really help you with your other questions (well, vacation solves itself when you use maildrop, but the performance and all that)
johannes
Johannes Berg writes:
There's just a slight performance increase if you use dovecot's lda, if you don't dovecot indexes the mails on the next access.
I would categorize this and say "it depends". Without dovecot's LDA new mail will need to be indexed.
Depending on how much new mail is waiting since the lat time it was all indexed and the speed of the machine it can produce some noticeable results.
Consider an account dedicated for handling lists. You could have hundreds of new emails since the last time you checked. In such a I case I find that it made a very noticeable difference. Specially in my lowly home machine.
Rick Hazey writes:
I'm making a case to use Dovecot over Courier and would like some
feedback. I've searched the web extensively and have been unable to
find any hard information regarding Dovecot's IMAP performance. I've
got some pressure to use Courier but my gut tells me that Dovecot
should be much faster.
You don't have a test machine you can try it on?
In particular, I will have typical users with
mailboxes exceeding 10,000 emails and in some cases up to 100,000.
Courier might be the "safe" choice but I'm not confident it can
handle inboxes with that many messages.
Courier will be near useless on a heavy usage machine with mailboxes with 5,000+ emails. It highly depends on how busy the machine is, but courier does not handle large amount of mails well.
Dovecot, with its indexes,
should be much, much faster. Can anyone point me to some benchmarks
or production uses of dovecot that bear this out?
I am working on testing and preparing a Courier to Dovecot move. As a test I put 15,000 emails in two mailboxes.. one in a Courier machine and one on a (much smaller machine) with Dovecot. For that test I was using a webmail program to connect to both. Dovecot took under a minute the first time (when it created the idexes).. and was near instantaneous thereafter (as long as no new mail came in). Courier just timed out the webmail client.. after several minutes of waiting.
From my research, Courier maildrop appears to be what I need for
local delivery.
I recommend you consider Dovecot's LDA "deliver".
Is it possible to use Maildrop for local delivery
with Dovecot?
Yes, it is possible.
I also need the capability for users to set their own vacation
messages via a web page. Maildrop looks to handle this easily (by
allowing me to link the maildrop filter to a file with the vacation
message)
Not sure how to do it with deliver, but I believe it is possible, but I have not looked at that yet.
Is this a new install or something you are migrating?
On Nov 28, 2006, at 4:48 PM, Francisco Reyes wrote:
You don't have a test machine you can try it on?
I do have a test machine but getting enough email into it for testing
will be a daunting task. (see below) I'm primarily in the research
phase now and testing will come next, after I have some solid info
that will support Dovecot.
Courier will be near useless on a heavy usage machine with
mailboxes with 5,000+ emails. It highly depends on how busy the
machine is, but courier does not handle large amount of mails well.I am working on testing and preparing a Courier to Dovecot move. As a test I put 15,000 emails in two mailboxes.. one in a Courier
machine and one on a (much smaller machine) with Dovecot. For that
test I was using a webmail program to connect to both. Dovecot took
under a minute the first time (when it created the idexes).. and
was near instantaneous thereafter (as long as no new mail came in).
Courier just timed out the webmail client.. after several minutes
of waiting.
Your experience is what I was looking for. It confirms what I
suspected. Thanks.
I recommend you consider Dovecot's LDA "deliver".
Are you aware of any vacation message solutions that work with the
Sieve plugin? If the vacation message could come from a file or a
database, that would be ideal. Or is it possible for Sieve to call a
shell script or accept the vacation message via a pipe?
Is this a new install or something you are migrating?
It's a migration but I'm migrating from EIMS, which makes things
problematic. EIMS is a Mac email server that is quite good.
Unfortunately, it does not handle IMAP very efficiently. It also
stores email in it's own file format and that format uses Mac
resource forks. Probably the only way I'll get existing email moved
is to connect via IMAP to both servers and copy the files. I'm
expecting the whole process to be somewhat painful....
El Miércoles, 29 de Noviembre de 2006 00:36, Rick Hazey escribió:
It's a migration but I'm migrating from EIMS, which makes things problematic. EIMS is a Mac email server that is quite good. Unfortunately, it does not handle IMAP very efficiently. It also stores email in it's own file format and that format uses Mac resource forks. Probably the only way I'll get existing email moved is to connect via IMAP to both servers and copy the files. I'm expecting the whole process to be somewhat painful....
This is exactly what imapsync (http://www.linux-france.org/prj/imapsync/) does. It's a bit slow, but you can split the userlist in multiple parts and use multiple machines to increase performance.
HTH.
Aaaaaaaaagur.
-- Trying to make bits uncopyable is like trying to make water not wet. The sooner people accept this, and build business models that take this into account, the sooner people will start making money again. - Bruce Schneier
You can setup an alias to a virtual user (shell script) that will
make the required database (or file) calls to retrieve your vacation
messages. See here:
http://dev.high5.net/trac/browser/trunk/VIRTUAL_VACATION/INSTALL.TXT
Im using Dovecot (with Dovecot LDA) and Postfix, I've been very happy
with this setup. Im also have all my users, aliases, mailboxes etc
stored in MySQL for easy maintenance. We are using posftixadmin.
HTH
-Will
On Nov 28, 2006, at 3:36 PM, Rick Hazey wrote:
Are you aware of any vacation message solutions that work with the
Sieve plugin? If the vacation message could come from a file or a
database, that would be ideal. Or is it possible for Sieve to call
a shell script or accept the vacation message via a pipe?
-Will
On Nov 28, 2006, at 4:48 PM, Francisco Reyes wrote:
As a test I put 15,000 emails in two mailboxes.. one in a Courier
machine and one on a (much smaller machine) with Dovecot. For that
test I was using a webmail program to connect to both. Dovecot took
under a minute the first time (when it created the idexes).. and
was near instantaneous thereafter (as long as no new mail came in).
Courier just timed out the webmail client.. after several minutes
of waiting.
Just to be clear, are you using maildir with Dovecot in this test?
Rick Hazey wrote:
Hi all.
From my research, Courier maildrop appears to be what I need for local delivery.
It wouldn't hurt the list for you to tell us which criteria you judged the delivery agents by. Never know, someone might have an even better alternative.
Or someone might just add in that feature you want to Dovecot. :)
-- Curtis Maloney cmaloney@cardgate.net
On Nov 29, 2006, at 2:16 AM, Curtis Maloney wrote:
It wouldn't hurt the list for you to tell us which criteria you
judged the delivery agents by. Never know, someone might have an
even better alternative.
There are two must-have features:
- capability to move emails marked as spam into a junk mail folder
- capability for users to set their own vacation message via a web page
Sieve can do both but for #2 it appears that the vacation message
text must be contained within the Sieve script itself. I'm planning
on using mySQL for the authentication database and the vacation
message should be stored in mySQL. I don't see any way to access the
vacation message text from mySQL using Sieve.
Of course, there are unknown needs that will popup in the future and
Maildrop appears to be flexible enough to handle most situations. I
have no experience with either, so this opinion is based on what I've
been reading and subject to change.
Rick Hazey wrote:
- capability for users to set their own vacation message via a web page
Sieve can do both but for #2 it appears that the vacation message text must be contained within the Sieve script itself. I'm planning on using mySQL for the authentication database and the vacation message should be stored in mySQL. I don't see any way to access the vacation message text from mySQL using Sieve.
Properly handling vacation messages is beyond the current capabilities of SIEVE, IMNSHO. The problem is that it is surprisingly tricky to correctly figure out what messages *not* to reply to: listserv messages, bounces, etc. Indeed, there is no way in the SIEVE language to limit the vacation messages by number either; it is considered best practices to only respond with the vacation message a limited number of times (configurable).
Personally (and professionally) I use qmail-autoresponder:
http://untroubled.org/qmail-autoresponder/which, despite its name, isn't just limited to qmail, but can be placed in a .forward file.
HTH
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 29 Nov 2006, John Peacock wrote:
Personally (and professionally) I use qmail-autoresponder:
It has quite nice features, but two:
- Check SENDER rate
- Don't respond if:
- SENDER exceded the rate limit (defaults to 1 message every hour)
I'd say that one message per vacation is enough. :-)
- It responses to BCC'ed messages as well, as http://untroubled.org/qmail-autoresponder/procedure.txt does not state that the recipient must be part of To/CC.
I use sendmail's vacation. It has its own limits, when it comes to the way to construct the message and to detect the BCC case, one needs to whitelist all recipient addresses. Also, it uses a DB database.
Bye,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux)
iQEVAwUBRW2SGS9SORjhbDpvAQJWHAf/XSSOVBW6vY8aPQKDD3Lpj/btu1WiFEno iTTMGlz8gg0QHMk1m/GfIFOUS0Ud8fSuy5cFqzMMLxEsrWma9qSpU0IHzJAHhjlt gGQ7Cf7DzAU7M79npJwGKcBwsMqwA2ckltsxgM68TfFPk+yZIVxzMyTwahsw0WKe 7iBnJ0mO/QOiimjevUvPXekHjqZJ6Pk2i+q1/rKyy1wqkQ94cG85jlR+KxzmpwLD YLgB/dKB86yaN2xIgTWh+CjbSd/MfQoc5qiXn6ptVMWraEYaJn8u/+ECGTnWnvOU lVsGK3A5/EkIVTMfM9X4jQV2alj+6+aCaUi8T014bLl/MekGu80uqg== =nn6X -----END PGP SIGNATURE-----
Steffen Kaiser wrote:
- It responses to BCC'ed messages as well, as http://untroubled.org/qmail-autoresponder/procedure.txt does not state that the recipient must be part of To/CC.
I don't see this as being a problem (those contacts need to be aware the message isn't going to necessarily be read, too). If it was a problem for anyone, the code is trivially easy to change (it's very clean code).
I use sendmail's vacation. It has its own limits, when it comes to the way to construct the message and to detect the BCC case, one needs to whitelist all recipient addresses. Also, it uses a DB database.
qmail-autoresponder-mysql does what you might expect... ;-)
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
Hi,
(IMO Vacation messages aren't really part of this mailing list forum, as I belive it's a MTA's job)
I have had Vacation Messages implemtented for a few years now using Exim & MySQL with a Squirelmail plugin I wrote. Users would login to their webmail profile, using the plugin, add, edit or delete their vacation message which is stored in a MySQL DB.
On successful delivery of the mail through the MTA, Exim would check to see if the user has set a message in the vacation database (MySQL), and if so, respond to the sender with the body of custom text. This would only happen once per sender per day (predefined period I decided on). So If you ended up mailing the user 10 mails, you would only receive one message back informing you the recipient is "on vacation". One can set a global exclude list of addresses not to reply to, ie postmaster etc, but undoubtably some will get through.
I have plans to implement a time/date schedule for the vacation messages as well, so that users can pre-book a vacation message for a particular duration ahead of time.
So.. To some it up. I prefer using Exim (MTA) to do the dirty work.
Regards Richard
Sieve can do both but for #2 it appears that the vacation message text
must be contained within the Sieve script itself. I'm planning on using mySQL for the authentication database and the vacation message should be stored in mySQL. I don't see any way to access the vacation message text from mySQL using Sieve.
Properly handling vacation messages is beyond the current capabilities of SIEVE, IMNSHO. The problem is that it is surprisingly tricky to correctly figure out what messages *not* to reply to: listserv messages, bounces, etc. Indeed, there is no way in the SIEVE language to limit the vacation messages by number either; it is considered best practices to only respond with the vacation message a limited number of times (configurable).
Personally (and professionally) I use qmail-autoresponder:
http://untroubled.org/qmail-autoresponder/which, despite its name, isn't just limited to qmail, but can be placed in a .forward file.
HTH
John
-- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Blvd Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5747
On Nov 29, 2006, at 11:42 AM, Richard Mayhew - Nashua Mobile
Broadband Division wrote:
(IMO Vacation messages aren't really part of this mailing list
forum, as I belive it's a MTA's job)So.. To some it up. I prefer using Exim (MTA) to do the dirty work.
I hear what you're saying but it seems to me that the vacation
message function belongs in Sieve (via Dovecot Deliver). I want to
move spam into a junk folder and only auto-respond to the remaining
email. I'm certainly no expert; this just seemed the logical way to
do it and the way that Sieve and Maildrop handle it.
Are you able to avoid auto-responding to spam with your configuration
using Exim?
Hey,
Anything is possible with Exim :) What I decided to do in the past with "Quarantining" mail was to get Exim to deliver the message directly to the specified Junk Folder, and based on a Spam Score in the message headers one could ignore the auto-responder.
Much the same as I am not an expert using Sieve, but I prefer ,IMO, that the MTA handle's anything to do with mail delivery, black/white lists, rbl, parsing, scanning (to some degree) etc. and let Dovecot (IMAP/POP3) serve the mail back to the client. This is in no way pushing aside Dovecot, just my personal taste. I do however see an overlap with mail quota's as Exim and Dovecot both are setup to maintain them in synergy. One wont accept mail if the user is over quota, much the same with Dovecot maintaining the quota from an IMAP perspective.
I prefer to blackhole/delete the mail if its classified as spam, so the need to worry about whether or not to send a auto-responder based on if the mail is spam or not is somewhat void in this case. The MTA parses the mail through SpamAssassin and based on a returned result, the MTA either rejects the message (550) or accepts it for delivery. This way we don't have rouge mail running around our systems using precious resources.
Regards Richard.
-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] On Behalf Of Rick Hazey Sent: 29 November 2006 10:17 PM To: dovecot@dovecot.org Subject: Re: [Dovecot] considering dovecot
On Nov 29, 2006, at 11:42 AM, Richard Mayhew - Nashua Mobile Broadband Division wrote:
(IMO Vacation messages aren't really part of this mailing list forum, as I belive it's a MTA's job)
So.. To some it up. I prefer using Exim (MTA) to do the dirty work.
I hear what you're saying but it seems to me that the vacation message function belongs in Sieve (via Dovecot Deliver). I want to move spam into a junk folder and only auto-respond to the remaining email. I'm certainly no expert; this just seemed the logical way to do it and the way that Sieve and Maildrop handle it.
Are you able to avoid auto-responding to spam with your configuration using Exim?
On Wed, Nov 29, 2006 at 08:30:22AM -0500, John Peacock wrote:
Rick Hazey wrote:
- capability for users to set their own vacation message via a web page
Sieve can do both but for #2 it appears that the vacation message text must be contained within the Sieve script itself. I'm planning on using mySQL for the authentication database and the vacation message should be stored in mySQL. I don't see any way to access the vacation message text from mySQL using Sieve.
Possibly one could use a wrapper to get the message from SQL. Then again, if you have the code, you can add something to query a database. My own MDA (which I haven't plugged here lately :) but which I do use with dovecot) supports reading the text from a .cdb file, but since I am not a mySQL user, and haven't been asked by any mySQL users, I haven't added the capability for that kind of lookup. I imagine it would be pretty straightforward, though.
Properly handling vacation messages is beyond the current capabilities of SIEVE, IMNSHO. The problem is that it is surprisingly tricky to correctly figure out what messages *not* to reply to: listserv messages, bounces, etc.
Really? Looking at http://untroubled.org/qmail-autoresponder/procedure.txt (which you mentioned later on) I would say that this is a subset of the requirements for Sieve's vacation facility. See http://www.ietf.org/internet-drafts/draft-ietf-sieve-vacation-06.txt section 4.6, among others. What am I missing?
Indeed, there is no way in the SIEVE language to limit the vacation messages by number either; it is considered best practices to only respond with the vacation message a limited number of times (configurable).
What do you mean? The big point of the vacation verb in Sieve is to do exactly that; the responses are throttled to once per sender per interval, where interval is specified in days. Maybe you mean something else by "limit the vacation messages by number?"
mm
On Wed, 2006-11-29 at 12:47 -0500, Mark E. Mallett wrote:
My own MDA (which I haven't plugged here lately :) but which I do use with dovecot)
Have you tried if you could plug it into Dovecot's deliver as a replacement plugin to CMU sieve? I'd like to get an alternative to it, but I don't really want to spend time coding yet another Sieve implementation. And last I checked, your code wasn't exactly library/plugin-friendly at least to my eyes.. :)
On Wed, Nov 29, 2006 at 10:38:16PM +0200, Timo Sirainen wrote:
On Wed, 2006-11-29 at 12:47 -0500, Mark E. Mallett wrote:
My own MDA (which I haven't plugged here lately :) but which I do use with dovecot)
Have you tried if you could plug it into Dovecot's deliver as a replacement plugin to CMU sieve?
No, but that might be an interesting experiment. I *have* looked over the interface between dovecot lda and libsieve.
I'd like to get an alternative to it, but I don't really want to spend time coding yet another Sieve implementation. And last I checked, your code wasn't exactly library/plugin-friendly at least to my eyes.. :)
You're right there, but I don't think it would take much to make it so. I have "plugged" mfl (the language) into multiple applications, but those have been written with using the language in mind from the beginning.
I see only a few obstacles to making it more generically usable (most of you can probably hit delete now...):
mfl/sieve code makes calls to specifically-named functions to do actions (like "fileinto" etc). These could fairly easily be turned into callbacks or calls via function pointers.
mfl code thinks it knows how messages are stored in memory, and makes some direct use of structure elements; it also makes calls to specifically-named functions to access things like headers and mime parts. These things could also be turned into callbacks.
mfl code makes use of a bunch of my own suite of utility functions, which would have to be added to the library references. Undoubtedly some of these utility functions duplicate those in your application (or any that would want to use it), so it would add some size to the executable. It also creates another library dependancy to be figured out at configure time. Neither of those are all that big of a deal, but they might offend some purist nature.
I dunno, maybe I'll give it a shot, e.g. over the upcoming holiday. I've been thinking about trying to make the mfl piece more standalone anyway: the main downside being that it creates yet another library piece to maintain and for users to download.
mm
I use YAA! (yet another autoresponder) to handle vacation messages,, but I feel the system could be made more efficient.
When a user wants to set a vacation message, they go to a web page I created, and login (ldap, apache, authentication via apache's mod_auth_ldap). When they read the page, it reads the vacation message if any from mysql, and allows them to modify it. (PHP)
When mail is received, postfix forks it to the mailbox and Yaa!. This means Yaa! Touches every message in the system.
I wanted to do it all in LDAP, but I figured out I wasn't smart enough to make YAA!'s ldap support work, so I broke down and used Mysql.
In other news, Novell stopped contributing developers to Hula, the open source exchange alternative today.
-ejay
-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] On Behalf Of Rick Hazey Sent: Tuesday, November 28, 2006 3:01 PM To: dovecot@dovecot.org Subject: [Dovecot] considering dovecot
Hi all.
I'm making a case to use Dovecot over Courier and would like some feedback.
I've searched the web extensively and have been unable to find any hard
information regarding Dovecot's IMAP performance. I've got some pressure to
use Courier but my gut tells me that Dovecot should be much faster. In
particular, I will have typical users with mailboxes exceeding 10,000 emails
and in some cases up to 100,000.
Courier might be the "safe" choice but I'm not confident it can handle
inboxes with that many messages. Dovecot, with its indexes, should be much,
much faster. Can anyone point me to some benchmarks or production uses of
dovecot that bear this out?
From my research, Courier maildrop appears to be what I need for local delivery. Is it possible to use Maildrop for local delivery with Dovecot? Or is the Dovecot LDA needed to update the maildir indexes?
I also need the capability for users to set their own vacation messages via a web page. Maildrop looks to handle this easily (by allowing me to link the maildrop filter to a file with the vacation message) but I don't see similar capability with the Sieve plugin. I do see where I can create a vacation message with Sieve but the text of the message seems to be required to reside within the Sieve script. Perhaps I'm missing something?
Any feedback would be greatly appreciated.
On Nov 29, 2006, at 10:09 AM, Ejay Hire wrote:
I use YAA! (yet another autoresponder) to handle vacation
messages,, but I feel the system could be made more efficient.
Could you give me an overview of how your setup works using YAA? I
can't seem to wrap my brain around how you would do something like
this in a dovecot environment.
Sure. The trick is in getting the message to the Autoresponder system. This can be done by the delivery agent (maildrop), or further upstream in the MTA. I use Postfix as my MTA. When I receive a message via smtp, it (pipes it through a virus filter) (pipes it through a spam filter) and drops the message in the maildir. It also "forks" the message, sending a copy to YAA!. YAA runs it's own little pseudo smtp server, and recieves the message from postfix. (on port 3333 I think..). Then Yaa looks at the rcpt to, looks it up in the database, and generates the vacation message. If no vacation message exists, the message is discarded.
-ejay
-----Original Message----- From: dovecot-bounces@dovecot.org [mailto:dovecot-bounces@dovecot.org] On Behalf Of Rick Hazey Sent: Wednesday, November 29, 2006 11:18 AM To: dovecot@dovecot.org Subject: Re: [Dovecot] considering dovecot
On Nov 29, 2006, at 10:09 AM, Ejay Hire wrote:
I use YAA! (yet another autoresponder) to handle vacation messages,, but I feel the system could be made more efficient.
Could you give me an overview of how your setup works using YAA? I can't seem to wrap my brain around how you would do something like this in a dovecot environment.
participants (12)
-
Curtis Maloney
-
Ejay Hire
-
Francisco Reyes
-
Johannes Berg
-
John Peacock
-
Joseba Torre
-
Mark E. Mallett
-
Richard Mayhew - Nashua Mobile Broadband Division
-
Rick Hazey
-
Steffen Kaiser
-
Timo Sirainen
-
Will Prater