[Dovecot] Planning a dovecot deployment: issues and questions
Hi folks,
I am currently evaluating Dovecot for deployment to ~200 software engineers (so, heavy email users); I'm hoping this list could provide wisdom about how best to do the configuration, evaluation and deployment, and what testing folks have done to determine whether dovecot is right for their site. Below I've listed the diffs between the dovecot-example.conf and our config.
I've been playing with 1.0rc19 running on Solaris Nevada. We have set it up on our server on an alternate port from the existing IMAPd, which is something ancient, horrible and problematic. We're using sendmail and mail.local as our MTA and MDA, respectively, as these ship with the OS, and we need to test them.
I have some users who have volunteered to experiment with the server, and they are starting to do so. I've also seen the "flags" problems others have mentioned on the list, and I do not think I could deploy to production with that bug affecting the users.
I also have some issues which I'm not sure have been previously reported: One user (of my 4 test users) is occasionally seeing a bug in which mutt 1.4.2.1 claims to be "[downloading xxx/30724]" headers. Upon getting very near to the 30724 total, the total rises to something like: "[downloading xxx/50132]". And on and on. The actual number of messages in the box is 30724. This is not very reproducible, and we do not yet know if it's a problem with mutt or dovecot. Another, using Apple's Mail.App reported:
"Everything was working fine until i quit out of my mail app (Mac mail) and restarted. Then i saw weird things:
- messages wouldn't get marked unread
- messages from eons ago started showing up in my Drafts folder
- Inbox would keep spinning/loading
- then eventually Inbox just hung showing me no messages"Any thoughts about how to approach these issues? I don't really have a feel for how to go from user-observed-app-doing-X to imapd-is-doing-Y.
The big challenge for our site may be mailbox size: the median mailbox size is 62M, the average is 176M and the largest is 1.1GB. I'd be curious to know what sort of mailbox sizes other sites have experienced, and how widespread the use of the Dovecot MDA is-- is this likely to make a big difference for us, and will we have to migrate our (approximately 25) procmail users to sieve?
Having read the recent discussions about "1.0", I think my team would really appreciate knowing how to obtain something that was thoroughly vetted and worked highly reliably. If major changes are coming, I think it'd be great to have a 1.0.x stability focused tree, and a 1.1.x (or 2.x.x) feature tree.
Thanks in advance for your comments,
-dp[note that IMAPS is off for now, but that's just temporary]
$ diff dovecot.conf dovecot-example.conf 21d20 < protocols = imap 42,45d40 < protocol imap { < listen = *:150 < } < 50c45 < disable_plaintext_auth = no
#disable_plaintext_auth = yes 89c84 < ssl_disable = yes
#ssl_disable = no 214,215d208 < mail_location = mbox:~:INBOX=/var/mail/%u:INDEX=%h/Mail/imapd-indices < 258d250 < mail_extra_groups = mail 290d281 < dotlock_use_excl = yes 296d286 < lock_method = fcntl 441,444d430 < # dp: we'll see. < mbox_read_locks = dotlock fcntl < mbox_write_locks = dotlock fcntl <
-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp
On 02/01/2007 04:44 PM, Dan Price wrote:
Hi folks,
$ diff dovecot.conf dovecot-example.conf
< mail_location = mbox:~:INBOX=/var/mail/%u:INDEX=%h/Mail/imapd-indices
That one does sound a bit fishy. Are the index files even compatible to each other?
Other than that, a little bit of info: like MySQL DB integration, autoresponders, spam/virusscan and so on and
- Dovecot running on two different servers, providing IMAP and POP3 (both with and without SSL)
- Storage format is maildir
- MTA is sendmail, MDA is a custom bit of software doing funky stuff
so forth
- LDA is dovecots delivery agent, plopping mails pre-sorted into various folders
- One server has about 50 users out of which 15 use IMAP, the rest uses POP3, the other server has about 10 IMAP-only users.
- Mail quota is set to 5GByte for both servers
- Clients use Outlook/Outlook Express/Thunderbird - the latter with check_quota plugin
Biggest single folder size is around 1.5GByte, no idea how many mails. Average access time to that folder from Thunderbird is about 15 seconds or so for initial loading, pretty much instant when opening any mail in there. Indexing it from scratch takes quite a bit longer though, about 7 minutes.
Hope that helps Jen
On Thu 01 Feb 2007 at 06:08PM, Jens Knoell wrote:
On 02/01/2007 04:44 PM, Dan Price wrote:
Hi folks,
$ diff dovecot.conf dovecot-example.conf
< mail_location = mbox:~:INBOX=/var/mail/%u:INDEX=%h/Mail/imapd-indices
That one does sound a bit fishy. Are the index files even compatible to each other?
I'm sorry-- I made up the name Mail/imapd-indices myself. Only dovecot is using that directory.
Biggest single folder size is around 1.5GByte, no idea how many mails. Average access time to that folder from Thunderbird is about 15 seconds or so for initial loading, pretty much instant when opening any mail in there. Indexing it from scratch takes quite a bit longer though, about 7 minutes.
It does help some :) I should have mentioned that we are fairly well tied to mbox format for now (and hence my comment about scalability-- a 1.1GB mbox box is a sad thing; I wish I could remember all the reasons why we felt compelled to stay with mbox-- I'll have to go ask the sendmail guru.
Thanks,
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp
Dan Price wrote:
On Thu 01 Feb 2007 at 06:08PM, Jens Knoell wrote:
Biggest single folder size is around 1.5GByte, no idea how many mails. Average access time to that folder from Thunderbird is about 15 seconds or so for initial loading, pretty much instant when opening any mail in there. Indexing it from scratch takes quite a bit longer though, about 7 minutes.
It does help some :) I should have mentioned that we are fairly well tied to mbox format for now (and hence my comment about scalability-- a 1.1GB mbox box is a sad thing; I wish I could remember all the reasons why we felt compelled to stay with mbox-- I'll have to go ask the sendmail guru.
Thanks,
-dp
In all honesty, that's what we used before as well. The main issue I had with that setup were waaaay too many support calls because people could not figure out why they were not able to store mails AND subfolders in any folder. I've only actually switched to maildir a few weeks ago after some testing. Not sure if that limitation on mbox still exists in current Dovecot though - haven't tried.
Jen
I have 2+Gb mailboxes (mbox format) with my dovecot-1.0-0.beta8.2.fc5. Works fine.
FiL
Jens Knoell wrote:
In all honesty, that's what we used before as well. The main issue I had with that setup were waaaay too many support calls because people could not figure out why they were not able to store mails AND subfolders in any folder. I've only actually switched to maildir a few weeks ago after some testing. Not sure if that limitation on mbox still exists in current Dovecot though - haven't tried.
Jen
On Thu, 2007-02-01 at 15:44 -0800, Dan Price wrote:
I have some users who have volunteered to experiment with the server, and they are starting to do so. I've also seen the "flags" problems others have mentioned on the list, and I do not think I could deploy to production with that bug affecting the users.
I'll release rc20 in a few hours, and it's fixed there.
I also have some issues which I'm not sure have been previously reported: One user (of my 4 test users) is occasionally seeing a bug in which mutt 1.4.2.1 claims to be "[downloading xxx/30724]" headers. Upon getting very near to the 30724 total, the total rises to something like: "[downloading xxx/50132]".
Never heard of this before..
"Everything was working fine until i quit out of my mail app (Mac mail) and restarted. Then i saw weird things:
- messages wouldn't get marked unread - messages from eons ago started showing up in my Drafts folder
I think I once saw something similar to this when the mailbox's UIDVALIDITY changed. Mail.app didn't understand that change, and instead of fetching the new message with UID 1234, it looked up the old UID 1234 from its cache and showed it to me instead.
The reason for the UIDVALIDITY change then has probably something to do with you still running UW-IMAP(?) also. Or was the user only using Dovecot?
- Inbox would keep spinning/loading - then eventually Inbox just hung showing me no messages"
Don't know about these. I've been using Mail.app fine..
Any thoughts about how to approach these issues? I don't really have a feel for how to go from user-observed-app-doing-X to imapd-is-doing-Y.
rawlog could be helpful in knowing what exactly is happening. http://dovecot.org/bugreport.html
The big challenge for our site may be mailbox size: the median mailbox size is 62M, the average is 176M and the largest is 1.1GB. I'd be curious to know what sort of mailbox sizes other sites have experienced, and how widespread the use of the Dovecot MDA is-- is this likely to make a big difference for us, and will we have to migrate our (approximately 25) procmail users to sieve?
mbox_very_dirty_syncs=yes would at least help if you're not going to use Dovecot MDA.
Having read the recent discussions about "1.0", I think my team would really appreciate knowing how to obtain something that was thoroughly vetted and worked highly reliably. If major changes are coming, I think it'd be great to have a 1.0.x stability focused tree, and a 1.1.x (or 2.x.x) feature tree.
There are major changes in CVS HEAD already. I've mostly been trying to fix the last few bugs, but looks like I managed to break some things with those fixes in rc19..
Quoting Dan Price dp@eng.sun.com:
I am currently evaluating Dovecot for deployment to ~200 software engineers (so, heavy email users); I'm hoping this list could provide wisdom about how best to do the configuration, evaluation and deployment, and what testing folks have done to determine whether dovecot is right for their site. Below I've listed the diffs between the dovecot-example.conf and our config.
I would say it is a great IMAP/POP server for almost any site. Best would be to wait for the release version of course, but most (not all) of the RC releases are stable.
Any thoughts about how to approach these issues? I don't really have a feel for how to go from user-observed-app-doing-X to imapd-is-doing-Y.
Report them individually to the mailing list, and you will get instructions on how to debug/repor them...
The big challenge for our site may be mailbox size: the median mailbox size is 62M, the average is 176M and the largest is 1.1GB. I'd be curious to know what sort of mailbox sizes other sites have experienced, and how widespread the use of the Dovecot MDA is-- is this likely to make a big difference for us, and will we have to migrate our (approximately 25) procmail users to sieve?
We use mbox format and have no problems, though our largest mailboxe is 600 MB, so I have no idea how this would work with a 1.1 GB mailbox. I'd be sure you have lots of memory in the machine if you want to be handling mailboxes that size.
We use sendmail for MTA, procmail for filters, dovecot for pop3/imap, Horde/IMP for webmail, and have no problems. You don't have to move to maildir, or sieve, or anything else. dovecot plays nice with almost any software...
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
On Fri 02 Feb 2007 at 10:53AM, Eric Rostetter wrote:
I would say it is a great IMAP/POP server for almost any site. Best would be to wait for the release version of course, but most (not all) of the RC releases are stable.
Thanks Eric! I'm going to pull down rc21 today; hopefully that will be something I can give to a wider group of testers.
We use mbox format and have no problems, though our largest mailboxe is 600 MB, so I have no idea how this would work with a 1.1 GB mailbox. I'd be sure you have lots of memory in the machine if you want to be handling mailboxes that size.
Well, we should have 32GB of RAM.
We use sendmail for MTA, procmail for filters, dovecot for pop3/imap, Horde/IMP for webmail, and have no problems. You don't have to move to maildir, or sieve, or anything else. dovecot plays nice with almost any software...
Super. The endorsement helps :) It's always nice to hear from someone with the same setup you have.
-dp-- Daniel Price - Solaris Kernel Engineering - dp@eng.sun.com - blogs.sun.com/dp
participants (5)
-
Dan Price
-
Eric Rostetter
-
FiL
-
Jens Knoell
-
Timo Sirainen