[Dovecot] Multi Server Exim/Dovecot and NFS
In reading the Dovecot NFS WIKI entry:
It mentions a FreeBSD NFS client caching bug. I believe that bug only exists pre-8.x ? It seems that a patch in the PR notes was MFC'd and made it into 8.x at some point. (at least I checked the source in 8.3 and it was there). So I assume that FreeBSD 8.x mitigates the issue? Maybe not?
So our deployment is multi server Dovecot/Exim running on FreeBSD servers, frontended by a layer3 load balancing switch and a NetApp NFSv3 datastore.
The only real concern we have right now is race conditions and file write contention on the Dovecot index files. Our original plan was to use Dovecot LMTP on the servers to deliver to the inbox, but that doesn't seem to be recommended as we'll have a number of servers all acting as primary MX for domains and delivering to the same set of mailboxes. So are the Dovecot dotlock file locking mechanisms not enough for multiple servers trying to write/access index files in Maildirs?
If we have to deliver straight to the ~/Maildir/new out of Exim, that's fine, also. We're just wondering if there is a added benefit of actually using LMTP. (other than making sure the index are kept the most recent).
The other concern we have is client mailbox access. While we can make sessions "sticky" from one IP so they hit the same server, it doesn't help with the person that checks their email via POP from their PC while having their mobile phone access it via IMAP. (obviously from different IP addresses). So I guess there is some possibly index file write contention there as well. (though hopefully this should be rare).
I'm curious as to know others experiences with a similar deployment.
TIA
-- Robert Blayzor INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/
Am 16.07.2012 14:58, schrieb Robert Blayzor:
In reading the Dovecot NFS WIKI entry:
It mentions a FreeBSD NFS client caching bug. I believe that bug only exists pre-8.x ? It seems that a patch in the PR notes was MFC'd and made it into 8.x at some point. (at least I checked the source in 8.3 and it was there). So I assume that FreeBSD 8.x mitigates the issue? Maybe not?
So our deployment is multi server Dovecot/Exim running on FreeBSD servers, frontended by a layer3 load balancing switch and a NetApp NFSv3 datastore.
The only real concern we have right now is race conditions and file write contention on the Dovecot index files. Our original plan was to use Dovecot LMTP on the servers to deliver to the inbox, but that doesn't seem to be recommended as we'll have a number of servers all acting as primary MX for domains and delivering to the same set of mailboxes. So are the Dovecot dotlock file locking mechanisms not enough for multiple servers trying to write/access index files in Maildirs?
If we have to deliver straight to the ~/Maildir/new out of Exim, that's fine, also. We're just wondering if there is a added benefit of actually using LMTP. (other than making sure the index are kept the most recent).
The other concern we have is client mailbox access. While we can make sessions "sticky" from one IP so they hit the same server, it doesn't help with the person that checks their email via POP from their PC while having their mobile phone access it via IMAP. (obviously from different IP addresses). So I guess there is some possibly index file write contention there as well. (though hopefully this should be rare).
I'm curious as to know others experiences with a similar deployment.
TIA
There where hot disputs relating such setups, read list archive
i guess nfs is simply not the ideal solution, however its used in real existing setups, also maildir may not ideal
for good advice there should be more info about how many mailboxes , how much traffic, mailbox size , loadbalancing solution etc
dont think exim is critical envolved in this ( but i dont know exim very good )
however someone from the list with equal setup may share his experience with you
as far i remember, Timo posted good advice also ,for nfs setups in the past
-- Best Regards MfG Robert Schetterer
On Jul 16, 2012, at 9:15 AM, Robert Schetterer wrote:
There where hot disputs relating such setups, read list archive
i guess nfs is simply not the ideal solution, however its used in real existing setups, also maildir may not ideal
for good advice there should be more info about how many mailboxes , how much traffic, mailbox size , loadbalancing solution etc
dont think exim is critical envolved in this ( but i dont know exim very good )
however someone from the list with equal setup may share his experience with you
as far i remember, Timo posted good advice also ,for nfs setups in the past
We're talking about tens of thousands of mailboxes and at least half a dozen servers involved. There is quite a bit of traffic. We've been running Maildir via NFS with another mailserver product for some time, and it's been ok as far as the shared file system standpoint.
I don't know why Maildir would be "not ideal". It seems to be the only option when it comes to NFS.
NFS is used extensively in our network and we already have a major investment with NetApp, and have been very happy with them.
The only issue I see right now is Dovecot index file write bashing if a user accesses their mailbox from multiple locations at the same time, and I gave that example in the previous post. That seems to be more and more common these days but still not the norm for 80%+ of the mailboxes we have.
From the MTA side of things, it looks like having Exim write directly into the ~/Maildir/new/ is the only sane choice since mail can be coming into a mailbox from a number of servers at the same time. We've used this extensively in the past with no issues.
We use a Brocade ADX to load balancing POP and IMAP sessions to the server. Stickiness doesn't really buy you anything on POP3 since a session connects, does it's thin and disconnects. IMAP on the other hand, I believe can have multiple connections open from the same client, so I think sticky would help. But neither of those help when the client is connecting from a different IP address.
Robert Blayzor INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/
The only issue I see right now is Dovecot index file write bashing if a user accesses their mailbox from multiple locations at the same time, and I gave that example in the previous post. That seems to be more and more common these days but still not the norm for 80%+ of the mailboxes we have.
We have a similar setup and solved the problem for us this way:
- The delivering mail servers (postfix+dovecot) don't write any index files, so we can use round robin here.
- We have an additional layer of proxy servers between the loadbalancers and the dovecot IMAP servers, that try to make sessions sticky. For this we use the IMAP/POP3 proxy code of nginx with a PHP script that does some database voodoo, normalizes the user name and based on all this decides which IMAP backend server to use. This is based on the username (not the IP), so it is sticky for multiple IMAP and POP3 connections from different IPs. I think dovecot director should be able to do similar things (our setup is a little bit older and needs some more flexibility).
Tschoeeee
Roland-- Roland Rosenfeld - Content Delivery - NED - Technik NetCologne Gesellschaft für Telekommunikation mbH - HRB 25580, AG Köln Am Coloneum 9 50829 Köln Tel.: +49 221 2222-373 Fax: +49 221 2222-7373 Geschäftsführer: Dr. Hans Konle (Sprecher), Karl-Heinz Zankel
Am 16.07.2012 16:19, schrieb Robert Blayzor:
We're talking about tens of thousands of mailboxes and at least half a dozen servers involved. There is quite a bit of traffic. We've been running Maildir via NFS with another mailserver product for some time, and it's been ok as far as the shared file system standpoint.
jep ,thats a big setup
I don't know why Maildir would be "not ideal". It seems to be the only option when it comes to NFS.
mdbox may be more performant
http://wiki2.dovecot.org/MailboxFormat/dbox
NFS is used extensively in our network and we already have a major investment with NetApp, and have been very happy with them.
i believe you, but good cluster file systems may better fit with imap
The only issue I see right now is Dovecot index file write bashing if a user accesses their mailbox from multiple locations at the same time, and I gave that example in the previous post. That seems to be more and more common these days but still not the norm for 80%+ of the mailboxes we have.
in modern times, people use to have many clients open ,same time, over imap i.e with mobiles comming over different ips, your setup solution should honor this, indexes then may get a problem, which you have to handle with
--snip index files
If you keep the index files stored on NFS, you'll need to set mmap_disable=yes. If you're not running lockd you'll have to set lock_method=dotlock, but this degrades performance. Note that some NFS installations have problems with lockd. If you're beginning to get all kinds of locking related errors, try if the problems go away with dotlocking.
With mbox/Maildir format it's also possible to store index files on local disk instead of on NFS. If the user gets redirected to different servers, the local indexes are automatically created/updated. If the user is (nearly) always redirected to the same server this should be fine and you would likely get higher performance than indexes stored on NFS, but if the server changes it can be slow to recreate the index/cache files. --snip
From the MTA side of things, it looks like having Exim write directly into the ~/Maildir/new/ is the only sane choice since mail can be coming into a mailbox from a number of servers at the same time. We've used this extensively in the past with no issues.
dont know exim, i use postfix lmtp, and a cluster file system storage drbd, no failures so far
We use a Brocade ADX to load balancing POP and IMAP sessions to the server.
dont know brocade, but good working loadbalancers is a must have , you may also think about using directors additional
http://wiki2.dovecot.org/Director
shouldnt harm you
Stickiness doesn't really buy you anything on POP3 since a session connects, does it's thin and disconnects. IMAP on the other hand, I believe can have multiple connections open from the same client, so I think sticky would help. But neither of those help when the client is connecting from a different IP address.
Robert Blayzor INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/
i think your setup is quite common, anyway it may not be ideal i would recommand to hire dovecot specialist and/or Timos Company directly to do consulting with such big setups
-- Best Regards MfG Robert Schetterer
On 2012-07-16 10:19 AM, Robert Blayzor rblayzor.bulk@inoc.net wrote:
The only issue I see right now is Dovecot index file write bashing if a user accesses their mailbox from multiple locations at the same time, and I gave that example in the previous post. That seems to be more and more common these days but still not the norm for 80%+ of the mailboxes we have.
My understanding is that this (problems with NFS) is precisely what Timo wrote the Director functionality for...
http://wiki2.dovecot.org/Director
--
Best regards,
Charles
On Jul 16, 2012, at 8:58 AM, Robert Blayzor wrote:
In reading the Dovecot NFS WIKI entry:
It mentions a FreeBSD NFS client caching bug. I believe that bug only exists pre-8.x ? It seems that a patch in the PR notes was MFC'd and made it into 8.x at some point. (at least I checked the source in 8.3 and it was there). So I assume that FreeBSD 8.x mitigates the issue? Maybe not?
So our deployment is multi server Dovecot/Exim running on FreeBSD servers, frontended by a layer3 load balancing switch and a NetApp NFSv3 datastore.
Our deployment is similar. Dovecot/Exim on Oracle Linux 6.2 (RHEL derivative), layer3 load balancing (pacemaker/ldirectord managing LVS, "sticky" sessions per client IP) and NetApp NFSv3 datastore (maildir).
The only real concern we have right now is race conditions and file write contention on the Dovecot index files. Our original plan was to use Dovecot LMTP on the servers to deliver to the inbox, but that doesn't seem to be recommended as we'll have a number of servers all acting as primary MX for domains and delivering to the same set of mailboxes. So are the Dovecot dotlock file locking mechanisms not enough for multiple servers trying to write/access index files in Maildirs?
I can't speak to FreeBSD but we have had no problems using the NetApp NFS lock manager (lock_method = fcntl) instead of dotlock. Previously we were using dotlock and any skew in clocks on the Dovecot servers would cause higher CPU usage and tons of extra logging. If it works for you, try to use the NetApp NFS lock manager instead of dotlock.
It sounds like you have a bunch of hosts running both Exim and Dovecot and they are all your MX hosts? If so, it sounds like you could use Dovecot's LDA binary or LMTP. Since you are running Dovecot also on your Exim hosts and thus don't need to go over the network to a different host to see the mailstore/deliver the message, I would stick with LDA but both options will work.
If we have to deliver straight to the ~/Maildir/new out of Exim, that's fine, also. We're just wondering if there is a added benefit of actually using LMTP. (other than making sure the index are kept the most recent).
I believe the latest word from Timo is that the benefit of index updating from LDA/LMTP delivery for maildir is minimal, especially if your indexes are on NFS. We do use LDA for delivery with indexes on NFS, and have not had problems (~7000 user site). The important settings are:
# Don't use mmap() at all. This is required if you store indexes to shared # filesystems (NFS or clustered filesystem). #mmap_disable = no mmap_disable = yes
# When to use fsync() or fdatasync() calls: # optimized (default): Whenever necessary to avoid losing important data # always: Useful with e.g. NFS when write()s are delayed # never: Never use it (best performance, but crashes can lose data) #mail_fsync = optimized mail_fsync = always
# Mail storage exists in NFS. Set this to yes to make Dovecot flush NFS caches # whenever needed. If you're using only a single mail server this isn't needed. #mail_nfs_storage = no mail_nfs_storage = yes # Mail index files also exist in NFS. Setting this to yes requires # mmap_disable=yes and fsync_disable=no. #mail_nfs_index = no mail_nfs_index = yes
The other concern we have is client mailbox access. While we can make sessions "sticky" from one IP so they hit the same server, it doesn't help with the person that checks their email via POP from their PC while having their mobile phone access it via IMAP. (obviously from different IP addresses). So I guess there is some possibly index file write contention there as well. (though hopefully this should be rare).
We make sessions "sticky" and have the same "different client IP, same username" pattern. We haven't had problems with this setup since switching away from dotlock to NFS locks.
I'm curious as to know others experiences with a similar deployment.
I should mention we are hoping to change this deployment soon. As others have said, you would probably be better served at least with Dovecot Director (to ensure client access and Exim deliveries using LMTP for user X always hits server Y) and possibly mdbox instead of maildir. We have been testing Director and I feel good enough about it to move indexes off of NFS onto local disk and setting a scheduled task to update indexes on every server at some reasonable interval so a server failure doesn't require a complete index rebuild when users tied to the failed server get sent to a different one. Migrating from maildir to mdbox is on my to-investigate list once Director is implemented but without having tested a migration there is a lot to like in terms of Alternate Storage and reduced IOPS v Maildir. You just have to *really* trust those index files because they are the only place flags/keywords are stored.
TIA
-- Robert Blayzor INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/
HTH, David Warden
On Jul 16, 2012, at 11:34 AM, David Warden wrote:
I can't speak to FreeBSD but we have had no problems using the NetApp NFS lock manager (lock_method = fcntl) instead of dotlock. Previously we were using dotlock and any skew in clocks on the Dovecot servers would cause higher CPU usage and tons of extra logging. If it works for you, try to use the NetApp NFS lock manager instead of dotlock.
Right. I believe Pre 7.x on FreeBSD, NFS client side locking was rather dodgy or non-existent. I believe in more recent version 8.x+ (and perhaps even some in 7.x), NFS locks client to server locks are supported. But this is rather encouraging and we'll do some testing with it. I wasn't a fan of going with dotlock...
It sounds like you have a bunch of hosts running both Exim and Dovecot and they are all your MX hosts? If so, it sounds like you could use Dovecot's LDA binary or LMTP. Since you are running Dovecot also on your Exim hosts and thus don't need to go over the network to a different host to see the mailstore/deliver the message, I would stick with LDA but both options will work.
Yes, Exim/Dovecot same servers, multiple servers. I had LDA/LMTP working fine until I read the notes that if mail is being delivered simultaneously from multiple servers that this should not be used as it's not safe for the indexing. If fcntl locking fixes this, then we'll look at it again. I was weighing the benefits of doing index updating at STMP time vs just doing it when the user logs in, and I'm leaning toward just having Exim deliver directly to the ~/Maildir/new instead of using LDA. If a user gets 100 emails in their box, but doesn't login for days, no sense in updating all of the index files 100 times at SMTP time, and just do it once when they login. Flawed logic? Also LMTP is a bit more protocol over and is another database lookup on the Dovecot side, etc.
I believe the latest word from Timo is that the benefit of index updating from LDA/LMTP delivery for maildir is minimal, especially if your indexes are on NFS. We do use LDA for delivery with indexes on NFS, and have not had problems (~7000 user site). The important settings are:
And that's exactly what I'm thinking.
# Don't use mmap() at all. This is required if you store indexes to shared # filesystems (NFS or clustered filesystem). #mmap_disable = no mmap_disable = yes
# When to use fsync() or fdatasync() calls: # optimized (default): Whenever necessary to avoid losing important data # always: Useful with e.g. NFS when write()s are delayed # never: Never use it (best performance, but crashes can lose data) #mail_fsync = optimized mail_fsync = always
# Mail storage exists in NFS. Set this to yes to make Dovecot flush NFS caches # whenever needed. If you're using only a single mail server this isn't needed. #mail_nfs_storage = no mail_nfs_storage = yes # Mail index files also exist in NFS. Setting this to yes requires # mmap_disable=yes and fsync_disable=no. #mail_nfs_index = no mail_nfs_index = yes
Righto, we've got all that. Except right now we're just testing with dotlocking... I'm going to have to try fcntl locking and see what that gets me. Perhaps someone here is using FreeBSD with nfs locking now and can share experiences as well.
We make sessions "sticky" and have the same "different client IP, same username" pattern. We haven't had problems with this setup since switching away from dotlock to NFS locks.
This is encouraging news. Keeping a client sticky is important I think, especially for IMAP... no so much for POP. I guess in the load balancer you could create a client protocol group and make them all hit the same server, but doing pop/IMAP from the same client would be unusual unless it was several different client types behind NAT.
I should mention we are hoping to change this deployment soon. As others have said, you would probably be better served at least with Dovecot Director (to ensure client access and Exim deliveries using LMTP for user X always hits server Y) and possibly mdbox instead of maildir. We have been testing Director and I feel good enough about it to move indexes off of NFS onto local disk and setting a scheduled task to update indexes on every server at some reasonable interval so a server failure doesn't require a complete index rebuild when users tied to the failed server get sent to a different one. Migrating from maildir to mdbox is on my to-investigate list once Director is implemented but without having tested a migration there is a lot to like in terms of Alternate Storage and reduced IOPS v Maildir. You just have to *really* trust those index files because they are the only place flags/keywords are stored.
I'm not really up to speed on the dbox format, but I did look at it briefly. I'd like to stick with maildir if at all possible. It's worked for us for years, even over NFS, with little or no problems. I think your always going to have problems with clients accessing from multiple locations/servers, etc... It's just how complex you want to get. Director seems like a good step, but also adds another layer for us to test and maintain. We're looking at that too..
Thanks for your input!
-- Robert Blayzor INOC, LLC rblayzor@inoc.net http://www.inoc.net/~rblayzor/
participants (5)
-
Charles Marcus
-
David Warden
-
Robert Blayzor
-
Robert Schetterer
-
Roland Rosenfeld