[Dovecot] nfs director
Halo, Please can you explain why this is advantage over a hardware load balancer. I fail to see advantage if anything it add in more point of failure, with several hundred thousand user, we can ill afford to mess around or add to complexity, sometime keeping it simple is simply way to be, when use qmail/vpopmail, we never had one failure or problem, ever, we very proud of this record so our users.
if director service assign 60K user to each front end, how it handle if 5K simultaneous user login, but all 5K happen to be assign to that one machine, it do all work whilst other 7 server sit there do nothing negating what the LB is design for?
Is it really worth it? Do we really need this, or just let foundry switch handle it as it does now. We also have 24 front end SMTP server, these deliver mail to netapp filer, all 24 plus 8 pop3 server and 2 webmail imap server all mount /vmail, so all access same maildir. it seem work very effective thus far and for many many year when we use qmail and vpopmail.
Thaks
On 26.8.2010, at 23.54, Edward avanti wrote:
Please can you explain why this is advantage over a hardware load balancer.
It guarantees that the same user is accessed via the same server. Hardware LB can at best assign the connections from the same IP to the same server (but not e.g. new mail deliveries, or if user has multiple clients like home/work/mobile, or simply accesses webmail at the same time as has a client open).
I fail to see advantage if anything it add in more point of failure, with several hundred thousand user, we can ill afford to mess around or add to complexity, sometime keeping it simple is simply way to be, when use qmail/vpopmail, we never had one failure or problem, ever, we very proud of this record so our users.
Are you using Dovecot? For how long? Do you see any errors in your logs?
if director service assign 60K user to each front end, how it handle if 5K simultaneous user login, but all 5K happen to be assign to that one machine, it do all work whilst other 7 server sit there do nothing negating what the LB is design for?
The users are distributed based on the MD5 hash of their username and that gives a pretty good distribution of where the users go. Unless 5k of your users suddenly decide to coordinate such attack, I doubt you'll ever see anything even close to that happening. But sure, there is some variation. I don't have any real world numbers, but my guess is it's normally less than 20%.
Also with some more work it would be possible to dynamically adjust how many new users are getting assigned to servers, but I wasn't planning on implementing that unless it becomes clear that it's needed.
Is it really worth it? Do we really need this, or just let foundry switch handle it as it does now. We also have 24 front end SMTP server, these deliver mail to netapp filer, all 24 plus 8 pop3 server and 2 webmail imap server all mount /vmail, so all access same maildir. it seem work very effective thus far and for many many year when we use qmail and vpopmail.
If you don't mind random Dovecot errors about index corruption I guess you're fine with how it works now. I guess your mails are delivered to maildirs by qmail? If you ever switch to Dovecot LDA you'll probably start getting more errors. And if you ever plan to switch to dbox format then at latest you'll need director.
On Fri, Aug 27, 2010 at 9:31 AM, Timo Sirainen tss@iki.fi wrote:
On 26.8.2010, at 23.54, Edward avanti wrote:
Please can you explain why this is advantage over a hardware load balancer.
It guarantees that the same user is accessed via the same server. Hardware LB can at best assign the connections from the same IP to the same server (but not e.g. new mail deliveries, or if user has multiple clients like home/work/mobile, or simply accesses webmail at the same time as has a client open).
But how does it help when new mail arrive on one of the 24 SMTP server, where there can be no direction that I can see. You might control where user login to get mail, but not which server accept mail.
I fail to see advantage if anything it add in more point of failure, with several hundred thousand user, we can ill afford to mess around or add to complexity, sometime keeping it simple is simply way to be, when use qmail/vpopmail, we never had one failure or problem, ever, we very proud of this record so our users.
Are you using Dovecot? For how long? Do you see any errors in your logs?
for not very long, few month.
if director service assign 60K user to each front end, how it handle if 5K simultaneous user login, but all 5K happen to be assign to that one machine, it do all work whilst other 7 server sit there do nothing negating what the LB is design for?
The users are distributed based on the MD5 hash of their username and that gives a pretty good distribution of where the users go. Unless 5k of your users suddenly decide to coordinate such attack, I doubt you'll ever see anything even close to that happening. But sure, there is some variation. I don't have any real world numbers, but my guess is it's normally less than 20%.
there is a reason we have 8 pop server, in evening, pop mail is very busy by many user, many server often have thousand of concurrent user session but not multiple session per user.
Also with some more work it would be possible to dynamically adjust how many new users are getting assigned to servers, but I wasn't planning on implementing that unless it becomes clear that it's needed.
Is it really worth it? Do we really need this, or just let foundry switch handle it as it does now. We also have 24 front end SMTP server, these deliver mail to netapp filer, all 24 plus 8 pop3 server and 2 webmail imap server all mount /vmail, so all access same maildir. it seem work very effective thus far and for many many year when we use qmail and vpopmail.
If you don't mind random Dovecot errors about index corruption I guess you're fine with how it works now. I guess your mails are delivered to maildirs by qmail? If you ever switch to Dovecot LDA you'll probably start
we have convert many qmail to postfix, sunday morning we plan migrate remaining 14 to postfix.
getting more errors. And if you ever plan to switch to dbox format then at latest you'll need director.
So this all because of index? what is ramification for user if this corruption occur? is it self healing over new pop or delivery? Can index be disable? We thought using dovecot deliver, but this seem not good idea now? As dovecot not know how qmail or postfix put mail in, since just put it in, can not dovecot deliver work same? or have only dovecot on pop3 server update index?
On 27.8.2010, at 1.47, Edward avanti wrote:
Please can you explain why this is advantage over a hardware load balancer.
It guarantees that the same user is accessed via the same server. Hardware LB can at best assign the connections from the same IP to the same server (but not e.g. new mail deliveries, or if user has multiple clients like home/work/mobile, or simply accesses webmail at the same time as has a client open). But how does it help when new mail arrive on one of the 24 SMTP server, where there can be no direction that I can see. You might control where user login to get mail, but not which server accept mail.
If you're delivering mails using standard Maildir then yeah, it's ok that your 24 SMTP server writes it. If you touch any of Dovecot's index files then you can't do it on the SMTP server 100% safely. Instead you'd have to have your SMTP server deliver it to the correct destination server using LMTP protocol via LMTP/director proxy.
there is a reason we have 8 pop server, in evening, pop mail is very busy by many user, many server often have thousand of concurrent user session but not multiple session per user.
Yeah, with POP3 there aren't much problems since there is almost always only a single client. With IMAP it's different.
getting more errors. And if you ever plan to switch to dbox format then at latest you'll need director.
So this all because of index?
Pretty much, yes. But also the dovecot-uidlist file that keeps the UID <-> filename mapping. That's less likely to break though.
what is ramification for user if this corruption occur?
Random .. usually slight performance loss, maybe kicking out one user session but the next one working. But I can't guarantee that it won't be worse.
is it self healing over new pop or delivery?
There have also been corruption bugs that Dovecot hasn't managed to fix automatically, but it's been a long time since those happened with maildir.
Can index be disable?
In delivery time or everywhere? In delivery yes, you can disable indexes, but you can't disabled dovecot-uidlist updates and those can also break.
We thought using dovecot deliver, but this seem not good idea now? As dovecot not know how qmail or postfix put mail in, since just put it in, can not dovecot deliver work same? or have only dovecot on pop3 server update index?
Yes, it would be possible to change Dovecot code so that deliver won't update dovecot-uidlist either, but that makes the code just more complex so I haven't wanted to waste my time adding workarounds for a few NFS users..
Hi,
If you don't mind random Dovecot errors about index corruption I guess you're fine with how it works now. I guess your mails are delivered to maildirs by qmail? If you ever switch to Dovecot LDA you'll probably start getting more errors. And if you ever plan to switch to dbox format then at latest you'll need director.
Ah! This probably explains why we're not being hit as hard as some others. We dont use dovecot lda. So we only see problems when users have 2 or more clients open and happen to hit the exact same polling times. We do have plans to move to dovecot lda so good to know our problems would have increased a lot. Should start testing with the director this weekend. Got 4 servers to play with.
Cor
On Fri, 2010-08-27 at 08:54 +1000, Edward avanti wrote:
Halo, Please can you explain why this is advantage over a hardware load balancer.
it is no advantage over a dedicated hardware solution, but director does not do the exact same thing.
I fail to see advantage if anything it add in more point of failure, with
i agree with this and it is why we dont use it
we use dovecots deliver with postfix and have noticed no problems, not to say there was none, but if so, we dont notice it. postfix looks up the user, it determines if it accepts the mail, if it does, it queues it for mailscanner to do its stuff, then gives it back to postfix, which is then told to give it to dovecots deliver, it makes no sense to me that it should then be sent to another machine just to be stored on a remote file server, the same remote file server the initial server assigned that conenction by a true load balancer has mounted and would store it to as well.... would be miuch easier to have deliver ignore the index file by an option, eliminating the corruption risks to the index file and just storing the darm thing. or am i only one who thinks mail systems do not need to be complex to run faultlessly, I think those who feel the need to make it very complex are not only looking for trouble, but further trying to justify their position to their employer that they are indispensable.
if director service assign 60K user to each front end, how it handle if 5K simultaneous user login, but all 5K happen to be assign to that one machine,
that would be rare, but, technically speaking, if you are that large in user numbers, it is a possible scenario
Is it really worth it? Do we really need this, or just let foundry switch handle it as it does now. We also have 24 front end SMTP server, these deliver mail to netapp filer, all 24 plus 8 pop3 server and 2 webmail imap server all mount /vmail, so all access same maildir. it seem work very effective thus far and for many many
Sounds similar setup to us, smtp, pop3 and webmail all mounting /var/vmail/ on a FAS2050, I've asked if it can avoid touching the index files before (see a thread as recent as a few weeks back), Timo is just not interested, to much work apparently for so little users (although I never in all hte years ive been on this list, ever seen a poll taken/question asked to users - about it, plus, well, every single dovecot user is on this list, right? <sarcasm> anyway, mostly I guess although it has risks, it seems to work for everyone who uses NFS anyway and has done for very many years :) , maybe one day when Timo is so bored and cant think of anything to add, he will give us an option, or a dedicated deliver binary separate to normal deliver that does this)
/rant ( but its nice to know im not the only one here who feels this way) Cheers
Noel,
On 8/26/10 9:59 PM, "Noel Butler" noel.butler@ausics.net wrote:
I fail to see advantage if anything it add in more point of failure, with
i agree with this and it is why we dont use it
we use dovecots deliver with postfix and have noticed no problems, not to say there was none, but if so, we dont notice it.
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop.
Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy.
Timo has been saying for YEARS that you need user-node affinity if you're doing NFS, and now he's done something about it. If you've already got a load balancer, then just point the balancer at a pool of directors, and then point the directors at your existing mailserver pool.
<shameless plug> For health monitoring on the directors, check out: http://github.com/brandond/poolmon </shameless plug>
-Brad
Brandon, I just fail to see why adding more complexity, and essentially making $9K load balancers redundant, is the way of the future, Timo has said its very safe for index's if non dovecot programs write to the maildir, so why the hell is it deliberately left risky using dovecots deliver, I've seen this all before in other setups/software, adding extras that depend on this that and whatever, to make it nifty and play nice when it can be done a simpler way, and it always leads to higher downtime in the end, hence my refusal to go the director way, the simplest and easiest out is to stop using deliver and use postfix's virtual which is what Ill look at if it gives us problems that way there will be no risk (according to Timo) and without added programs running and depending on each other, thus keeping our points of failure low which is why our mail servers have not had one single bit of downtime since I took over.
point in case is with hte OP's initial comment:
"if director service assign 60K user to each front end, how it handle if 5K simultaneous user login, but all 5K happen to be assign to that one machine, it do all work whilst other 7 server sit there do nothing negating what the LB is design for?"
makes perfect sense if he is that big that it assings 60K to each director that in peak periods theres a real risk, no mater how low, that everyone logging in, is in one particular directors list, flooring that box with I/O whilst his others sit there with one or two users on it.
I really thought we got over the NFS corruption stuff when Daniel wrote Maildir ... *sigh*
On Thu, 2010-08-26 at 22:28 -0700, Brandon Davidson wrote:
Noel,
On 8/26/10 9:59 PM, "Noel Butler" noel.butler@ausics.net wrote:
I fail to see advantage if anything it add in more point of failure, with
i agree with this and it is why we dont use it
we use dovecots deliver with postfix and have noticed no problems, not to say there was none, but if so, we dont notice it.
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop.
Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy.
Timo has been saying for YEARS that you need user-node affinity if you're doing NFS, and now he's done something about it. If you've already got a load balancer, then just point the balancer at a pool of directors, and then point the directors at your existing mailserver pool.
<shameless plug> For health monitoring on the directors, check out: http://github.com/brandond/poolmon </shameless plug>
-Brad
Noel,
On 8/26/10 11:28 PM, "Noel Butler" noel.butler@ausics.net wrote:
I just fail to see why adding more complexity, and essentially making $9K load balancers redundant, is the way of the future.
To each their own. If your setup works without it, then fine, don't use it... but I don't see why you feel the need to disparage it either. It's hardly bloat; those of us with larger installations do find it useful. IIRC it was sponsored development, and was running in production for a large ISP from the very moment it was released.
-Brad
On Fri, 2010-08-27 at 04:04 -0700, Brandon Davidson wrote:
To each their own. If your setup works without it, then fine, don't use it... but I don't see why you feel the need to disparage it either. It's
I'll some it up put well by someone who mailed me offlist... mx-in-1 gets the connection, postfix looks up user in mysql, mysql says " hey i know him " posfix says to sender " send away", then, postfix applies its filters/clamav/spamassassin,(so by now all the REAL hard work has been done) so now postfix says OK dovecot-lda here it is so you can deliver to the NFS mounted dir, but WAIT says dovecot-lda, my director says no i'm not the driveway you want, pop over and drive in using to mx-in-2, so that server then gets it and whatever else it wants to do with it now before giving it off to hte same NFS server that mx-in-1 had.. now., this might not be so funny when you have two boxes, but if you have many, or 20 or so like the OP... *shakes head* All they are doing FFS is passing it along. regardless of if mx-in-2 does anything else with it, it seems kinda strange and very backward routing mail to another server, just to deliver on yet another device, double handling comes to mind, even if it doesnt rescan msg and go through all the filters again, its still an unnecessary step to send it to another box, just to be stored on, yet another... I'd like someone to sanely justify that to me.
hardly bloat; those of us with larger installations do find it useful. IIRC
I dont know how large your operation is, but I suspect my 118K mailbox's and yours together still dont match the OP's 400K And anything that adds to requirements of a server that is not needed in other aspects, is bloat, maybe some setups this is fine, I can not justify modifying mine to include extra points of failure when it all works fine.
If it becomes a problem all I need to do is modify all MTA postfix main.cf's to not use dovecot as virtual delivery, thats commenting out one single line, thats it, (tested already), the only difference is postfix is still in dark ages and uses Maildir, not Maildir++, but that is hardly a problem :)
ah well, its the weekend, so i'm out of this madness now for a few days.
I dont think we are living in the 19th century now, I think its time for the html to txt conversion to be scrapped, its screwed up the paragraph formatting ( and few other things in recent times I've seen) more than once, making it look like an a5 size book page.
how about it?
On Sat, 2010-08-28 at 13:11 +1000, Noel Butler wrote:
On Fri, 2010-08-27 at 04:04 -0700, Brandon Davidson wrote:
To each their own. If your setup works without it, then fine, don't use it... but I don't see why you feel the need to disparage it either. It's
I'll some it up put well by someone who mailed me offlist... mx-in-1 gets the connection, postfix looks up user in mysql, mysql says " hey i know him " posfix says to sender " send away", then, postfix applies its filters/clamav/spamassassin,(so by now all the REAL hard work has been done) so now postfix says OK dovecot-lda here it is so you can deliver to the NFS mounted dir, but WAIT says dovecot-lda, my director says no i'm not the driveway you want, pop over and drive in using to mx-in-2, so that server then gets it and whatever else it wants to do with it now before giving it off to hte same NFS server that mx-in-1 had.. now., this might not be so funny when you have two boxes, but if you have many, or 20 or so like the OP... *shakes head* All they are doing FFS is passing it along. regardless of if mx-in-2 does anything else with it, it seems kinda strange and very backward routing mail to another server, just to deliver on yet another device, double handling comes to mind, even if it doesnt rescan msg and go through all the filters again, its still an unnecessary step to send it to another box, just to be stored on, yet another... I'd like someone to sanely justify that to me.
hardly bloat; those of us with larger installations do find it useful. IIRC
I dont know how large your operation is, but I suspect my 118K mailbox's and yours together still dont match the OP's 400K And anything that adds to requirements of a server that is not needed in other aspects, is bloat, maybe some setups this is fine, I can not justify modifying mine to include extra points of failure when it all works fine.
If it becomes a problem all I need to do is modify all MTA postfix main.cf's to not use dovecot as virtual delivery, thats commenting out one single line, thats it, (tested already), the only difference is postfix is still in dark ages and uses Maildir, not Maildir++, but that is hardly a problem :)
ah well, its the weekend, so i'm out of this madness now for a few days.
On 8/27/10 11:15 PM, Noel Butler wrote:
I dont think we are living in the 19th century now, I think its time for the html to txt conversion to be scrapped, its screwed up the paragraph formatting ( and few other things in recent times I've seen) more than once, making it look like an a5 size book page.
how about it?
Oh right, the 20th century is the "century of protocol abuse" for people who think everything on the network should be a web page, and everything on the net should be accessed with a web browser.
If this change is made, I for one will ditch this list and just rely on searching the archives. I get enough HTML garbage from clueless morons all day long, I don't need more of it from a supposedly clueful group.
-Dave-- Dave McGuire Port Charlotte, FL
Dave McGuire put forth on 8/27/2010 10:43 PM:
If this change is made, I for one will ditch this list and just rely on searching the archives. I get enough HTML garbage from clueless morons all day long, I don't need more of it from a supposedly clueful group.
I use 'mailnews.display.prefer_plaintext;true' and I never see HTML formatted emails. Once this feature started working reliably (years ago) in Thunderbird, I stopped posting complaints such as yours to the lists I participate in. At one point I was a text-only zealot like you. MUA technology solved the problem for me. You might try it. :)
There will be no HTML everywhere conversion in the future. The internet and the web are both built upon RFCs. There is even an RFC defining the format of RFCs. They are to be plain text only, formatted with exactly x columns and rows per page, fixed 10 point type, courier--ASCII. If the HTML zealots ever attempt an overthrow of the net, the Sun will go supernova and destroy the Earth, and there will be no one left to worry about it. So feel safe in knowing that text only will survive forever in many corners of the web, and the world. :)
-- Stan
I am with Dave on this one.
"Dave McGuire" mcguire@neurotica.com wrote:
On 8/27/10 11:15 PM, Noel Butler wrote:
I dont think we are living in the 19th century now, I think its time for the html to txt conversion to be scrapped, its screwed up the paragraph formatting ( and few other things in recent times I've seen) more than once, making it look like an a5 size book page.
how about it?
Oh right, the 20th century is the "century of protocol abuse" for people who think everything on the network should be a web page, and everything on the net should be accessed with a web browser.
If this change is made, I for one will ditch this list and just rely on searching the archives. I get enough HTML garbage from clueless morons all day long, I don't need more of it from a supposedly clueful group.
-Dave-- Dave McGuire Port Charlotte, FL
-- Richard
Sent from my Dell Streak with K-9 Mail. Please excuse my brevity.
Noel, I think you just dont quite understand the problem the director is solving.
The issue is that NFS is not lock-safe over multiple servers. We have 35 imap servers accessing a central NFS cluster. (we have over a million mailboxes) We offer IMAP to end user clients, and through webmail. This means that users are more and more likely to have multiple mail clients open.
they have a mail client open at home, lets say Thunderbird or OSX Mail. When they go to work they leave it on, so the software keeps looking for mail.
At work, they open webmail, so they can access their private email through their companies firewall. They leave this webmail session open in a browser tab.
They also have an iphone, and it's continuously checking their mail as well.
This is not farfetched. This is normal behavior when you start offering IMAP access.
We of course have a hardware loadbalancer (foundry) that directs incoming connections. But this loadbalancer does not know the 3 connections above are from the same user. So each gets directed to a different imap server.
All these 3 sessions want to update the index files. (im current not including dovecot LDA, which also wants to update the index files). Because of issues inherent in NFS, the 3 imap servers that handle these connections may all think they have an exclusive lock. They may end up writing to the index files at the same time because of this. So the end result is a corrupt index file, which causes problems in the clients. The chances of this happening get higher as you have more users and more servers.
Now there is a workaround. NFS locking can be made to work better if all processes trying to access the NFS indexes are on the same machine. So if all 3 clients in the above example happen to end up on the same imap server, there wouldnt be a problem. That imap server can safely say 'sorry, you cant write to the index file right now, another process is already writing to it'.
What the dovecot director is doing is ensuring that sessions from the same user all get directed to the same imap server, so NFS locking works safely.
I wish my foundry could do this, so I wouldnt need the director, but alas, it cant. If you operate a situation like im describing above, you WILL need the director, or accept that your users may run into problems reading their mail.
If you dont see this problem, you either are not running the same situation im describing, or you do have this problem but just dont know :) I am very happy that Timo implemented this, so those of us that run this setup and are experiencing this issue, have a way to work around it.
Regards,
Cor
On Sat, 2010-08-28 at 09:18 +0200, Cor Bosman wrote:
Noel, I think you just dont quite understand the problem the director is solving.
The issue is that NFS is not lock-safe over multiple servers. We have 35 imap servers accessing a central NFS cluster. (we have over a million mailboxes) We offer IMAP to end user clients, and through webmail. This means that users are more and more likely to have multiple mail clients open.
we have a some total of 2 imap servers, only used for webmail, so it is not a problem for us, at all, period. director will not solve those inbound mail server issues, the REAL issue at hand here being discussed, and anyone not knowing this should not be partaking in this thread.
This is not farfetched. This is normal behavior when you start offering IMAP access.
imap access is not common in this country, it is in fact extremely rare, common only for webmail servers.
All these 3 sessions want to update the index files. (im current not including
not an issue as I previously stated (twice I believe) as we do not permit multi session logins
dovecot LDA, which also wants to update the index files). Because of issues
funny how postfix doesnt have this issue, qmail doesnt have this issue nor does exim (never ran sendmail in this config only a crazy person would use mbox over NFS)
Now there is a workaround. NFS locking can be made to work better if all processes trying to access the NFS indexes are on the same machine. So
there certainly is a workaround, its called, in our case, use postfix's deliver, which under my instruction has already happened on half the servers since it is clear this issue will not ever be solved in dovecot.
What the dovecot director is doing is ensuring that sessions from the same user all get directed to the same imap server, so NFS locking works safely.
yes, this has been mentioned some 40 times already, you know, i do actually know this, however that will NOT solve the problem as outlined earlier, the director by Timos admission will NOT load balance users in its current state, nor will it load balance inbound connections.
I wish my foundry could do this, so I wouldnt need the director, but alas, it cant. If you operate a situation like im describing above, you WILL need the director, or accept that your users may run into problems reading their mail.
well they aint needed it since 2003, nothings died, broken, crashed, caused hurricanes, droughts, whatever, so, cant see why all of a sudden they will now.
Hi Noel, if you dont need the director, then thats great right? Why does anyone need to justify anything? Just dont use it, end of discussion. Those of us that do have a need for it, can use it anyways. Even without your agreement? Is that such a big problem?
we have a some total of 2 imap servers, only used for webmail, so it is not a problem for us, at all, period.
Awesome.
imap access is not common in this country, it is in fact extremely rare, common only for webmail servers.
It's common here. We have tens of thousands of direct imap connections, and they're often from the same user. Seeing 3 or more connections from the same user on different remote IP numbers is not uncommon.
not an issue as I previously stated (twice I believe) as we do not permit multi session logins
Not for you perhaps, but it is for us, and it is for others. So im glad you dont have this issue, we do, and the director solves it. No need to bash the director service just because you dont like it.
dovecot LDA, which also wants to update the index files). Because of issues
funny how postfix doesnt have this issue, qmail doesnt have this issue nor does exim (never ran sendmail in this config only a crazy person would use mbox over NFS)
We dont use mbox. We use maildir. I agree only a crazy person would use mbox in a large scale environment.
Postfix has this issue as well. So does qmail. So does exim. It has nothing to do with the software being used. It is a problem in the NFS protocol. (it's not a locking issue as I mentioned, but a caching issue as timo corrected.. i knew that, but in my haste to respond had locking in my mind. same principle applies. Multiple servers accessing the indexes can cause corruption due to inherent problems in the NFS protocol).
The reason you dont see this, is because you dont allow simultaneous logins and dont offer direct imap access. Lets have this discussion again once you offer that :)
there certainly is a workaround, its called, in our case, use postfix's deliver, which under my instruction has already happened on half the servers since it is clear this issue will not ever be solved in dovecot.
Good for you. Im happy it works for you.
yes, this has been mentioned some 40 times already, you know, i do actually know this, however that will NOT solve the problem as outlined earlier, the director by Timos admission will NOT load balance users in its current state, nor will it load balance inbound connections.
Huh? Guess I dont understand what you outlined earlier. The director does load balance users. It uses an MD5 hash to do this, which in theory could cause uneven loadbalancing, but I know for a fact that in practice it poses no problem as we've been using an MD5 hash to loadbalance our webmail frontend to our imap servers for years. Thats 20.000+ simultaneous users and they get very evenly spread.
Maybe I just dont understand what your issues with other people using the director are. If you have a solid system without the director, more power to you. Thats great. Congrats.
Regards,
Cor
Hi,
Postfix has this issue as well. So does qmail. So does exim. It has nothing to do with the software being used. It is a problem in the NFS protocol.
Just to be clear. Of course these programs wont have this issue when used with dovecot-imap, because obviously they wont be updating any dovecot index files. You can tell dovecot-lda to not update index files, and you'd have exactly the same behavior. Combine that with not allowing multiple logins, and indeed, you wont ever need the director.
What I meant was, if qmail, or exim, or postfix, would use some kind of indexing like dovecot, and would allow that over NFS, they'd have the same underlying issue to deal with. It's actually quite amazing and cool that Timo has addressed this problem for those that are using dovecot-lda with indexing over NFS, and/or dovecot-imap with multiple logins by the same user over NFS.
Cor
On Wed, Sep 1, 2010 at 8:53 PM, Noel Butler noel.butler@ausics.net wrote:
On Sat, 2010-08-28 at 09:18 +0200, Cor Bosman wrote:
imap access is not common in this country, it is in fact extremely rare, common only for webmail servers.
Much of Asia also no use imap as primary mail for customer
All these 3 sessions want to update the index files. (im current not including
not an issue as I previously stated (twice I believe) as we do not permit multi session logins
dovecot LDA, which also wants to update the index files). Because of issues
funny how postfix doesnt have this issue, qmail doesnt have this issue nor does exim (never ran sendmail in this config only a crazy person would use mbox over NFS)
Now there is a workaround. NFS locking can be made to work better if all processes trying to access the NFS indexes are on the same machine. So
there certainly is a workaround, its called, in our case, use postfix's deliver, which under my instruction has already happened on half the servers since it is clear this issue will not ever be solved in dovecot.
This easy yes, since with Qmail, we have Maildir SQL column in mailtable, simple to have postfix select Maildir same as old Qmail, we change very little to use this configuration which is why we use.
What the dovecot director is doing is ensuring that sessions from the
same
user all get directed to the same imap server, so NFS locking works safely.
yes, this has been mentioned some 40 times already, you know, i do actually know this, however that will NOT solve the problem as outlined earlier, the director by Timos admission will NOT load balance users in its current state, nor will it load balance inbound connections.
I wish my foundry could do this, so I wouldnt need the director, but alas, it cant. If you operate a situation like im describing above, you WILL need the director, or accept that your users may run into problems reading their mail.
well they aint needed it since 2003, nothings died, broken, crashed, caused hurricanes, droughts, whatever, so, cant see why all of a sudden they will now.
We use Dovecot only recent, but many I speak use for very many year, if director was really need, why it only come about now and not 5 or more year ago, all many mail network run broken for so many year? I no think so. It might compliment some situation, but not suitable or advantage for all. Imap not likely happen here for user, hardware no so cheap here,such is storage space not big deal, not until we buy hardware for near cheap as U.S., which 50-70% less than we pay.
Hi,
We use Dovecot only recent, but many I speak use for very many year, if director was really need, why it only come about now and not 5 or more year ago, all many mail network run broken for so many year? I no think so. It might compliment some situation, but not suitable or advantage for all. Imap not likely happen here for user, hardware no so cheap here,such is storage space not big deal, not until we buy hardware for near cheap as U.S., which 50-70% less than we pay.
The director is not needed in most cases. If you feel you dont need the director, then just forget about its existence and ignore all emails about it. Just pretend it doesnt exist :)
I'll try to explain why the director is needed in some situations.
dovecot has an index file. If you look in your configuration file, you'll see something like:
mail_location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u
The index file holds an index so dovecot can search in your mailbox really fast. (sorry if you know all this). So, if for instance you want to sort your mailbox on subject, dovecot doesnt have to read through all your emails to find all the subjects. Instead, it can read the index file really fast, and return a sorted mailbox basically instantly. A lot of IMAP servers dont even have this feature! Imagine the alternative, where dovecout would have to read through 25.000 emails, just so it can sort your mailbox on subject. A nightmare for performance, but like I said, some imap servers still do that! Dovecot luckily uses an index, and is therefore faster than a lot of other imap servers.
So, we have concluded that dovecot has a really cool performance feature in the form of an index file.
Now imagine what happens if you have more than 1 imap server. We have 35 imap servers right now. Each server needs access to the index file. One popular way to do this is by using NFS. With NFS you have a central file server, and each imap server uses the remote fileserver for storage. So now, all 35 servers have access to a cool central index file.
But, there is a slight snag. NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. The index file can then end up being corrupted. This causes dovecot to give faulty data back to the client, which in practice ends up looking like the mailbox of the user is empty. Not good. This causes calls to support desks about why a user has just lost all his mail. It's not lost, it's just that the index file is corrupt, and now dovecot thinks the mail is gone.
You may wonder rightly so, why this is a problem only now. Well, it hasnt been a problem only now. We've been dealing with this issue for years. We've had discussions with Timo over the years on how to solve this problem. But Timo has only so much time, and this problem was not important enough to fix. He had more important bugs to fix. So, we lived with the problem, or worked around it in ugly ways.
Recently Timo has found the time (I believe an ISP paid Timo to solve this problem) to think about ways to solve it. It's really not an easy problem to fix. We've known for years that the best way to fix this would be to have a user always end up on the same server, because thats when that specific NFS problem doesnt occur. If 2 sessions of the same user both end up on Server 1, then both sessions will see the same data, and no problems can occur.
But how to get all sessions of a user on the same server? Most loadbalancers dont support this feature. Our Foundries dont, and our previous Alteons didnt either. So Timo thought of a way how dovecot itself could make sure all sessions of a user end up on the same server. Basically by turning dovecot into a proxy, that proxies connections to another server. It maintains an internal state on what user belongs on what server, and redirects connections appropriately. And thats all the director is. A proxy with an internal state of where to proxy a connection to.
So why dont you see this problem with your setup? Like I said, this is only a problem with specific setups.
you need to be using NFS for central index storage. If you dont use NFS for your indexes, you wont see this problem, and you wont need the director. carry on as you always have.
even if you use NFS, you may not run into this problem. Because you also need to have situations where multiple servers could be updating the indexes at the same time. There are only 2 situations where this could happen:
- you're using dovecot-lda, and have index updating turned on. Index updating in the lda is cool, because it means dovecot will always have the latest information about your email in the index files.
and/or
- you allow users to have sessions to multiple servers. For instance, they connect with Outlook and their iPhone at the same time. And maybe even a webmail connection as well.
If you dont have one of these 2 situations, you wont ever see a problem and you wont need the director. carry on as you always have.
But, for those (mostly larger) setups where you are using NFS for indexes, AND you can have multiple sessions of the same user on different servers (and they can be either LDA or IMAP sessions), you're probably better off using the director.
Again, if this doesnt apply to you, simply..... do not use the director.
Regards,
Cor
On 2010-09-02 9:08 AM, Cor Bosman cor@xs4all.nl wrote:
NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file.
Just curious, but is there no way to just disable the caching on the NFS share where the indexes are stored?
--
Best regards,
Charles
On Thu, 2010-09-02 at 09:47 -0400, Charles Marcus wrote:
On 2010-09-02 9:08 AM, Cor Bosman cor@xs4all.nl wrote:
NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file.
Just curious, but is there no way to just disable the caching on the NFS share where the indexes are stored?
Disabling attribute caching solves most of this, but increases the NFS traffic by 10x or so (and still isn't perfect).
Dovecot v1.1 added a setting where it tried to flush the NFS cache only when necessary. It worked better than nothing, but not perfectly. I tried fixing it for a long time, spent a lot of time reading Linux and FreeBSD NFS kernel code and writing test programs, trying to get Linux to fix some things .. then I finally gave up.
why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time.
On Thu, Sep 2, 2010 at 11:08 PM, Cor Bosman cor@xs4all.nl wrote:
Hi,
We use Dovecot only recent, but many I speak use for very many year, if director was really need, why it only come about now and not 5 or more year ago, all many mail network run broken for so many year? I no think so. It might compliment some situation, but not suitable or advantage for all. Imap not likely happen here for user, hardware no so cheap here,such is storage space not big deal, not until we buy hardware for near cheap as U.S., which 50-70% less than we pay.
The director is not needed in most cases. If you feel you dont need the director, then just forget about its existence and ignore all emails about it. Just pretend it doesnt exist :)
I'll try to explain why the director is needed in some situations.
dovecot has an index file. If you look in your configuration file, you'll see something like:
mail_location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u
The index file holds an index so dovecot can search in your mailbox really fast. (sorry if you know all this). So, if for instance you want to sort your mailbox on subject, dovecot doesnt have to read through all your emails to find all the subjects. Instead, it can read the index file really fast, and return a sorted mailbox basically instantly. A lot of IMAP servers dont even have this feature! Imagine the alternative, where dovecout would have to read through 25.000 emails, just so it can sort your mailbox on subject. A nightmare for performance, but like I said, some imap servers still do that! Dovecot luckily uses an index, and is therefore faster than a lot of other imap servers.
So, we have concluded that dovecot has a really cool performance feature in the form of an index file.
Now imagine what happens if you have more than 1 imap server. We have 35 imap servers right now. Each server needs access to the index file. One popular way to do this is by using NFS. With NFS you have a central file server, and each imap server uses the remote fileserver for storage. So now, all 35 servers have access to a cool central index file.
But, there is a slight snag. NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. The index file can then end up being corrupted. This causes dovecot to give faulty data back to the client, which in practice ends up looking like the mailbox of the user is empty. Not good. This causes calls to support desks about why a user has just lost all his mail. It's not lost, it's just that the index file is corrupt, and now dovecot thinks the mail is gone.
You may wonder rightly so, why this is a problem only now. Well, it hasnt been a problem only now. We've been dealing with this issue for years. We've had discussions with Timo over the years on how to solve this problem. But Timo has only so much time, and this problem was not important enough to fix. He had more important bugs to fix. So, we lived with the problem, or worked around it in ugly ways.
Recently Timo has found the time (I believe an ISP paid Timo to solve this problem) to think about ways to solve it. It's really not an easy problem to fix. We've known for years that the best way to fix this would be to have a user always end up on the same server, because thats when that specific NFS problem doesnt occur. If 2 sessions of the same user both end up on Server 1, then both sessions will see the same data, and no problems can occur.
But how to get all sessions of a user on the same server? Most loadbalancers dont support this feature. Our Foundries dont, and our previous Alteons didnt either. So Timo thought of a way how dovecot itself could make sure all sessions of a user end up on the same server. Basically by turning dovecot into a proxy, that proxies connections to another server. It maintains an internal state on what user belongs on what server, and redirects connections appropriately. And thats all the director is. A proxy with an internal state of where to proxy a connection to.
So why dont you see this problem with your setup? Like I said, this is only a problem with specific setups.
you need to be using NFS for central index storage. If you dont use NFS for your indexes, you wont see this problem, and you wont need the director. carry on as you always have.
even if you use NFS, you may not run into this problem. Because you also need to have situations where multiple servers could be updating the indexes at the same time. There are only 2 situations where this could happen:
- you're using dovecot-lda, and have index updating turned on. Index updating in the lda is cool, because it means dovecot will always have the latest information about your email in the index files.
and/or
- you allow users to have sessions to multiple servers. For instance, they connect with Outlook and their iPhone at the same time. And maybe even a webmail connection as well.
If you dont have one of these 2 situations, you wont ever see a problem and you wont need the director. carry on as you always have.
But, for those (mostly larger) setups where you are using NFS for indexes, AND you can have multiple sessions of the same user on different servers (and they can be either LDA or IMAP sessions), you're probably better off using the director.
Again, if this doesnt apply to you, simply..... do not use the director.
Regards,
Cor
On Fri, 3 Sep 2010 07:22:00 +1000 Edward avanti edward.avanti@gmail.com articulated:
why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time.
If you are going to go off on a temper tantrum, at least post it in an acceptable fashion; id est, lose the "top posting" urge. While you are at it, trim the superfluous text from the replied to post.
-- Jerry ✌ Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
There is no sadder sight than a young pessimist.
have you been told where you might go lately and do with some part your anatomy? this Timo list, not you list, best remember this since you nobody this list
On Fri, Sep 3, 2010 at 8:19 AM, Jerry dovecot.user@seibercom.net wrote:
On Fri, 3 Sep 2010 07:22:00 +1000 Edward avanti edward.avanti@gmail.com articulated:
why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time.
If you are going to go off on a temper tantrum, at least post it in an acceptable fashion; id est, lose the "top posting" urge. While you are at it, trim the superfluous text from the replied to post.
-- Jerry ✌ Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
There is no sadder sight than a young pessimist.
On 9/2/2010 4:32 PM, Edward avanti wrote:
have you been told where you might go lately and do with some part your anatomy? this Timo list, not you list, best remember this since you nobody this list
On Fri, Sep 3, 2010 at 8:19 AM, Jerry dovecot.user@seibercom.net wrote:
On Fri, 3 Sep 2010 07:22:00 +1000 Edward avanti edward.avanti@gmail.com articulated:
why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time.
Edward,
He was not ranting, or advocating using the director for anyone that did not need it. He was simply (finally) attempting to make it completely clear what the purpose of the director was and who it made sense for. He correctly stated that the reason it is an issue at all is because dovecot uses indexes when all of the other IMAP servers do not. It is one of the sources of the huge performance gain in the product.
As I read down the thread, I realized that many of the posters were pointing out that it shouldn't be used because postfix or qmail can deliver without problems. But in doing that, you lose the performance gain (and in my case the use of sieve filters) that the dovecot-lda brings to the table. Even without using the dovecot-lda, the indexes for the IMAP server are very worth it. Keeping the indexes in good shape creates less work on the server, and good performance for the end user.
I find it ridiculous that you lashed out at him as a ranter and a troll. If anything, you labeled yourself as one with those posts.
Tim
On Thu, 2010-09-02 at 22:36 -0700, Tim Traver wrote:
As I read down the thread, I realized that many of the posters were pointing out that it shouldn't be used because postfix or qmail can deliver without problems. But in doing that, you lose the performance gain (and in my case the use of sieve filters) that the dovecot-lda
Not entirely true, you lose the performance gain maybe if you stopped caching for pop3/imap side there has been no performance degradation on the "delivery" side, and deliver not updating caches has not affected in any noticable way how the clients receive their mail.
That said, this is from a purely pop3 stand point, in testing webmail, no difference was noted either, I can not speak for those who consider imap the primary use where it may or may not affect performance at a noticeable level.
brings to the table. Even without using the dovecot-lda, the indexes for
The only thing I see as a setback is (and again, I've not looked into this so there may be a plain sight solution within postfix itself) no sieve support. This is why I myself some time ago asked about an added option to deliver to ignore it and just deliver the darn message file, letting pop3/imap discover any changed and do updates.
ahhh 40 mins to friday beer oclock
On 9/2/2010 11:20 PM, Noel Butler wrote:
On Thu, 2010-09-02 at 22:36 -0700, Tim Traver wrote:
As I read down the thread, I realized that many of the posters were pointing out that it shouldn't be used because postfix or qmail can deliver without problems. But in doing that, you lose the performance gain (and in my case the use of sieve filters) that the dovecot-lda
Not entirely true, you lose the performance gain maybe if you stopped caching for pop3/imap side there has been no performance degradation on the "delivery" side, and deliver not updating caches has not affected in any noticable way how the clients receive their mail.
That said, this is from a purely pop3 stand point, in testing webmail, no difference was noted either, I can not speak for those who consider imap the primary use where it may or may not affect performance at a noticeable level.
Noel,
yes, there is no change on the delivery performance, as dovecot-lda is just a delivery program to the Maildir just like any other. The thing that it does different is update the dovecot uidlist and update the index with a new mail. As you mentioned, it also has the sieve filter now built into it, which is important to me, but maybe not everyone depending on their setup.
Not using dovecot-lda is just postponing the work that it does until the user attempts to fetch the mail using POP or IMAP. I doubt that it is "noticeable" when you test it, but when dovecot has to reindex when you log in to IMAP or POP, you're adding several things that it has to do before you get your messages, and if you really are trying to keep performance high on a LARGE user base, then this could make a difference. I'd rather it take the time to do the indexing and uid stuff on the deliver than on the pickup, wouldn't you?
t
brings to the table. Even without using the dovecot-lda, the indexes for
The only thing I see as a setback is (and again, I've not looked into this so there may be a plain sight solution within postfix itself) no sieve support. This is why I myself some time ago asked about an added option to deliver to ignore it and just deliver the darn message file, letting pop3/imap discover any changed and do updates.
ahhh 40 mins to friday beer oclock
Tim,
On Fri, 2010-09-03 at 12:26 -0700, Tim Traver wrote:
yes, there is no change on the delivery performance, as dovecot-lda is just a delivery program to the Maildir just like any other. The thing that it does different is update the dovecot uidlist and update the index with a new mail. As you mentioned, it also has the sieve filter now built into it, which is important to me, but maybe not everyone depending on their setup.
Not using dovecot-lda is just postponing the work that it does until the user attempts to fetch the mail using POP or IMAP. I doubt that it is
but as I've said (many times now) imap is not an issue here, as primary use is pop3 and rarely is leave on server used based on some useage tests over the mail store directories so it is not a problem, besides, it has to reindex anyway every access be it pop3 imap or deliver, we just remove some risks to add reliability since Timo says it is risky allowing multiple servers to access it... hell, maybe we too should revert back to Qmail and vpopmail where this was never an issue as well as the OP :)
My priorities as stability and reliability... performance last
Cheers N
On 7.9.2010, at 11.41, Noel Butler wrote:
but as I've said (many times now) imap is not an issue here, as primary use is pop3 and rarely is leave on server used based on some useage tests over the mail store directories
BTW. Dovecot is primarily an IMAP server and optimized for that. For POP3-only or mostly-POP3 traffic another server might work better.
FFS Gerard keep your self appointed net copping to hte other usual lists, even though on one of them (MailSCaner) you were told by JF that anyone can post in anyway they want, including top post. your not a moderator here, stop playing one.
On Thu, 2010-09-02 at 18:19 -0400, Jerry wrote:
On Fri, 3 Sep 2010 07:22:00 +1000 Edward avanti edward.avanti@gmail.com articulated:
If you are going to go off on a temper tantrum, at least post it in an acceptable fashion; id est, lose the "top posting" urge. While you are at it, trim the superfluous text from the replied to post.
On Fri, 2010-09-03 at 07:22 +1000, Edward avanti wrote:
why you reply with 7 page of rubbish?
nasty, he was trying to explain it, even though you (i, and some others) feel its irrelevant , perhaps this subject could have bene modified to remove director :)
By a quick skim of this (I didnt get his original - likely because xs-for-all is not a welcome domain here for spam/idiots in the past it is score +7 automatically in SA, so other things must have triggered the spam rules and it was eaten, in fact no messgae since the one I replied to on Wed, sorry Cor, your mail service is bad hehe) he seems to be selling the director, but didnt comprehend this really has nothing to do with it since your question clearly stated why, but no need to be so nasty towards him.
On Fri, Aug 27, 2010 at 3:28 PM, Brandon Davidson brandond@uoregon.eduwrote:
Noel,
On 8/26/10 9:59 PM, "Noel Butler" noel.butler@ausics.net wrote:
I fail to see advantage if anything it add in more point of failure, with
i agree with this and it is why we dont use it
we use dovecots deliver with postfix and have noticed no problems, not to say there was none, but if so, we dont notice it.
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop.
Sendmail and procmail? This mean you use mbox? This always bad for NFS anyway
Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a
backend is not problem. it front end it where mail arrives, these are server we should be able turn off indexing, other front end type server for pop3, can have index on since no multi login allowed
new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy.
Timo has been saying for YEARS that you need user-node affinity if you're doing NFS, and now he's done something about it. If you've already got a load balancer, then just point the balancer at a pool of directors, and then point the directors at your existing mailserver pool.
<shameless plug> For health monitoring on the directors, check out: http://github.com/brandond/poolmon </shameless plug>
-Brad
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop.
Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy.
We are in the exact same position as Brad. We also use sendmail's LDA, we also use a metrocluster, and we also have hordes of imap and webmail users.
We see the exact same thing Brad sees. And I see it myself about once a week as well. The index gets corrupted due to access by 2 different clients, and to the user it then looks like their mail disappears. The user totally freaks out, because they'll invariably have really really important mail that has to be recovered right now. Usually a law firm as well. They call the helpdesk, keeping a support person busy with something thats really just a known bug.
It probably isnt much of an issue if you use POP. But in large scale IMAP setups, where people are getting used to having access to all their email server-side (and thus mailboxes growing, needing larger indexes, increasing the chances of problems) from a myriad of clients this WILL happen if you're using NFS.
Ive even considered moving away from NFS again for indexes due to this problem. But it really is noticable if you have a lot of email that your index isnt up to date as you move across our dozens and dozens of imap servers.
Cor
On Sat, 28 Aug 2010, Cor Bosman wrote:
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop.
Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy.
We are in the exact same position as Brad. We also use sendmail's LDA, we also use a metrocluster, and we also have hordes of imap and webmail users.
We see the exact same thing Brad sees. And I see it myself about once a week as well. The index gets corrupted due to access by 2 different clients, and to the user it then looks like their mail disappears. The user totally freaks out, because they'll invariably have really really important mail that has to be recovered right now. Usually a law firm as well. They call the helpdesk, keeping a support person busy with something thats really just a known bug.
It probably isnt much of an issue if you use POP. But in large scale IMAP setups, where people are getting used to having access to all their email server-side (and thus mailboxes growing, needing larger indexes, increasing the chances of problems) from a myriad of clients this WILL happen if you're using NFS.
Ive even considered moving away from NFS again for indexes due to this problem. But it really is noticable if you have a lot of email that your index isnt up to date as you move across our dozens and dozens of imap servers.
Any idea how Rackspace has implemented the director? They have to be using some kind of shared storage, it wouldn't make sense to make storage local to each host in such a large environment.
Charles
Cor
We're a similar installation (60-70k users, FAS3050 cluster).
We have been using "perdition" (IMAP/POP redirector) software for a while. The IMAP/POP.ourdomain A records point to 2 front ends, which all they do is to redirect the IMAP/POP session to the a specific mail server for each user, based on their LDAP mailhost entry.
We use postfix to deliver mail, and procmail is the LDA. We are running a background process on each mail server (3 mailservers which do SMTP/POP/IMAP, and barely sweat at it - 2xquad core Xeons with 8gig each), which monitors the maillog, and if dovecot sees a index corruption, the monitor fixes the problem (we used to see these errors when we still used mbox, not anymore though).
We run a periodic process, which gets statistics of usage from the mail servers, and reassign the users to mail servers in order to better distribute the load. Each new user that is created gets his mail server by a random function which choses one of the three.
Each mail server in the user LDAP entry is in fact a virtual address on a load balancer, pointing to the real mail server behind it, BUT also having a backup server for each in case the real server crashes, so assuming mailsrv1 crashes, mailsrv2 will take its clients.
The setup works rather well, within the limitations of maildir and netapp (mainly full body search being slowish with very large mailboxes made out of 10s of thousands of files).
We used to not use the "perdition" directors in the past, and once we started using them, we saved alot of problems on a few fronts:
- Index corruption issues
- SSL termination - since the front ends to the SSL termination, the backend servers access from the front ends is clear text, saving CPU cycles from the backends servers.
I haven't taken a look yet at Dovecot's solution for the director, but I am writing this since I do think that it is addressing a real life problem for any medium++ or larger installation that uses NFS.
Oh, and Timo, I don't think we are just "a couple of NFS users". Maildir and NFS are not as uncommon as you'd think, even in very large installations.
--Ariel
Brandon Davidson wrote:
Noel,
On 8/26/10 9:59 PM, "Noel Butler" noel.butler@ausics.net wrote:
I fail to see advantage if anything it add in more point of failure, with
i agree with this and it is why we dont use it
we use dovecots deliver with postfix and have noticed no problems, not to say there was none, but if so, we dont notice it.
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop.
Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy.
Timo has been saying for YEARS that you need user-node affinity if you're doing NFS, and now he's done something about it. If you've already got a load balancer, then just point the balancer at a pool of directors, and then point the directors at your existing mailserver pool.
<shameless plug> For health monitoring on the directors, check out: http://github.com/brandond/poolmon </shameless plug>
-Brad
--
Ariel Biener e-mail: ariel@post.tau.ac.il PGP: http://www.tau.ac.il/~ariel/pgp.html
On 8/31/2010 10:22, Ariel Biener wrote:
Oh, and Timo, I don't think we are just "a couple of NFS users". Maildir and NFS are not as uncommon as you'd think, even in very large installations.
NFS with maildir has been the gold standard for a long time, whether the NFS server be <insert favorite distro> boxes or a huge hardware appliance. It's certainly not uncommon.
~Seth
On Thu, Sep 2, 2010 at 11:09 AM, Seth Mattinen sethm@rollernet.us wrote:
On 8/31/2010 10:22, Ariel Biener wrote:
Oh, and Timo, I don't think we are just "a couple of NFS users". Maildir and NFS are not as uncommon as you'd think, even in very large installations.
NFS with maildir has been the gold standard for a long time, whether the NFS server be <insert favorite distro> boxes or a huge hardware appliance. It's certainly not uncommon.
~Seth
Sad I think most assume single machine use or two machine use, if dovecot not good, we have sub 1 hour rollback time plan in place, since it only daemon and configuration change.
On Fri, 2010-09-03 at 07:16 +1000, Edward avanti wrote:
On Thu, Sep 2, 2010 at 11:09 AM, Seth Mattinen sethm@rollernet.us wrote:
On 8/31/2010 10:22, Ariel Biener wrote:
Oh, and Timo, I don't think we are just "a couple of NFS users". Maildir and NFS are not as uncommon as you'd think, even in very large installations.
NFS with maildir has been the gold standard for a long time, whether the NFS server be <insert favorite distro> boxes or a huge hardware appliance. It's certainly not uncommon.
~Seth
Sad I think most assume single machine use or two machine use, if dovecot not good, we have sub 1 hour rollback time plan in place, since it only daemon and configuration change.
I don't think most assume that at all, dovecot is, has and will, be used in many different setups from a single SOHO to large numbers like yours, NFS seems to be more prevalent than others apparently realise, maybe because , like here, it has worked, and we dont bitch about things (normally) I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if this list was flooded by dovecot NFS users asking he'd quickly change his mind, but as everyone here knows, there is only a few hundred memberfs on this lsit, and how many tens of thousands of dovecot installs :) Some NFS users here are of small size where they can see beneifits of using another program (director) to manage things, and dont care about it impacting their inbound servers, some do care, but yes when you get to large user numbers, well, we are engineers, not PR/Sales ppl, so we can by our nature see through the upselling of using it.
Anyways the caching from pop will help a bit, so i doubt you'll have to rollback that part anytime soon. Its not been an issue here in years of use, i'm pretty pedantic, so if it was an issue im sure i'd have come accross it by now, so can not see why it will be for you now. I'cve accepted Timo wont code that option for deliver, so if your worried about it use postfix's deliver, if doing the same keeps you happy then thats all there is to it. we know we don t answer to Timo. or the supporters or fanbois, we answer to the ones who pay our salaries. And, since we do not pay for Dovecot we are free to use any, all or no part of it, obviously courier didnt do for you what you wanted or you wouldnt be here, i know it did what we wanted, albeit a bit slower.
Hi Noel,
I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if
I guess he told you this in private? The way i understand it the index-over-nfs problem can not be fixed 100%. I know for a fact Timo tried very hard to fix it, but NFS is what it is. And even if you can fix it in one OS, say Linux, you'd have a dozen other OSs to cope with. It is not only a dovecot problem, it is actually more of an NFS protocol problem.
installs :) Some NFS users here are of small size where they can see beneifits of using another program (director) to manage things, and dont care about it impacting their inbound servers, some do care, but yes when you get to large user numbers, well, we are engineers, not PR/Sales ppl, so we can by our nature see through the upselling of using it.
We dont use the director on our inbound servers either. We dont even use dovecot-lda. In our experience this isnt really a problem. The imap processes keep the indexes working well enough. So to answer Edward's original question, No, i dont think you need to replace your Foundries.
xs-for-all is not a welcome domain here for spam/idiots in the past it
We have over 300,000 DSL users. There's always idiots. But we're very active in the anti-spam community. We even run multiple public mirrors of all kinds of blacklists. If you send us a complaint, it will get acted on. Scoring any largish ISP +7 in SA because you got some spam in the past is kinda silly :) I cant imagine blocking a whole ISP because you got spam from them. Wow. Our users would revolt.
Im not selling the director at all. Quite the contrary. I think most people wont need it. And I wish we didnt need it. It adds complexity, and that sucks. I hate it :) I think we actually agree there! :)
You're right, turning off indexing in the LDA has no impact whatsoever on the delivery side. (again, we also dont use it there). But it does have a slight impact on the frontend side. It means the imap servers need to update the indexes every time they're out of date (which is basically every time new email arrives). For us, this impact doesnt seem to be a problem.
Not having indexes does impact imap at a noticable level, especially for large mailboxes. Ive seen 10+ second delays on large mailboxes while the imap server is sorting the mailbox without the index.
Basically, you're damned if you do, and damned if you dont.
Cor
For those that are worried about the md5 hash sending out most users to 1 server and thus loadbalancing badly. This doesnt seem to be the case. I just pointed one of our test webmail environments to a director cluster (2 director servers behind a foundry, pointing to 2 real imap servers), and this is what the load spread was after a few minutes:
imapdirector1:/usr/home/cor# doveadm director status mail server ip vhosts users 194.109.26.171 100 698 194.109.26.172 100 701
It doesnt get any better. It looks like Timo's choice of hash is spot on.
Regards,
Cor
On Fri, 2010-09-03 at 09:50 +0200, Cor Bosman wrote:
Hi Noel,
I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if
I guess he told you this in private? The way i understand it the
yep, we had a discussion off-list weeks ago
We dont use the director on our inbound servers either. We dont even use dovecot-lda. In our experience this isnt really a problem. The imap
thats going to be the way to go to avoid any problems, its never been a problem anyway for the servers I manage, but currently running half only as postix's virtual, nearly 2 weeks now? no problems, so if we need to go all postfix on delivery there will be no issue.
xs-for-all is not a welcome domain here for spam/idiots in the past it
We have over 300,000 DSL users. There's always idiots. But we're very active in the anti-spam community. We even run multiple public mirrors of all kinds of blacklists. If you send us a complaint, it will get acted on. Scoring any largish ISP +7 in SA because you got some spam in the past is kinda silly :) I cant imagine blocking a whole ISP because you got spam from them. Wow. Our users would revolt.
I recall copious amounts of it, I'd certainly never call it "some", though, this was a few years back, if you have cleaned up your users (yes I too know all well there will always be idiots having also managed a DSL based (read as: windows infected malware weenies) ISP) then I am prepared to suspend those rules, as you know, when a domain has problems and only gets bigger over time, its harder to imagine there will be less of a problem, but, that said, a leopard although rarely changing its spots, can occasionally do so.
Our users have also become accustomed to the fact that we operate as, protecting the majority, not letting any minority dictate our decisions, we'd rather have 1K peaved off users for not getting mail, than 100K peaved off for being constantly spammed.
You're right, turning off indexing in the LDA has no impact whatsoever on the delivery side. (again, we also dont use it there). But it does have a slight impact on the frontend side. It means the imap servers need to update the indexes every time they're out of date (which is basically every time new email arrives). For us, this impact doesnt seem to be a problem.
Thats one thing I was not able to reproduce, but, again, not having a real imap userbase we wouldnt see issue, as most users use pop3 only, if we changed policy and if users actually used imap, perhaps we would see a degredation.
Cheers
On 3.9.2010, at 7.00, Noel Butler wrote:
I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if this list was flooded by dovecot NFS users asking he'd quickly change his mind, but as everyone here knows, there is only a few hundred
I never said I don't care about NFS users or think there aren't enough of them. I guess I could have been a bit clearer:
I don't consider the number of users who want to use Dovecot LDA without indexes to be worth the trouble to add extra complexity to maildir code.
(Once someone who pays my salary thinks otherwise, it probably becomes worth the trouble.)
On Fri, Sep 3, 2010 at 7:26 PM, Timo Sirainen tss@iki.fi wrote:
On 3.9.2010, at 7.00, Noel Butler wrote:
I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if this list was flooded by dovecot NFS users asking he'd quickly change his mind, but as everyone here knows, there is only a few hundred
I never said I don't care about NFS users or think there aren't enough of them. I guess I could have been a bit clearer:
I don't consider the number of users who want to use Dovecot LDA without indexes to be worth the trouble to add extra complexity to maildir code.
Why to Maildir code, is not easier to not call that part of code by command line -switch to deliver simple method and would this no be beneficial to all NFS user, not just the one who ask you.
On 10/09/2010 07:50, Edward avanti wrote:
On Fri, Sep 3, 2010 at 7:26 PM, Timo Sirainentss@iki.fi wrote:
On 3.9.2010, at 7.00, Noel Butler wrote:
I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if this list was flooded by dovecot NFS users asking he'd quickly change his mind, but as everyone here knows, there is only a few hundred
I never said I don't care about NFS users or think there aren't enough of them. I guess I could have been a bit clearer:
I don't consider the number of users who want to use Dovecot LDA without indexes to be worth the trouble to add extra complexity to maildir code.
Why to Maildir code, is not easier to not call that part of code by command line -switch to deliver simple method and would this no be beneficial to all NFS user, not just the one who ask you.
Writing messages to Maildir(++) (without indexes) is so straightforward that systems tend to have support for that built-in.
Exim, Postfix, and Procmail have support for delivery to Maildir built-in.
If you just want a program which does simple deliveries to Maildir then you can use Procmail.
Bill
On Fri, Aug 27, 2010 at 2:59 PM, Noel Butler noel.butler@ausics.net wrote:
On Fri, 2010-08-27 at 08:54 +1000, Edward avanti wrote:
Halo, Please can you explain why this is advantage over a hardware load balancer.
it is no advantage over a dedicated hardware solution, but director does not do the exact same thing.
I fail to see advantage if anything it add in more point of failure, with
i agree with this and it is why we dont use it
we use dovecots deliver with postfix and have noticed no problems, not to say there was none, but if so, we dont notice it. postfix looks up the user, it determines if it accepts the mail, if it does, it queues it for mailscanner to do its stuff, then gives it back to postfix, which is then told to give it to dovecots deliver, it makes
I have offlist discussion with Timo, he said help with I/O, you make good case, not more I/O intense than scanning mail, delivery just like router
no sense to me that it should then be sent to another machine just to be stored on a remote file server, the same remote file server the initial server assigned that conenction by a true load balancer has mounted and would store it to as well.... would be miuch easier to have deliver ignore the index file by an option, eliminating the corruption risks to the index file and just storing the darm thing. or am i only one who thinks mail systems do not need to be complex to run faultlessly, I think those who feel the need to make it very complex are not only looking for trouble, but further trying to justify their position to their employer that they are indispensable.
If operation is simple, is little to go wrong, when nothing go wrong, boss happy and my job safe
if director service assign 60K user to each front end, how it handle if
5K
simultaneous user login, but all 5K happen to be assign to that one machine,
that would be rare, but, technically speaking, if you are that large in user numbers, it is a possible scenario
We have 418K mailbox users
Is it really worth it? Do we really need this, or just let foundry switch handle it as it does now. We also have 24 front end SMTP server, these deliver mail to netapp filer, all 24 plus 8 pop3 server and 2 webmail imap server all mount /vmail, so all access same maildir. it seem work very effective thus far and for many many
Sounds similar setup to us, smtp, pop3 and webmail all mounting /var/vmail/ on a FAS2050, I've asked if it can avoid touching the index files before (see a thread as recent as a few weeks back), Timo is just not interested, to much work apparently for so little users
Oh my, so i waste time talking asking him for extra switch to deliver to ignore indexing, drat.
(although I never in all hte years ive been on this list, ever seen a poll taken/question asked to users - about it, plus, well, every single dovecot user is on this list, right? <sarcasm> anyway, mostly I guess although it has risks, it seems to work for everyone who uses NFS anyway and has done for very many years :) , maybe one day when Timo is so bored and cant think of anything to add, he will give us an option, or a dedicated deliver binary separate to normal deliver that does this)
Maybe not many people here use time proven setup
/rant ( but its nice to know im not the only one here who feels this way) Cheers
On 27.8.2010, at 5.59, Noel Butler wrote:
I've asked if it can avoid touching the index files before (see a thread as recent as a few weeks back),
You can avoid touching indexes:
protocol lda { mail_location = maildir:~/Maildir:INDEX=MEMORY }
But you still have the problem of dovecot-uidlist file that gets updated. Well .. maybe you could do something ugly like:
protocol lda { mail_location = maildir:~/Maildir:INDEX=MEMORY:CONTROL=/tmp/controls/%u }
And then once in a while rm -rf /tmp/controls, but I don't know how badly that'll work out. I guess it's possible that LDA even goes and scans through the existing cur/ directory to build a new dovecot-uidlist.
participants (16)
-
Ariel Biener
-
Brad Davidson
-
Brandon Davidson
-
Charles Marcus
-
Charles Sprickman
-
Cor Bosman
-
Dave McGuire
-
Edward avanti
-
Jerry
-
Noel Butler
-
Richard B. Pyne
-
Seth Mattinen
-
Stan Hoeppner
-
Tim Traver
-
Timo Sirainen
-
William Blunn