Re: dovecot-lda without starting dovecot?
you could try setting protocols = in config file to disable (most) listeners. ---Aki TuomiDovecot oy -------- Original message --------From: Stephan von Krawczynski <skraw.ml@ithnet.com> Date: 03/11/2017 19:39 (GMT+02:00) To: Dovecot Mailing List <dovecot@dovecot.org> Subject: Re: dovecot-lda without starting dovecot? On Fri, 3 Nov 2017 19:30:22 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 3 Nov 2017, at 18.23, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote: Hello Aki,
let me explain this a bit more. We do not intend to use only some copied binary. Of course we would do a full installation of dovecot and pidgeonhole, only we question if it is necessary to start the dovecot init-file bringing up the dovecot imap/imap-login/pop/pop-login/auth and other processes. Indeed we have virtual users.
While it might be possible to disable all the other services except master I must ask why? How would the users be accessing their mails then?
Sami
Hello Sami,
you did not read my first post. We are talking about a multihost installation where one host does SMTP and LDA, and the other does POP and IMAP (with dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP host_. So there is no need for open POP or IMAP ports and the corresponding running services.
-- Regards, Stephan
While it might be possible to disable all the other services except master I must ask why? How would the users be accessing their mails then?
Sami
Hello Sami,
you did not read my first post. We are talking about a multihost installation where one host does SMTP and LDA, and the other does POP and IMAP (with dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP host_. So there is no need for open POP or IMAP ports and the corresponding running services.
Again that does not answer my question why? Why do you want all the locking problems and multi-access problems that come with setup like that? What is the actual problem that you are trying to solve?
Sami
On 11/3/2017 4:57 PM, Sami Ketola wrote:
While it might be possible to disable all the other services except master I must ask why? How would the users be accessing their mails then?
Sami Hello Sami,
you did not read my first post. We are talking about a multihost installation where one host does SMTP and LDA, and the other does POP and IMAP (with dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP host_. So there is no need for open POP or IMAP ports and the corresponding running services. Again that does not answer my question why? Why do you want all the locking problems and multi-access problems that come with setup like that? What is the actual problem that you are trying to solve?
Sami
I think you have to run dovecot on the server receiving mails, just don't open imap/pop3 ports so users can't get there. Then publish the other server as your imap/pop3 server. Both servers probably should have the same setup for postfix and dovecot, but just kill the ports on the one that doesn't need it. To run lda you're going to have to run dovecot on your mail relay.
On Sat, 4 Nov 2017 01:57:31 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
While it might be possible to disable all the other services except master I must ask why? How would the users be accessing their mails then?
Sami
Hello Sami,
you did not read my first post. We are talking about a multihost installation where one host does SMTP and LDA, and the other does POP and IMAP (with dovecot). Now we want to use dovecot-lda for the local delivery _on the SMTP host_. So there is no need for open POP or IMAP ports and the corresponding running services.
Again that does not answer my question why? Why do you want all the locking problems and multi-access problems that come with setup like that? What is the actual problem that you are trying to solve?
Sami
Really, I can hardly believe you don't now large loadbalancing ISP setups with multiple nodes per single service ...? The simple problem: massive numbers of emails
-- Regards, Stephan
On 4 Nov 2017, at 10.31, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
On Sat, 4 Nov 2017 01:57:31 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
Again that does not answer my question why? Why do you want all the locking problems and multi-access problems that come with setup like that? What is the actual problem that you are trying to solve?
Sami
Really, I can hardly believe you don't now large loadbalancing ISP setups with multiple nodes per single service ...? The simple problem: massive numbers of emails
Nope. Has never been done. Has never been recommended way. You will get more problems with that setup that you are seeking to solve.
Use multiple dovecot backends with director ring in front and switch to lmtp delivery via the director ring if you have scalability problems. Then you can just increase number of backends in case they are overloaded.
Sami
On Sun, 5 Nov 2017 10:44:25 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 4 Nov 2017, at 10.31, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
On Sat, 4 Nov 2017 01:57:31 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
Again that does not answer my question why? Why do you want all the locking problems and multi-access problems that come with setup like that? What is the actual problem that you are trying to solve?
Sami
Really, I can hardly believe you don't now large loadbalancing ISP setups with multiple nodes per single service ...? The simple problem: massive numbers of emails
Nope. Has never been done. Has never been recommended way. You will get more problems with that setup that you are seeking to solve.
Use multiple dovecot backends with director ring in front and switch to lmtp delivery via the director ring if you have scalability problems. Then you can just increase number of backends in case they are overloaded.
Sami
Sorry to say this setup works flawlessly for years. The only addition we will make now is to do the delivery with dovecot-lda. Everything else (including multiple dovecot pop/imap servers) will stay as is. Hopefully dovecot-lda does not fiddle around with the indexes too much, as we then would have to delete this part of the code out. It is not needed as we found out during the last 10 years of delivering mails into the maildirs by atomic rename action while dovecot is presenting them over imap.
-- Regards, Stephan
On 5 Nov 2017, at 12.55, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote: Sorry to say this setup works flawlessly for years. The only addition we will make now is to do the delivery with dovecot-lda. Everything else (including multiple dovecot pop/imap servers) will stay as is. Hopefully dovecot-lda does not fiddle around with the indexes too much, as we then would have to delete this part of the code out. It is not needed as we found out during the last 10 years of delivering mails into the maildirs by atomic rename action while dovecot is presenting them over imap.
Feel free to do anything you like. I'm just going to mention to people later reading these from the achives not to take this kind of strange hack as an example of recommended dovecot clustering. Instead consider it as an opposite of any best practices cluster setup.
Sami
On Mon, 6 Nov 2017 08:37:11 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 5 Nov 2017, at 12.55, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote: Sorry to say this setup works flawlessly for years. The only addition we will make now is to do the delivery with dovecot-lda. Everything else (including multiple dovecot pop/imap servers) will stay as is. Hopefully dovecot-lda does not fiddle around with the indexes too much, as we then would have to delete this part of the code out. It is not needed as we found out during the last 10 years of delivering mails into the maildirs by atomic rename action while dovecot is presenting them over imap.
Feel free to do anything you like. I'm just going to mention to people later reading these from the achives not to take this kind of strange hack as an example of recommended dovecot clustering. Instead consider it as an opposite of any best practices cluster setup.
Sami
Maybe they are interested that it runs like charm ;-) Of course it is overly complex to let dovecot-lda look up the mail_location since we already know that when starting it, but at least it shows a nice output in the logs. Still we are not content with it touching/locking dovecot.index.log. If someone pointed at one location in the code where this could be disabled we would implement a new param for switching that off.
-- Regards, Stephan
On 11/6/2017, 4:01:19 AM, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
Still we are not content with it touching/locking dovecot.index.log. If someone pointed at one location in the code where this could be disabled we would implement a new param for switching that off.
?????????????????????????
Dovecot's indexing is one of its main features, and WHY it is so much faster than others.
And you want to just turn it off? Good luck...
On Mon, 6 Nov 2017 09:50:16 -0500 Tanstaafl <tanstaafl@libertytrek.org> wrote:
On 11/6/2017, 4:01:19 AM, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
Still we are not content with it touching/locking dovecot.index.log. If someone pointed at one location in the code where this could be disabled we would implement a new param for switching that off.
?????????????????????????
Dovecot's indexing is one of its main features, and WHY it is so much faster than others.
And you want to just turn it off? Good luck...
It seems you have not understood what I am talking about. Our pre-dovecot lda did not touch the index either. And it did not harm the imap/pop procedure in any way. So we know there is no need to fiddle with the index in the process of delivery into the maildirs to keep our performance as it was before.
-- Regards, Stephan
On 07/11/2017 09:18, Stephan von Krawczynski wrote:
On Mon, 6 Nov 2017 09:50:16 -0500 Tanstaafl <tanstaafl@libertytrek.org> wrote:
On 11/6/2017, 4:01:19 AM, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote: Still we are not content with it touching/locking dovecot.index.log. If someone pointed at one location in the code where this could be disabled we would implement a new param for switching that off.
?????????????????????????Dovecot's indexing is one of its main features, and WHY it is so much faster than others.
And you want to just turn it off? Good luck...
It seems you have not understood what I am talking about. Our pre-dovecot lda did not touch the index either. And it did not harm the imap/pop procedure in any way. So we know there is no need to fiddle with the index in the process of delivery into the maildirs to keep our performance as it was before.
mail_location Optionally disable indexes using :INDEX=MEMORY
don't use this on IMAP boxes, but is safe to use on SMTP and POP3's boxes though
eg:
mail_location = maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY
-- Kind Regards,
Noel Butler
This Email, including any attachments, may contain legally privilegedinformation, therefore remains confidential and subject to copyright protected under international law. You may not disseminate, discuss, or reveal, any part, to anyone, without the authors express written authority to do so. If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments, immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message. Only PDF [1] and ODF [2] documents accepted, please do not send proprietary formatted documents
Links:
[1] http://www.adobe.com/ [2] http://en.wikipedia.org/wiki/OpenDocument
On Tue, 07 Nov 2017 13:19:12 +1000 Noel Butler <noel.butler@ausics.net> wrote:
mail_location Optionally disable indexes using :INDEX=MEMORY
don't use this on IMAP boxes, but is safe to use on SMTP and POP3's boxes though
eg:
mail_location = maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY
-- Kind Regards,
Noel Butler
Hello Noel,
this sounds interesting. Can you please elaborate why you think this is no good idea for IMAP? We used a different LDA-scheme before (simply created the mail in maildir/tmp, then renamed it to maildir/new, just like Bernstein suggests to do) and it worked very well, no matter if the box was used whith IMAP or POP3. Why should there be any difference in using dovecot-lda without indexes? Does dovecot-lda "create" the new mail by atomic rename from tmp, too?
-- Regards, Stephan
On 7 Nov 2017, at 9.15, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
On Tue, 07 Nov 2017 13:19:12 +1000 Noel Butler <noel.butler@ausics.net> wrote:
mail_location Optionally disable indexes using :INDEX=MEMORY
don't use this on IMAP boxes, but is safe to use on SMTP and POP3's boxes though
eg:
mail_location = maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY
Hello Noel,
this sounds interesting. Can you please elaborate why you think this is no good idea for IMAP? We used a different LDA-scheme before (simply created the mail in maildir/tmp, then renamed it to maildir/new, just like Bernstein suggests to do) and it worked very well, no matter if the box was used whith IMAP or POP3. Why should there be any difference in using dovecot-lda without indexes? Does dovecot-lda "create" the new mail by atomic rename from tmp, too?
Although the above disables updating indexes, there's still dovecot-uidlist file that is always updated by dovecot-lda. It might also cause corruption problems when multiple servers are accessing it at the same time. There's some old code that attempts to avoid updating the uidlist when it's not necessary (MAILBOX_TRANSACTION_FLAG_ASSIGN_UIDS==0), but I don't know if that works. There's also .dovecot.lda-dupes file that is written. And the Sieve scripts' compiled versions that you wanted to start using. In short: This might work, but I have a feeling you'll run into random corruption problems. I gave up trying to support simultaneous access in multiple servers via NFS long time ago, because no matter what I did kernel always cached too much and caused corruption (or alternatively it didn't cache enough and caused performance problems).
On Tue, 7 Nov 2017 19:19:23 +0200 Timo Sirainen <tss@iki.fi> wrote:
On 7 Nov 2017, at 9.15, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
On Tue, 07 Nov 2017 13:19:12 +1000 Noel Butler <noel.butler@ausics.net> wrote:
mail_location Optionally disable indexes using :INDEX=MEMORY
don't use this on IMAP boxes, but is safe to use on SMTP and POP3's boxes though
eg:
mail_location = maildir:/var/vmail/%Ld/%1Ln/%1.1Ln/%2.1Ln/%Ln/Maildir:INDEX=MEMORY
Hello Noel,
this sounds interesting. Can you please elaborate why you think this is no good idea for IMAP? We used a different LDA-scheme before (simply created the mail in maildir/tmp, then renamed it to maildir/new, just like Bernstein suggests to do) and it worked very well, no matter if the box was used whith IMAP or POP3. Why should there be any difference in using dovecot-lda without indexes? Does dovecot-lda "create" the new mail by atomic rename from tmp, too?
Although the above disables updating indexes, there's still dovecot-uidlist file that is always updated by dovecot-lda. It might also cause corruption problems when multiple servers are accessing it at the same time. There's some old code that attempts to avoid updating the uidlist when it's not necessary (MAILBOX_TRANSACTION_FLAG_ASSIGN_UIDS==0), but I don't know if that works. There's also .dovecot.lda-dupes file that is written. And the Sieve scripts' compiled versions that you wanted to start using. In short: This might work, but I have a feeling you'll run into random corruption problems. I gave up trying to support simultaneous access in multiple servers via NFS long time ago, because no matter what I did kernel always cached too much and caused corruption (or alternatively it didn't cache enough and caused performance problems).
Hello Timo,
Me again have to stress that our former implementation of the lda process did do exactly nothing to all the dovecot files, and everything worked pretty well. We had no problems in years. So I really wonder if it wouldn't be the best way to simply cut away all the heavy dovecot magic from the lda as an option. All it really needs to do is pipe the mail into a temp file, do the sieve stuff and learn from it where to rename this temp file to, basically. On the other hand you have lots of config parameters in dovecot dealing with different approaches to lock files (from fcntl to dotlock). I would expect at least one of them to work over NFS. And we do see stuff like:
2017-11-07T00:46:59.102961+01:00 mail-a05 dovecot: lda(somebody@somedomain.com): Warning: Locking transaction log file <nfs-path>/dovecot.index.log took 40 seconds (appending)
So some locking seems to work.
Regards, Stephan
On 7 Nov 2017, at 21.33, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
Me again have to stress that our former implementation of the lda process did do exactly nothing to all the dovecot files, and everything worked pretty well. We had no problems in years. So I really wonder if it wouldn't be the best way to simply cut away all the heavy dovecot magic from the lda as an option. All it really needs to do is pipe the mail into a temp file, do the sieve stuff and learn from it where to rename this temp file to, basically. On the other hand you have lots of config parameters in dovecot dealing with different approaches to lock files (from fcntl to dotlock). I would expect at least one of them to work over NFS.
I suppose you could create a unique new per-delivery temporary directory where dovecot-lda writes the mail, so it knows nothing about the user's real home/mail directory. Then move it from there to Maildir/tmp/ -> rename to Maildir/new/. I suppose you'd need to copy the Sieve script to the temp directory. Not sure if something would have to be done with .dovecot.lda-dupes.
And we do see stuff like:
2017-11-07T00:46:59.102961+01:00 mail-a05 dovecot: lda(somebody@somedomain.com): Warning: Locking transaction log file <nfs-path>/dovecot.index.log took 40 seconds (appending)
So some locking seems to work.
The problem isn't locking, but caching of data. Especially because Dovecot doesn't lock when reading files.
On Tue, 7 Nov 2017 21:40:55 +0200 Timo Sirainen <tss@iki.fi> wrote:
On 7 Nov 2017, at 21.33, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
Me again have to stress that our former implementation of the lda process did do exactly nothing to all the dovecot files, and everything worked pretty well. We had no problems in years. So I really wonder if it wouldn't be the best way to simply cut away all the heavy dovecot magic from the lda as an option. All it really needs to do is pipe the mail into a temp file, do the sieve stuff and learn from it where to rename this temp file to, basically. On the other hand you have lots of config parameters in dovecot dealing with different approaches to lock files (from fcntl to dotlock). I would expect at least one of them to work over NFS.
I suppose you could create a unique new per-delivery temporary directory where dovecot-lda writes the mail, so it knows nothing about the user's real home/mail directory. Then move it from there to Maildir/tmp/ -> rename to Maildir/new/. I suppose you'd need to copy the Sieve script to the temp directory. Not sure if something would have to be done with .dovecot.lda-dupes.
And we do see stuff like:
2017-11-07T00:46:59.102961+01:00 mail-a05 dovecot: lda(somebody@somedomain.com): Warning: Locking transaction log file <nfs-path>/dovecot.index.log took 40 seconds (appending)
So some locking seems to work.
The problem isn't locking, but caching of data. Especially because Dovecot doesn't lock when reading files.
The problem with data caching would not be existant if dovecot used O_DIRECT like proposed in the nfs (linux) docs.
---- From nfs(5) manpage ----
The NFS protocol is not designed to support true cluster file system cache coherence without some type of application serialization. If absolute cache coherence among clients is required, applications should use file locking. Alternatively, applications can also open their files with the O_DIRECT flag to disable data caching entirely.
----- end ----
AFAIK no complex director stuff would be needed then, right? The second sentence implies that using file locking should also be enough, which dovecot does.
-- Regards, Stephan
On 20 Nov 2017, at 10.50, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
AFAIK no complex director stuff would be needed then, right? The second sentence implies that using file locking should also be enough, which dovecot does.
You are building such a complex system and then you think that creating director layer would be complex?
Just out of curiosity may I ask what size of an environment is this? As it's very hard for me to believe that this would scale.
Sami
On Mon, 20 Nov 2017 12:05:46 +0200 Sami Ketola <sami.ketola@dovecot.fi> wrote:
On 20 Nov 2017, at 10.50, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
AFAIK no complex director stuff would be needed then, right? The second sentence implies that using file locking should also be enough, which dovecot does.
You are building such a complex system and then you think that creating director layer would be complex?
Just out of curiosity may I ask what size of an environment is this? As it's very hard for me to believe that this would scale.
Sami
If I had lets say 10 nodes handling incoming SMTP and delivery to local maildirs in parallel and independantly on a network storage and imapping this from another independant 4 nodes, then building a not needed bottleneck by shifting over the local delivery via director and LMTP to fewer nodes is just braindead creation of high loads on few boxes.
And btw it scales perfectly because all that is needed if load increases is up'ing additional nodes for SMTP/delivery or IMAP _which are all the same (of two types)_.
The loadbalancer needed for this can be equally used for e.g. web, ftp and other services. So there is absolutely no extra stuff needed for the mail setup.
-- Regards, Stephan
On 20 Nov 2017, at 12.54, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
If I had lets say 10 nodes handling incoming SMTP and delivery to local maildirs in parallel and independantly on a network storage and imapping this from another independant 4 nodes, then building a not needed bottleneck by shifting over the local delivery via director and LMTP to fewer nodes is just braindead creation of high loads on few boxes.
And btw it scales perfectly because all that is needed if load increases is up'ing additional nodes for SMTP/delivery or IMAP _which are all the same (of two types)_.
The loadbalancer needed for this can be equally used for e.g. web, ftp and other services. So there is absolutely no extra stuff needed for the mail setup.
Thanks for the info. But I was more after some real performance figures such as LMTP deliveries per second, IMAP logins per second or number of concurrent IMAP connections or mails FETCH/sec.
Sami
On 11/6/2017, 6:18:43 PM, Stephan von Krawczynski <skraw.ml@ithnet.com> wrote:
On Mon, 6 Nov 2017 09:50:16 -0500 Tanstaafl <tanstaafl@libertytrek.org> wrote:
Dovecot's indexing is one of its main features, and WHY it is so much faster than others.
And you want to just turn it off? Good luck...
It seems you have not understood what I am talking about. Our pre-dovecot lda did not touch the index either. And it did not harm the imap/pop procedure in any way. So we know there is no need to fiddle with the index in the process of delivery into the maildirs to keep our performance as it was before.
Apparently I'm still missing something...
Sure, you may be keeping your performance at pre-dovecot levels - but why on earth would that even be a goal?
One of the main reasons I switched from Courier to Dovecot a very long time ago was for the expected performance boost from the use of the indexes, which were automatically updated at delivery time (if you used the LDA), and the boost was huge, I was extremely pleased with the results.
There is no 'fiddling', it just works.
participants (7)
-
Aki Tuomi
-
Noel Butler
-
Sami Ketola
-
Stephan Herker
-
Stephan von Krawczynski
-
Tanstaafl
-
Timo Sirainen