On 10/25/2012 10:54 PM, Ben Morrow wrote:
At 7PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
On 10/25/2012 4:24 PM, Ben Morrow wrote:
At 1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of the Postfix local/virtual delivery agent. Using Dovecot LDA eliminates the file locking issue. Thus it also increases throughput as lock latency is eliminated.
Nonsense. deliver and imap are still separate processes accessing the same mbox, so they still need to use locks. The only difference is that since they are both dovecot programs, they will automatically be using the *same* locking strategies, and things will Just Work.
"Nonsense" implies what I stated was factually incorrect, which is not the case. There's a difference between factual incorrectness and simply staying out of the weeds.
What you stated was factually incorrect.
If you want to get into the weeds, and have me call you out for "nonsense", LDA/deliver is not a separate UNIX process. The LDA code runs within the imap process for the given user.
Nonsense. dovecot-lda runs in its own process, and does not involve the imap process in any way. As such it has to do locking.
You apparently know your tools better than I do. Neither ps nor top show a 'dovecot-lda' or similarly named process on my systems. When I send a test message from gmail through Postfix I only see CPU or memory activity in an imap process. When I close the MUA to end the imap processes and then send a test message I don't see any CPU or memory activity in any dovecot processes, only Postfix processes, including local, and spamd. So is devecot-lda running as a sub-process or thread of Postfix' local process? Or is it part of the 'dovecot' process, and the message goes through so quick that top doesn't show any CPU usage by the 'dovecot' process?
If I have the following in my dovecot.conf: ... <snipped for readability> ...
I'm not sure what you mean by 'processes of [one's own] program' but
I.e. Dovecot has its own set of processes, Postfix has its processes, etc. With "one's one processes" I'd think it makes more sense to use IPC and other tricks to accomplish concurrent access to a file rather than filesystem locking features.
it's extremely common for a process to have to take locks against another copy of itself. All traditional Unix LDAs and MUAs do this; for instance, procmail will take locks in part so that if another instance of procmail is delivering another mail to the same user at the same time the mbox won't end up corrupted.
I guess I've given MDAs w/mbox too much credit, without actually looking at the guts. Scalable databases such Oracle, db2, etc, are far more intelligent about this, and can have many thousands of processes reading and writing the same file concurrently, usually via O_DIRECT, not buffered IO, so they have complete control over IO. This is accomplished with a record lock manager and IPC, preventing more than one process from accessing one record concurrently, but allowing massive read/write concurrency to multiple records in a file. I'd think the same concurrency optimization could be done with Dovecot.
However, as Timo has pointed out, so few people use mbox these days that he simply hasn't spent much, if any, time optimizing mbox. Implementing some kind of lock manager and client code just for mbox IO concurrency simply wouldn't be worth the time. Unless he's already done something similar with mdbox. If he has, maybe that could be 'ported' to mbox as well. But again, it's probably not worth the effort given the number of mbox users, and the fact that nobody is complaining about mbox performance. I'm certainly not. It works great here.
-- Stan