[Dovecot] Locking /var/mail/user issue with postfix and dovecot

[Stan Hoeppner]

On 10/25/2012 10:54 PM, Ben Morrow wrote:
> At  7PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
>> On 10/25/2012 4:24 PM, Ben Morrow wrote:
>>> At  1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
>>>>
>>>> Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of
>>>> the Postfix local/virtual delivery agent.  Using Dovecot LDA eliminates
>>>> the file locking issue.  Thus it also increases throughput as lock
>>>> latency is eliminated.
>>>
>>> Nonsense. deliver and imap are still separate processes accessing the
>>> same mbox, so they still need to use locks. The only difference is that
>>> since they are both dovecot programs, they will automatically be using
>>> the *same* locking strategies, and things will Just Work.
>>
>> "Nonsense" implies what I stated was factually incorrect, which is not
>> the case.  There's a difference between factual incorrectness and simply
>> staying out of the weeds.
> 
> What you stated was factually incorrect.
> 
>> If you want to get into the weeds, and have me call you out for
>> "nonsense", LDA/deliver is not a separate UNIX process.  The LDA code
>> runs within the imap process for the given user.
> 
> Nonsense. dovecot-lda runs in its own process, and does not involve the
> imap process in any way. As such it has to do locking.

You apparently know your tools better than I do.  Neither ps nor top
show a 'dovecot-lda' or similarly named process on my systems.  When I
send a test message from gmail through Postfix I only see CPU or memory
activity in an imap process.  When I close the MUA to end the imap
processes and then send a test message I don't see any CPU or memory
activity in any dovecot processes, only Postfix processes, including
local, and spamd.  So is devecot-lda running as a sub-process or thread
of Postfix' local process?  Or is it part of the 'dovecot' process, and
the message goes through so quick that top doesn't show any CPU usage by
the 'dovecot' process?

> If I have the following in my dovecot.conf:
...
<snipped for readability>
...

> I'm not sure what you mean by 'processes of [one's own] program' but

I.e. Dovecot has its own set of processes, Postfix has its processes,
etc.  With "one's one processes" I'd think it makes more sense to use
IPC and other tricks to accomplish concurrent access to a file rather
than filesystem locking features.

> it's extremely common for a process to have to take locks against
> another copy of itself. All traditional Unix LDAs and MUAs do this; for
> instance, procmail will take locks in part so that if another instance
> of procmail is delivering another mail to the same user at the same time
> the mbox won't end up corrupted.

I guess I've given MDAs w/mbox too much credit, without actually looking
at the guts.  Scalable databases such Oracle, db2, etc, are far more
intelligent about this, and can have many thousands of processes reading
and writing the same file concurrently, usually via O_DIRECT, not
buffered IO, so they have complete control over IO.  This is
accomplished with a record lock manager and IPC, preventing more than
one process from accessing one record concurrently, but allowing massive
read/write concurrency to multiple records in a file.  I'd think the
same concurrency optimization could be done with Dovecot.

However, as Timo has pointed out, so few people use mbox these days that
he simply hasn't spent much, if any, time optimizing mbox.  Implementing
some kind of lock manager and client code just for mbox IO concurrency
simply wouldn't be worth the time.  Unless he's already done something
similar with mdbox.  If he has, maybe that could be 'ported' to mbox as
well.  But again, it's probably not worth the effort given the number of
mbox users, and the fact that nobody is complaining about mbox
performance.  I'm certainly not.  It works great here.

-- 
Stan