[Dovecot] Locking /var/mail/user issue with postfix and dovecot

[Ben Morrow]

At  7PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
> On 10/25/2012 4:24 PM, Ben Morrow wrote:
> > At  1PM -0500 on 25/10/12 you (Stan Hoeppner) wrote:
> >>
> >> Switch Postfix to use the Dovecot Local Deliver Agent (LDA) in place of
> >> the Postfix local/virtual delivery agent.  Using Dovecot LDA eliminates
> >> the file locking issue.  Thus it also increases throughput as lock
> >> latency is eliminated.
> > 
> > Nonsense. deliver and imap are still separate processes accessing the
> > same mbox, so they still need to use locks. The only difference is that
> > since they are both dovecot programs, they will automatically be using
> > the *same* locking strategies, and things will Just Work.
> 
> "Nonsense" implies what I stated was factually incorrect, which is not
> the case.  There's a difference between factual incorrectness and simply
> staying out of the weeds.

What you stated was factually incorrect.

> If you want to get into the weeds, and have me call you out for
> "nonsense", LDA/deliver is not a separate UNIX process.  The LDA code
> runs within the imap process for the given user.

Nonsense. dovecot-lda runs in its own process, and does not involve the
imap process in any way. As such it has to do locking.

If I have the following in my dovecot.conf:

    mbox_read_locks = fcntl
    mbox_write_locks = dotlock fcntl

    namespace {
        location = mbox:~/mbox
        separator = /
        type = private
        list = yes
        prefix = MBOX/
    }

and I run

    ktrace dovecot-lda -f mauzo at localhost -m MBOX/foo < mail

then the ktrace contains

 <snip>
 44973 dovecot-lda CALL  access(0x8021f5f68,0x6<W_OK|R_OK>)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo"
 44973 dovecot-lda RET   access 0
    [Check the mbox exists and is accessible]

 <snip>
 44973 dovecot-lda CALL  lstat(0x8020196c0,0x7fffffffcb60)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo.lock"
 44973 dovecot-lda RET   lstat -1 errno 2 No such file or directory
    [Look for a .lock file]

 <snip>
 44973 dovecot-lda CALL  open(0x8020196c0,0xa02<O_RDWR|O_CREAT|O_EXCL>,
    0x1b6<S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH>)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo.lock"
 44973 dovecot-lda RET   open 9
 44973 dovecot-lda CALL  write(0x9,0x802019830,0x19)
 44973 dovecot-lda RET   write 25/0x19
 44973 dovecot-lda CALL  clock_gettime(0xd,0x7fffffffcbf0)
 44973 dovecot-lda RET   clock_gettime 0
 44973 dovecot-lda CALL  fstat(0x9,0x7fffffffcd90)
 44973 dovecot-lda RET   fstat 0
 44973 dovecot-lda CALL  close(0x9)
 44973 dovecot-lda RET   close 0
 44973 dovecot-lda CALL  lstat(0x8020196c0,0x7fffffffce60)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo.lock"
 44973 dovecot-lda RET   lstat 0
    [Create a .lock file, and check it was successful]

 <snip>
 44973 dovecot-lda CALL  open(0x8021f5f68,0x2<O_RDWR>,<unused>0xfac3c0)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo"
 44973 dovecot-lda RET   open 9
    [Open the mbox file itself]

 <snip>
 44973 dovecot-lda CALL  fcntl(0x9,F_SETLKW,0xffffffffffffcfd0)
 44973 dovecot-lda RET   fcntl 0
    [Set a fcntl lock on the mbox file]

 <snip>
 44973 dovecot-lda CALL  pread(0,0x802031000,0x1000,0)
 44973 dovecot-lda RET   pread 43/0x2b
 <snip>
 44973 dovecot-lda CALL  write(0x9,0x802228000,0xf5)
 44973 dovecot-lda RET   write 245/0xf5
 <snip>
 44973 dovecot-lda CALL  fsync(0x9)
 44973 dovecot-lda RET   fsync 0
    [Read from stdin and write to the mbox file]

 <snip>
 44973 dovecot-lda CALL  lstat(0x802028440,0x7fffffffd010)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo.lock"
 44973 dovecot-lda RET   lstat 0
 44973 dovecot-lda CALL  unlink(0x802028440)
 44973 dovecot-lda NAMI  "/home/mauzo/mbox/foo.lock"
 44973 dovecot-lda RET   unlink 0
    [Remove the .lock file]

 <snip>
 44973 dovecot-lda CALL  fcntl(0x9,F_SETLK,0xffffffffffffd160)
 44973 dovecot-lda RET   fcntl 0
    [Clear the fcntl lock]

 <snip>
 44973 dovecot-lda CALL  close(0x9)
 44973 dovecot-lda RET   close 0
    [Close the mbox file]

> This is what allows
> Dovecot to perform 'simultaneous' reads/writes to an mbox file, avoiding
> filesystem level locking latency.  Using filesystem level locking to
> control read/write access between processes of own's program would be
> insane on many levels.

I'm not sure what you mean by 'processes of [one's own] program' but
it's extremely common for a process to have to take locks against
another copy of itself. All traditional Unix LDAs and MUAs do this; for
instance, procmail will take locks in part so that if another instance
of procmail is delivering another mail to the same user at the same time
the mbox won't end up corrupted.

Ben