[Dovecot] Locking /var/mail/user issue with postfix and dovecot

[Stan Hoeppner]

You are a well of accessible knowledge Ben.  (How have I missed your
posts in the past?)

On 10/26/2012 3:11 AM, Ben Morrow wrote:

> Assuming you have 
> 
>     mailbox_command = /.../dovecot-lda -a "${RECIPIENT}"

I'm setup for system users so it's a simpler, but yes.

> or something equivalent in your Postfix configuration, dovecot-lda runs
> as a subprocess of local(8) under the uid of the delivered-to user.

Of course that makes sense given Postfix is doing the calling.  I would
have assumed this but my feeble use of tools wasn't showing anything.

> Filesystem locking, at least if NFS is not involved, is not that
> expensive. Successfully acquiring a flock or fcntl lock takes only a
> single syscall which doesn't have to touch the disk, and any form of IPC
> is going to need to do that. (Even something like a shared memory region
> will need a mutex for synchronisation, and acquiring the mutex has to go
> through the kernel.)

Thanks for this.  I was under the assumption flock/fcntl were more
expensive than they are.  Probably because all I'd read about them was
in relation to NFS (which I don't use, but I read alot like many do).

> Dotlocking *is* expensive, because acquiring a dotlock is a complicated
> process requiring lots of syscalls, some of which have to write to disk;
> and any scheme involving acquiring several locks on the same file is
> going to be more so, especially if you can end up getting the first lock
> but finding you can't get the second, so then you have to undo the first
> and try again.

Yeah, I knew dotlocks were the worst due to disk writes, but didn't know
the other details.

> More importantly, the biggest problem with mbox as a mailbox format is
> that any access at all has to lock the whole mailbox. If the LDA is
> trying to deliver a new message at the same time as an IMAP user is
> fetching a completely different message, or if two instances of the LDA
> are trying to deliver at the same time, they will be competing for the
> same lock even though they don't really need to be. A file-per-message
> format like Maildir avoids this, to the point of being mostly lockless,
> but that brings its own efficiency problems; the point of dbox is to
> find the compromise between these positions that works best.

mbox locking hasn't been problem here as I split the INBOX from the user
mailboxes containing IMAP folders (mbox files).  We make heavy use of
sieve scripts to sort on delivery, so there's not much concurrent access
to any one mbox file.

The efficiency issue is why I chose mbox over maildir.  Users here keep
a lot of (list) mail and FTS often.  The load on the spindles with
maildir is simply too great and would bog down all users.  The IOPS
benefit of mbox in this scenario outweighs any locking issues.

> I wouldn't look too hard at the details of the various ways there are of
> locking and parsing mbox files, or the ways in which they can go wrong.
> It's enough to make anyone swear off email for life :).

Heheh.

> The only reason for using mbox is for compatibility with other systems
> which use mbox, 

Not necessarily true.  See above.  I'm sure I'm not the only one using
mbox for this reason.  Dovecot is my only app hitting these mbox files.

> which means you have to do the locking the same way as
> they do (assuming you can work out what that is). If you're going to
> change the locking rules you might as well change the file format at the
> same time, both to remove the insanity and to make it actually suitable
> for use as an IMAP mailstore. That's what Timo did with dbox, so if
> you've got your systems to the point where nothing but Dovecot touches
> the mail files you should seriously consider switching.

If/when I do switch mailbox formats it'll be to mdbox so FTS doesn't
drop a big hammer on the spindles.

Thanks for the informative discussion Ben.

-- 
Stan