[Dovecot] Improving lmtp performance

[Stan Hoeppner]

Joseba Torre put forth on 2/3/2011 3:53 AM:

> and some occasional delivery
> Feb  1 12:43:23 server1 postfix/lmtp[9549]: 0C5866CF37: 
> to=<cdmorales001 at server1.mydomain.com>, 
> relay=server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp], 
> delay=854, delays=37/668/140/9.8, dsn=2.0.0, status=sent (250 2.0.0 
> <cdmorales001 at server1.mydomain.com> upb1FNftR03cIgAAl0Wliw Saved)

Since you're using lmtp and not local(8) for delivery, setting the following is
unnecessary and shouldn't have changed any Postfix behavior:
local_destination_concurrency_limit = 10

"local_destination_concurrency_limit (default: 2)

    The maximal number of parallel deliveries via the local mail delivery
transport to the same recipient (when "local_destination_recipient_limit = 1")
or the maximal number of parallel deliveries to the same local domain (when
"local_destination_recipient_limit > 1"). This limit is enforced by the queue
manager. The message delivery transport name is the first field in the entry in
the master.cf file.

    A low limit of 2 is recommended, just in case someone has an expensive shell
command in a .forward file or in an alias (e.g., a mailing list manager). You
don't want to run lots of those at the same time."

If you do any deliveries via local, to say, root/etc, you probably want to
change this back to the default.


The default for lmtp_destination_concurrency_limit =
default_destination_concurrency_limit

default_destination_concurrency_limit (default: 20)

    The default maximal number of parallel deliveries to the same destination.
This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and
virtual(8) delivery agents. With per-destination recipient limit > 1, a
destination is a domain, otherwise it is a recipient.

Thus, if concurrency is what fixed your problem, merely changing Dovecot
process_min_avail = 10 solved the problem entirely, as Postfix was already
configured to pump mail through 20 parallel connections, though it would seem
you're only doing 10, if that is what this Dovecot setting does.  Again, I can't
find "process_min_avail" documented anywhere.

> or delivery error
> Feb  1 12:43:21 server1 postfix/lmtp[9554]: 1C2946CF92: 
> to=<aanton007 at server1.mydomain.com>, 
> relay=server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp], 
> delay=815, delays=1.6/664/141/8
> .9, dsn=4.2.2, status=SOFTBOUNCE (host 
> server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp] said: 552 5.2.2 
> <aanton007 at server1.mydomain.com> Quota exceeded (mailbox for user is full) (
> in reply to end of DATA command))

^^ This obviously isn't due to the performance problem.

> (softbounce was activated to prevent bounces because of misconfigurations)

> (this was the actual delivery rate: in this case, 3 messages in 14 secs)

I still can't quite grasp this.  Even with a single lmtp connection you should
be able to push mail as fast as the disks can write.  Dovecot doesn't appear to
be blocking for any reason.  This long delay between messages makes me think
something is blocking and timing out.

relay=server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp]

Note the "relay=server1.mydomain.com".  You should put brackets [] around the
nexthop host definition in your transport table (or use localhost or the
loopback address) or wherever you're defining the lmtp nexthop.  If for some
reason there's a name lookup delay/timeout, that could potentially be the cause
of your issue.  If not name resolution timing out, the problem is almost
definitely some kind of timeout.  Increasing parallelism simply allows more
timeouts to occur in parallel, increasing total deliveries, and masking the
source of the problem.

> My question was if this is the expected behavior, or if lmtp was expected to 
> behave a lot better with just 1 process (that was what I expected, at least), 
> and if my solution was usual or at least correct. It was not a practical 
> question -my system is working fine now-, but a more teorical one. I was 
> expecting answers like

> "I'm using a very simple conf for lmtp in a busy server, something similar to

I think very few people use lmtp for delivery.  I'm guessing most use Dovecot
LDA, a few procmail, maildrop, etc.

> protocol lmtp {
>   mail_plugins = quota sieve
> }
> service lmtp {
>   unix_listener /var/spool/postfix/private/dovecot-lmtp {
>     group = postfix
>     mode = 0660
>     user = postfix
>   }
> }
> 
> and it's working fine for me, so look in other place".

I think you should look for the source of the timeout and squash it.  Increasing
the Postfix logging level should show you where the timeout is occurring.

> I agree. That's why a say "if this is a common problem".

I'd say probably not common.  I don't see lmtp discussed here very often.  That
could simply be my perception, however.  I see lmtp discussed more often on the
Postfix list, but that's usually with a mailbox daemon other than Dovecot, or,
maybe Dovecot but lmtp over the network, not on the same box.

> Yes, everything was in the same VM guest, no elaborated sieve scripts that I 
> know (I've just recreated them with horde's ingo, and I'm sure they are quite 
> basic), no AV/AS, and system's load was normal, with some i/o wait, but in my 
> experience this is usual in VM guests.

These facts further my suspicion that it's a timeout issue.  Increase your
Postfix logging level and see what you find.

-- 
Stan