Re: [Dovecot] Improving lmtp performance

Joseba Torre put forth on 2/3/2011 3:53 AM:

and some occasional delivery Feb 1 12:43:23 server1 postfix/lmtp[9549]: 0C5866CF37: to=cdmorales001@server1.mydomain.com, relay=server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp], delay=854, delays=37/668/140/9.8, dsn=2.0.0, status=sent (250 2.0.0 cdmorales001@server1.mydomain.com upb1FNftR03cIgAAl0Wliw Saved)

Since you're using lmtp and not local(8) for delivery, setting the following is unnecessary and shouldn't have changed any Postfix behavior: local_destination_concurrency_limit = 10

"local_destination_concurrency_limit (default: 2)

The maximal number of parallel deliveries via the local mail delivery

transport to the same recipient (when "local_destination_recipient_limit = 1") or the maximal number of parallel deliveries to the same local domain (when "local_destination_recipient_limit > 1"). This limit is enforced by the queue manager. The message delivery transport name is the first field in the entry in the master.cf file.

A low limit of 2 is recommended, just in case someone has an expensive shell

command in a .forward file or in an alias (e.g., a mailing list manager). You don't want to run lots of those at the same time."

If you do any deliveries via local, to say, root/etc, you probably want to change this back to the default.

The default for lmtp_destination_concurrency_limit = default_destination_concurrency_limit

default_destination_concurrency_limit (default: 20)

The default maximal number of parallel deliveries to the same destination.

This is the default limit for delivery via the lmtp(8), pipe(8), smtp(8) and virtual(8) delivery agents. With per-destination recipient limit > 1, a destination is a domain, otherwise it is a recipient.

Thus, if concurrency is what fixed your problem, merely changing Dovecot process_min_avail = 10 solved the problem entirely, as Postfix was already configured to pump mail through 20 parallel connections, though it would seem you're only doing 10, if that is what this Dovecot setting does. Again, I can't find "process_min_avail" documented anywhere.

or delivery error Feb 1 12:43:21 server1 postfix/lmtp[9554]: 1C2946CF92: to=aanton007@server1.mydomain.com, relay=server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp], delay=815, delays=1.6/664/141/8 .9, dsn=4.2.2, status=SOFTBOUNCE (host server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp] said: 552 5.2.2 aanton007@server1.mydomain.com Quota exceeded (mailbox for user is full) ( in reply to end of DATA command))

^^ This obviously isn't due to the performance problem.

(softbounce was activated to prevent bounces because of misconfigurations)

(this was the actual delivery rate: in this case, 3 messages in 14 secs)

I still can't quite grasp this. Even with a single lmtp connection you should be able to push mail as fast as the disks can write. Dovecot doesn't appear to be blocking for any reason. This long delay between messages makes me think something is blocking and timing out.

relay=server1.mydomain.com[/var/spool/postfix/private/dovecot-lmtp]

Note the "relay=server1.mydomain.com". You should put brackets [] around the nexthop host definition in your transport table (or use localhost or the loopback address) or wherever you're defining the lmtp nexthop. If for some reason there's a name lookup delay/timeout, that could potentially be the cause of your issue. If not name resolution timing out, the problem is almost definitely some kind of timeout. Increasing parallelism simply allows more timeouts to occur in parallel, increasing total deliveries, and masking the source of the problem.

My question was if this is the expected behavior, or if lmtp was expected to behave a lot better with just 1 process (that was what I expected, at least), and if my solution was usual or at least correct. It was not a practical question -my system is working fine now-, but a more teorical one. I was expecting answers like

"I'm using a very simple conf for lmtp in a busy server, something similar to

I think very few people use lmtp for delivery. I'm guessing most use Dovecot LDA, a few procmail, maildrop, etc.

protocol lmtp { mail_plugins = quota sieve } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } }

and it's working fine for me, so look in other place".

I think you should look for the source of the timeout and squash it. Increasing the Postfix logging level should show you where the timeout is occurring.

I agree. That's why a say "if this is a common problem".

I'd say probably not common. I don't see lmtp discussed here very often. That could simply be my perception, however. I see lmtp discussed more often on the Postfix list, but that's usually with a mailbox daemon other than Dovecot, or, maybe Dovecot but lmtp over the network, not on the same box.

Yes, everything was in the same VM guest, no elaborated sieve scripts that I know (I've just recreated them with horde's ingo, and I'm sure they are quite basic), no AV/AS, and system's load was normal, with some i/o wait, but in my experience this is usual in VM guests.

These facts further my suspicion that it's a timeout issue. Increase your Postfix logging level and see what you find.

-- Stan