[Dovecot] LMTP in 2.0: concurrency, inet sockets
I didn't really have much time to look at the beta versions of dovecot 2.0 (apart from installing it once and migrating most of my current configuration, purely for testing purposes).
An exciting new feature is the LMTP support. In the default configuration I have seen, that one only listened on a Unix socket - will it support network (inet) sockets, too?
What about concurrent deliviers? As of now, when using the deliver LDA together with Postfix, you have to make sure that only one recipient is passed to deliver at a time, so for multi-recipient mail, the system might spawn a large number of processes (two per recipient, therefore two forks/execve calls). Is that limit still imposed with the new LMTP server?
Stefan
On 3.2.2010, at 9.41, Stefan Foerster wrote:
An exciting new feature is the LMTP support. In the default configuration I have seen, that one only listened on a Unix socket - will it support network (inet) sockets, too?
Yes. In v2.0 all services can run on both unix or inet sockets.
What about concurrent deliviers? As of now, when using the deliver LDA together with Postfix, you have to make sure that only one recipient is passed to deliver at a time, so for multi-recipient mail, the system might spawn a large number of processes (two per recipient, therefore two forks/execve calls). Is that limit still imposed with the new LMTP server?
Nope. Each LMTP session can handle multiple recipients. And each LMTP process can handle multiple sessions. That also unfortunately means that if you're using multiple UNIX UIDs, lmtp processes need to run as root for now.
- Timo Sirainen <tss@iki.fi>:
On 3.2.2010, at 9.41, Stefan Foerster wrote:
What about concurrent deliviers? As of now, when using the deliver LDA together with Postfix, you have to make sure that only one recipient is passed to deliver at a time, so for multi-recipient mail, the system might spawn a large number of processes (two per recipient, therefore two forks/execve calls). Is that limit still imposed with the new LMTP server?
Nope. Each LMTP session can handle multiple recipients. And each LMTP process can handle multiple sessions. That also unfortunately means that if you're using multiple UNIX UIDs, lmtp processes need to run as root for now.
The new LMTP support seems to deliver an amazing performance. I've done a test where I delivered 100k messages with strict test script, with concurrency raning from 1 for 92% up to 18 for 0.2% of all deliveries. Recipient concurrency was 1 for 84% up to 9 for 1.3% of all messages (I did some logmining to get real data).
Both tests were using Postfix 2.7-20101117 with recipient data stored in Berkeley DB ("hash:") maps, and I provided the userdb and passdb files for both Dovecot 1.2.10 and 2.0b2 as flat text files, too.
For 100k messages, delivery to a Maildir store using Postifx's pipe(8) and Dovecot's deliver took 4319s, which equals 23.15 delivieries per second - and that's with the pipe(8) delivery agent haning around for up to 100 seconds, being reused for multiple transactions.
Dovecot 2.0, with LMTP, was finished after 4007s, which equals 24.96 deliveries per second.
Apart from an almost 10% difference, the system felt a lot less "sluggish" (read: I/O saturated) with 2.0b2 - unfortunately, I didn't run sysstat during the test, but I'll get that done next weekend.
Keep up the good work!
Stefan
Stefan Foerster put forth on 2/7/2010 6:39 AM:
- Timo Sirainen <tss@iki.fi>:
On 3.2.2010, at 9.41, Stefan Foerster wrote:
What about concurrent deliviers? As of now, when using the deliver LDA together with Postfix, you have to make sure that only one recipient is passed to deliver at a time, so for multi-recipient mail, the system might spawn a large number of processes (two per recipient, therefore two forks/execve calls). Is that limit still imposed with the new LMTP server?
Nope. Each LMTP session can handle multiple recipients. And each LMTP process can handle multiple sessions. That also unfortunately means that if you're using multiple UNIX UIDs, lmtp processes need to run as root for now.
The new LMTP support seems to deliver an amazing performance. I've done a test where I delivered 100k messages with strict test script, with concurrency raning from 1 for 92% up to 18 for 0.2% of all deliveries. Recipient concurrency was 1 for 84% up to 9 for 1.3% of all messages (I did some logmining to get real data).
Both tests were using Postfix 2.7-20101117 with recipient data stored in Berkeley DB ("hash:") maps, and I provided the userdb and passdb files for both Dovecot 1.2.10 and 2.0b2 as flat text files, too.
For 100k messages, delivery to a Maildir store using Postifx's pipe(8) and Dovecot's deliver took 4319s, which equals 23.15 delivieries per second - and that's with the pipe(8) delivery agent haning around for up to 100 seconds, being reused for multiple transactions.
Dovecot 2.0, with LMTP, was finished after 4007s, which equals 24.96 deliveries per second.
Apart from an almost 10% difference, the system felt a lot less "sluggish" (read: I/O saturated) with 2.0b2 - unfortunately, I didn't run sysstat during the test, but I'll get that done next weekend.
Keep up the good work!
Thanks for the test info Stefan. What are the hardware specs of the system you ran this test on?
-- Stan
- Stan Hoeppner <stan@hardwarefreak.com>:
Stefan Foerster put forth on 2/7/2010 6:39 AM:
For 100k messages, delivery to a Maildir store using Postifx's pipe(8) and Dovecot's deliver took 4319s, which equals 23.15 delivieries per second - and that's with the pipe(8) delivery agent haning around for up to 100 seconds, being reused for multiple transactions.
Dovecot 2.0, with LMTP, was finished after 4007s, which equals 24.96 deliveries per second.
Apart from an almost 10% difference, the system felt a lot less "sluggish" (read: I/O saturated) with 2.0b2 - unfortunately, I didn't run sysstat during the test, but I'll get that done next weekend.
Keep up the good work!
Thanks for the test info Stefan. What are the hardware specs of the system you ran this test on?
KVM guest, dedicated LV using virtio, underlying single SATA disk. The system had 1024Mb of RAM and two CPU cores (out of eight) assigned.
Stefan
participants (3)
-
Stan Hoeppner
-
Stefan Foerster
-
Timo Sirainen