Massive alias / bulk delivery problem
Hello list,
we are experimenting with using Dovecot as part of a simplified mail stack with SQL for a internal messaging / notification system for our customers. Sometimes we need to send a message to thousands of customers at once and we do it by having aliases resolve to 10k and 20k+ mail accounts.
When we send such a message the server load (obviously) is very high for a few minutes (in case of an alias containing 10k addresses its around 6-7 minutes for a 2 core machine with load around 20-30).
Are there any optimizations or ways how to design the system around this problem ? Eg. is there a way how to deliver messages more slowly, and spread the load over longer time ?
The stack is very simple with Postfix receiving the messages and Dovecot acting as a LMTP and IMAP server. Mailboxes are in the Maildir format on disk. Customers get the messages on login, a PHP script connects to the mail account associated with their ID via IMAP and sends unseen messages in the form of a JSON.
Best regards,
gnd/
gnd> we are experimenting with using Dovecot as part of a simplified gnd> mail stack with SQL for a internal messaging / notification gnd> system for our customers. Sometimes we need to send a message to gnd> thousands of customers at once and we do it by having aliases gnd> resolve to 10k and 20k+ mail accounts.
gnd> When we send such a message the server load (obviously) is very gnd> high for a few minutes (in case of an alias containing 10k gnd> addresses its around 6-7 minutes for a 2 core machine with load gnd> around 20-30).
gnd> Are there any optimizations or ways how to design the system gnd> around this problem ? Eg. is there a way how to deliver messages gnd> more slowly, and spread the load over longer time ?
This is not a dovecot issue, but a postfix issue. You need to setup some queues in postfix to handle mail floods like this. You should take this to the postfix mailing list on postfix.org, and first look in the archives, there have been recent discussions about this type of issue.
gnd> The stack is very simple with Postfix receiving the messages and gnd> Dovecot acting as a LMTP and IMAP server. Mailboxes are in the gnd> Maildir format on disk. Customers get the messages on login, a gnd> PHP script connects to the mail account associated with their ID gnd> via IMAP and sends unseen messages in the form of a JSON.
Postfix is also sending the emails too. :-)
John
Yes you are right, i realized that just after i sent the email, that it was a bit in haste.
anyway, if there is anyway how to optimize dovecot for effective delivery of 20k+ emails within a few minutes, id be glad to know.
.. and i will have to ask the same q at postfix mailinglists
gnd/
On 7/14/20 8:12 PM, John Stoffel wrote:
gnd> we are experimenting with using Dovecot as part of a simplified gnd> mail stack with SQL for a internal messaging / notification gnd> system for our customers. Sometimes we need to send a message to gnd> thousands of customers at once and we do it by having aliases gnd> resolve to 10k and 20k+ mail accounts.
gnd> When we send such a message the server load (obviously) is very gnd> high for a few minutes (in case of an alias containing 10k gnd> addresses its around 6-7 minutes for a 2 core machine with load gnd> around 20-30).
gnd> Are there any optimizations or ways how to design the system gnd> around this problem ? Eg. is there a way how to deliver messages gnd> more slowly, and spread the load over longer time ?
This is not a dovecot issue, but a postfix issue. You need to setup some queues in postfix to handle mail floods like this. You should take this to the postfix mailing list on postfix.org, and first look in the archives, there have been recent discussions about this type of issue.
gnd> The stack is very simple with Postfix receiving the messages and gnd> Dovecot acting as a LMTP and IMAP server. Mailboxes are in the gnd> Maildir format on disk. Customers get the messages on login, a gnd> PHP script connects to the mail account associated with their ID gnd> via IMAP and sends unseen messages in the form of a JSON.
Postfix is also sending the emails too. :-)
John
On Tue, 14 Jul 2020, gnd wrote:
[...] anyway, if there is anyway how to optimize dovecot for effective delivery of 20k+ emails within a few minutes, id be glad to know.
.. but in the first message you wrote:
Are there any optimizations or ways how to design the system around this problem ? Eg. is there a way how to deliver messages more slowly, and spread the load over longer time ?
So it is not clear whether you want to actually deliver the messages as fast as possible ("effective delivery .. within a few minutes") or if you'd rather have the process spread over longer time.
Generally it is recommended to let the system work as fast as it can, as otherwise you'll just make it slower for no reason (you will use your resources less efficiently).
Cheers.
On 7/15/20 6:26 PM, Bernardo Reino wrote:
On Tue, 14 Jul 2020, gnd wrote:
[...] anyway, if there is anyway how to optimize dovecot for effective delivery of 20k+ emails within a few minutes, id be glad to know.
.. but in the first message you wrote:
Are there any optimizations or ways how to design the system around this problem ? Eg. is there a way how to deliver messages more slowly, and spread the load over longer time ?
So it is not clear whether you want to actually deliver the messages as fast as possible ("effective delivery .. within a few minutes") or if you'd rather have the process spread over longer time.
i see how this can sound confusing. what i meant is that in the ideal case i would sacrifice the delivery time (the mails can be delivered within one hour not a few minutes) as long as the server load stays 'normal'. currently the delivery of 20k mails on a 2core server takes around 90 seconds, but the load reaches a peak of ~15.
It was worse, but I have turned on these settings in Dovecot:
mail_fsync = never maildir_very_dirty_syncs=yes
The server is still unused, so i'm afraid that under normal circumstances (having a constant stream of IMA traffic from those 20k users) things will be much worse.
Generally it is recommended to let the system work as fast as it can, as otherwise you'll just make it slower for no reason (you will use your resources less efficiently).
id prefer to slow down / spread the load over longer time as immediate message delivery is not so important.. but as written earlier, im crying on the wrong grave, as this is probably better to approach from postfix side.
Cheers.
On Tue, 14 Jul 2020, gnd wrote:
anyway, if there is anyway how to optimize dovecot for effective delivery of 20k+ emails within a few minutes, id be glad to know.
It depends on what you mean by "optimize". Arguably, letting postfix/dovecot hammer its brains out for a few minutes might have the lowest overall performance/complexity cost, but may also be DoS'ing your mail system for a small time window.
Configuring postfix to concurrently send message to a single set-UID LMTP will probaby help with minimizing process overhead. It might be especially useful if you also need to de-duping large attachments.
If your current setup can cope with this mail load, just let it. If it ain't broke, why fix it?
If the intensity is causing problems, you could offload mail delivery to an auxilliary process outside your mail system by aliasing to a handler script ( e.g. |remail.sh), which accepts the message, then sends it to your 20k+ recipients in small batches with small delays.
Joseph Tam jtam.home@gmail.com
participants (5)
-
Aki Tuomi
-
Bernardo Reino
-
gnd
-
John Stoffel
-
Joseph Tam