[Dovecot] architecture to handle 1000 messages per second?

[Ed W]

On 04/01/2010 03:28, Bob Eastbrook wrote:
> Hi all,
>
> Thanks for the suggestions.  It sounds like the consensus is that I
> should avoid polling with POP or IMAP and deal with incoming messages
> directly.
>    

I think it depends on your expected load and the results of a few 
benchmarks...

The benefit of "push" mail, ie deliver to a command is much faster 
processing of requests, with lower latency.  The disadvantage is that if 
requests come in faster then the backend can process them then they will 
either overwhelm your backend or start to queue in (Postfix) and this 
will lead to loss of control over the order of processing and fairly 
arbitrary delays to certain messages (out of order)

The benefit of deliver to a mailbox and poll is that you can use the 
mailbox as a primitive queue system and just pull messages out as fast 
as you like and in-order.

Note both methods are pretty inefficient compared with a dedicated 
message queue system, eg Amazon SQS, rabbit MQS, etc, etc.

Personally I like the idea of using the mailserver as a basic message 
queue system and I use it for several applications.  I don't expect high 
loads and out of order processing is not an issue so I deliver to a 
command which in turn uses curl to deliver to a backend process via 
http.  I have no real protection against being overwhelmed by quantity 
of requests in this architecture, but the expected message speed is only 
a few per hour so it's not yet a big deal

I think that while 1,000 messages a sec is not yet a big deal you should 
do some serious analysis of your needs.  Something which becomes 10-100x 
larger will need a serious look at the architecture and you will wish 
you did it now and not later

Good luck!

Ed W