On 04/01/2010 03:28, Bob Eastbrook wrote:
Hi all,
Thanks for the suggestions. It sounds like the consensus is that I should avoid polling with POP or IMAP and deal with incoming messages directly.
I think it depends on your expected load and the results of a few benchmarks...
The benefit of "push" mail, ie deliver to a command is much faster processing of requests, with lower latency. The disadvantage is that if requests come in faster then the backend can process them then they will either overwhelm your backend or start to queue in (Postfix) and this will lead to loss of control over the order of processing and fairly arbitrary delays to certain messages (out of order)
The benefit of deliver to a mailbox and poll is that you can use the mailbox as a primitive queue system and just pull messages out as fast as you like and in-order.
Note both methods are pretty inefficient compared with a dedicated message queue system, eg Amazon SQS, rabbit MQS, etc, etc.
Personally I like the idea of using the mailserver as a basic message queue system and I use it for several applications. I don't expect high loads and out of order processing is not an issue so I deliver to a command which in turn uses curl to deliver to a backend process via http. I have no real protection against being overwhelmed by quantity of requests in this architecture, but the expected message speed is only a few per hour so it's not yet a big deal
I think that while 1,000 messages a sec is not yet a big deal you should do some serious analysis of your needs. Something which becomes 10-100x larger will need a serious look at the architecture and you will wish you did it now and not later
Good luck!
Ed W