Timo Sirainen tss@iki.fi wrote:

On Sun, 2009-08-09 at 22:20 +0200, Andrzej Adam Filip wrote:

...

Could you offer some suggestion how to fetch mailbox content over high RTT link (with negligible packet loss)?

Currently I use IMAP+IDLE *but* it fails to use full available bandwidth due to high RTT and "send command wait for response" nature of POP3 and IMAP4 protocols.

I'm not entirely sure what you want. Download all new messages automatically whenever they show up?

It is "maximum version" of what I would like to get. ( maximum: IMAP+IDLE equivalent, acceptable: faster POP3)

And by "mailbox" do you mean "user" or "folder"? So would you want to download all new mails in all folders?

One folder is acceptable. "All folders" would be nice.

And is it ok to create your own software to do the downloading?

I was thinking about creating perl script for the task.

If all of them were "yes", the best you could right now would be to set up a virtual "all" mailbox containing all mailboxes. Then in IDLE you'd see whenever new messages pop up and then issue FETCH n:* BODY.PEEK[] or whatever. There's also IMAP NOTIFY extension that would allow your client to ask server to immediately send the message body instead of the client having to ask for it. But Dovecot doesn't currently support that.

Or if you just always wanted to download all mails, again a virtual mailbox and FETCH 1:* BODY.PEEK[] gives you all mails.

-- [pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu Men never make passes at girls wearing glasses. -- Dorothy Parker

Ben Winslow rain@bluecherry.net wrote:

On Sun, 09 Aug 2009 22:20:41 +0200 Andrzej Adam Filip andrzej.filip@gmail.com wrote:

...

Could you offer some suggestion how to fetch mailbox content over high RTT link (with negligible packet loss)?

Currently I use IMAP+IDLE *but* it fails to use full available bandwidth due to high RTT and "send command wait for response" nature of POP3 and IMAP4 protocols.

The entire purpose of the command tag in IMAP is to facilitate batching or concurrent commands. See RFC3501 section 5.5. It shouldn't be too difficult to batch requests for new messages, or even select all the new messages in one request for a single folder.

Could you recommend tool/client program capable to use "IMAP pipelining"?

-- [pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu Virtue does not always demand a heavy sacrifice -- only the willingness to make it when necessary. -- Frederick Dunn

Ed W lists@wildgooses.com wrote:

Andrzej Adam Filip wrote:

...

Could you offer some suggestion how to fetch mailbox content over high RTT link (with negligible packet loss)?

Currently I use IMAP+IDLE *but* it fails to use full available bandwidth due to high RTT and "send command wait for response" nature of POP3 and IMAP4 protocols.

Is this a satellite phone question?

I build a bunch of products to help with this, but for a fairly specialist market. You can probably contact me offlist via this email address or any of them on the website below?

http://www.mailasail.com

However, we use lots of compression, pipelining and some predictive fetches to wring every bit of performance out of a 2.4Kbit connection with 1.5s+ latency

Good luck

Ed W

It is not a a satellite phone question. :-) It is "imap over (compressed) ssh" with "link merely one category above modem speed" to access "shared mailbox". Fetchamil's IMAP+IDLE works "almost acceptable" except pretty long periods of pretty high loads on the link.

It seems that significant part of problems is sufficiently similar to sat-phone situation.

BTW Do you suffer sufficient packet loss to consider using UDP instead of TCP based protocols?

-- [pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu There is no likelihood man can ever tap the power of the atom. -- Robert Millikan, Nobel Prize in Physics, 1923