Is atomic MOVING of messages between IMAP folders possible?
I would like to use a shared IMAP account, with multiple users accessing it simultaneously. The users would take ownership of messages by first attempting to MOVE the messages from the Inbox, into their private IMAP folder, still within the same account. Now, since there will be multiple users competing for the same messages, I naturally want only ONE of the simultaneous moves to be successful at a time.
So far, this isn't working. If I do the move from two clients, simultaneously, the messages can go to *both *destination folders - duplicates can result.
Is it possible to configure Dovecot and/or an IMAP client to behave the way I want it to? If the answer to this is YES, then I'll offer my config details. If the answer is NO, the next question is - do any email systems at all behave the way I want? (I tried a hosted Exchange/OWA service - it has the same problem)
Thanks & regards, Greg.
On 04 Aug 2014, at 10:44, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
I would like to use a shared IMAP account, with multiple users accessing it simultaneously. The users would take ownership of messages by first attempting to MOVE the messages from the Inbox, into their private IMAP folder, still within the same account. Now, since there will be multiple users competing for the same messages, I naturally want only ONE of the simultaneous moves to be successful at a time.
So far, this isn't working. If I do the move from two clients, simultaneously, the messages can go to *both *destination folders - duplicates can result.
Is it possible to configure Dovecot and/or an IMAP client to behave the way I want it to? If the answer to this is YES, then I'll offer my config details. If the answer is NO, the next question is - do any email systems at all behave the way I want? (I tried a hosted Exchange/OWA service - it has the same problem)
Dovecot doesn't even attempt to do atomic MOVEs. I don't think any server will. If you can change the client code, you could use CONDSTORE instead, which does give atomic STOREs.
Am 04.08.2014 um 14:00 schrieb Timo Sirainen:
On 04 Aug 2014, at 10:44, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
I would like to use a shared IMAP account, with multiple users accessing it simultaneously. The users would take ownership of messages by first attempting to MOVE the messages from the Inbox, into their private IMAP folder, still within the same account. Now, since there will be multiple users competing for the same messages, I naturally want only ONE of the simultaneous moves to be successful at a time.
So far, this isn't working. If I do the move from two clients, simultaneously, the messages can go to *both *destination folders - duplicates can result.
Is it possible to configure Dovecot and/or an IMAP client to behave the way I want it to? If the answer to this is YES, then I'll offer my config details. If the answer is NO, the next question is - do any email systems at all behave the way I want? (I tried a hosted Exchange/OWA service - it has the same problem)
Dovecot doesn't even attempt to do atomic MOVEs. I don't think any server will. If you can change the client code, you could use CONDSTORE instead, which does give atomic STOREs.
Internet Message Access Protocol (IMAP) - MOVE Extension http://tools.ietf.org/html/rfc6851
well, both, client and server would need to support it rely on that is unlikely for many years
Yes, both client and server support IMAP MOVE, and both also support CONDSTORE.
I have tried both with and without CONDSTORE enabled in the client, with the same result.
I am very confident IMAP MOVE is actually being invoked, because intra-account moves occur extremely rapidly. (much faster than inter-account moves, which of course is a copy & delete)
Thanks so far.
Client is Postbox & Thunderbird on Windows. (I realise Postbox is based on Thunderbird)
Greg.
On 4 August 2014 22:03, Reindl Harald <h.reindl@thelounge.net> wrote:
Am 04.08.2014 um 14:00 schrieb Timo Sirainen:
On 04 Aug 2014, at 10:44, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
I would like to use a shared IMAP account, with multiple users accessing it simultaneously. The users would take ownership of messages by first attempting to MOVE the messages from the Inbox, into their private IMAP folder, still within the same account. Now, since there will be multiple users competing for the same messages, I naturally want only ONE of the simultaneous moves to be successful at a time.
So far, this isn't working. If I do the move from two clients, simultaneously, the messages can go to *both *destination folders - duplicates can result.
Is it possible to configure Dovecot and/or an IMAP client to behave the way I want it to? If the answer to this is YES, then I'll offer my config details. If the answer is NO, the next question is - do any email systems at all behave the way I want? (I tried a hosted Exchange/OWA service - it has the same problem)
Dovecot doesn't even attempt to do atomic MOVEs. I don't think any server will. If you can change the client code, you could use CONDSTORE instead, which does give atomic STOREs.
Internet Message Access Protocol (IMAP) - MOVE Extension http://tools.ietf.org/html/rfc6851
well, both, client and server would need to support it rely on that is unlikely for many years
On 04 Aug 2014, at 14:12, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
Yes, both client and server support IMAP MOVE, and both also support CONDSTORE.
I have tried both with and without CONDSTORE enabled in the client, with the same result.
With CONDSTORE I was thinking you could do it something like:
1 FETCH 1 (FLAGS MODSEQ)
- 1 FLAGS () MODSEQ 12345 2 STORE (UNCHANGEDSINCE 12345) 1 +FLAGS $AtomicMove 3 MOVE 1 elsewhere
If another client attempts the same, either 1 will return $AtomicMove in flags -> abort or 2 will fail with NO. But you should still handle failures if the client/connection dies between 2 and 3 or 3 fails for some reason.
But, of course if you can't change the client code to do this then it doesn't help.
I am very confident IMAP MOVE is actually being invoked, because intra-account moves occur extremely rapidly. (much faster than inter-account moves, which of course is a copy & delete)
Inter-account physically copies the data (FETCH + APPEND + EXPUNGE). Alternative to MOVE is COPY + EXPUNGE, which is just as fast as MOVE. Dovecot actually implements MOVE by internally doing a COPY + EXPUNGE.
Thanks Timo, and no, I can't (easily) change the code of the client.
I must say I am extremely disappointed that intra-account moves are not atomic. As far as I can tell, IMAP was designed to allow shared access, so in my opinion this operation should be atomic. Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)
Looks like a bloatware - sorry - helpdesk system - is what I will need to use.
Greg.
On 4 August 2014 22:44, Timo Sirainen <tss@iki.fi> wrote:
On 04 Aug 2014, at 14:12, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
Yes, both client and server support IMAP MOVE, and both also support CONDSTORE.
I have tried both with and without CONDSTORE enabled in the client, with the same result.
With CONDSTORE I was thinking you could do it something like:
1 FETCH 1 (FLAGS MODSEQ)
- 1 FLAGS () MODSEQ 12345 2 STORE (UNCHANGEDSINCE 12345) 1 +FLAGS $AtomicMove 3 MOVE 1 elsewhere
If another client attempts the same, either 1 will return $AtomicMove in flags -> abort or 2 will fail with NO. But you should still handle failures if the client/connection dies between 2 and 3 or 3 fails for some reason.
But, of course if you can't change the client code to do this then it doesn't help.
I am very confident IMAP MOVE is actually being invoked, because intra-account moves occur extremely rapidly. (much faster than inter-account moves, which of course is a copy & delete)
Inter-account physically copies the data (FETCH + APPEND + EXPUNGE). Alternative to MOVE is COPY + EXPUNGE, which is just as fast as MOVE. Dovecot actually implements MOVE by internally doing a COPY + EXPUNGE.
Note that MOVE isn't atomic even between moving within one user's folders. The MOVE RFC itself also doesn't say anything about it ever having to be atomic. Although if by atomicity you mean that you simply want to make sure that the same source mail can't be MOVEd twice, that would be doable with some work I think. Even for full conversations (without partial failures).
On 05 Aug 2014, at 13:19, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
Thanks Timo, and no, I can't (easily) change the code of the client.
I must say I am extremely disappointed that intra-account moves are not atomic. As far as I can tell, IMAP was designed to allow shared access, so in my opinion this operation should be atomic. Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)
Looks like a bloatware - sorry - helpdesk system - is what I will need to use.
Greg.
On 4 August 2014 22:44, Timo Sirainen <tss@iki.fi> wrote:
On 04 Aug 2014, at 14:12, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
Yes, both client and server support IMAP MOVE, and both also support CONDSTORE.
I have tried both with and without CONDSTORE enabled in the client, with the same result.
With CONDSTORE I was thinking you could do it something like:
1 FETCH 1 (FLAGS MODSEQ)
- 1 FLAGS () MODSEQ 12345 2 STORE (UNCHANGEDSINCE 12345) 1 +FLAGS $AtomicMove 3 MOVE 1 elsewhere
If another client attempts the same, either 1 will return $AtomicMove in flags -> abort or 2 will fail with NO. But you should still handle failures if the client/connection dies between 2 and 3 or 3 fails for some reason.
But, of course if you can't change the client code to do this then it doesn't help.
I am very confident IMAP MOVE is actually being invoked, because intra-account moves occur extremely rapidly. (much faster than inter-account moves, which of course is a copy & delete)
Inter-account physically copies the data (FETCH + APPEND + EXPUNGE). Alternative to MOVE is COPY + EXPUNGE, which is just as fast as MOVE. Dovecot actually implements MOVE by internally doing a COPY + EXPUNGE.
That's promising that it should be doable. (yes, all I want is for the move to only occur once - duplicate messages is not a "move" at all) I'll forward your suggestions to the Thunderbird & Postbox teams.
In the meantime I'll continue to evaluate helpdesk systems and "collaborative inbox" products.
Greg. On 06/08/2014 2:11 am, "Timo Sirainen" <tss@iki.fi> wrote:
Note that MOVE isn't atomic even between moving within one user's folders. The MOVE RFC itself also doesn't say anything about it ever having to be atomic. Although if by atomicity you mean that you simply want to make sure that the same source mail can't be MOVEd twice, that would be doable with some work I think. Even for full conversations (without partial failures).
On 05 Aug 2014, at 13:19, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
Thanks Timo, and no, I can't (easily) change the code of the client.
I must say I am extremely disappointed that intra-account moves are not atomic. As far as I can tell, IMAP was designed to allow shared access, so in my opinion this operation should be atomic. Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)
Looks like a bloatware - sorry - helpdesk system - is what I will need to use.
Greg.
On 4 August 2014 22:44, Timo Sirainen <tss@iki.fi> wrote:
On 04 Aug 2014, at 14:12, Greg Sullivan <greg.sullivan@sullivang.net> wrote:
Yes, both client and server support IMAP MOVE, and both also support CONDSTORE.
I have tried both with and without CONDSTORE enabled in the client, with the same result.
With CONDSTORE I was thinking you could do it something like:
1 FETCH 1 (FLAGS MODSEQ)
- 1 FLAGS () MODSEQ 12345 2 STORE (UNCHANGEDSINCE 12345) 1 +FLAGS $AtomicMove 3 MOVE 1 elsewhere
If another client attempts the same, either 1 will return $AtomicMove in flags -> abort or 2 will fail with NO. But you should still handle failures if the client/connection dies between 2 and 3 or 3 fails for some reason.
But, of course if you can't change the client code to do this then it doesn't help.
I am very confident IMAP MOVE is actually being invoked, because intra-account moves occur extremely rapidly. (much faster than inter-account moves, which of course is a copy & delete)
Inter-account physically copies the data (FETCH + APPEND + EXPUNGE). Alternative to MOVE is COPY + EXPUNGE, which is just as fast as MOVE. Dovecot actually implements MOVE by internally doing a COPY + EXPUNGE.
On 05/08/2014 17:29, Greg Sullivan wrote:
That's promising that it should be doable. (yes, all I want is for the move to only occur once - duplicate messages is not a "move" at all) I'll forward your suggestions to the Thunderbird & Postbox teams.
In the meantime I'll continue to evaluate helpdesk systems and "collaborative inbox" products.
Greg.
I agree with the goal though. I have extremely simple needs for a helpdesk/CMS type system, and some plugins to Thunderbird would be quite satisfactory for my needs
Really I need:
- Enhanced addressbook, possibly reading via vcard from my main business system (bring in customer details and links to their orders on main system)
- Ability to force breaking and rejoin of specific message threading (because customers find an old invoice and hit reply to it to send us a support request + other customers who send you 15 emails (without hitting reply to trigger threading) to describe a single problem). Note I believe this requires rewriting the message, so it couldn't be atomic with current IMAP?
- Enhanced use of flags to mark whether thread needs further input or is closed
Nice to have would be: though?)
- Logging these state changes to somewhere else so that you can get statistics (can probably be done by polling the state of the IMAP server
- Atomic "locking" of threads so that we don't get two people answer something. Could be handled through use of flags perhaps?
Thunderbird is helpful in that in theory all one needs to do is write the above in javascript and drop appropriate display buttons on the email inbox, so even if some external lock manager is needed to arbitrate access, then this is no great problem. In practice I lack the time to work on this, but I'm vaguely interested to find out if there is a way to hire "plugin developers" for Thunderbird?
Good luck
Ed W
On -10.01.-28163 20:59, Greg Sullivan wrote:
I must say I am extremely disappointed that intra-account moves are not atomic. As far as I can tell, IMAP was designed to allow shared access, so in my opinion this operation should be atomic. Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)
How would it be of any use to the passive client that the *operation* is atomic when (as far as I can see, which admittedly mightn't be much) there is no way defined in the IMAP protocol to atomically *notify* it of said change?
IMAP IDLE, for example, may inform it that one message disappeared from mailbox X and one popped up in mailbox Y - not that these two are actually the same message, still have the same set of flags set, etc.. That's for the client to find out by specific requests - which already breaks the atomicity and allows for a race condition between clients.
Regards, J. Bern
*NEU* - NEC IT-Infrastruktur-Produkte im <http://www.linworks-shop.de/>: Server--Storage--Virtualisierung--Management SW--Passion for Performance Jochen Bern, Systemingenieur --- LINworks GmbH <http://www.LINworks.de/> Postfach 100121, 64201 Darmstadt | Robert-Koch-Str. 9, 64331 Weiterstadt PGP (1024D/4096g) FP = D18B 41B1 16C0 11BA 7F8C DCF7 E1D5 FAF4 444E 1C27 Tel. +49 6151 9067-231, Zentr. -0, Fax -299 - Amtsg. Darmstadt HRB 85202 Unternehmenssitz Weiterstadt, Geschäftsführer Metin Dogan, Oliver Michel
Jochen, I don't have any in-depth knowledge of the IMAP protocol. I'm just saying that given that IMAP is designed for concurrent access from multiple clients, I would have expected it to behave much better when more than one person attempts to move a message, that's all. I was gobsmacked when I discovered that duplicates could easily occur!
Quote from the IMAP wikipedia page: Internet Message Access Protocol (IMAP) is a protocol for e-mail retrieval and storage developed by Mark Crispin in 1986 at Stanford University as an alternative to POP. IMAP unlike POP, specifically allows multiple clients simultaneously connected to the same mailbox, and through flags stored on the server, different clients accessing the same mailbox at the same or different times can detect state changes made by other clients.
Regards, Greg.
On 6 August 2014 04:00, Jochen Bern <Jochen.Bern@linworks.de> wrote:
On -10.01.-28163 20:59, Greg Sullivan wrote:
I must say I am extremely disappointed that intra-account moves are not atomic. As far as I can tell, IMAP was designed to allow shared access, so in my opinion this operation should be atomic. Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)
How would it be of any use to the passive client that the *operation* is atomic when (as far as I can see, which admittedly mightn't be much) there is no way defined in the IMAP protocol to atomically *notify* it of said change?
IMAP IDLE, for example, may inform it that one message disappeared from mailbox X and one popped up in mailbox Y - not that these two are actually the same message, still have the same set of flags set, etc.. That's for the client to find out by specific requests - which already breaks the atomicity and allows for a race condition between clients.
Regards, J. Bern
*NEU* - NEC IT-Infrastruktur-Produkte im <http://www.linworks-shop.de/>: Server--Storage--Virtualisierung--Management SW--Passion for Performance Jochen Bern, Systemingenieur --- LINworks GmbH <http://www.LINworks.de/> Postfach 100121, 64201 Darmstadt | Robert-Koch-Str. 9, 64331 Weiterstadt PGP (1024D/4096g) FP = D18B 41B1 16C0 11BA 7F8C DCF7 E1D5 FAF4 444E 1C27 Tel. +49 6151 9067-231, Zentr. -0, Fax -299 - Amtsg. Darmstadt HRB 85202 Unternehmenssitz Weiterstadt, Geschäftsführer Metin Dogan, Oliver Michel
Am 06.08.2014 um 12:15 schrieb Greg Sullivan:
Jochen, I don't have any in-depth knowledge of the IMAP protocol. I'm just saying that given that IMAP is designed for concurrent access from multiple clients, I would have expected it to behave much better when more than one person attempts to move a message, that's all. I was gobsmacked when I discovered that duplicates could easily occur!
Quote from the IMAP wikipedia page: Internet Message Access Protocol (IMAP) is a protocol for e-mail retrieval and storage developed by Mark Crispin in 1986 at Stanford University as an alternative to POP. IMAP unlike POP, specifically allows multiple clients simultaneously connected to the same mailbox, and through flags stored on the server, different clients accessing the same mailbox at the same or different times can detect state changes made by other clients.
that's the theory
"can detect" -> when and how in case of concurrency things are not that easy because you have anyways network latency as part of the game
On 6 August 2014 04:00, Jochen Bern <Jochen.Bern@linworks.de> wrote:
On -10.01.-28163 20:59, Greg Sullivan wrote:
I must say I am extremely disappointed that intra-account moves are not atomic. As far as I can tell, IMAP was designed to allow shared access, so in my opinion this operation should be atomic. Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)
How would it be of any use to the passive client that the *operation* is atomic when (as far as I can see, which admittedly mightn't be much) there is no way defined in the IMAP protocol to atomically *notify* it of said change?
IMAP IDLE, for example, may inform it that one message disappeared from mailbox X and one popped up in mailbox Y - not that these two are actually the same message, still have the same set of flags set, etc.. That's for the client to find out by specific requests - which already breaks the atomicity and allows for a race condition between clients.
Quoting Greg Sullivan <greg.sullivan@sullivang.net>:
I was gobsmacked when I discovered that duplicates could easily occur!
Quote from the IMAP wikipedia page: Internet Message Access Protocol (IMAP) is a protocol for e-mail retrieval and storage developed by Mark Crispin in 1986 at Stanford University as an alternative to POP. IMAP unlike POP, specifically allows multiple clients simultaneously connected to the same mailbox, and through flags stored on the server, different clients accessing the same mailbox at the same or different times can detect state changes made by other clients.
Disagree. I'm not "gobsmacked" due to the fact that IMAP was designed
to ensure that no unintentional DESTRUCTIVE actions take place. I'd
be "gobsmacked" if it was the opposite - preferring duplicate
prevention over message loss.
What's worse:
- Concurrent users where one accidentally deletes (i.e. EXPUNGE) a
message due to inconsistent mailbox state between the two sessions. - Concurrent users where one accidentally creates a duplicate message
during a move operation by the other user.
1 is worse. (Although 1 becomes mostly irrelevant once UIDs are used
and UIDPLUS is available.) But 2, in real-world practice, simply
doesn't happen enough to make it a critical issue.
michael
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Quoting Greg Sullivan <greg.sullivan@sullivang.net>:
I was gobsmacked when I discovered that duplicates could easily occur!
"move" is not atomic in a filesystem either. Under circumstances it has to be implemented by "cp && rm". Unless you protect the move's source by locking - as Timo suggested with the message at Tue, 5 Aug 2014 21:19:46 - the same duplication can occur.
In your reply you've wrote: "Heaven FORBID that I should ask for entire conversation moves to be atomic as well. (which is really what I want)" Actually you want to use database-like transactions for IMAP commands or sessions.
Actually, what should happen _during_ the MOVE, if another client queries for the source message or launches commands on it? Now, for a short period of time the copy takes the message is available at the source location only, then it disappears if the move was successful. IMHO, you could write a plugin that monitors or wraps "MOVE" commands and denies second simultaneous move attempts. In a way, that plugin would do the locking Timo suggested as a server-side implementation. How many simultaneous MOVE do you expect? Maybe you could lock the source mailbox wholly during a move.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQEVAwUBU+Mp13z1H7kL/d9rAQLiAAgAqryldRY7xyLxhULj+nmABbspsXsauip5 KNGYx1lPmTh1JqhvGZ/1mxn80ImidXa2U3poa8ltHPthjHW1NvqsDCMFXTzpUC36 l6zIP7VOEwMSLAq0/KmsbvJIh13h9IiZRzG3zZDO7AaDyxru171qctuEwOKO3EWp W4kicYne3fAj/FXEI4+Dnm7a6cJabIRL2DTzkU33JC0S+/seq/8ZXj9XO99ocwJV 86xjdtaD90H3akIA1G7/Q1Ielw5tEPGMD7Ye4RF+ra1DYOsj9RYXisVqNhYLDh84 qUsBPXm7Ap4rgp07iar10ex3UYQbGEq0VR9bxUxna4IMS67EtBwLpQ== =7Rkz -----END PGP SIGNATURE-----
participants (7)
-
Ed W
-
Greg Sullivan
-
Jochen Bern
-
Michael M Slusarz
-
Reindl Harald
-
Steffen Kaiser
-
Timo Sirainen