[Dovecot] dsync - one or two ways?
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations. Some things it can be used for:
- Initially transfer a mailbox to another server via SSH
- A faster sync done to an existing mailbox, sending only changes
- A superfast sync based on modification sequences.
- Source and destination mailboxes can use different formats (convert-tool will be history)
dsync can handle all kinds of conflicts in mailboxes, handle mailbox deletions, renames, etc. So it's safe to sync even if both source and destination mailboxes have had all kinds of changes.
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
Timo Sirainen wrote:
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations. Some things it can be used for:
- Initially transfer a mailbox to another server via SSH
- A faster sync done to an existing mailbox, sending only changes
- A superfast sync based on modification sequences.
- Source and destination mailboxes can use different formats (convert-tool will be history)
dsync can handle all kinds of conflicts in mailboxes, handle mailbox deletions, renames, etc. So it's safe to sync even if both source and destination mailboxes have had all kinds of changes.
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
Yes; moving mail from an old source (for example, my old home dovecot server which still occasionally gets mail, but otherwise I don't really use it). I wouldn't want all my current mail sent to it for this purpose.
~Seth
On Fri, 2009-07-17 at 11:59 -0700, Seth Mattinen wrote:
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
Yes; moving mail from an old source (for example, my old home dovecot server which still occasionally gets mail, but otherwise I don't really use it). I wouldn't want all my current mail sent to it for this purpose.
dsync probably wouldn't be such a good idea for this use case anyway. dsync assumes that an IMAP client has accessed the mailboxes in both source and destination, so if there are any UID conflicts (and in the case you describe there would be UID conflicts all the time) it gives new UIDs for the conflicting messages.
Timo Sirainen wrote:
On Fri, 2009-07-17 at 11:59 -0700, Seth Mattinen wrote:
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
Yes; moving mail from an old source (for example, my old home dovecot server which still occasionally gets mail, but otherwise I don't really use it). I wouldn't want all my current mail sent to it for this purpose.
dsync probably wouldn't be such a good idea for this use case anyway. dsync assumes that an IMAP client has accessed the mailboxes in both source and destination, so if there are any UID conflicts (and in the case you describe there would be UID conflicts all the time) it gives new UIDs for the conflicting messages.
So is this targeted as a kind of replication solution then?
~Seth
On Fri, 2009-07-17 at 12:06 -0700, Seth Mattinen wrote:
So is this targeted as a kind of replication solution then?
Either continuous replication or just one-off user moves/conversions without locking the mailbox and without losing changes done during the move/conversion.
I think even using dsync for backups should be considered as two-way replication. If there really are some new mails in your backup server that aren't in the primary server, it most likely means you just lost some mails on your primary server and you need to get them back.
Sure if you can think of some other uses for it I'd like to hear. But I if you were just planning on copying new mails from a server with completely different mails, I don't really think dsync should be used for that. Wouldn't it be easier to just have the mails forwarded directly there on SMTP level?
On 7/17/2009, Timo Sirainen (tss@iki.fi) wrote:
Either continuous replication or just one-off user moves/conversions without locking the mailbox and without losing changes done during the move/conversion.
This just sounds great, and yes, I can definitely see a need for both.
I think even using dsync for backups should be considered as two-way replication. If there really are some new mails in your backup server that aren't in the primary server, it most likely means you just lost some mails on your primary server and you need to get them back.
Or you just deleted them and *don't* want them back... ;)
--
Best regards,
Charles
On Fri, 2009-07-17 at 15:57 -0400, Charles Marcus wrote:
I think even using dsync for backups should be considered as two-way replication. If there really are some new mails in your backup server that aren't in the primary server, it most likely means you just lost some mails on your primary server and you need to get them back.
Or you just deleted them and *don't* want them back... ;)
Ah, but dsync catches those too!
If a message exists in one side but doesn't exist on the other side, dsync asks for a list of recently expunged messages. If it finds the message from there with same IMAP UID and same GUID, it's expunged from the other side too.
Basically the same will work with mailbox deletions and unsubscribes too. It'll just need to store a list of mailbox GUIDs there were deleted/unsubscribed.
On 7/17/2009 3:59 PM, Timo Sirainen wrote:
On Fri, 2009-07-17 at 15:57 -0400, Charles Marcus wrote:
I think even using dsync for backups should be considered as two-way replication. If there really are some new mails in your backup server that aren't in the primary server, it most likely means you just lost some mails on your primary server and you need to get them back. Or you just deleted them and *don't* want them back... ;)
Ah, but dsync catches those too!
If a message exists in one side but doesn't exist on the other side, dsync asks for a list of recently expunged messages. If it finds the message from there with same IMAP UID and same GUID, it's expunged from the other side too.
Basically the same will work with mailbox deletions and unsubscribes too. It'll just need to store a list of mailbox GUIDs there were deleted/unsubscribed.
Somehow I knew you would have already thought of that...
So, when will 2.0 be in any kind of semi-usable state (by non-programmer types)? ;)
--
Best regards,
Charles
On 7/17/2009 4:34 PM, Timo Sirainen wrote:
So, when will 2.0 be in any kind of semi-usable state (by non-programmer types)? ;)
Dunno. And I'm still hoping to break its index files code completely before stabilizing it :)
You're obviously having way too much fun... time for a break... ;)
Seriously though...
Will this be configurable on a per mailbox basis? Or is it intended to be server <-> server / all or nothing?
--
Best regards,
Charles
On Fri, 2009-07-17 at 17:01 -0400, Charles Marcus wrote:
Will this be configurable on a per mailbox basis? Or is it intended to be server <-> server / all or nothing?
You'll need to call it one user at a time in the proper environment (UNIX UID and GID set up right). How to do that and multiple users is your problem. :)
Here's an example command line for it:
dsync -e 'ssh hostname dsync dest' source
Or conversion to dbox (see how v2.0 has Postfix-like -o parameter for overriding settings):
dsync -e 'dsync -o mail_location=dbox:~/dbox dest' source
If only two-way sync is supported those source/dest parameters will go away.
Timo Sirainen wrote:
On Fri, 2009-07-17 at 12:06 -0700, Seth Mattinen wrote:
So is this targeted as a kind of replication solution then?
Either continuous replication or just one-off user moves/conversions without locking the mailbox and without losing changes done during the move/conversion.
Cool. I just built a master/slave redundant dovecot last night with DRBD and heartbeat and this sounds easier. ;)
~Seth
On Jul 17, 2009, at 9:16 PM, Seth Mattinen wrote:
Timo Sirainen wrote:
So is this targeted as a kind of replication solution then? Either continuous replication or just one-off user moves/conversions without locking the mailbox and without losing changes done during
On Fri, 2009-07-17 at 12:06 -0700, Seth Mattinen wrote: the move/conversion.
Cool. I just built a master/slave redundant dovecot last night with
DRBD and heartbeat and this sounds easier. ;)
Well, by "continuous" I didn't actually really mean "continuous" but
"periodic" or something :) Meaning you'd have to run dsync every time
after something changes and separately for each user. So you could do
a bit laggy replication using it, but it's not really the permanent or
the most efficient solution for that.
On Jul 17, 2009, at 2:47 PM, Timo Sirainen wrote:
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox
in two locations. Some things it can be used for:
- Initially transfer a mailbox to another server via SSH
- A faster sync done to an existing mailbox, sending only changes
- A superfast sync based on modification sequences.
- Source and destination mailboxes can use different formats (convert-tool will be history)
dsync can handle all kinds of conflicts in mailboxes, handle mailbox deletions, renames, etc. So it's safe to sync even if both source and destination mailboxes have had all kinds of changes.
Wow, that will be very handy! Thanks!!
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I
think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
I think a two-way sync would be useful functionality to have,
perhaps tied to a command line option, at least for the sake of
completeness. It would also open up possibilities for having an
"offline" IMAP server, which might be useful for mobile workgroups
and such that may not always have full connectivity.
-Dave-- Dave McGuire Port Charlotte, FL
Timo Sirainen tss@iki.fi writes:
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations. Some things it can be used for:
- Initially transfer a mailbox to another server via SSH
- A faster sync done to an existing mailbox, sending only changes
- A superfast sync based on modification sequences.
- Source and destination mailboxes can use different formats (convert-tool will be history)
dsync can handle all kinds of conflicts in mailboxes, handle mailbox deletions, renames, etc. So it's safe to sync even if both source and destination mailboxes have had all kinds of changes.
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
This sounds really awesome! How testable is it? My setup would be: the server dovecot. I read mail there with Gnus.
- one server dovecot that is online all the time, filtering email to several
mailboxes with sieve scripts
- the server dovecot would receive changes through a webmail instance
- a dovecot running on my laptop that I would like to be in sync with
Theres only a one user(me) that would have this setup, but there are a handful of other users as well. I have a couple of hundred mailboxes to sync. Can all mailboxes be synced automatically, or do I need to make a list of them for dsync?
-- Joakim Verona
On Fri, 2009-07-17 at 22:31 +0200, joakim@verona.se wrote:
This sounds really awesome! How testable is it?
By "testable" do you mean "is it working now"? It should work except for mailbox deletions and renames. But I haven't really tested it, just ran unit tests. :) And it pretty much requires using Dovecot v2.0 for IMAP so that it can handle expunges properly. And I wouldn't really recommend doing that yet..
My setup would be: the server dovecot. I read mail there with Gnus.
- one server dovecot that is online all the time, filtering email to several mailboxes with sieve scripts
- the server dovecot would receive changes through a webmail instance
- a dovecot running on my laptop that I would like to be in sync with
Yeah, should work with that setup.
Theres only a one user(me) that would have this setup, but there are a handful of other users as well. I have a couple of hundred mailboxes to sync. Can all mailboxes be synced automatically, or do I need to make a list of them for dsync?
User's all mailboxes are synced automatically.
Timo Sirainen tss@iki.fi writes:
On Fri, 2009-07-17 at 22:31 +0200, joakim@verona.se wrote:
This sounds really awesome! How testable is it?
By "testable" do you mean "is it working now"? It should work except for mailbox deletions and renames. But I haven't really tested it, just ran unit tests. :) And it pretty much requires using Dovecot v2.0 for IMAP so that it can handle expunges properly. And I wouldn't really recommend doing that yet..
Would this setup work then?
- dovecot 1.2 on master and slave machines
- dsync on the slave machine, 2 way sync
- propagate expunges by some other means, like removing maildir mail files with unison or similar?
How would I handle mailbox addition, renaming and deletions?
My setup would be: the server dovecot. I read mail there with Gnus.
- one server dovecot that is online all the time, filtering email to several mailboxes with sieve scripts
- the server dovecot would receive changes through a webmail instance
- a dovecot running on my laptop that I would like to be in sync with
Yeah, should work with that setup.
Theres only a one user(me) that would have this setup, but there are a handful of other users as well. I have a couple of hundred mailboxes to sync. Can all mailboxes be synced automatically, or do I need to make a list of them for dsync?
User's all mailboxes are synced automatically.
-- Joakim Verona
On Sat, 2009-07-18 at 01:43 +0200, joakim@verona.se wrote:
Timo Sirainen tss@iki.fi writes:
On Fri, 2009-07-17 at 22:31 +0200, joakim@verona.se wrote:
This sounds really awesome! How testable is it?
By "testable" do you mean "is it working now"? It should work except for mailbox deletions and renames. But I haven't really tested it, just ran unit tests. :) And it pretty much requires using Dovecot v2.0 for IMAP so that it can handle expunges properly. And I wouldn't really recommend doing that yet..
Would this setup work then?
- dovecot 1.2 on master and slave machines
- dsync on the slave machine, 2 way sync
- propagate expunges by some other means, like removing maildir mail files with unison or similar?
Actually there shouldn't be problems with expunges after all. Conflicts just are detected less reliably. For example if:
Server 1 saves message with UID=123 Server 2 saves a different message with UID=123 Server 1 expunges messages UID=123
dsync is run to move changes from server 2 to server 1. It notices that "oh, UID=123 has been expunged from server 1 already" and server 2's UID=123 message isn't copied there. Or if you're syncing changes from 1 to 2, it would expunge the UID=123 from server 2.
If server 1 had been running Dovecot v2.0, the expunge record would have message's GUID stored and dsync would notice that there was a GUID conflict and copy the server 2's message to server 1 as UID=124.
How would I handle mailbox addition, renaming and deletions?
The reason why dsync can't do that without Dovecot v2.0 is because it can't be done reliably without storing extra information.. (And I haven't yet even implemented that to v2.0.) So the answer is: Not easily.
Timo Sirainen tss@iki.fi writes:
On Sat, 2009-07-18 at 01:43 +0200, joakim@verona.se wrote:
Timo Sirainen tss@iki.fi writes:
On Fri, 2009-07-17 at 22:31 +0200, joakim@verona.se wrote:
This sounds really awesome! How testable is it?
By "testable" do you mean "is it working now"? It should work except for mailbox deletions and renames. But I haven't really tested it, just ran unit tests. :) And it pretty much requires using Dovecot v2.0 for IMAP so that it can handle expunges properly. And I wouldn't really recommend doing that yet..
Would this setup work then?
- dovecot 1.2 on master and slave machines
- dsync on the slave machine, 2 way sync
- propagate expunges by some other means, like removing maildir mail files with unison or similar?
Actually there shouldn't be problems with expunges after all. Conflicts just are detected less reliably. For example if:
Server 1 saves message with UID=123 Server 2 saves a different message with UID=123 Server 1 expunges messages UID=123
dsync is run to move changes from server 2 to server 1. It notices that "oh, UID=123 has been expunged from server 1 already" and server 2's UID=123 message isn't copied there. Or if you're syncing changes from 1 to 2, it would expunge the UID=123 from server 2.
If server 1 had been running Dovecot v2.0, the expunge record would have message's GUID stored and dsync would notice that there was a GUID conflict and copy the server 2's message to server 1 as UID=124.
In my case mail would only arrive on the master. If I interpret you correctly this case will be very simple for dsync and will work already.
How would I handle mailbox addition, renaming and deletions?
The reason why dsync can't do that without Dovecot v2.0 is because it can't be done reliably without storing extra information.. (And I haven't yet even implemented that to v2.0.) So the answer is: Not easily.
Will addition work automatically? Can renaming and deletion be made by renaming/deleting maildir folders manually?
I guess I can do the setup and test it myself :)
-- Joakim Verona
Not really.. Another future thing might, though. :)
On Sat, 2009-07-18 at 00:20 +0200, Christian Felsing wrote:
Does this mean true multi master operation ? That would make configs with load balancers much easier :-)
Timo Sirainen schrieb:
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations. Some things it can be used for:
Timo Sirainen wrote:
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations.
I am very, very pleased to hear that you are working on this! Just at this moment, I am busy replacing a Dovecot IMAP based server with one with more powerful hardware, and until the tests are complete I want to operate both machines simultaneosly, so synchronisation is a big issue.
Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync?
If you want a decision "one-way XOR two-way", I'd definitely prefer the two-way sync option. If you can offer both sync types, all the better.
BTW, if you have not yet planned to include them, I suggest exception lists. The default would be to sync all messages in all folders for a given list of users, right? It would be nice to be able to configure dsync to achieve the following, too:
"Don't sync folders Trash, Junk, *.Newsletter, Boring.*" "Don't sync messages of a size greater than 2 MB"
What do you think?
-R
On Jul 18, 2009, at 2:20 AM, Ralph Seichter wrote:
BTW, if you have not yet planned to include them, I suggest exception lists. The default would be to sync all messages in all folders for a given list of users, right? It would be nice to be able to configure dsync to achieve the following, too:
"Don't sync folders Trash, Junk, *.Newsletter, Boring.*"
This wouldn't be difficult to add.
"Don't sync messages of a size greater than 2 MB"
Why would you want that? Also if you later then wanted to sync it
anyway, it would have to get a new IMAP UID since messages can't be
inserted, and if client had already downloaded that it would have to
do it again. So I don't think this is such a good idea.
Timo Sirainen wrote:
"Don't sync folders Trash, Junk, *.Newsletter, Boring.*"
This wouldn't be difficult to add.
That's good to hear.
"Don't sync messages of a size greater than 2 MB"
Why would you want that?
To save space? I was thinking about which message properties might be useful for filtering, and message size bubbled up first. Lotus Notes/ Domino allows replicating databases with a maximum document size filter, which is useful for low-bandwidth connections or if you can't afford a volume flat rate for your mobile phone.
Also if you later then wanted to sync it anyway, it would have to get a new IMAP UID since messages can't be inserted, and if client had already downloaded that it would have to do it again. So I don't think this is such a good idea.
Well, I wouldn't want a feature to cause trouble. It was just an idea. The folder based filtering however is important to me, so I am glad that you consider it a feasible suggestion.
-R
participants (7)
-
Charles Marcus
-
Christian Felsing
-
Dave McGuire
-
joakim＠verona.se
-
Ralph Seichter
-
Seth Mattinen
-
Timo Sirainen