[Dovecot] Replication via sneakernet
Is there a way to get a running transaction log from a Dovecot instance to synchronize (and I use that word very, very loosely) multiple instances that don't share a network?
Why I would want to do such a thing: I have a couple way stations along a long, winding road through very hilly country, and at each of these stations, I have a few people (and many, many more at one end of the road). Which people are where varies from hour to hour (but they're all members of a known, larger, group of people with accounts). It would be helpful to get communication (however limited) along this road (and establishing direct communication, either through radio or landline, is not feasible within the budget constraints I have), and it is possible to store a complete copy of the mail spool to a small box at each station (the total mail spool over the period I care about is in the tens of megs).
There are trams shuttling back and forth along this road (stopping at each station), and adding a small box (such as a weatherproofed Raspberry Pi with a wifi dongle) to transport files up and down the road is pretty simple. Doing incremental copies of the mail spool is also simple, keeping track of files that get moved or deleted is a bit harder (but still not that bad), but I have no idea how hard it is to track what messages have been seen, especially for the shared folders (which we use instead of mailing lists).
I realize this is not a "normal" use of Dovecot, but given this, does anyone who knows the software (and how it keeps track of messages) better have suggestions on how to approach the problem?
On 28.11.2013, at 5.17, David Bishop dovecot@dpe.lusars.net wrote:
Is there a way to get a running transaction log from a Dovecot instance to synchronize (and I use that word very, very loosely) multiple instances that don't share a network?
dsync replication was designed in a way that would allow transferring the state over e.g. USB sticks, although it’s missing some code to make it actually possible.
There are trams shuttling back and forth along this road (stopping at each station), and adding a small box (such as a weatherproofed Raspberry Pi with a wifi dongle) to transport files up and down the road is pretty simple.
But if you do it this way and you can keep a full copy of the shared mail storage on your Raspberry, that would be possible already with dsync I think. dsync supports quick incremental updates by keeping track of the previous state between the servers. This state is saved in a file, so you could keep a different state for each different dsynced server.
David Bishop dovecot@dpe.lusars.net writes:
Is there a way to get a running transaction log from a Dovecot instance to synchronize (and I use that word very, very loosely) multiple instances that don't share a network?
I think what you want is not synchronize the state of one mail server across multiple computers, but have mail transport from one system to another via sneakernet. In other words, you want UUCP. It might be ancient, but it should do exactly what you want.
In short, you'd configure your MTA to deliver mails to your sneakernet via pipe to uucico (not sure if I got the command 100% right), and configure your individual boxes to push and pull mail using uusomething whenever your moving Raspberry comes in range.
Hope that helps, Nikolaus
-- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
»Time flies like an arrow, fruit flies like a Banana.«On Fri, Dec 6, 2013 at 10:28 PM, Nikolaus Rath Nikolaus@rath.org wrote:
David Bishop dovecot@dpe.lusars.net writes:
Is there a way to get a running transaction log from a Dovecot instance to synchronize (and I use that word very, very loosely) multiple instances that don't share a network?
I think what you want is not synchronize the state of one mail server
across multiple computers, but have mail transport from one system to another via sneakernet. In other words, you want UUCP. It might be ancient, but it should do exactly what you want.
I've looked at UUCP, and while it solves a related problem, it doesn't seem like it solves my problem (which is why I originally asked about copying files directly out of the spool). It is, I think, possible to force UUCP to do what I want, but dsync seems an easier solution. (Sadly, it'll be a bit before I have time to work up a testbed.)
As an example: If Bob-from-maintenance is working his way up the line, he may be emailing updates to the maintenance list (stuff like "this needs replacement but I don't have the part" and "This problem has been sorted") before hopping a tram to the next station. At the next station (please note: different server with its own mail spool pretending to be the same server), the local IMAP server should look (to Bob, and Bob's phone) exactly the same (down to message IDs, because otherwise, his phone is going to get confused about the locally-cached email). If an email someone sent in response to one of his emails catches up with Bob ("I have the part and will sort it -- Ann-from-engineering" or "new problem reported", for instance), it should show up as a new message, but his original message should show up as having been read.
If I do this with dsync (and correctly understand dsync), this works the way (I think) it should. If I do this with UUCP, every station is going to have its own copy of each message, but not copies of read flags (and no way of conveying that a message has been read, or, more importantly, deleted, which becomes a problem when some mail recipients never go all the way to the end of the line).
On 06 Dec 2013, at 21:09 , David Bishop dovecot@dpe.lusars.net wrote:
If I do this with UUCP, every station is going to have its own copy of each message, but not copies of read flags (and no way of conveying that a message has been read, or, more importantly, deleted, which becomes a problem when some mail recipients never go all the way to the end of the line).
Worse than that, if you are using something sensible like Maildir, you will end up with multiple copies of the same email. You might have the original, the read one, the forwarded one, the replied to one (since maildir uses the filename for metadata flags).
-- Humans are always slightly lost. It's a basic characteristic. It explains a lot about them.
David Bishop dovecot@dpe.lusars.net writes:
On Fri, Dec 6, 2013 at 10:28 PM, Nikolaus Rath Nikolaus@rath.org wrote:
David Bishop dovecot@dpe.lusars.net writes:
Is there a way to get a running transaction log from a Dovecot instance to synchronize (and I use that word very, very loosely) multiple instances that don't share a network?
I think what you want is not synchronize the state of one mail server
across multiple computers, but have mail transport from one system to another via sneakernet. In other words, you want UUCP. It might be ancient, but it should do exactly what you want.
I've looked at UUCP, and while it solves a related problem, it doesn't seem like it solves my problem (which is why I originally asked about copying files directly out of the spool). It is, I think, possible to force UUCP to do what I want, but dsync seems an easier solution. (Sadly, it'll be a bit before I have time to work up a testbed.)
As an example: If Bob-from-maintenance is working his way up the line, he may be emailing updates to the maintenance list (stuff like "this needs replacement but I don't have the part" and "This problem has been sorted") before hopping a tram to the next station. At the next station (please note: different server with its own mail spool pretending to be the same server), the local IMAP server should look (to Bob, and Bob's phone) exactly the same (down to message IDs, because otherwise, his phone is going to get confused about the locally-cached email).
Ah, well, your original email never mentioned phones :-). In that case UUCP is not going to work well.
I was assuming that Bob (and everyone else) has a laptop. In that case, he would carry his mail spool around with him, and send and receive emails via UUCP whenever he is in the vicinity of another system.
Best, Nikolaus
-- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
»Time flies like an arrow, fruit flies like a Banana.«On Sat, Dec 7, 2013 at 1:49 PM, Nikolaus Rath Nikolaus@rath.org wrote:
Ah, well, your original email never mentioned phones :-). In that case UUCP is not going to work well.
I was assuming that Bob (and everyone else) has a laptop. In that case, he would carry his mail spool around with him, and send and receive emails via UUCP whenever he is in the vicinity of another system.
Bob is already lugging almost 10kg of kit and parts via tram. I can only presume that any proposal to add to that (especially something comparatively delicate) would be met with rather less enthusiasm than you might think (it would be replacing a pad of post-it notes and a biro; messages currently get relayed via tram drivers, either verbally or via written note). His phone already gets used for games/music/etc, so nothing new would be added to what Bob must carry.
Also, even if I were to add a lappy to Bob's cart, I would still want to be able to recreate the mail spool (should Bob drop it off the tram and onto rocks while attempting to email on the move), complete with "read" and "deleted" flags, and UUCP does not seem to be the simple way of doing this.
participants (4)
-
David Bishop
-
LuKreme
-
Nikolaus Rath
-
Timo Sirainen