On 2/21/2013 8:43 AM, Odhiambo Washington wrote:

I've been running 2.2beta1 in production since it got out and I haven't had any issues. However, my setup is simply IMAP/POP3 without any complicated configurations. This is on FreeBSD 8.3-STABLE with about 100 users. I've just updated to 2.2b2...

The same applies to me except it's CentOS 5 and I have only a half dozen POP3 users and a couple of IMAP users with a total of about 2 dozen mailboxes. No problems with 2.2beta2 so far.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 21.2.2013, at 22.12, Michael Grimm <trashcan@odo.in-berlin.de> wrote:

I am running v2.2beta1 for almost a week now (handful of users, ~1000 mails/day). And yes, I can confirm that replication works ...

...

Please test and report any bugs found!

... there is only a "feature request" left:

I did run a lot of stress tests as mentioned before (simultaneously injected local mail).

Whenever I do run those tests with a delay of 1 second between every injection, I do observe, that not all mails injected become visible in my MUAs (Mail.app and Roundcube), immediately. All "get new mail" functionality or MUA restarts fails to fetch those missing messages from both servers.

But, if I do restart any one of both dovecot servers involved, those remaining missing messages are fetched instantaneously. It seems to me, as if those missing messages were kept in the "replicator queue" and not delivered to the partner server, and as if that queue became flushed during restart. After restart all injected and replicated messages are accessible by the MUAs.

Possibly a bug, see if it still happens with beta2.

I never waited longer then 10 minutes before restarting dovecot, thus I do not know if I didn't wait long enough. But I can say that an additionally injected local mail shows up in both MUAs but doesn't become replicated.

Thus, if I am not mistaken that something like a "flushing" really takes place, I wonder if there is a "doveadm flush replicator-queue" functionality is available already (that I might have over-seen), and if not, would that be a big deal to implement?

Well, you can sync a user with e.g.: doveadm sync -d -l 30 -u user@domain

or with -A parameter to sync everyone. And with -f parameter to do a full sync if needed. The -l 30 parameter adds locking so two dsyncs won't run at the same time for the user.

Hello Michael,

Can you post to the list your working configurations for dovecot?

I've been fighting with the dsync replication a while ago and was super buggy so I've put that project on ice.

If now it finally works I would like to get it moving again.

Best, Andrei

On 21.02.2013, at 21:17, Timo Sirainen <tss@iki.fi> wrote:

...

On 21.2.2013, at 22.12, Michael Grimm <trashcan@odo.in-berlin.de> wrote:

...

...

But, if I do restart any one of both dovecot servers involved, those remaining missing messages are fetched instantaneously.

Possibly a bug, see if it still happens with beta2.

Sorry, I did forget to mention that this is happening with v2.2.beta2 (3fb9a8bc35aa) as well.

...

...

Thus, if I am not mistaken that something like a "flushing" really takes place, I wonder if there is a "doveadm flush replicator-queue" functionality is available already (that I might have over-seen), and if not, would that be a big deal to implement?

Well, you can sync a user with e.g.: doveadm sync -d -l 30 -u user@domain or with -A parameter to sync everyone. And with -f parameter to do a full sync if needed.

I needed to add that "-f" for full sync, and that "flushed" the queue. Thanks for pointing me to that.

I do have a "replication_full_sync_interval = 1 hours" set. Does that setting equals to the same like running a "doveadm dsync ..." every other hour?

Regards, Michael !DSPAM:512687b9301841449136444!

Thank you very much Michael.

I'll set it up again, because I think that initially I was not using any aggregator service. My setup is a little atypical as I have servers spread-out on 2 (soon 3) continents that are connected via slow uplinks.

Hopefully the v2.2 is more stable and less buggy for the synch ;)

Best regards, Andrei

Hi --

On 2013-02-21 Michescu Andrei wrote:

...

Can you post to the list your working configurations for dovecot?

(This is based on http://dovecot.org/list/dovecot/2012-March/064513.html)

My design:

     single user "vmail" to run dsync over ssh
     (one may use root instead)

Thus, my prerequisites are:

     create "vmail" user accounts at both servers (example: mx1 and

mx2) exchange ssh-keys for ssh authentication between both servers involved

My relevant parts from dovecot.conf, identical for both servers:

     ## --- DSYNC REPLICATION

---

     # ssh command line used in dsync replication
     # added:
     #       -p xxx                  (ssh port)
     #       -o mail_plugins=        (omit mail_log plugins for

dsync) # dsync_remote_cmd = ssh -p 44488 -l%{login} %{host} doveadm -omail_plugins= dsync-server -u%u -n%{namespace}

     # aggregator, replicator, doveadm, and config needed, and

dsync_remote_cmd (see above) # service aggregator { # give enough permissions for mail processes # fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } } service replicator { # start replication at startup # process_min_avail = 1 } service doveadm { # if you're using a single virtual user, set this to start ssh as vmail (not root) # user = vmail } service config { # needed to grant access to /var/run/dovecot/config for service doveadm # unix_listener config { user = vmail } }

The following part is for server mx1, only:

     # dsync replication plugin
     #
     plugin {
             # this host replicates to remote host
             #
             mail_replica = remote:vmail@mx2.FQDN

             # run full synchronization mode every other hour
             #
             replication_full_sync_interval = 1 hours
     }

The following part is for server mx2, only:

     # dsync replication plugin
     #
     plugin {
             # this host replicates to remote host
             #
             mail_replica = remote:vmail@mx1.FQDN

             # run full synchronization mode every other hour
             #
             replication_full_sync_interval = 1 hours
     }

HTH, Michael

!DSPAM:51271e3f42781904211018!

On 2013-02-25 15:58, Timo Sirainen wrote:

On 21.2.2013, at 22.12, Michael Grimm <trashcan@odo.in-berlin.de> wrote:

...

Whenever I do run those tests with a delay of 1 second between every injection, I do observe, that not all mails injected become visible in my MUAs (Mail.app and Roundcube), immediately. All "get new mail" functionality or MUA restarts fails to fetch those missing messages from both servers.

So .. what exactly do you mean by this? That in both servers you run a script that delivers a mail once per second to the same user?

Yes. In my tests I do inject 200 messages at every server simultaneously with a delay of 1 second.

And at some point the replication just stops replicating those mails to the other server?

Yes. I would expect 400 messages at every inbox, but normally I do end up with around 270 in an inbox, and both inboxes do show slightly different numbers (e.g. 245 and 297). (Looks like stopping.)

I can see how that would happen with regular "doveadm sync" command, but replicator uses stateful syncing where that shouldn't be possible.

I did repeat this test appr. 10 times, always the same. Restarting both dovecot servers or running "doveadm dsync -d -l 30 -u test -f" leads to an instantaneous appearence of all 400 messages in every inbox.

What is puzzeling me most: If I do inject both 200 messages *without* any delay, I cannot see this behavior. All 400 messages appear without delay.

Regards, Michael

On 2013-02-26 10:55, Timo Sirainen wrote:

On 25.2.2013, at 17.38, Michael Grimm <trashcan@odo.in-berlin.de> wrote:

...

Yes. I would expect 400 messages at every inbox, but normally I do end up with around 270 in an inbox, and both inboxes do show slightly different numbers (e.g. 245 and 297). (Looks like stopping.)

I can't reproduce this. Some interesting questions:

• If you include hostname+counter in the message, what do the mailboxes look like in the different sides? Did they skip over some numbers or did they both stop at some specific remote counter and continue the local counters until the end?

• Is it even trying to run doveadm sync commands at the end? (e.g. make dsync_remote_cmd execute some wrapper script that logs something)

• If the doveadm syncs continue, try saving rawlogs from them to see what they're doing (-r /tmp/rawlog parameter to doveadm dsync-server).

I will investigate this further, but that will take some time.

...

I did repeat this test appr. 10 times, always the same. Restarting both dovecot servers or running "doveadm dsync -d -l 30 -u test -f" leads to an instantaneous appearence of all 400 messages in every inbox.

It probably works even without -f parameter?

In the meantime I can confirm that it will work without that parameter as well.

Thanks and regards, Michael

On 26.2.2013, at 22.20, Michael Grimm <trashcan@odo.in-berlin.de> wrote:

BUT: It look as if I haven't waited long enough for replication to become finished, sorry :-(

Actually, while going through all those files and writing this mail, all missing messages appeared in my MUA, and I do find in both maillogs:

@mx1: | dovecot: dsync-local(test): Error: dsync(vmail@mx2.TLD): I/O has stalled, no activity for 600 seconds | dovecot: dsync-local(test): Error: Remote command process isn't dying, killing it

@mx2: | dovecot: dsync-local(test): Error: dsync(vmail@mx1.TLD): I/O has stalled, no activity for 600 seconds | dovecot: dsync-local(test): Error: Remote command process isn't dying, killing it

Ah, this explains the behavior. I had hoped that with the redesign there was practically no way to cause this kind of I/O stalling.

Do you have any idea what I should do next?

Send me the last rawlogs just before it stalls, from both servers? They should show what each side thought they sent to the other, and what the other really received, and from that I can hopefully find out more easily why it stalled.