March 2015 - dovecot - dovecot.org

Problem with replication ("Fatal: -N parameter requires syncing with remote host")
by Vanja 16 Mar '15

16 Mar '15

I have been fighting, for 2 days, with a Dovecot setup and I now officially need help :) Basically, I am trying to setup replication between 2 identical Dovecot servers, on different physical servers (different hostnames/etc, of course), and although I've managed to solve many problems I can't overcome the final hurdle. When a mail comes in to one of the servers, replication over TCP should kick in and mail should be replicated, but instead I get this in the mail log: ------------------------------------------------------------------------------------------------- dovecot2 dovecot: doveadm(user1(a)test1.int): Fatal: -N parameter requires syncing with remote host ------------------------------------------------------------------------------------------------- Mail is delivered to dovecot2 server, Postfix delivers it via LMTP, and everything seems to work ok. However, I can't figure out the cause for the error that I am seeing. I have looked into source code and found where this error is triggered (line 834 in doveadm/dsync/doveadm-dsync.c, inside Dovecot 2.2.9 source code), but I am afraid that I don't understand the context (ctx->run_type == DSYNC_RUN_TYPE_LOCAL) and it would take me a lot of time to debug the whole thing from the scratch. I've never looked into Dovecot internals before, so it's all a mystery to me at this point. I've tried stracing the process but the flow does not make sense to me, I can't understand what exactly happens before error is displayed. Also, there is no traffic to port 12345 on other node, so execution fails before syncing even starts (and error message is not related with remote node config/setup/etc). Triggering replication manually, from command line, works as expected: # doveadm sync -A tcp:192.168.1.100 (at this point user1's mail folder is properly updated on remote server) I've tried searching around, but it appears that noone else has this problem(or isn't complaining about it :). I am not sure if I am doing something wrong (apart from replication). I will need to support multiple virtual domains and I am basically using passwd-file for authentication, so I am using a single passwd file for all virtual domain users which allows user lookup (doveadm user '*') to work, which is required by replication. This is configuration file from dovecot2 server (which I use for testing and delivering mails): --- config start --- # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.16.0-30-generic x86_64 Ubuntu 14.04.2 LTS ext4 debug_log_path = /var/log/dovecot_debug.log doveadm_password = 67890 doveadm_port = 12345 mail_debug = yes mail_location = maildir:/vmail/mail/%d/%n/Maildir mail_plugins = " notify replication" namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = username_format=%u /vmail/auth/passwd driver = passwd-file } plugin { mail_replica = 192.168.1.100 } protocols = imap lmtp service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth-worker { unix_listener auth-worker { user = dovecot-auth } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = dovecot-auth } service doveadm { inet_listener { port = 12345 } } service imap-login { inet_listener imap { port = 0 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 10024 } process_min_avail = 3 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = vmail } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem userdb { args = username_format=%u /vmail/auth/passwd default_fields = uid=vmail gid=vmail home=/vmail/mail/%d/%n driver = passwd-file } protocol lmtp { postmaster_address = root@localhost } --- config end --- Configuration file on other node is exactly the same, except different IP address in mail_replica statement. Any help is much appreciated. Thank you.

1 1

Re: Problem with replication ("Fatal: -N parameter requires syncing with remote host")
by Vanja 16 Mar '15

16 Mar '15

Sorry to post to the list again, but just to confirm that I am seeing same problem with 2.2.16 that I compiled from sources. Same error message appears in the log, can't figure out what in the setup makes replication break. Philipp, can you tell me which auth backend are you using? passwd-file, SQL database or something else? Would you consider posting your 'sanitized' config file here, or emailing it to me privately? Thanks. Vanja On 03/16/2015 03:23 PM, Philipp wrote: > Am 16.03.2015 09:12 schrieb Vanja: >> doveadm_port = 12345 > >> plugin { >> mail_replica = 192.168.1.100 >> } > > would you try: > mail_replica = tcp:192.168.1.100:12345 > which works for me. > > check for firewall settings (both ways) anyway - and I can only advise > to go for 2.2.16; there have been > numerous fixes after 2.2.9 to the whole dsync/replica parts.

1 0

Re: Problem with replication ("Fatal: -N parameter requires syncing with remote host")
by Vanja 16 Mar '15

16 Mar '15

Philipp, I already tried explicitly specifying port, it does not make any difference. I tried quite many 'obvious' (to put it this way) configuration changes and tried changing many things in setup, but it would be too much to list all of them in a post. Basically, I was hoping that someone would know what exactly triggers this behavior, so I could fix it. At this point in time, I would like to see whether I can make this work with 2.2.9 that comes with Ubuntu 14.04, to avoid having to compile anything from sources (for easier maintenance, later on, since I probably won't be the one maintaining this). I am also not sure whether there is a reliable (semi-official?) PPA with latest Dovecot. I saw this one ( https://launchpad.net/~mamarley/+archive/ubuntu/updates ) but I am not sure whether I can rely on it not breaking anything at some point. Thanks. On 03/16/2015 03:23 PM, Philipp wrote: > Am 16.03.2015 09:12 schrieb Vanja: >> doveadm_port = 12345 > >> plugin { >> mail_replica = 192.168.1.100 >> } > > would you try: > mail_replica = tcp:192.168.1.100:12345 > which works for me. > > check for firewall settings (both ways) anyway - and I can only advise > to go for 2.2.16; there have been > numerous fixes after 2.2.9 to the whole dsync/replica parts.

1 0

Overriding dovecot.conf from Userdb Extras
by E.B. 16 Mar '15

16 Mar '15

Hi, I thought I read that anything from dovecot.conf can be overridden in a userdb lookup. Or a passdb lookup with "userdb_" prefix. But I tried for fun change log_path but it never worked. Is that because logging is special, already started logging before it comes to the passdb/userdb lookups? So are there some dovecot.conf settings that cannot be overridden? Thanks!

3 4

Re: How to detect out-of-sync condition
by Joseph Tam 16 Mar '15

16 Mar '15

> I tried to look at a dovecot.index.log and it was unreadable in a text > editor. I didn't see anything in the wiki link about how to view the log. It's not supposed to be human readable -- the point of this exercise was to check whether your indices are really out of date. You can perhaps compare the mtime of the index files and the mailbox. If they're close or the same, then your indices are not out of sync and don't require sync'ing. > So I guess I'll just be guilty of excessive optimizing. > Should I run a daily doveadm index or resync? You've skipped over the first 3 steps (is it really out of sync, is there a better way to avoid it, does the un-sync really cause probems) and went right to a solution for a possibly non-existent problem. Be that as it may, the frequency of re-syncing is up to you. I have no idea how large your userbase is or how intensive this operation will be. If it takes only a minute, you could run it more frequently. If you're going through terabytes of data, less frequently. I'm not quite convince you need it at all: I never had to do a global re-indexing except for a one-time dovecot initializing, or dealing with the odd crash that left indices in a unfixable state (which is more a bug than an unsync problem). Joseph Tam <jtam.home(a)gmail.com>

1 0

Why is Sieve trying to re-compile global scripts?
by E.B. 16 Mar '15

16 Mar '15

I have some global scripts that were running nicely. Then I opened one in an editor and (probably, but not 100% sure) mindlessly saved the file, even though I hadn't made any changes. Shortly after, Sieve errors started showing in the log: Error: 4k5JA74R/1TlIwABG/SpMA: sieve: binary save: failed to create temporary file: open(/usr/local/var/dovecot/sieve/script2.svbin.example.com.4139.) failed: Permission denied... Error: 4k5JA74R/1TlIwABG/SpMA: sieve: The LDA Sieve plugin does not have permission to save global Sieve script binaries; global Sieve scripts like `/usr/local/var/dovecot/sieve/script2.sieve' need to be pre-compiled using the sievec tool Well, OK, is it going by the timestamp on the files? Fine. I recompiled it by hand. Yet, I STILL got these errors! I triple and quadruple checked that the timestamp on the svbin files was more recent. And Sieve was only complaining about one of the two scripts in the directory. I restarted dovecot. No change. So I removed read permission on the .sieve files and only left read permission on the .svbin files. THIS WORKED. No more error. I can live with that, but why was it not complaining before, why was it only complaining about one of my scripts and why would it complain at all when the timestamps on the svbin should have indicated on compilation is needed?

3 14

sieve script not applied
by Joan Aymà 15 Mar '15

15 Mar '15

Hi everyone, I installed all-new server postfix+amavis+dovecot and pigeonhole. Clients, thunderbird and roundcube creates sieve scripts ok and changed ones are seen by eatch other, so server at 4190 accepts the scripts and ensure they are created. But filters onto new mail isn't applied. Example: # rule:[guifinet] if header :contains :comparator "i;octet" "subject" "[guifi-" { fileinto "INBOX.guifinet"; } Isn't the rule correct? Didn't find any troubleshooting on this topic. Any ideas? Thanks.

1 0

Re: How to detect out-of-sync condition
by Joseph Tam 14 Mar '15

14 Mar '15

On Fri, 13 Mar 2015, Cliff Hayes writes: >> By closing off other avenues other than dovecot imap/pop/lda/etc., >> the indices will stay sync'd. > > I use dovecot's lda and dovecot's sieve filter. Then I'm not sure how mailboxes ever get out of sync. > So it looks like I need to compare the index/mailbox mtimes as you suggest. > > What am I looking for? > I see that the indexes are updated when I run the resync. > > I checked my mailbox (that was not resynced) and noticed that > dovecot.index last update was 16 days ago. Just to be clear, you're talking "doveadm index ...", yes? You might have to check dovecot.index.log* as well as they could contain the latest changes that have yet to propagate to the main index file. http://wiki2.dovecot.org/Design/Indexes/TransactionLog > So am I resyncing if the gap is over x days? > If so, is there a way to resync just those mailboxes with a doveadm > command or do I have to write a program look for that condition and run > doveadm when matched? It's possible that "doveadm index" checks modification times as an optimization measure. My recommended game plan: 1) Are your indices *really* out of date (checks logs as Steffen recommends)? 2) If so, how do they get out of date and can you avoid it? 3) If you can't avoid it, does it cause real problems? Most of the time, dovecot will seemlessly rebuild it and it's transparent to the user. 4) If you got this far, run a trial "doveadm index -A INBOX" (assuming you're just concerned about INBOXs) to see if it's really such a bad operation. As Knuth (hallowed be thy name!) said, "premature optimization is the root of all evil." Only after all these steps are found to be unsatisfactory would I consider writing your own scan and fix tool. Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: How to detect out-of-sync condition
by Joseph Tam 14 Mar '15

14 Mar '15

Cliff Hayes writes: > I didn't see anything but the log is huge and I could have easily missed > it. > Is there a certain error or phrase I should look for? > If so please advise. In my setup using mbox, typical log entries would be like (Common) Warning: UIDVALIDITY changed ({int} -> {int}) in mbox file ... Error: Next message unexpectedly corrupted in mbox file ... (Rare) Error: Corrupted index cache file {cache-file}: Broken physical size for mail UID ... Error: read({file}) failed: Next message unexpectedly lost from mbox file ... Error: read({file}) failed: Cached message size larger than expected ... Joseph Tam <jtam.home(a)gmail.com>

1 0

JMAP support
by List 13 Mar '15

13 Mar '15

Just found http://jmap.io/ which is a JSON based RPC protocol for synchronizing messages/contacts/calendars. Any plans to support this protocol in Dovecot?

1 0