- dovecot - dovecot.org

[Dovecot] BlackBerry will not setup my POP3 email, all other mail clients fine
by Ian Solecki 05 Mar '12

05 Mar '12

Hello, hoping someone here might be able to help me or at least point me in the right direction. My company recently (last week) moved to a new dedicated server for website and email hosting. It is a fairly run-of-the-mill Linux machine running cPanel and Dovecot (with the BlackBerry Fastmail service enabled) as a mail server. When we first set up the server, I was able to create a mailbox, access it via any desktop mail client, webmail, AND by setting it up as a basic POP3 account on my BlackBerry. Mail was running fine to and from the BlackBerry, no problems. I deleted that account from my BlackBerry as it was a test account, and went to add my actual account but was unable to do so. Received the "Cannot log in. Verify your email address, user name and password. If the error persists, contact (my domain name)" message. I tried the test account that had been running successfully and sending/receiving emails not minutes earlier, and it would not set up either. I have since tried multiple accounts on multiple BlackBerry devices (different models) on multiple carriers on several of my different domains (all of which point to the same server, of course), to no avail. Yet, any of these accounts still work flawlessly in any desktop mail client (Outlook, OE, Thunderbird, iPad, iPhone, etc). Also, any of these BlackBerrys work flawlessly with any other mail server. The username to log in to these mailboxes is not standard, it's mailboxname+ example.com if the email address is mailboxname(a)example.com and the mail server is mail.example.com. So, I know I have to access the "Advanced Settings" in BlackBerry email setup in order to put this username in. Still, no effect. So, there is something wrong with how mail SETUP works (not sending/receiving, though that may also not work, I have no way of knowing now) between RIM and my server, and it's something that has changed since the server was set up a week ago. My carrier(s) are clueless, my dedicated server provider (Lunarpages) is clueless. Can anyone help?

3 2

[Dovecot] doveconf: Warning: Obsolete settings.... in /etc/dovecot/dovecot.conf: ...
by D Chen 05 Mar '12

05 Mar '12

Got duplicated Warnings from the "doveconf -n" output ! Can anyone explain and fix them ? thx! admin@server:/etc/dovecot$ doveconf -n # 2.0.13: /etc/dovecot/dovecot.conf doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:716: protocol managesieve {} has been replaced by protocol sieve { } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:888: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:926: passdb pam {} has been replaced by passdb { driver=pam } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1039: userdb passwd {} has been replaced by userdb { driver=passwd } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1101: auth_user has been replaced by service auth { user } # OS: Linux 3.0.0-16-server x86_64 Ubuntu 11.10 log_timestamp = "%Y-%m-%d %H:%M:%S " mail_location = maildir:~/Maildir mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date passdb { driver = pam - Ignored: } passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap pop3 sieve service auth { unix_listener /var/spool/postfix/private/auth-client { group = postfix mode = 0660 user = postfix } unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0660 user = postfix } user = root } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_cipher_list = ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM ssl_key = </etc/ssl/private/dovecot.pem userdb { driver = passwd } userdb { driver = passwd } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 10 } protocol pop3 { mail_max_userip_connections = 10 pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { deliver_log_format = msgid=%m: %$ mail_plugins = sieve postmaster_address = postmaster quota_full_tempfail = yes rejection_reason = Your message to <%t> was automatically rejected:%n%r }

4 4

[Dovecot] Log sybnch error
by Stephen Davies 05 Mar '12

05 Mar '12

My mail log has many entries like: Mar 2 12:34:13 server dovecot: imap(john): Error: Log synchronization error at seq=2,offset=4264 for /home/john/Mail/INBOX/.imap/SMS Emails/dovecot.index: Extension header update points outside header size Version 2.1.1 dovecot -n gives: # 2.1.1: /usr/etc/dovecot/dovecot.conf # OS: Linux 2.6.33.5-desktop-2mnb i686 Mandriva Linux 2010.2 mail_access_groups = mail mail_privileged_group = mail passdb { driver = pam } protocols = pop3 imap service anvil { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = anvil extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 1 protocol = service_count = 0 type = anvil unix_listener anvil-auth-penalty { group = mode = 0600 user = } unix_listener anvil { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service auth-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = auth -w extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener auth-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service auth { chroot = client_limit = 0 drop_priv_before_exec = no executable = auth extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener auth-client { group = mode = 0600 user = } unix_listener auth-login { group = mode = 0600 user = $default_internal_user } unix_listener auth-master { group = mode = 0600 user = } unix_listener auth-userdb { group = mode = 0666 user = } unix_listener login/login { group = mode = 0666 user = } user = root vsz_limit = 18446744073709551615 B } service config { chroot = client_limit = 0 drop_priv_before_exec = no executable = config extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = config unix_listener config { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service dict { chroot = client_limit = 1 drop_priv_before_exec = no executable = dict extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dict { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service director { chroot = . client_limit = 0 drop_priv_before_exec = no executable = director extra_groups = fifo_listener login/proxy-notify { group = mode = 00 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener director-admin { group = mode = 0600 user = } unix_listener login/director { group = mode = 00 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service dns_client { chroot = client_limit = 1 drop_priv_before_exec = no executable = dns-client extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = unix_listener dns-client { group = mode = 0666 user = } unix_listener login/dns-client { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service doveadm { chroot = client_limit = 1 drop_priv_before_exec = no executable = doveadm-server extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 1 type = unix_listener doveadm-server { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service imap-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = imap-login extra_groups = group = idle_kill = 0 inet_listener imap { address = port = 143 ssl = no } inet_listener imaps { address = port = 993 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = imap service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service indexer-worker { chroot = client_limit = 1 drop_priv_before_exec = no executable = indexer-worker extra_groups = group = idle_kill = 0 privileged_group = process_limit = 10 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer-worker { group = mode = 0600 user = $default_internal_user } user = vsz_limit = 18446744073709551615 B } service indexer { chroot = client_limit = 0 drop_priv_before_exec = no executable = indexer extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener indexer { group = mode = 0666 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } service ipc { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = ipc extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener ipc { group = mode = 0600 user = } unix_listener login/ipc-proxy { group = mode = 0600 user = $default_login_user } user = $default_internal_user vsz_limit = 18446744073709551615 B } service lmtp { chroot = client_limit = 1 drop_priv_before_exec = no executable = lmtp extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = lmtp service_count = 0 type = unix_listener lmtp { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service log { chroot = client_limit = 0 drop_priv_before_exec = no executable = log extra_groups = group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = log unix_listener log-errors { group = mode = 0600 user = } user = vsz_limit = 18446744073709551615 B } service pop3-login { chroot = login client_limit = 0 drop_priv_before_exec = no executable = pop3-login extra_groups = group = idle_kill = 0 inet_listener pop3 { address = port = 110 ssl = no } inet_listener pop3s { address = port = 995 ssl = yes } privileged_group = process_limit = 0 process_min_avail = 0 protocol = pop3 service_count = 1 type = login user = $default_login_user vsz_limit = 18446744073709551615 B } service pop3 { chroot = client_limit = 1 drop_priv_before_exec = no executable = pop3 extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = pop3 service_count = 1 type = unix_listener login/pop3 { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service ssl-params { chroot = client_limit = 0 drop_priv_before_exec = no executable = ssl-params extra_groups = group = idle_kill = 0 privileged_group = process_limit = 0 process_min_avail = 0 protocol = service_count = 0 type = startup unix_listener login/ssl-params { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } service stats { chroot = empty client_limit = 0 drop_priv_before_exec = no executable = stats extra_groups = fifo_listener stats-mail { group = mode = 0600 user = } group = idle_kill = 4294967295 secs privileged_group = process_limit = 1 process_min_avail = 0 protocol = service_count = 0 type = unix_listener stats { group = mode = 0600 user = } user = $default_internal_user vsz_limit = 18446744073709551615 B } ssl_cert = </etc/pki/tls/certs/dovecot.pem ssl_key = </etc/pki/tls/private/dovecot.pem userdb { driver = passwd } -- ============================================================================= Stephen Davies Consulting P/L Voice: 08-8177 1595 Adelaide, South Australia. Fax : 08-8177 0133 Records & Collections Management. Mobile:040 304 0583

2 4

[Dovecot] doveadm -A stops processing at first uid<first_valid_uid
by Joseph Tam 04 Mar '12

04 Mar '12

I would like to run various doveadm commands that involves all (mail) users like doveadm expunge -A mailbox Trash savedbefore 30d but any doveadm command that uses "-A" to iterate through all users will stop processing at the first account with UID<first_valid_uid. doveadm(sysdaemon): Error: user sysdaemon: Couldn't drop privileges: Mail access for users with GID 5551 not permitted (see first_valid_gid in config file, gid from userdb lookup). doveadm(sysdaemon): Error: User init failed doveadm: Error: Failed to iterate through some users However, these accounts are system accounts (locked password, no shell) and are in userdb to provide UID<->name mapping for utilities like ls, chown, etc. There are various workaround like iterating manually, or setting first_valid_uid to zero, or even reordering users to put all system accounts at the end, but is there a better way to do this? Joseph Tam <jtam.home(a)gmail.com>

2 3

[Dovecot] Multiple namespaces seems to be used at the same time
by Steve Campbell 04 Mar '12

04 Mar '12

I've just converted from an old Centos 3 box to a Centos 6.2 box. I've switched from UW-imap to dovecot in the process. In my configurations, I've placed the multiple namespace sections as suggested by the "Backward compatability" part of the wiki. I use mbox since I mostly copied the home directories from the old to the new server. On some of the clients, it appears that the client is using multiple namespaces at the same time. When they view their subscribed folders, they see multiple "mail" folders instead of just the single "mail" folder under their home directory. The .subscription files are more than likely not correct (haven't looked yet, but will fix them as a user calls), but should this ever happen? I'm also sure the client's prefix isn't set since the old system never required it and there are just so many other things that are required right now on this conversion. Speaking of prefixes, I'd like to get the default of "" (nothing entered) to work for the majority of the users to avoid having to add this to the multiple users we have. Hopefully, by going through each user's home directory and copying the old .mailboxlist to a new .subscriptions file and ensuring the imap folders are in ~/mail will do this. Does this sound resonable? Thanks for any help. The learning curve is getting less steep, but it's still a ways off to the top of the hill. steve campbell

2 1

[Dovecot] doveadm fetch prints duplicate results in 2.1
by Christoph Bußenius 04 Mar '12

04 Mar '12

Hi, when the private namespace has "prefix = INBOX." and you use doveadm fetch to search for "mailbox INBOX", then it prints every message twice: # doveadm fetch -u testuser 'mailbox uid guid' mailbox INBOX mailbox: INBOX uid: 1 guid: 1757c403a4ec4d4fbf040000463889b5 mailbox: INBOX uid: 1 guid: 1757c403a4ec4d4fbf040000463889b5 Without "prefix = INBOX." in the configuration, the messages are only printed once. In 2.0, they were never printed twice. doveconf -n # 2.1.1: /usr/local/dovecot/etc/dovecot/dovecot.conf # OS: Linux 2.6.32-35-server x86_64 Ubuntu 10.04.4 LTS disable_plaintext_auth = no mail_gid = vmail mail_location = mdbox:~/mdbox mail_plugins = acl mail_uid = vmail namespace { inbox = yes location = prefix = INBOX. separator = . } passdb { args = scheme=CRYPT username_format=%u /usr/local/dovecot/etc/dovecot/users driver = passwd-file } protocols = imap service auth { unix_listener auth-userdb { group = vmail mode = 0660 } } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { args = username_format=%u /usr/local/dovecot/etc/dovecot/users driver = passwd-file } protocol imap { mail_plugins = acl imap_acl } Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching

2 2

[Dovecot] 2.1.1: Incorrect quoting of RFC 2822 personal parts in ENVELOPE data
by Michael M Slusarz 04 Mar '12

04 Mar '12

I'm seeing this: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({22} XXXXX \"X-XX\" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001@scheduler>")) It should be: 1 UID FETCH 31734 (ENVELOPE) * 23 FETCH (UID 31734 ENVELOPE ("Fri, 2 Mar 2012 19:05:24 -0500 (EST)" "XXXXXX" (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXX")) (({20} XXXXX "X-XX" XXXXXX NIL "XXXXXXX" "XXXXXXXXX.XXXXXX.XXX")) ((NIL NIL "XXXXXXX" "XXXXXXXXX.XXX")) ((NIL NIL "slusarz" "curecanti.org")) NIL NIL NIL "<1109380587237.1109118788902.20323.7.35190001@scheduler>")) since the RFC 2822 quoting characters must be removed. michael

2 1

[Dovecot] LIST-STATUS issue
by Michael M Slusarz 04 Mar '12

04 Mar '12

Looked at the hg commits since 2.1.1, but didn't see anything that was exactly on-point to this... UNSEEN counts seem to be broken when done in a LIST-STATUS call. I see this, for example: >> Timestamp: Thu, 01 Mar 2012 10:55:47 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 0) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. 3 seconds later... >> Timestamp: Thu, 01 Mar 2012 10:55:50 -0700 S: * OK [CAPABILITY IMAP4rev1 LITERAL+ LOGIN-REFERRALS ID ENABLE AUTH=LOGIN XIMAPPROXY] Dovecot ready. C: [LOGIN Command - username: slusarz] S: * OK [XPROXYREUSE] IMAP connection reused by squirrelmail-imap_proxy S: 1 OK User logged in C: 2 EXAMINE IN.dovecot (QRESYNC (1254166849 25800 18484,20260,20393,20705,20860:20868)) S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft impflag1 impflag2 impflag0 Junk seen NonJunk) S: * OK [PERMANENTFLAGS ()] Read-only mailbox. S: * 13 EXISTS S: * 0 RECENT S: * OK [UNSEEN 5] First unseen. S: * OK [UIDVALIDITY 1254166849] UIDs valid S: * OK [UIDNEXT 20869] Predicted next UID S: * OK [HIGHESTMODSEQ 25800] Highest S: 2 OK [READ-ONLY] Select completed. [...] C: 7 SEARCH RETURN (COUNT) UNSEEN S: * ESEARCH (TAG "7") COUNT 8 S: 7 OK Search completed (0.000 secs). I can confirm that I did not receive 8 dovecot list messages in 3 seconds. :) Once the mailbox is accessed, I am seeing correct counts in LIST-STATUS return again: >> Timestamp: Thu, 01 Mar 2012 10:57:28 -0700 [...] C: 3 LIST "" (INBOX IN.dovecot) RETURN (STATUS (UNSEEN)) S: * LIST () "." "IN.dovecot" S: * STATUS "IN.dovecot" (UNSEEN 8) S: * LIST () "." "INBOX" S: * STATUS "INBOX" (UNSEEN 3) S: 3 OK List completed. I am using imapproxy for these connections, so all access are happening in the same Dovecot access. michael

2 8

[Dovecot] directly addressable public folders issues
by Tom Fernandes 04 Mar '12

04 Mar '12

Hi, I would like to have an address info(a)example.com whose mails are stored in a public folder. I also want certain users to be able to create sieve-filter-rules and subfolders. From what I understand it's a good idea to have a separate location for home and for mail_location. For my normal accounts I have: home = /var/vmail/<user> mail_location = ~/Maildir Is there a way to have the same for public folders? This are my current settings: namespace public { separator = / prefix = public/ location = maildir:/var/vmail/public subscriptions = no } user_attrs = homeDirectory=home=/var/vmail/%$, =mail=maildir:~/Maildir The LDAP-homeDirectory-attribute for info(a)example.com is "public/.info" Like this an incoming mail is stored below /var/vmail/public/.info/Maildir which is good. The MUA seems to read to read from /var/vmail/public/.info though. How can I make the MUA read from /var/vmail/public/.info/Maildir instead? And in case this works - is it possible to have only certain users modify the sieve-rule for this public folder or do I definitely need a passdb-entry for info(a)example.com? Modifying scripts via commandline is not an option... I'm using dovecot 1.2. regards, Tom Fernandes

2 1

[Dovecot] Dovecot clustering with dsync-based replication
by Timo Sirainen 03 Mar '12

03 Mar '12

This document describes a design for a dsync-replicated Dovecot cluster. This design can be used to build at least two different types of dsync clusters, which are both described here. Ville has also drawn overview pictures of these two setups, see http://www.dovecot.org/img/dsync-director-replication.png and http://www.dovecot.org/img/dsync-director-replication-ssh.png First of all, why dsync replication instead of block level filesystem replication? - dsync won't replicate filesystem corruption. - A cold restart of replication won't go through all of the data in the disks, but instead quickly finds out what has changed. - Split brain won't result in downtime or losing any data. If both sides did changes, the changes are merged without data loss. - If using more than 2 storages, the users' replicas can be divided among the other storages. So if one storage goes down, the extra load is shared by all the other storages, not just one. Replication mail plugin ----------------------- This is a simple plugin based on notify plugin. It listens for all changes that happen to mailboxes (new mails, flag changes, etc.) Once it sees a change, it sends an asynchronous (username, priority) notification to replication-notify-fifo. The priority can be either high (new mails) or low (everything else). Optionally the replication plugin can also support synchronous replication of new mail deliveries. In this way it connects to replication-notify UNIX socket, tells it to replicate the user with sync (=highest) priority and waits until it is done or replication_sync_timeout occurs. The IMAP/LMTP client won't see an "OK" reply until the mail is replicated (or the replication has failed). The synchronous replication probably adds a noticeable delay, so it might not be acceptable for IMAP, but might be for LMTP. So, what is listening in those replication-notify* sockets? It depends on if Dovecot is running on director-based setup or not. Aggregator ---------- When running in Dovecot director-based setup, all of the Dovecot backends (where replication plugin runs) also run "aggregator" process. Its job is very simple: It proxies the notifications from mail plugin and sends them via a single TCP connection to the replicator process running in Dovecot proxies. This is simply an optimization to avoid tons of short lived TCP connections directly from replication plugin to director server. When not running in Dovecot director setup (i.e. there is only a single Dovecot instance that handles all of the users), there is no point in having an aggregator proxy, because the replicator process is running on the same server. In this kind of setup the replicator process directly listens on the replication-notify* sockets. Replicator ---------- The initial design for replicator isn't very complex either: It keeps a priority queue of all users, and replicates those users at the top of the queue. Notifications about changes to user's mailboxes (may) move the user up in the priority queue. If the user at the top of the queue already has been replicated "recently enough", the replicator stops its work until new changes arrive or the "recently enough" is no longer that. dsync can do two types of syncs: quick syncs and full syncs. A quick sync trusts indexes and does the replication with the least amount of work and network traffic. A quick sync is normally enough to replicate all changes, but just in case something has gone wrong there's also the full sync option, which guarantees that the mailboxes end up being fully synced. A full sync is slower though, and uses more network traffic. The priority queue is sorted by: 1. Priority (updated by a notification from replication plugin) 2. If priority!=none: Last fast sync (those users are replicated first whose last replication time is oldest) 2. If priority=none: Last full sync (these users should already be fully synced, but do a full sync for them once in a while anyway) All users get added to the replication queue at replicator startup with "none" priority. The list of users is looked up via userdb iteration. If the previous replication state is found from a disk dump, it's used to update the priorities, last_*_sync timestamps and other replication state. Replicator process creates such dumps periodically [todo: every few mins? maybe a setting?]. Replicator starts replicating users at the top of the queue, setting their priorities to "none" before starting. This means that if another change notification arrives during replication, the priority is bumped up and no changes get lost. replication_max_conns setting specifies how many users are replicated simultaneously. If the user's last_full_sync is older than replication_full_sync_interval setting, a full sync is done instead of a fast sync. If the user at the top of the queue has "none" priority and the last_full_sync is newer than replication_full_sync_interval, the replication stops. [todo: it would be nice to prefer doing all the full syncs at night when there's hopefully less disk I/O] (A global replication_max_conns setting isn't optimal in proxy-based setup, where different backend servers are doing the replication. There it should maybe be a per-backend setting. Then again, it doesn't account for the replica servers that also need to do replication work. Also to properly handle this each backend should have its own replication queue, but this requires doing a userdb lookup for each user to find out their replication server, and this would need to be done periodically in case the backend changes, which can easily happen often with director-based setup. So all in all, none of this is being done in the initial implementation. Ideally the users are distributed in a way that a global replication queue would work well enough.) In director-based setup each director runs a replicator server, but only one of them (master) actually asks the backends to do the replication. The rest of them just keep track of what's happening, and if the master dies or hangs, one of the others becomes the new master. The server with lowest IP address is always the master. The replicators are connected to a ring like the directors, using the same director_servers setting. The communication between them is simply about notifications of what's happening to users' priorities. Preferably the aggregators would always connect to the master server, but this isn't required. In general there's not much that can go wrong, since it's not a problem if two replicators request a backend to start replication for the same user or if the replication queue states aren't identical. If the replication is running too slowly [todo: means what exactly?], log a warning and send an email to admin. So, how does the actual replication happen? Replicator connects to doveadm server and sends a "sync -u user@domain" command. In director-based setup the doveadm server redirects this command to the proper backend. doveadm sync ------------ This is an independent feature from all of the above. Even with none of it implemented, you could run this to replicate a user. Most of this is already implemented. The only problem is that currently you need to explicitly tell it where to sync. So, when the destination isn't specified, it could do a userdb lookup and use the returned "mail_replica" field as the destination. Multiple (sequentially replicated) destinations could be supported by returning "mail_replica2", "mail_replica3" etc. field. In NFS-based (or shared filesystem-based in general) setup the mail_replica setting is identical to mail_location setting. So your primary mail_location would be in /storage1/user/Maildir, while the secondary mail_replica would be in /storage2/user/Maildir. Simple. In non-NFS-based setup two Dovecot servers talk dsync protocol to each others. Currently dsync already supports SSH-based connections. It would also be easy to implement direct TCP-based connections between two doveadm servers. In future these connections could be SSL-encrypted. Initially I'm only supporting SSH-based connections, as they're already implemented. So what does the mail_replica setting look like in this kind of a setup? I'm not entirely sure. I'm thinking that it could be either "ssh:host" or "ssh:user@host", where user is the SSH login user (this is opposite of the current doveadm sync command line usage). In future then it could support also tcp:host[:port]. Both of these ssh: and tcp: prefixes would also be supported by doveadm sync command line usage (and perhaps the prefixless user@domain be deprecated). dsync can run without any long lived locking and it typically works fine. In case mailbox was modified during dsync, the replicas may not end up being identical, but nothing breaks. dsync currently usually notices this and logs a warning. When these conflicting changes was caused by imap/pop3/lda/etc. this isn't a problem, they've already notified replicator already to perform another sync that will fix it. Running two dsyncs at the same time is more problematic though, mainly related to new emails. Both dsyncs notice that mail X needs to be replicated, so both save it and it results in having a duplicate. To avoid this, there should be a dsync-lock. If this lock exists, dsync should wait until the previous dsync is done and then do it again, just in case there were more changes since the previous sync started. This should conclude everything needed for replication itself. High-availability NFS setup --------------------------- Once you have replication, it's of course nice if the system automatically recovers from a broken storage. In NFS-based setups the idea is to do soft mounts, so if the NFS server goes away things start failing with EIO errors, which Dovecot notices and switches to using the secondary storage(s). In v2.1.0 Dovecot already keeps track of mounted filesystems. Initially they're all marked as "online". When multiple I/O errors occur in a filesystem [todo: how many exactly? where are these errors checked, all around in the code or checking the log?] the mountpoint is marked as "offline" and the connections accessing that storage are killed [todo: again how exactly?]. Another job for replication plugin is to hook into namespace creation. If mail_location points to a mountpoint marked as "offline", it's replaced with mail_replica. This way the user can access mails from the secondary storage without downtime. If the replica isn't fully up to date, this means that some of the mails (or other changes) may temporarily be lost. These will come back again after the original storage has come back up and replication has finished its job. So as long as mails aren't lost in the original storage, there won't be any permanent mail loss. When an offline storage comes back online, its mountpoint's status is initially changed to "failover" (as opposed to "online"). During this state the replication plugin works a bit differently when the user's primary mail_location is in this storage: It first checks if the user is fully replicated, and if so uses the primary storage, otherwise it uses the replica storage. Long running IMAP protocesses check the replication state periodically and kill themselves once the user is replicated, to move back to primary storage. Once replicator notices that all users have been replicated, it tells the backends' to change the "failover" state to "online" (via doveadm server). High-availability non-NFS setup ------------------------------- One possibility is to use Dovecot proxies, which know which servers are down. Instead of directing users to those servers, it would direct them to replica servers. The server states could be handled similar to NFS setup's online vs. failover vs. offline states. Another possibility would be to do the same as above, except without separate proxy servers. Just make "mail.example.com" DNS point to two IP addresses, and if one Dovecot notices that it's not the user's primary server, it proxies to the secondary server, unless it's down. If one IP is down, clients hopefully connect to the other.

3 5