- dovecot - dovecot.org

replication of a big mailbox
by Tim Kuijsten 16 Jun '15

16 Jun '15

I'm running dovecots replicator successfully on 33 mailboxes. There is only one mailbox (the biggest one, 167000 mails) that is not replicating successfully. After 10 minutes the "source" starts logging these messages: dovecot: dsync-local(foo(a)example.com): Error: dsync(imap.netsend.nl): I/O has stalled, no activity for 600 seconds dovecot: dsync-local(foo(a)example.com): Error: Timeout during state=sync_mails (send=mail_requests recv=changes) dovecot: dsync-local(foo(a)example.com): Error: Remote command process isn't dying, killing it dovecot: dsync-local(foo(a)example.com): Error: kill(17459, SIGKILL) failed: Operation not permitted dovecot: doveadm: Error: Pseudo-terminal will not be allocated because stdin is not a terminal. dovecot: doveadm: Error: dsync-remote(foo(a)example.com): Error: Couldn't lock //.dovecot-sync.lock: Timed out after 30 seconds dovecot: doveadm: Error: Pseudo-terminal will not be allocated because stdin is not a terminal. dovecot: doveadm: Error: dsync-remote(foo(a)example.com): Error: Couldn't lock //.dovecot-sync.lock: Timed out after 30 seconds The 30 seconds timeout keeps repeating, while the doveadm process on the destination keeps on going for hours and hours. Destination imap.netsend.nl is creating a lot of hardlinks (at least one hardlink per message) in .temp.... files. This takes about 1,5 hour, after that the number of hardlinks decline again. But it looks like the connection with source is already lost. I've patched the source to overcome the timeout by changing it from 10 minutes to 3 hours[1] but now I'm running into a memory error: dovecot: dsync-local(foo(a)example.com): Error: dsync(imap.netsend.nl): I/O has stalled, no activity for 10800 seconds dovecot: dsync-local(foo(a)example.com): Error: Timeout during state=sync_mails (send=done recv=mails) dovecot: dsync-local(foo(a)example.com): Error: mremap_anon(211677184) failed: Cannot allocate memory The allocate memory error shows up every 10 seconds. Next I'll try to look into the memory allocation code, but maybe someone already has a better idea? Thanks. -Tim Config on source: # 2.2.15: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.7 amd64 auth_debug = yes auth_mechanisms = plain login auth_verbose = yes dsync_remote_cmd = /usr/bin/ssh imap.netsend.nl first_valid_gid = 30000 first_valid_uid = 30000 last_valid_gid = 30100 last_valid_uid = 30100 mail_debug = yes mail_home = /srv/vmail/%d/%n mail_location = maildir:~/Maildir mail_plugins = " notify replication" passdb { args = /passwd driver = passwd-file } passdb { args = /passwd_smtponly driver = passwd-file } plugin { mail_replica = remoteprefix:imap.netsend.nl } protocols = imap lmtp service aggregator { fifo_listener replication-notify-fifo { mode = 0666 } unix_listener replication-notify { mode = 0666 } } service auth-worker { chroot = /etc/mail unix_listener /etc/mail/auth-worker { mode = 0600 user = $default_internal_user } user = $default_internal_user } service auth { chroot = /etc/mail unix_listener /var/spool/postfix/private/dovecot-auth { group = _postfix mode = 0600 user = _postfix } } service imap-login { inet_listener imap { port = 0 } process_min_avail = 1 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = _postfix mode = 0600 user = _postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/ssl/private/2015/chain.crt ssl_cipher_list = ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA; ssl_dh_parameters_length = 2048 ssl_key = </etc/ssl/private/the.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { args = /passwd default_fields = uid=30000 gid=30000 home=/srv/vmail/%d/%n/./ driver = passwd-file } valid_chroot_dirs = /srv/vmail verbose_ssl = yes protocol imap { mail_max_userip_connections = 20 } config on destination imap.netsend.nl: # 2.2.15: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.7 amd64 auth_debug = yes auth_mechanisms = plain login auth_verbose = yes first_valid_gid = 30000 first_valid_uid = 30000 last_valid_gid = 30100 last_valid_uid = 30100 mail_debug = yes mail_home = /srv/vmail/%d/%n mail_location = sdbox:~/dbox passdb { args = /passwd driver = passwd-file } passdb { args = /passwd_smtponly driver = passwd-file } protocols = imap lmtp service auth-worker { chroot = /etc/mail unix_listener /etc/mail/auth-worker { mode = 0600 user = $default_internal_user } user = $default_internal_user } service auth { chroot = /etc/mail unix_listener /var/spool/postfix/private/dovecot-auth { group = _postfix mode = 0600 user = _postfix } } service imap-login { inet_listener imap { port = 0 } process_min_avail = 1 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = _postfix mode = 0600 user = _postfix } } ssl = required ssl_cert = </etc/ssl/private/2015/chain.crt ssl_cipher_list = ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA; ssl_dh_parameters_length = 2048 ssl_key = </etc/ssl/private/the.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { args = /passwd default_fields = uid=30000 gid=30000 home=/srv/vmail/%d/%n/./ driver = passwd-file } valid_chroot_dirs = /srv/vmail verbose_ssl = yes protocol imap { mail_max_userip_connections = 20 } [1] --- src/doveadm/dsync/dsync-ibc-stream.c 2014-09-29 13:07:07.000000000 +0200 +++ src/doveadm/dsync/dsync-ibc-stream.c2 2015-06-15 12:28:56.000000000 +0200 @@ -24,7 +24,7 @@ #include <stdlib.h> -#define DSYNC_IBC_STREAM_TIMEOUT_MSECS (60*10*1000) +#define DSYNC_IBC_STREAM_TIMEOUT_MSECS (60*180*1000) #define DSYNC_IBC_STREAM_OUTBUF_THROTTLE_SIZE (1024*128) #define DSYNC_PROTOCOL_VERSION_MAJOR 3

1 0

Problem logging in during testing
by Steve Matzura 16 Jun '15

16 Jun '15

Instructions at http://wiki2.dovecot.org/BasicConfiguration say the next thing to test after getting IMAP and Telnet working is to create a dummy user file with the following command: echo "$USER:{PLAIN}password:$UID:$GID::$HOME" > users And place it in /etc/dovecot and protect it appropriately. It was also edited to change the plain-text password from `password' to what it really is. The next section describes changes to 10-auth.conf and other things: If you used the example configuration files, switch to passwd-file by modifying conf.d/10-auth.conf: # Add '#' to comment out the system user login for now: #!include auth-system.conf.ext # Remove '#' to use passwd-file: !include auth-passwdfile.conf.ext In conf.d/auth-passwdfile.conf.ext you should have: passdb { driver = passwd-file args = scheme=CRYPT username_format=%u /etc/dovecot/users } userdb { driver = passwd-file args = username_format=%u /etc/dovecot/users } Verify with doveconf -n passdb userdb that the output looks like above (and there are no other passdbs or userdbs). All done successfully. Now, when I telnet 127.0.0.1 143 and attempt to log in with the string: a login <any username> <any password> whether it's one in my users file or not, I get: a NO [AUTHENTICATIONFAILED] Authentication failed.

4 11

Re: More about my SSL certificate problem
by Joseph Tam 16 Jun '15

16 Jun '15

Steve Matzura writes: > What is the reasoning behind that `<' anyway? It just appears so odd > that a path should have that at its front. I think this was a special syntax-hack that Timo used to mean "read into a buffer", or some different way to handle file data, probably for security reasons. Joseph Tam <jtam.home(a)gmail.com>

1 0

What became of dovecot.passwd?
by Steve Matzura 15 Jun '15

15 Jun '15

Since I've decided to go to version 2, which is a lot more difficult to set up because I have to look in a lot more places than I did with version 1.0.15, what do I do with the old dovecot/passwd or its contents? I see no references to it in any of the stuff in conf.d or doc/dovecot/wiki. As always, thanks in advance.

2 3

More about my SSL certificate problem
by Steve Matzura 15 Jun '15

15 Jun '15

Someone, I don't remember whether it was Alex or Urban, asked me to check on my private keyfile to see if it began with "Begin RSA private key". I reported that it does not. Now here's where things get strange: When I ran the command: $ openssl req -nodes -newkey rsa:2048 -sha1 -keyout myserver.key -out server.csr on an old Ubuntu system, `myserver.key' does begin with "Begin RSA private key". But when I run that same command on a Fedora 20 system, the word "RSA" isn't there. Could this have anything to do with my 0906D06C error?

4 8

Testin new installation
by Steve Matzura 15 Jun '15

15 Jun '15

The first place I went to for test advice and planning recommendations was http://wiki2.dovecot.org/TestInstallation - The very first test failed. Doc says: Next check that Dovecot is listening for connections: # telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN] Dovecot ready. If you got "connection refused", make sure that Dovecot is configured to serve the imap protocol and listening on the expected interfaces/addresses. The simplest way to do that would be using doveconf(1): # doveconf protocols listen protocols = imap pop3 lmtp sieve listen = *, :: If the protocols setting shouldn't contain imap so add it. Also make sure, that relevant !include or !try_include configuration lines are not commented. *** End doc extract *** All of the above specified settings are correct. Yet, when I telnet localhost 143, I get: Trying ::1... # this is certainly suspect Escape character is '^['. Connection closed by foreign host.

7 21

dsync replication issues with shared mailboxes
by Sergey Schwartz 15 Jun '15

15 Jun '15

Gentlemen, I've setup 2 servers with dsync replication and hit a serious issue - some messages got duplicated thousands times in some shared mailboxes(~5). There is actually no reason to replicate anything from shared name space and I've tried to limit replication scope with just 'inbox' name space but it didn't help. dovecot version ||2.2.18 (2de3c7248922) errors in the logs May 26 13:23:38 mx10 dovecot: doveadm: Error: dsync-remote(korobova@mydomain): Error: Mailbox turkey: Save commit failed: Message has been copied too many times (24498 + 8270) May 26 13:25:16 mx10 dovecot: doveadm: Error: dsync-remote(korobova@mydomain): Error: Mailbox turkey: Save commit failed: Message has been copied too many times (24498 + 8270) May 26 13:26:40 mx10 dovecot: dsync-local(zhelonkina@mydomain): Error: Mailbox booking: Save commit failed: Message has been copied too many times (0 + 32768) May 26 13:30:01 mx10 dovecot: dsync-local(kashchits@mydomain): Error: Mailbox shared/davydjanc@mydomain/russia: Save commit failed: Message has been copied too many times (17139 + 15629) May 26 13:30:01 mx10 dovecot: dsync-local(davydjanc@mydomain): Error: Mailbox russia: Save commit failed: Message has been copied too many times (17139 + 15629) May 26 13:30:01 mx10 dovecot: dsync-local(kapitanova@mydomain): Error: Mailbox shared/davydjanc@mydomain/russia: Save commit failed: Message has been copied too many times (17139 + 15629) May 26 13:30:03 mx10 dovecot: dsync-local(grinenko@mydomain): Error: Mailbox shared/davydjanc@mydomain/russia: Save commit failed: Message has been copied too many times (17139 + 15629) May 26 13:30:03 mx10 dovecot: dsync-local(maslova@mydomain): Error: Mailbox shared/davydjanc@mydomain/russia: Save commit failed: Message has been copied too many times (17139 + 15629) Data was not lost and I've been able to run deduplicate for most of mailboxes, but one mailbox got broken and I had to manually move message to another mailbox force-resync failed fot that mailbox doveadm(l.davydjanc@mydomain): Error: mdbox map /srv/vmail/current/mydomain/davydjanc/storage/dovecot.map.index corrupted: map_uid=361735 refcount too low Hope you guys have an idea how to work it around :) replication.conf dsync_remote_cmd = /usr/bin/ssh -i /home/vmail/.ssh/id_rsa -l %{login} %{host} doveadm dsync-server -u %u service aggregator { fifo_listener replication-notify-fifo { user = vmail mode = 0600 } unix_listener replication-notify { user = vmail mode = 0600 } process_min_avail = 1 } service replicator { unix_listener replicator-doveadm { user = vmail mode = 0600 } process_min_avail = 1 } service doveadm { user = vmail process_min_avail = 1 } service config { unix_listener config { user = vmail } process_min_avail = 1 } replication_max_conns = 50 replication_dsync_parameters = -d -n inbox -l 30 -U plugin { mail_replica = remote:vmail@mx20 } doveconf -n # 2.2.18 (2de3c7248922): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 # OS: Linux 3.13.0-34-generic x86_64 Ubuntu 14.04.2 LTS xfs auth_cache_negative_ttl = 0 auth_cache_size = 512 M auth_default_realm = mydomain auth_failure_delay = 5 secs auth_worker_max_count = 4096 default_client_limit = 40960 default_process_limit = 8192 default_vsz_limit = 2 G disable_plaintext_auth = no first_valid_gid = 500 mail_location = mdbox:/srv/vmail/current/%d/%n mail_plugins = acl zlib notify replication mail_shared_explicit_inbox = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mdbox_rotate_size = 32 M namespace { list = yes location = mdbox:/srv/vmail/current/%%d/%%n prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/virtual/base.conf.ext driver = sql } passdb { args = /etc/dovecot/virtual/master.conf.ext driver = sql } plugin { acl = vfile acl_shared_dict = file:/srv/vmail/current/shared.db/000-shared-mailboxes.db sieve = file:/srv/vmail/current/%d/%n/sieve;active=/srv/vmail/current/%d/%n/active.sieve sieve_max_redirects = 0 zlib_save = xz zlib_save_level = 6 } protocols = " imap lmtp sieve" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service imap-login { process_min_avail = 24 } service imap { process_limit = 4096 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } ssl_cert = </etc/ssl/mxbgcom/2015/mydomain.com.crt.chained ssl_key = </etc/ssl/mxbgcom/2015/mydomain.com.key syslog_facility = local0 userdb { args = /etc/dovecot/virtual/base.conf.ext driver = sql } protocol lmtp { mail_plugins = acl zlib notify replication sieve } protocol imap { mail_max_userip_connections = 15 mail_plugins = acl zlib notify replication imap_acl } -- Best regards, Sergey Schwartz Senior System Administrator Biblio Globus Tour Operator www.bgoperator.ru T: +7 495 5042500 ext 1532 E: sergey.schwartz(a)bgoperator.com

1 0

Sorry, another faq
by Martin S 14 Jun '15

14 Jun '15

I've installed a mailserver according to these instructions: http://www.server-world.info/en/note?os=CentOS_7&p=httpd&f=13 When I try to login to the server through Roundcube webmail I get Connection to storage server failed. So checking on this [12-Jun-2015 11:28:53 +0000]: <6jap13r2> IMAP Error: Login failed for martin from 83.251.209.249. Could not connect to ssl://myserver:993: Permission denied in /usr/share/roundcubemail/program/lib/Roundcube/rcube_imap.php on line 198 (POST /roundcubemail/?.... /etc/roundcubemail/config.inc.php Has the following: $config['default_host'] = 'ssl://mail.myserver.tld'; [address obfuscated] $config['default_port'] = 993; as I understand should be correct. I've been looking at various posts on the net that says the problem is a permission on dovecot, but I fail to find anything to how to fix this as it looks right to me. Any suggestions? /Martin S -- Regards, Martin S

5 5

dovecot only partially indexing ... fts
by Infoomatic 14 Jun '15

14 Jun '15

hi, I have a setup using solr as fts, however it seems that when I move messages it does not automatically index the stuff. Also, when using sieve scripts to store messages in different folders I get: dovecot: indexer-worker(user): Indexed 0 messages in testfolder If I get a mail to normal INBOX I get dovecot: indexer-worker(user): Indexed 1 messages in INBOX doveadm index -u user '*' does the trick, however, I have set "fts_autoindex = yes" ... the relevant part of doveconf -n: mail_plugins = fts fts_solr zlib acl quota zlib fts fts_solr acl plugin { fts = solr fts_autoindex = yes fts_solr = url=http://localhost:8983/solr/ break-imap-search sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +notify +imapflags } protocol lda { mail_plugins = fts fts_solr zlib sieve acl } protocol imap { mail_plugins = fts fts_solr zlib acl quota imap_acl imap_quota } is this normal behaviour or did I screw something up? TIA, infoomatic

1 0

fts_solr Bad Request / BUG: Unknown internal error
by Nick Rosier 14 Jun '15

14 Jun '15

Hi, yesterday I created a new mailbox; ever since (and only for the new mailbox) I am getting these errors: 2015-06-12T00:00:29.808526+02:00 mail dovecot: indexer-worker(xxx(a)example.com): Error: fts_solr: Lookup failed: Bad Request 2015-06-12T00:00:29.808560+02:00 mail dovecot: indexer-worker(xxx(a)example.com): Error: Mailbox INBOX: Status lookup failed: BUG: Unknown internal error Is this a known issue? Rgds, N.

2 2