November 2015 - dovecot

Re: Connection between postfix and dovecot failing
by Sytse Kuijk 28 Nov '15

28 Nov '15

Well, post fix tries, but cannot connect. Sounds to me like problem at the receiving side. In any case, one question would be: is amavis part of dovecot or of postfix? sytse -- ------------------------------------------ - Sytse Kuijk - ___ - R.S.R. n.v. - o|o - Gijzelaarsstraat 16 - \_____(.)_____/ - 2000 Antwerpen - - GSM: +32 468 20 00 00 - ___ - Phone: +32 3 216 4328 - o|o - E-mail: sytse(a)pobox.com - \_____(.)_____/ ------------------------------------------ iThink therefore iMac > On 28 nov. 2015, at 22:24, Edgar Pettifogging <edgar(a)pettijohn-web.com> wrote: > > This should probably move to postfix list. Doesn't look like dovecot gets a chance to deliver. > > Sent from my iPhone > >> On Nov 28, 2015, at 11:34 AM, Sytse Kuijk <sytse(a)pobox.com> wrote: >> >> Well, I tried that. I can connect a telnet session on that port and that does not refuse the connection, it sits there waiting, but I do not know the protocol so I cannot check if it does what it is supposed to. >> >> I also checked who is listening on that port, here is the dialog: >> >> sh-3.2# sudo lsof -i -n -P | grep 10024 >> perl5.18 75844 _amavisd 5u IPv4 0x866f6119e39f833d 0t0 TCP 127.0.0.1:10024 (LISTEN) >> perl5.18 75844 _amavisd 6u IPv6 0x866f6119e03f253d 0t0 TCP [::1]:10024 (LISTEN) >> >> looks sort of correct to me… >> >> sytse >> >> -- >> >> ------------------------------------------ >> - Sytse Kuijk - ___ >> - R.S.R. n.v. - o|o >> - Gijzelaarsstraat 16 - \_____(.)_____/ >> - 2000 Antwerpen - >> - GSM: +32 468 20 00 00 - ___ >> - Phone: +32 3 216 4328 - o|o >> - E-mail: sytse(a)pobox.com - \_____(.)_____/ >> ------------------------------------------ >> >> iThink therefore iMac >> >>> On 28 nov. 2015, at 18:31, Edgar Pettifogging <edgar(a)pettijohn-web.com> wrote: >>> >>> I would bet that whatever is listening at 127.0.0.1 port 10024 didn't get restarted when the power came back on. Amavis perhaps or another proxy server of some sort. >>> >>> Sent from my iPhone >>> >>>> On Nov 28, 2015, at 11:14 AM, Sytse Kuijk <sytse(a)pobox.com> wrote: >>>> >>>> Hi all, >>>> >>>> Hope someone has been here and done that before. >>>> I am running a Mac mini with OSX server version 10.5, that is on El Capitan, but the problem I have started on Yosemite (the previous version) and did not change at all in the upgrade (I had hoped it would have solved itself). >>>> >>>> As you might know, OSX server uses postfix and dovecot as mailing engines, good choice. >>>> Since we suffered a short power outage a little over a week ago, the mail has failed in delivering messages to the local mailboxes. >>>> We can send mail out, check the mailboxes (they never change right now),etc. but incoming mail was first just not delivered and after a week warning are sent back to the senders, telling there is a temporary problem (hope that this message is right and I can solve the issue). >>>> >>>> What seems to go wrong is that postfix cannot connect to the local MTA. The error message that appears in several places is: >>>> >>>> recipient=ann.cools(a)surikata.be >>>> offset=769 >>>> dsn_orig_rcpt=rfc822;ann.cools(a)surikata.be >>>> status=4.4.1 >>>> action=delayed >>>> reason=delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]:10024: Connection refused >>>> >>>> I find all sorts of explanations and cures, but none work up to now. >>>> I tried to find lock files that might be blocking, deleted some files ending in lck with zero size, with no result. >>>> I tried to switch off the spam/virus/junk features, no difference (some mail said that messages in the queue will not get through, but new ones would. Not so, not a single message up to now). >>>> I also get alerts in the Server application, warning that: >>>> >>>> An unassigned mail directory has been detected and is scheduled to be deleted. >>>> >>>> The unassigned mail directory located in: >>>> /Library/Server/Mail/Data/mail/702C16A8-6B2D-42D8-9362-659DD2949AFC >>>> is not associated with any user account and is scheduled to be deleted in 1 days. >>>> >>>> In the mean time no mail, and unfortunately, we need it badly. >>>> >>>> Any suggestions? >>>> >>>> Million thanks for your kind support. >>>> >>>> sytse >>>> -- >>>> ------------------------------------------ >>>> - Sytse Kuijk - ___ >>>> - R.S.R. n.v. - o|o >>>> - Gijzelaarsstraat 16 - \_____(.)_____/ >>>> - 2000 Antwerpen - >>>> - GSM: +32 468 20 00 00 - ___ >>>> - Phone: +32 3 216 4328 - o|o >>>> - E-mail: sytse(a)pobox.com <mailto:sytse@pobox.com> - \_____(.)_____/ >>>> ------------------------------------------ >>>> >>>> iThink therefore iMac

2 1

Re: Connection between postfix and dovecot failing
by Sytse Kuijk 28 Nov '15

28 Nov '15

Well, I tried that. I can connect a telnet session on that port and that does not refuse the connection, it sits there waiting, but I do not know the protocol so I cannot check if it does what it is supposed to. I also checked who is listening on that port, here is the dialog: sh-3.2# sudo lsof -i -n -P | grep 10024 perl5.18 75844 _amavisd 5u IPv4 0x866f6119e39f833d 0t0 TCP 127.0.0.1:10024 (LISTEN) perl5.18 75844 _amavisd 6u IPv6 0x866f6119e03f253d 0t0 TCP [::1]:10024 (LISTEN) looks sort of correct to me… sytse -- ------------------------------------------ - Sytse Kuijk - ___ - R.S.R. n.v. - o|o - Gijzelaarsstraat 16 - \_____(.)_____/ - 2000 Antwerpen - - GSM: +32 468 20 00 00 - ___ - Phone: +32 3 216 4328 - o|o - E-mail: sytse(a)pobox.com - \_____(.)_____/ ------------------------------------------ iThink therefore iMac > On 28 nov. 2015, at 18:31, Edgar Pettifogging <edgar(a)pettijohn-web.com> wrote: > > I would bet that whatever is listening at 127.0.0.1 port 10024 didn't get restarted when the power came back on. Amavis perhaps or another proxy server of some sort. > > Sent from my iPhone > >> On Nov 28, 2015, at 11:14 AM, Sytse Kuijk <sytse(a)pobox.com> wrote: >> >> Hi all, >> >> Hope someone has been here and done that before. >> I am running a Mac mini with OSX server version 10.5, that is on El Capitan, but the problem I have started on Yosemite (the previous version) and did not change at all in the upgrade (I had hoped it would have solved itself). >> >> As you might know, OSX server uses postfix and dovecot as mailing engines, good choice. >> Since we suffered a short power outage a little over a week ago, the mail has failed in delivering messages to the local mailboxes. >> We can send mail out, check the mailboxes (they never change right now),etc. but incoming mail was first just not delivered and after a week warning are sent back to the senders, telling there is a temporary problem (hope that this message is right and I can solve the issue). >> >> What seems to go wrong is that postfix cannot connect to the local MTA. The error message that appears in several places is: >> >> recipient=ann.cools(a)surikata.be >> offset=769 >> dsn_orig_rcpt=rfc822;ann.cools(a)surikata.be >> status=4.4.1 >> action=delayed >> reason=delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]:10024: Connection refused >> >> I find all sorts of explanations and cures, but none work up to now. >> I tried to find lock files that might be blocking, deleted some files ending in lck with zero size, with no result. >> I tried to switch off the spam/virus/junk features, no difference (some mail said that messages in the queue will not get through, but new ones would. Not so, not a single message up to now). >> I also get alerts in the Server application, warning that: >> >> An unassigned mail directory has been detected and is scheduled to be deleted. >> >> The unassigned mail directory located in: >> /Library/Server/Mail/Data/mail/702C16A8-6B2D-42D8-9362-659DD2949AFC >> is not associated with any user account and is scheduled to be deleted in 1 days. >> >> In the mean time no mail, and unfortunately, we need it badly. >> >> Any suggestions? >> >> Million thanks for your kind support. >> >> sytse >> -- >> ------------------------------------------ >> - Sytse Kuijk - ___ >> - R.S.R. n.v. - o|o >> - Gijzelaarsstraat 16 - \_____(.)_____/ >> - 2000 Antwerpen - >> - GSM: +32 468 20 00 00 - ___ >> - Phone: +32 3 216 4328 - o|o >> - E-mail: sytse(a)pobox.com <mailto:sytse@pobox.com> - \_____(.)_____/ >> ------------------------------------------ >> >> iThink therefore iMac

1 0

ldap paging
by Andre Helwig 27 Nov '15

27 Nov '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi List, in ldapsearch you are able to use paging in case of limited results on ldap server side with "-E pr=10000/noprompt" Is there any way to get the same effect for the iterate query? Thanks Andre -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJWWHvwAAoJEAoTNwRDnEhRVmMH/0+vGRx7urQ49bHbzc4asD7r XlbwS7whkDKrJpflK39PLPw5BJoptKhwLNOueotFv9Pq8EYQBkN/gX69BzxeoFk8 DZpiP/ogYQIG1lUijes4UkbttVpnviy9/mLBKjIfIkvGDu0q3zwuq5876q6t2Qxx kJkpLSizxAfnGLeD03txVPAc8wGzxhxkyEdosQ+ARriYI1Ce+dfK28z6hSAEXMA+ 4JZEvIsy6P+xepFCFYILFWeRO2YBzqHtgtjKu8KCiyBdUPG0Nii4z2YmN2AoCj7m rQ8l4CeTDw+lk0pMy26kCsmh+PK2/OXCUGjxgYJOR3mxn2HNSIfUXFVWs4ri464= =qWS+ -----END PGP SIGNATURE-----

1 0

Sieve Max Redirect
by Shakeel Shahzad 27 Nov '15

27 Nov '15

Hello, I have configured iRedMail ( 0.9.1) on Ubuntu (14.04 LTS). In my dovecot.conf file, I have set sieve_max_redirects as 500. In dovecot (2.2.9) configuration file, I see sieve_max_redirect as 100 but when I add email addresses more than 32, I see that redirect fails. Can you please help me to see this issue and fix it through? If you need anything from my side, please let me know. Regards... Shakeel

3 4

doveadm-backup verbosity?
by Andreas Fischer 27 Nov '15

27 Nov '15

Hi, according to the man page doveadm-backup, the verbosity switch "-v" should do the following: "Enables verbosity, including progress counter" However, i see no difference running the following command with or without the "-v" switch: >>doveadm -v backup -R -A remote:myhost The command itself works without problem, two mailboxes are correctly back-upped, but the output is only an empty line. Not sure if this is expected behavior. Regards # 2.2.19: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.9 (357ac0a0e68b+) # OS: Linux 4.2.5-1-ARCH x86_64 dsync_remote_cmd = ssh -p 19022 -l root %{host} doveadm dsync-server -u%u first_valid_uid = 99 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate editheader passdb { args = nopassword=y driver = static } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve sieve_extensions = +editheader } protocols = service anvil { process_min_avail = 0 unix_listener anvil-auth-penalty { mode = 064 } } ssl = no userdb { args = /etc/dovecot/users default_fields = uid=nobody gid=nobody home=/backup/tmp/mail/%d/%n mail=sdbox:~/box driver = passwd-file }

1 0

fts_solr does not quote lucene's special characters
by Michael Stilkerich 27 Nov '15

27 Nov '15

Hello, I am using Dovecot 2.2.19 with the solr backend for full text search and experience the following issue. When I issue a search that includes characters that are part of lucene's query syntax (e.g. the double quote character), dovecot does not escape the special characters and issues a request with invalid syntax to the solr server. For example, I search for foo"bar, then my IMAP client (roundcube) sends the following to dovecot: A0003 UID THREAD REFS US-ASCII ALL UNDELETED BODY "foo\"bar" Dovecot in turn uses the following request URI to query SOLR: "GET /solr/select?fl=uid,score&rows=113739&sort=uid+asc&q=body:%22foo%22bar%22&fq=%2Bbox:4d86808c56d44282b35cea8700dda537+%2Buser:%22myuser%22 HTTP/1.1" 400 994 Solr responds with error 400 and the following body: type Status report message null description The request sent by the client was syntactically incorrect. This is because the " character in foo"bar is not quoted in the solr request. When I adapt the request to quote the " character solr succeeds with the expected result: "GET /solr/select?fl=uid,score&rows=113739&sort=uid+asc&q=body:%22foo%5C%22bar%22&fq=%2Bbox:4d86808c56d44282b35cea8700dda537+%2Buser:%22myuser%22 HTTP/1.1" 200 535 A fix for this would be highly appreciated. Cheers, Michael

2 1

auth_request_get_var_expand test fails on big endian arches.
by Rafael dos Santos 27 Nov '15

27 Nov '15

Hi, when running the tests in a big endian machine, I get the following error: 0 / 1 tests failed test-auth-request-var-expand.c:99: Assert failed: strcmp(str_c(str), test_output) == 0 test-auth-request-var-expand.c:104: Assert failed: strcmp(str_c(str), test_output) == 0 auth request var expand short and long ............................... : FAILED test: random seed #1 was 1446803094 auth request var expand flags ........................................ : ok test-auth-request-var-expand.c:158: Assert failed: strcmp(str_c(str), test_output) == 0 auth request var expand long-only .................................... : FAILED auth request var expand usernames .................................... : ok auth request var expand funcs ........................................ : ok 2 / 5 tests failed Makefile:1257: recipe for target 'check-test' failed Looking into it I found that auth_request_get_var_expand_table ends up calling the net_ip2addr function. This function, in turn, uses ntohl to convert from network byte order to host byte order. In big endian machines, that means that the string output will be ------------- big endian --------------- +user@+domain1@+domain2 +user +domain1@+domain2 +service 7.91.205.21 73.150.2.210 54321 +password +mech secured 21 210 valid instead of ---------- expected output --------- +user@+domain1@+domain2 +user +domain1@+domain2 +service 21.205.91.7 210.2.150.73 54321 +password +mech secured 21 210 valid Let me know in case you need help testing on big endian arches. Att. -- Rafael Fonseca

2 3

IMAP COPY creates invalid index data with zlib and mail_log plugins enabled but zlib_save off
by Robert L Mathews 27 Nov '15

27 Nov '15

I've noticed that maildir IMAP COPY commands can generate invalid dovecot.index entries when all seven of the following are true: - The zlib plugin is enabled; - The zlib_save/zlib_save_level options are NOT enabled; - The source message being copied is compressed; - The mail_log plugin is logging "copy" events; - The mail_log_fields setting includes at least one message header; - The destination mailbox folder has an index file that is recording the logged headers; - The source mailbox folder does NOT have an index file recording the logged headers. In this situation, copying a message results in an index entry in the destination that has all the headers blank, like this: RECORD: seq=5, uid=182415, flags=0x08 (Seen) - ext 1 modseq : 3 (0300000000000000) - ext 2 cache : 2352 (30090000) - cache offset=2352 size=120, prev_offset = 0 - hdr.FROM: - hdr.MESSAGE-ID: - hdr.SUBJECT: - hdr.DATE: - hdr.BCC: - hdr.CC: - hdr.CONTENT-TYPE: - hdr.IN-REPLY-TO: - hdr.NEWSGROUPS: - hdr.PRIORITY: - hdr.REFERENCES: - hdr.REPLY-TO: - hdr.TO: - hdr.X-PRIORITY: For these copies, the mail_log plugin records blank header data in the log, too, like this (despite the message having a valid message-ID, from, and subject): Oct 29 21:11:42 wheezy dovecot: imap[31763]: user=<1(a)example.com> ip=192.168.0.4 session=RV/MmUojyADAqAAE copy from INBOX.Saved: box=INBOX.Trash, uid=182412, msgid=, size=1357, vsize=1392, from=, subject=, flags=(\Seen) I've found this occurs in at least Dovecot 2.1.7, 2.2.13, and 2.2.18 (all packaged Debian versions). Changing any of the seven things I mentioned above fixes it. I've tried to find the code that causes this, but not yet been successful. My theory is that when the mail_log plugin calls mail_get_first_header(), and there's no index header record to read it from, it looks at the real message on disk. Perhaps it somehow operates on a compressed copy of the message although it expects uncompressed plaintext, so it doesn't find any of the headers. The "missing" headers then get cached to the destination index file as a side-effect. The copied maildir message itself is fine, by the way. It's an identical, still-compressed copy of the original message. And deleting the destination "dovecot.index*" files that contain the bad data causes Dovecot to rebuild the index with the correct header data. It's unusual to use the zlib plugin without the zlib_save/zlib_save_level options enabled, of course. That's probably why (as far as I can tell) there have been no reports of this before. I noticed it only because I was testing a small amount of manual mail compression before deploying it for all mail. Here's the "doveconf -n" of a system in the "has the problem" state: # 2.2.18: /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 (0c4ae064f307+) # OS: Linux 3.2.0-4-686-pae i686 Debian stretch/sid auth_mechanisms = plain login auth_username_format = auth_verbose = yes auth_verbose_passwords = sha1 default_vsz_limit = 1 G disable_plaintext_auth = no lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_log_format_elements = user=<%u> pid=[%{mail_pid}] method=%m rip=%r lip=%l %c mail_fsync = never mail_location = maildir:~/ mail_log_prefix = "%s[%{pid}]: user=<%u> ip=%{rip} session=%{session} " mail_max_userip_connections = 100 mail_plugins = mail_log notify zlib managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate spamtest spamtestplus namespace { inbox = yes location = prefix = INBOX. separator = . type = private } namespace { hidden = yes inbox = no list = no location = prefix = separator = . type = private } passdb { args = /etc/dovecot/dovecot.users driver = passwd-file } plugin { fts = lucene fts_lucene = whitespace_chars=@. mail_log_events = copy delete undelete expunge mailbox_delete mailbox_rename flag_change save mailbox_create mail_log_fields = vsize size flags box uid box msgid size flags vsize from subject sieve = %h/mailbox.sieve sieve_after = %h/../../domain-after.sieve sieve_before = %h/../../domain-before.sieve sieve_dir = %h/sieve sieve_extensions = -reject +spamtest +spamtestplus sieve_global_dir = %h/../../sieve-global-include-scripts sieve_spamtest_max_value = 7 sieve_spamtest_status_header = X-Spam-Level sieve_spamtest_status_type = strlen } protocols = pop3 imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { process_min_avail = 8 service_count = 0 } service imap { process_limit = 2000 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service pop3-login { process_min_avail = 8 service_count = 0 } service pop3 { process_limit = 2000 } ssl = no userdb { args = /etc/dovecot/dovecot.users driver = passwd-file } verbose_proctitle = yes protocol imap { imap_client_workarounds = delay-newmail imap_logout_format = %i bytes in, %o bytes out } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = ; %i bytes in, %o bytes out, %t top (%p bytes), %r retr (%b bytes), %d of %m deleted, mailbox size %s bytes pop3_uidl_format = UID%u-%v } protocol lda { mail_fsync = optimized mail_plugins = mail_log notify zlib sieve } protocol lmtp { mail_fsync = optimized mail_plugins = mail_log notify zlib sieve } -- Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

2 2

Problems Converting from Cyrus to Dovecot (cyrus2dovecot)
by Forrest 27 Nov '15

27 Nov '15

I have been attempting to use the cyrus2dovecot script, to no avail. I have many years of content that I want to convert from Cyrus to Dovecot; with the above not working, what are other options out there? Another idea I had is simply set up another IMAP server (using Dovecot) and drag-and-drop and just wait, which I may end up doing. In the above, I copied over my entire /var/imap and /var/spool/imap to another system; there is only one account (mine), so calling the script was fairly easy; it just doesn't work. inboxes=the "myaccount" that was copied over /home/myaccount/cyrus2dovecot --cyrus-inbox /home/myaccount/inboxes/%u \ --cyrus-seen /home/myaccount/varimap/user/%h/%u.seen \ --cyrus-sub /home/varimap/user/%h/%u.sub \ --dovecot-inbox /home/myaccount/dovecot/Maildir \ myaccount the log output complains of: cyrus2dovecot [myaccount]: (warning) Index record missing for: INBOX/62020. and correctly complains about squat indices, as that's not a file it would handle. There is no output into the Maildir, however. All directory paths are correct. Thanks.

6 9

logging imap searches ?
by mancyborg＠gmail.com 27 Nov '15

27 Nov '15

Hi thanks to this list support I'm using Dovecot's MailLog plugin http://wiki2.dovecot.org/Plugins/MailLog to log the following imap events: delete undelete expunge copy mailbox_delete mailbox_rename flag_change save mailbox_create and I'm very happy. Question: is it possible to also log imap searches https://tools.ietf.org/html/rfc3501#section-6.4.4 ? Thank you, Mike

1 0