October 2016 - dovecot

Maildir Expunged GUID mismatch for UID
by Steven Xu 14 Oct '16

14 Oct '16

Dovecot version:2.2.25 Since we used to keep our email files on widows server, I made the following changes in maildir-storage.h #define MAILDIR_EXTRA_SEP ',' #define MAILDIR_INFO_SEP_S ":" to "+". Everything seems working except EXPUNG, The dovecot log is flooded by messages like following: imap(xxxxx): Error: Mailbox INBOX: Expunged GUID mismatch for UID 7039 Then I read the source code, and found the following lines in maildir-sync-index.c T_BEGIN { guid = maildir_uidlist_lookup_ext(ctx->mbox->uidlist, uid, MAILDIR_UIDLIST_REC_EXT_GUID); if (guid == NULL) guid = t_strcut(filename, ':'); mail_generate_guid_128_hash(guid, guid_128); } T_END; I have to change the code to guid = t_strcut(filename, '+'); So, should MAILDIR_EXTRA_SEP be used here instead of ':'? Thanks, Steven

2 1

update dovecot-acl in all subfolders in a public folder
by Lukas Henrich 14 Oct '16

14 Oct '16

Hello everyone, right now I have problem and can't find a proper solution. But first: dovecot-version: 2.2.13 Now to my problem: A client of mine uses a public folder called "groups". In this Folder are several subfolders like "Archive", "projects", "sales" und so on. Unfortunately this client (and his employees) created thousands of subfolders within these folders. The folder "groups" is stored in /data/vmail/domain.com/. So, if i type "tree -a -L 1 /data/vmail/domain.com/groups" I get the following outpout: /data/vmail/domain.com/groups/ ??? .Archive ??? .Archive.subfolder1 ??? .Archive.subfolder2 ??? .Archive.subfolder2.subfolder3 ??? .Archive.subfolder2.subfolder4 (....) ??? .projects ??? .projects.subfolder1 ??? .projects.subfolder2 ??? .projects.subfolder2.subfolder3 ??? .projects.subfolder2.subfolder4 (...) In this groups-folder are at the moment more than 3400 folders! Now to the permissions: When the folders "Archive", "projects" and so on had been created every folder got a dovecot-acl for the permission for each user, e.g.: user=user1 kxeilprwts user=user2 kxeilprwts This worked finde, as the dovecot-acl got copied from the parent folder when the employees created new subfolders. Now to my problem I'm facing right now: This client got 2 new employees. So how can I edit all these dovecot-acl files in all subfolders where these 2 new employess schould get access to? Thank you in advance! Yours sincerely, Lukas Henrich Furthermore here is the output of dovecot -n: # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 4.4.6-1-pve x86_64 Debian 8.4 auth_username_format = %Ln disable_plaintext_auth = no lda_mailbox_autocreate = yes mail_home = /data/vmail/domain.com/%Ln mail_location = maildir:~ mail_plugins = " acl" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { hidden = no ignore_on_failure = no inbox = no list = children location = maildir:/data/vmail/domain.com/%%n:INDEXPVT=/data/vmail/domain.com/%n/share… prefix = shared/%%n/ separator = / subscriptions = yes type = shared } namespace { hidden = no ignore_on_failure = no inbox = no list = yes location = maildir:/data/vmail/domain.com/groups:INDEXPVT=/data/vmail/domain.com/%n/gr… prefix = groups/ separator = / subscriptions = yes type = public } namespace inbox { inbox = yes location = mailbox Archiv { special_use = \Archive } mailbox Archive { auto = subscribe special_use = \Archive } mailbox Archives { special_use = \Archive } mailbox "Deleted Messages" { special_use = \Trash } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Entwürfe { special_use = \Drafts } mailbox "Gelöschte Elemente" { special_use = \Trash } mailbox "Gelöschte Objekte" { special_use = \Trash } mailbox Gesendet { special_use = \Sent } mailbox "Gesendete Elemente" { special_use = \Sent } mailbox "Gesendete Objekte" { special_use = \Sent } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Papierkorb { special_use = \Trash } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Spam { special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / subscriptions = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = scheme=CRYPT username_format=%Ln /etc/dovecot/users driver = passwd-file } plugin { acl = vfile acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes.db sieve = ~/dovecot.sieve sieve_dir = ~/sieve } postmaster_address = admin(a)domain.com protocols = " imap lmtp sieve sieve" service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail user = vmail } } service imap-login { inet_listener imap { port = 143 } } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 24 } unix_listener /var/spool/postfix/private/lmtp-dovecot { group = postfix user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = </etc/ssl/certs/kmux-dovecot.intern.domain.com.crt ssl_key = </etc/ssl/private/kmux-dovecot.intern.domain.com.key userdb { args = /etc/dovecot/dovecot-ldap.conf.ext default_fields = uid=vmail gid=vmail home=/data/vmail/domain.com/%n driver = ldap } userdb { args = username_format=%Ln /etc/dovecot/users driver = passwd-file } protocol lmtp { mail_plugins = " acl sieve" } protocol lda { mail_plugins = " acl sieve" } protocol imap { mail_plugins = " acl imap_acl" }

1 0

fix SIS attachment errors
by Webert de Souza Lima 13 Oct '16

13 Oct '16

Hi, I've sent some e-mails about this before but since there was no answers I'll write it differently, with different information. I'm using SIS (Single Instance Attachment Storage). For any reason that is not relevant now, many attachments are missing and the messages can't be fetched: Error: read(attachments-connector(zlib(/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: open(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36) failed: No such file or directory in this specific case, the /dovecot/attach/bar.example/23/ae/ director doesn't exist. In other cases, just one file is missing so I would assume the hardlink could be recreated and it would work. If I create the missing file (with touch or whatever), I get the following errors: Error: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is smaller than expected (0 < 483065) Error: read(attachments-connector(zlib(/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is smaller than expected (0 < 483065) Error: fetch(body) failed for box=INBOX uid=15: BUG: Unknown internal error If I try to fill the file with the amount of bytes it complains about with the following command: $ dd if=/dev/zero of=/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36 bs=1 count=483065 then I get the following error: Error: read(/dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is larger than expected (483928 > 483065, eof=0) Error: read(attachments-connector(zlib(/srv/dovecot/mdbox/bar.example/foo/storage/m.1))) failed: read(//dovecot/attach/bar.example/23/ae/23aed008c1f32f048afd38d9aae68c5aeae2d17a9170e28c60c75a02ec199ef4e7079cd92988ad857bd6e12cd24cdd7619bd29f26edeec842a6911bb14a86944-fb0b6a214dfa63573c1f00009331bd36[base64:19 b/l]) failed: Stream is larger than expected (483928 > 483065, eof=0) Error: fetch(body) failed for box=INBOX uid=15: BUG: Unknown internal error Based on this I have a few questions: 1. Is there a way, or a tool to scan all mailboxes to get all the messages that have compromised attachments? 2. is there a way to "fix" the missing files (even if creating fake files or removing the attachments information from the messages) 3. What I need is to migrate these boxes using doveadm backup/sync, which fails when these errors occur. Is is possible to ignore them or is there another tool that would do it? Thank you. Webert Lima Belo Horizonte, Brasil

1 1

First steps in Dovecot; IMAP not working
by Marnaud 13 Oct '16

13 Oct '16

Hello, I'm new in Dovecot and am having troubles making it working. I'm trying using Outlook and Apple's Mail as the mail clients. Outlook says it can't establish a secured connection to the server (for the IMAP protocol). I'm guessing sending e-mails works but I can't check. This is my current configuration (using dovecot -n): # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-042stab116.1 x86_64 Debian 8.6 mail_location = mbox:~/mail:INBOX=/var/mail/%u namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } passdb { driver = pam } protocols = " imap" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } } service imap-login { inet_listener imaps { port = 993 ssl = yes } } ssl = no ssl_cert = </etc/ssl/certs/mail.crt ssl_key = </etc/ssl/private/mail.key userdb { driver = passwd } userdb { driver = passwd } I find abnormal I'm seeing "ssl = no" in this configuration despite the fact that I have "ssl = required" in the /etc/dovecot/conf.d/10-ssl.conf file, but I'm new to this... I have looked around the web; finally, I'm asking here, hoping it's the correct place to ask. Arnaud

2 1

SSL error
by arnaud gaboury 13 Oct '16

13 Oct '16

I run dovecot + postfix as my email server. Everything is working as expected, but I see an error in the dovecot logs: lmtp(7331): Error: SSL context initialization failed, disabling SSL: ENGINE_init(dynamic) failed Dovecot is running and emails are OK. I wonder why this error and how I can fix it? I see it is a SSL issue but no idea in which direction to look. Thank you for help

2 4

Strange subscriptions added by dsync backup
by Rik Theys 13 Oct '16

13 Oct '16

Hi, We're in the process of migrating our dovecot 1.x mail server to a Dovecot 2.2.25 server. During the migration I'm moving from mbox storage to mdbox. I use the following method to do a one-way sync from our current mail server to our new mail server (command executed on the new server): doveadm -v -o imapc_host=oldserver \ -o imapc_user=$u \ -o imapc_master_user=$masteruser \ -o imapc_password="$masterpass" \ -o imapc_port=993 -o imapc_ssl=imaps \ -o ssl_client_ca_file=/etc/pki/tls/certs/ca-bundle.crt \ -o imapc_ssl_verify=yes \ -o mail_fsync=never \ -o mail_prefetch_count=20 \ backup -R -u $u imapc: $u is replaced by the username I'm migrating. The dsync command runs and exits with code 0 (all OK). However, when I look at the subscriptions file on the new server, there seem to be additional subscriptions that are not in the subscriptions file on the old server: 7b0d681945d0fc5711560000ffff90ca 7c0d681945d0fc5711560000ffff90ca 7d0d681945d0fc5711560000ffff90ca 7e0d681945d0fc5711560000ffff90ca 7f0d681945d0fc5711560000ffff90ca 800d681945d0fc5711560000ffff90ca Where do these come from? Is something wrong with the migration? It also seems that subscriptions for IMAP folders that only contain subfolders[1] are no longer present in the new subscriptions file: # diff -u sub-old.sorted sub-new.sorted --- sub-old.sorted 2016-10-11 13:42:44.175070610 +0200 +++ sub-new.sorted 2016-10-11 13:47:53.973888462 +0200 @@ -1,26 +1,26 @@ -Archive/Administration/ +7b0d681945d0fc5711560000ffff90ca +7c0d681945d0fc5711560000ffff90ca +7d0d681945d0fc5711560000ffff90ca +7e0d681945d0fc5711560000ffff90ca +7f0d681945d0fc5711560000ffff90ca +800d681945d0fc5711560000ffff90ca Archive/Administration/Conferences Archive/Announcements -Archive/Education/ Archive/Education/E02N3A Archive/Education/I0D51A Archive/Politics -Archive/Research/ -Archive/Research/FET/ Archive/Research/FET/VPH - Virtual Physiological Human Archive/Research/Grants/ICON IBBT Call 2011 Archive/Research/Grants/Marie Curie ITN 2011 Archive/Research/Grants/Odysseus Archive/Research/Grants/SymBioSysII/Funding Archive/Research/Grants/SymBioSysII/JobApplications -Archive/Research/Manuscripts/ Archive/Research/Manuscripts/ruby-ensembl-api Archive/Research/Projects/GUNZ Archive/Research/Projects/MIQAS Archive/Research/Projects/MODY Archive/Research/Projects/ruby-ensembl-api Archives -Archive/Service/ Archive/Service/EditorORC Archive/Service/Reviewing Deleted Messages Is this expected behaviour? Regards, Rik [1] On the old server a folder can only contain either messages or subfolders, not both at the same time. -- Rik Theys System Engineer KU Leuven - Dept. Elektrotechniek (ESAT) Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee +32(0)16/32.11.07 ---------------------------------------------------------------- <<Any errors in spelling, tact or fact are transmission errors>>

1 0

Auto-archiving
by ＠lbutlr 13 Oct '16

13 Oct '16

I’d like to know if there is a way to tell dovecot to 1) move messages older than # days to the Archive folder 2) rebuild the indexes 3) remove any folders that are left with no mail Preferably, I’d like this to be a action I an schedule via crontab or something to fire off for any users that want it. So, I do not want it to do this across all users and mailboxes.

3 5

Disappearing message when move to Inbox subfolder
by Simon 12 Oct '16

12 Oct '16

We are using dovecot 2.0.9 on Centos 6.5 and having the following issues for one single user/single mailbox. When an Outlook (2016) user creates a folder under their Inbox, it does not create in their mailbox (checking with webmail the folder is not there). If they drag a message to that folder in Outlook, then it disappears. Ive searched their mailbox using doveadm and its simply gone. Ive tested with webmail doing essentially the same thing and it works with no issues. Any ideas here would be most well received :) Thanks in advance, Simon

1 0

dsync replication quota2 issue
by George Kontostanos 12 Oct '16

12 Oct '16

Hello list, We are testing a configuration with 2 mail servers using dsync replication (dovecot 2.2.25 ). Everything works fine except the quota2 which is calculated wrong only on one server. Quota2 resides on different databases since each server needs to update it. The problem: The local server always updates quota2 twice on each message it receives. This happens only on one server. Updates run fine on the second. SQL Debug: Query UPDATE quota2 SET bytes=bytes+2108,messages=messages+1 WHERE username = 'user(a)domain.org' Query UPDATE quota2 SET bytes=bytes+2108,messages=messages+1 WHERE username = 'user(a)domain.org' The result on the server that runs fine mysql> select * from quota2; +----------------------------+---------+----------+ | username | bytes | messages | +----------------------------+---------+----------+ | | user(a)domain.org | 2917126 | 17 | The result on the server that has the problem: mysql> select * from quota2; +----------------------------+---------+----------+ | username | bytes | messages | +----------------------------+---------+----------+ | | user(a)domain.org | 2920317 | 19 | dovecot -n is the same on both: root@mx2:/var/log # dovecot -n # 2.2.25 (7be1766): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.15 (97b3da0) # OS: FreeBSD 10.3-RELEASE amd64 ufs auth_mechanisms = plain login auth_verbose = yes default_client_limit = 2560 default_process_limit = 512 dict { acl = mysql:/usr/local/etc/dovecot/dovecot-dict-shares-sql.conf.ext quota = mysql:/usr/local/etc/dovecot/dovecot-dict-quota-sql.conf.ext } doveadm_password = # hidden, use -P to show it doveadm_port = 12345 log_path = /var/log/dovecot.log mail_debug = yes mail_home = /usr/local/vhosts/mail/%d/%n mail_location = maildir:/usr/local/vhosts/mail/%d/%n:LAYOUT=fs mail_max_userip_connections = 70 mail_plugins = quota acl notify replication mail_privileged_group = vmail mail_shared_explicit_inbox = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext mbox_write_locks = fcntl namespace { inbox = no list = children location = maildir:/usr/local/vhosts/mail/%%d/%%n:LAYOUT=fs:INDEX=/usr/local/vhosts/indexes/%d/%n/shared/%%u:INDEXPVT=/usr/local/vhosts/indexes/%d/%n/shared/%%u prefix = shared/%%d/%%n/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes list = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / type = private } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile acl_shared_dict = proxy::acl mail_replica = tcp:beta.sophimail.com:12345 quota = dict:User quota::proxy::quota quota_rule2 = Trash:storage=+100M sieve = /usr/local/vhosts/mail/%d/%n/.dovecot.sieve sieve_before = /usr/local/vhosts/sieve/before.d/ sieve_dir = /usr/local/vhosts/mail/%d/%n sieve_global_dir = /usr/local/vhosts/sieve/%d sieve_global_path = /usr/local/vhosts/sieve/%d/default.sieve } protocols = imap lmtp sieve sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = vmail } unix_listener replication-notify { mode = 0666 user = vmail } } service auth-worker { user = vmail } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-userdb { mode = 0600 user = vmail } user = dovecot } service config { unix_listener config { user = vmail } } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } user = vmail } service imap-login { inet_listener imap { port = 143 } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 1 vsz_limit = 64 M } service replicator { unix_listener replicator-doveadm { mode = 0666 } } ssl_cert = </usr/local/etc/postfix/keys/domain.com.crt ssl_key = </usr/local/etc/postfix/keys/domain.com.key userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = quota acl notify replication sieve notify replication } protocol imap { imap_client_workarounds = tb-extra-mailbox-sep mail_plugins = quota acl notify replication imap_quota imap_acl notify replication } protocol lda { mail_plugins = quota acl notify replication sieve acl postmaster_address = root } local 192.168.3.6 { protocol imap { ssl_cert = </usr/local/etc/postfix/keys/domain.com.crt ssl_key = </usr/local/etc/postfix/keys/domain.com.key } } dovecot-dict-quota-sql.conf.ext: connect = host=127.0.0.1 dbname=quota user=mailadmin password=********** map { pattern = priv/quota/storage table = quota2 username_field = username value_field = bytes } map { pattern = priv/quota/messages table = quota2 username_field = username value_field = messages } Sorry for the lengthy email, any help is very much appreciated. -- George Kontostanos ---

1 1

Detect IMAP server domain name in Dovecot IMAP proxy
by KT Walrus 12 Oct '16

12 Oct '16

I’m in the process of setting up a Dovecot IMAP proxy to handle a number of IMAP server domains. At the current time, I have my users divided into 70 different groups of users (call them G1 to G70). I want each group to configure their email client to access their mailboxes at a domain name based on the group they belong to (e.g., g1.example.com <http://g1.example.com/>, g2.example.com <http://g2.example.com/>, …, g70.example.com <http://g70.example.com/>). I will only support TLS encrypted IMAP connections to the Dovecot IMAP proxy (‘ssl=yes’ in the inet_listener). My SSL cert has alternate names for all 70 group domain names. I want the group domain to only support users that have been assigned to the group the domain name represents. That is, a user assigned to G23 would only be allowed to configure their email client for the IMAP server named g23.example.com <http://g23.example.com/>. My solution during testing has been to have the Dovecot IMAP proxy to listen on different ports: 9930-9999. I plan to purchase 70 IPs, one for each group, and redirect traffic on port 993 to the appropriate Dovecot IMAP proxy port based on the IP I assign to the group domain name in the site’s DNS. The SQL for handling the IMAP login uses the port number of the inet_listener I think this could work in production, but it will cost me extra to rent the 70 IPs and might be a pain to manage. Eventually, I would like to have over 5,000 groups so requiring an IP per group is less than ideal. I also think having Dovecot IMAP proxy have 5,000 inet_listeners might not work so well or might create too many threads/processes/ports to fit on a small proxy server. I would rather have 1 public IP for each Dovecot IMAP proxy and somehow communicate to the userdb which group domain name was configured in the email client so only the users assigned to this group can login with that username. Anyone have any ideas? For HTTP traffic, it is easy to query the host in the HTTP Request, but I don’t think IMAP traffic has such host info in it. Does the Dovecot IMAP proxy receive the hostname from the email client when exchanging SSL certs (like SNI for HTTPS)? Or, maybe I should have group domain in the username used to log in with (e.g., username+g23(a)example.com <mailto:username+g23@example.com> or username(a)g23.example.com <mailto:username@g23.example.com>). I don’t like to make the user configure their email client to log in with a name that is different than their mailbox address. It is simpler to just have them configure their email client with username(a)example.com <mailto:username@example.com> for both authorization and for the from/sender headers in the messages. Anyway, any ideas of how to set this up in production?

3 3