November 2015 - dovecot

Dsync problems
by Frank de Bot (lists) 26 Nov '15

26 Nov '15

Hi, I'm setting up a migration between 2 dovecot servers 2.1.12 to 2.2.19 . I've ran into some issues with dsync. I use the following command for syncing: doveadm -o mail_fsync=never -o imapc_host=%remote% -o imapc_master_user=master -o imapc_password=%masterpw% -o imapc_features="rfc822.size fetch-headers" -o mail_prefetch_count=20 -o imapc_user=%user% backup -1 -R -u %user% imapc: I execute a full sync first and when the mailserver are being migrated it only would have to do incremental. The first is run is without problems. But I find the following problem: - Flags on current mails aren't updated. If I set message to unread, the sync won't pick them up. - On the origin dovecot server I had removed some messages and purged the inbox. Those message are now in the Trash and are created again and again when I sync Is this normal behaviour? It renders it pretty much useless since data consistency isn't guaranteed. Regards, Frank de Bot

2 1

FTS, multiple virtualmailbox entries for single mailbox
by Frank de Bot (lists) 26 Nov '15

26 Nov '15

Hi, i have a dovecot setup with virtual users. Each user has 2 entries for his mailbox. One is with the username formatted as %username%@%default_realm% , the other %username%@%domain% I've done some tests with it, but I don't complete understand how it works - I've emptied all document in solr so it empty. Doveadm is instructed to do the same with 'doveadm -D -v fts rescan -A' - I reindex %username%@%default_realm% with 'doveadm -D index -u %username%@%default_realm% '*' - With fluke I can see 4 indexes documents. - I reindex %username%@%domain% - No documents are added to Solr, dovecot finds it's up-to-date - After 'doveadm -D -v fts rescan -A' the documents are added The part I don't understand is that a certain key used for indexing, but dovecot checks based on something else. If find this inconsistent. How can I manage to get virtualmailboxes where multiple logins are allowed on 1 mailbox and FTS is working? I think it won't work well this way if a user logs in with only a certain username with the default realm. Message received by dovecot will be indexed in Solr with %username%@%domain% , but there will be no indexed documents for %username%@%default_realm% A earlier threads suggested to be able to override the username for FTS, but it was tought to be not a good idea. Regards, Frank de Bot

2 1

Re: Questions about hardlinks, alternate storage and compression
by Gionatan Danti 26 Nov '15

26 Nov '15

Hi Javier, thanks for your reply. I already checked SIS and, while interesting, is not what I want, because: 1) it can be difficult to restore a single message/attachment from a backup 2) only the attachments, and not the entire messages, are deduped. Message-based hardlinks really exists for a reason. The good news is that I found _why_ they are not working: it depends from how dovecot and its sieve plugin (pigenhole) interact. Basically, if I define anything for the before_sieve and after_sieve variables, dovecot stops creating hardlinks for multiple copies of the same message. On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message. Does someone know if newer dovecot versions change anything in this regard? Thank you all. On 13/07/15 11:10, Javier Miguel Rodríguez wrote: > Search about "single instance storage dovecot". This is what you need. > > Regards > > Javier > > On 27/06/2015 18:18, Gionatan Danti wrote: >> Hi all, >> I have some questions about hardlinks, alternate storage and >> compression. I already scanned the list for related information and I >> have an idea of how things works, but I would like to have a definite >> answer. >> >> System spec: >> - CentOS 6.6 x64 >> - dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version >> - sdbox mail store >> >> 1) About hardlinks: when sending the same message to two different >> recipients, I see that the two u.x files are created as two different >> files. Diffing them, I see that the only difference is a single char >> (see [1] for an example). My questions are: >> a) it is possible to tell dovecot to create a single file + a single >> hardlink (linkref=2)? As other IMAP servers support that features (eg: >> Cyrus, CommunigatePro, etc) I am wondering if I missed something in >> configuring dovecot... >> b) If it is not possible, can I run a script that compare the >> various files and substitute equal ones (minus the changing line) with >> hardlinks, or it will confuse dovecot? As a side note, why the >> changing line ever exists? >> >> 2) About alternate storage and compression: actually, I use a single >> mail_location without compression. I would like to have an alternate >> storage and to enable compression on it only, leaving the main >> location without compression. I if understand it correctly, it >> _should_ be done using a command similar to "doveadm -Dv -o >> "plugin/zlib_save=gz" altmove -uu testuser sentbefore 8d". I'm right >> thinking that it should work? I will really end with a primary >> uncompressed mail store and an alternate, zlib-compressed one? >> >> Thank you all and sorry if I did some naive questions. >> >> [1] >> 63c63 >> < G2fd0811c64be8e553d970000eaa8309f >> --- >>> G2ed0811c64be8e553d970000eaa8309f >> >> >> > -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

3 6

LZ4 mailbox corruption
by Renaud Allard 25 Nov '15

25 Nov '15

Hello, When I look at email files on my server (mdbox format), they contain random LZ4 stuff. I have now changed the compression to gzip and new mails look cleaner. But older ones have problems. A mail file looks like this: 2 M1e C5655b6a8 ^A^BN 00000000000006F4 Dovecot-LZ4^M*<9B><C5>^@^A^@^@^@^@^F<DD><FF><83>Return-path: <nobody(a)winnie.projecthoneypot.org> Envelope-to: nikademus(a)llorien.org Delivery-date: Mon, 22 Nov 2004 09:50:58 +0100 Received: from }^@^G<F7> [66.135.39.84] by mail.}^@<FF>< with esmtp (TLS-1.0:DHE_RSA_3DES_EDE_CBC_SHA:24) id=1CW9uQ-0002gj-El for<DE>^@^C^_;<D0>^@8<B1>(localhost. ^@<F1>^Ddomain [127.0.0.1])<E5>^@^O^S^A^H<82>(8.12.8/^G^@^R)<FF>^@<F2>^HESMTP id iAM9gl4l021209<D6>^@^_<<D7>^@^B^_><D8>^@^A<B9>3:42:47 -06<A8>^Ac(from -^B^A<BA>^@Ohost<AC>^@^\<86>/Submit)<A8>^@!cd<A8>^@N7; 4^B <8C>^@^_DZ^B^D &^@<F2>^KMessage-Id: <200411220942.^Y^A^Dq^@^O<F1>^B <FF>^BTo: Renaud AllardB^A^E<B2> Then, I randomly see errors like this in maillog: truncated lz4 chunk at 555395 Given I am still able to read the mails using imap, I think there might be a way to recover them to a proper format. However, using dsync to make a mirror throws a lot of errors like below. # dsync -v -u recover -o "mail_plugins=zlib" mirror dbox:~/ndbox dsync(recover): Error: lz4.read(//home/vmail/recover/mdbox/storage/m.2): corrupted lz4 chunk at 43551 dsync(recover): Error: write(/home/vmail/recover/ndbox/mailboxes/INBOX/dbox-Mails/.temp.1448464321.P24341Q6M965521.isildur.arnor.org) failed: Invalid argument dsync(recover): Error: copy: i_stream_read(zlib(//home/vmail/recover/mdbox/storage/m.2)) failed: read(//home/vmail/recover/mdbox/storage/m.2) failed: lz4.read(//home/vmail/recover/mdbox/storage/m.2): corrupted lz4 chunk at 43551 dsync(recover): Error: read(zlib(//home/vmail/recover/mdbox/storage/m.2)) failed: read(//home/vmail/recover/mdbox/storage/m.2) failed: lz4.read(//home/vmail/recover/mdbox/storage/m.2): corrupted lz4 chunk at 43551 (uid=42028, box=INBOX) dsync(recover): Error: Mailbox INBOX: read(msg input) failed: read(//home/vmail/recover/mdbox/storage/m.2) failed: lz4.read(//home/vmail/recover/mdbox/storage/m.2): corrupted lz4 chunk at 43551 dsync(recover): Error: read(zlib(//home/vmail/recover/mdbox/storage/m.2)) failed: read(//home/vmail/recover/mdbox/storage/m.2) failed: lz4.read(//home/vmail/recover/mdbox/storage/m.2): corrupted lz4 chunk at 43551 (uid=42028, box=INBOX) dsync(recover): Error: unlink(��) failed: No such file or directory dsync(recover): Error: unlink(��//home/vmail/recover/mdbox/mailboxes/Drafts/dbox-Mails/dovecot.index.log.2) failed: No such file or directory dsync(recover): Error: unlink(Internal error occurred. Refer to server log for more information. [2015-11-25 16:12:01]) failed: No such file or directory dsync(recover): Error: unlink(��) failed: No such file or directory dsync(recover): Error: unlink(��) failed: File name too long dsync(recover): Error: unlink(Internal error occurred. Refer to server log for more information. [2015-11-25 16:12:01]) failed: No such file or directory Any help welcome

1 0

sieve is working/forwarding mail - but not for all users
by Götz Reinicke - IT Koordinator 25 Nov '15

25 Nov '15

Hi, we have dovecot-ee-2.2.18.2 and pigeonhole/managesieve 0.4.8 running for some time. Today some users informed us that they did not get mails from one project account forwarded to there personal accounts any more. This worked till one week ago and I cant think of any changes we made... The project account keeps a copy of received mails. I tried two different accounts to configure forwarding to internal and external mail addresses which is working. Question: Any hint or idea? How may I debug sieve forwarding? Thanks and regards . Götz

3 3

Disabling auth fallback to PAM
by martin f krafft 21 Nov '15

21 Nov '15

Hi folks, According to the wiki,¹ it's considered a feature of Dovecot and its ability to support multiple authentication sources that "if the password doesn't match in the first database, it checks the next one". ¹) http://wiki.dovecot.org/Authentication/MultipleDatabases I think it's great that Dovecot allows auth sources to be stacked like this, but I am not sold on the idea that the next database ought to be tried when a *password* does not match. Let me elaborate: If the first database has knowledge of a user, then it can (should) be considered authoritative, and if the provided password does not match, it's an authentication error right away. Only if the first source does not posess any knowledge about a given user, then should Dovecot proceed to query/check with the next database. Can this be configured somehow? If not, would it make sense to make this behaviour configurable? Thanks, -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ "the ships hung in the sky in much the same way that bricks don't." -- hitchhiker's guide to the galaxy spamtraps: madduck.bogus(a)madduck.net

2 2

doveadm import/sync with read-only source location
by Teodor Milkov 21 Nov '15

21 Nov '15

Hello, Currently doveadm tool can't be used with read-only source locations because it's trying to create lock files. I'd guess lock files shouldn't be necessary on a read-only media, because no one else is going to change anything while our doveadm instance is running anyway. Example: doveadm import -u test(a)example.com maildir:/backup.0/domains/example.com/postmaster/Maildir restored ALL doveadm(test(a)example.com): Error: open(/backup.0/domains/example.com/postmaster/Maildir/.Trash/dovecot-uidlis… failed: Read-only file system Perhaps this should be something like doveadm backup -R flag, but in the opposite direction – i.e. we have a [full] backup command, but no [partial] restore command right now. Or am I missing something? Best regards, Teodor

3 2

Shortcomings of Maildir++ layout
by Hadmut Danisch 21 Nov '15

21 Nov '15

Hi, I am trying to deal with the problem, that under regular mbox format a path can lead to either a file or a directory, but not both, while under IMAP, a path is usually both, a message folder containing subfolders. I found dovecot's approach to this problem, the Maildir++ layout described under http://wiki2.dovecot.org/MboxChildFolders , and was astonished to see, what problems come with Maildir++ and how complicated dovecot tries to deal with them. I wonder why dovecot is not using a much easier approach and just uses a configurable prefix and/or suffix like .dir or ____ for directory names, e.g. INBOX -> ~/mail/inbox abc -> ~/mail/abc abc/def -> ~/mail/abc.dir/def or ~/mail/___abc___/def which would make all that trouble with .subscriptions and .index much more easier. In my eyes the Maildir++ layout is overcomplicated because of the heap of workarounds to the self-created problems. regards Hadmut

4 5

bug in shared folders/lazy_expunge
by Infoo Matic 18 Nov '15

18 Nov '15

Hi list, I have a problem I was not able to resolve with my current setup (quite basic, the relevant part appended [1]). When userA shares a folder with userB then userB cannot access his mailbox. It's the same problem others have mentioned in *) http://www.dovecot.org/list/dovecot/2014-August/097449.html *) http://dovecot.org/pipermail/dovecot/2014-May/096355.html Could we please have a fix? TIA infoomatic [1] namespace { list = children location = mdbox:/var/lib/vmail/mail/%d/%%u prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace expunged { hidden = yes list = no location = mdbox:/var/lib/vmail/expunged/mail/%d/%u:INDEX=/var/lib/vmail/expunged/index/%d/%u:MAILBOXDIR=expunged prefix = .EXPUNGED/ subscriptions = no } namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / type = private }

1 0

usage of recipient delimiters
by fsantiago＠garbage-juice.com 15 Nov '15

15 Nov '15

Hello, I have a forum system which can potentially have users send email to it for posting purposes that arrives addressed to a particular user account with a recipient delimiter: user+1a2b3c4d5e6f(a)domain.com The forum server is supposed to check the "user" account mailbox's inbox for the messages and post them to the site automatically. This doesn't work right now because when the messages arrive, they get placed into a newly and automatically created folder named after the alpha numeric string after the + in the address (eg... 1a2b3c4d5e6f). I feel dovecot is likely doing this. any ideas why and how i can shut this off and just allow them to arrive and stay in the inbox? My version is: 2.2.10 My config is: # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 4.1.5-x86_64-linode61 x86_64 CentOS Linux release 7.1.1503 (Core) auth_master_user_separator = * auth_mechanisms = PLAIN LOGIN dict { acl = mysql:/etc/dovecot/dovecot-share-folder.conf quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf } first_valid_uid = 2000 last_valid_uid = 2000 listen = * [::] log_path = /var/log/dovecot.log mail_gid = 2000 mail_location = maildir:%Lh/Maildir/:INDEX=%Lh/Maildir/ mail_plugins = quota mailbox_alias acl mail_uid = 2000 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = no special_use = \Junk } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = / type = private } namespace { list = children location = maildir:%%Lh/Maildir/:INDEX=%%Lh/Maildir/Shared/%%u prefix = Shared/%%u/ separator = / subscriptions = yes type = shared } passdb { args = /etc/dovecot/dovecot-mysql.conf driver = sql } passdb { args = /etc/dovecot/dovecot-master-users driver = passwd-file master = yes } plugin { acl = vfile acl_shared_dict = proxy::acl auth_socket_path = /var/run/dovecot/auth-master mailbox_alias_new = Sent Messages mailbox_alias_new2 = Sent Items mailbox_alias_old = Sent mailbox_alias_old2 = Sent quota = dict:user::proxy::quotadict quota_rule = *:storage=1G quota_warning = storage=85%% quota-warning 85 %u quota_warning2 = storage=90%% quota-warning 90 %u quota_warning3 = storage=95%% quota-warning 95 %u sieve = %Lh/sieve/dovecot.sieve sieve_before = /var/vmail/sieve/dovecot.sieve sieve_dir = %Lh/sieve sieve_global_dir = /var/vmail/sieve } protocols = pop3 imap sieve lmtp service auth { unix_listener /var/spool/postfix/private/dovecot-auth { group = postfix mode = 0666 user = postfix } unix_listener auth-master { group = vmail mode = 0666 user = vmail } unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0660 user = vmail } } service imap-login { process_limit = 500 service_count = 1 } service lmtp { executable = lmtp -L inet_listener lmtp { address = 127.0.0.1 port = 24 } process_min_avail = 5 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } user = vmail } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } service pop3-login { service_count = 1 } service quota-warning { executable = script /usr/local/bin/dovecot-quota-warning.sh unix_listener quota-warning { group = vmail mode = 0660 user = vmail } } ssl = required ssl_cert = </etc/pki/tls/certs/iRedMail.crt ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_dh_parameters_length = 2048 ssl_key = </etc/pki/tls/private/iRedMail.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { args = /etc/dovecot/dovecot-mysql.conf driver = sql } protocol lda { auth_socket_path = /var/run/dovecot/auth-master lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes log_path = /var/log/dovecot-sieve.log mail_plugins = quota mailbox_alias acl sieve postmaster_address = root } protocol lmtp { info_log_path = /var/log/dovecot-lmtp.log lmtp_save_to_detail_mailbox = yes mail_plugins = quota sieve postmaster_address = postmaster recipient_delimiter = + } protocol imap { imap_client_workarounds = tb-extra-mailbox-sep mail_max_userip_connections = 20 mail_plugins = quota mailbox_alias acl imap_quota imap_acl } protocol pop3 { mail_max_userip_connections = 20 mail_plugins = quota mailbox_alias acl pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } Thank you. - Fabian S.

3 2