September 2014 - dovecot

Corrupted index cache after migrating from maildir to mdbox
by Tim 09 Jan '15

09 Jan '15

Hi guys, I'm running dovecot/postfix on my Debian Jessie maschine: # dovecot --version 2.2.13 doveconf -n output: http://p.bastelfreak.de/HsVwS/ today I migrated from maildir(located at /var/vmail) to mdbox with these commands: rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/ rm /home/vmail/*/*/Maildir -rf for i in $(doveadm user \*); do u=$(echo $i | cut -d@ -f1); d=$(echo $i |cut -d@ -f2); dsync -v -u $i mirror mdbox:/home/vmail/$d/$u/mdbox; done Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and did the following commands: service dovecot restart for i in $(doveadm user \*); do u=$(echo $i | cut -d@ -f1); d=$(echo $i |cut -d@ -f2); dsync -v -u $i mirror maildir:/var/vmail/$d/$u/Mail; done doveadm index -A all INBOX for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done unfortunately I'm getting the following two kinds of error messages for all mail boxes in my mail.log: Error: Corrupted index cache file /home/vmail/%domain/%user/mdbox/mailboxes/%maildir/dbox-Mails/dovecot.index.cache: Broken virtual size for mail UID 2564 and: Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108 ): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417 vs 6319 please let me know if you have any suggestions for fixing this issue or if you need more information. kind regards, Tim

6 19

sieve folder separator
by Thomas Koch 19 Oct '14

19 Oct '14

Hi, I'm moving from one email provider (me) to another (mailbox.org) and observed that the new provider uses "/" instead of "." as a folder separator. Now I need to adapt my sieve script. It's not too much work and I can even just define a variable at the beginning of my script: set "SEP" "/" But it would be great, if SIEVE would come with a predefined constant for the folder separator. Or does this already exists and I didn't find it? Thank you, Thomas Koch

2 1

[Dovecot] Disk Encryption
by Simon Brereton 17 Oct '14

17 Oct '14

Hi As I understand it email headers need to be unencrypted (otherwise DKIM doesn't work). From the MUA to either Postfix, or Dovecot the connection is (or can/should be) secured with TLS/SSL. What I would like to know is if it is possible to encrypt the mailstore? Postfix is using Dovecot for delivery so it's only Dovecot that would need to encrypt/decrypt the mailstore. Is this possible? Is there a terrible reason to do it even if it is possible? I realise that from MTA to MTA there's no guarantee of encryption (and in fact it's very unlikely unless keys have been exchanged), but my primary goal is supplement the physical security of the mail store of mails we already have or have sent. Mostly just idle curiosity as to what has been done, or what could be done. What is worth doing is a separate thread entirely. Thanks. Simon

11 28

replicating (manage)sieve - no symlink on replica?
by Philipp 17 Oct '14

17 Oct '14

2.2.13/0.4.2, both from source-tarball, no fancy options. tcp-replication between two nodes and following plugin parameters (same on both sides, except the ip-addr) plugin { sieve_before = /home/ssfn/etc/dovecot/sieve.default sieve_global_dir = /home/ssfn/etc/dovecot sieve = ~/.dovecot.sieve sieve_dir = ~/.sieve sieve_extensions = +vnd.dovecot.duplicate sieve_duplicate_period = 6h mail_replica = tcp:10.66.50.15:9993 } mail_location = sdbox:/home/ssfn/var/mail/boxes/%n mail_home = /home/ssfn/var/mail/boxes/%n Testbed: user getting first email delivered, the subdir %n/.sieve/tmp is automatically created as expected. User logs into managesieve on node1 and uploads a script. The %n/.sieve/managesieve.sieve and the symlink .dovecot.sieve -> .sieve/managesieve.sieve is created as expected. One node2 (via replica) happens: automatically create %n/.sieve/tmp automatically create/sync %n/.sieve/managesieve.sieve just NOT the symlink .dovecot.sieve -> .sieve/managesieve.sieve Logfiles for lmtp,doveadm,dsync-local dont indicate an error why it could not create the link or such, to me it looks like it wouldnt even try. mail-syncing works flawlessly (for delivery, just read about the expunge .. hum.) I tried to force a full-resync via doveadm replicator afterwards, to no change on the matter. Any input? Log from node2 after the moment managesieve saves everything correctly on node1: Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: auth input: user=ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: auth input: ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: Added userdb setting: plugin/=yes Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: Effective uid=993, gid=993, home=/home/ssfn/var/mail/boxes/ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=sdbox:/home/ssfn/var/mail/boxes/ssfntest:INDEX=/home/ssfn/var/mail/index/ssfntest:ALT=/home/ssfn/var/mail/alternate/ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: fs: root=/home/ssfn/var/mail/boxes/ssfntest, index=/home/ssfn/var/mail/index/ssfntest, indexpvt=, control=, inbox=, alt=/home/ssfn/var/mail/alternate/ssfntest Sep 11 15:20:12 dsync-server(ssfntest): Debug: Namespace : Using permissions from /home/ssfn/var/mail/boxes/ssfntest: mode=0700 gid=default Sep 11 15:20:12 dsync-server(ssfntest): Debug: doveadm-sieve: Iterating Sieve mailbox attributes Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve: Pigeonhole version 0.4.2 initializing Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: using active sieve script path: /home/ssfn/var/mail/boxes/ssfntest/.dovecot.sieve Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: using sieve script storage directory: /home/ssfn/var/mail/boxes/ssfntest/.sieve Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: permission lookup failed from /home/ssfn/var/mail/boxes/ssfntest/.sieve Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: created storage directory /home/ssfn/var/mail/boxes/ssfntest/.sieve/tmp Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: relative path to sieve storage in active link: .sieve/ Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve: script file /home/ssfn/var/mail/boxes/ssfntest/.sieve/managesieve.sieve not found Sep 11 15:20:12 dsync-server(ssfntest): Debug: doveadm-sieve: Value missing for key `vendor/vendor.dovecot/pvt/server/sieve/files/managesieve' (last change: 1970-01-01 01:00:00) Sep 11 15:20:13 dsync-server(ssfntest): Debug: doveadm-sieve: Assigned value for key `vendor/vendor.dovecot/pvt/server/sieve/files/managesieve' FS tree node2 after that: [..boxes]# find ssfntest -ls|grep -v mailb 24442 4 drwx------ 4 ssfnimp dlagimp 4096 Sep 11 15:20 ssfntest 21048 4 drwx------ 3 ssfnimp dlagimp 4096 Sep 11 15:20 ssfntest/.sieve 21052 4 drwx------ 2 ssfnimp dlagimp 4096 Sep 11 15:20 ssfntest/.sieve/tmp 21056 4 -rw------- 1 ssfnimp dlagimp 69 Sep 11 15:20 ssfntest/.sieve/managesieve.sieve 21016 0 lrwxrwxrwx 1 ssfnimp dlagimp 38 Sep 11 15:05 ssfntest/dbox-alt-root -> /home/dlag/var/mail/alternate/ssfntest FS tree node1 "before": [..boxes]# find ssfntest -ls|grep -v mailb 1835841 4 drwx------ 4 ssfnimp ssfnimp 4096 Sep 11 15:20 ssfntest 1835861 0 lrwxrwxrwx 1 ssfnimp ssfnimp 24 Sep 11 15:05 ssfntest/.dovecot.sieve -> .sieve/managesieve.sieve 1835854 4 -rw------- 1 ssfnimp ssfnimp 8 Sep 11 15:05 ssfntest/dovecot-uidvalidity 1835856 4 drwx------ 3 ssfnimp ssfnimp 4096 Sep 11 15:20 ssfntest/.sieve 1835857 4 drwx------ 2 ssfnimp ssfnimp 4096 Sep 11 15:20 ssfntest/.sieve/tmp 1835860 4 -rw------- 1 ssfnimp ssfnimp 69 Sep 11 15:20 ssfntest/.sieve/managesieve.sieve 1835846 0 lrwxrwxrwx 1 ssfnimp ssfnimp 38 Sep 11 15:05 ssfntest/dbox-alt-root -> /home/ssfn/var/mail/alternate/ssfntest

3 2

Sieve filter extracting wrong header value on certain mail messages
by gowen (dovecot) 09 Oct '14

09 Oct '14

# 2.2.12: /etc/dovecot/dovecot.conf # OS: Linux 3.15.4-x86_64-linode45 x86_64 openSUSE 12.3 (x86_64) Short version: 1) I filter out spam by using a dovecot sieve filter to act based on a header inserted by bogofilter 2) In some cases, sieve fails to correctly file based on this header 3) Testing with sieve-test indicates that it's testing the wrong header 4) Suggestions? Am I doing something wrong? Is there a problem with these headers that bends test-sieve out of shape? 5) Test files from below are also available via http://www.swynwyr.com/sieve-test/ Any help is appreciated!!! Long version with full details: Here is the simple sieve filter I'm using to test, and two different variants of an email which triggers this problem. I basically deleted things one by one from an original email (1) to get minimal testcases that triggered (2) or did not trigger (3) the problem. The filter: === begin test.sieve === require ["fileinto", "body", "regex"]; if header :contains "X-Bogosity" "Spam," { fileinto "SPAM.SPAM"; stop; } elsif header :contains "X-Bogosity" "Unsure," { fileinto "SPAM.UNSURE"; stop; } keep; stop; === end test.sieve === Here is an example email that will cause this filter to not function as expected: === begin message.2 === Return-Path: <alora(a)promosforusio.org> Subject: =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?= Subject: ÎHS KitÑhen ÎakÐµovÐµr GivÐµaway Subject: =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?= Subject: Subject: AHS Kitchen Makeover Giveaway X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3 Home Warranty Summer Giveaway === end message.2 === If I run test-sieve on this, we can clearly see in the trace log that when it's supposed to test for header "X-Bogosity" it in fact looks at one of the multiple "Subject" headers. Note that the result is storing in INBOX rather than SPAM.SPAM, and that the "matching value" for "X-Bogosity headers" is actually the contents of the final "Subject" header in the test on line 3. In the second test, on line 6, it uses a different and correct value for the same header! But too late to help as the first test was the one it should have had a match on. === begin session === $ sieve-test -t trace2.log -Tlevel=matching test.sieve message.2 Performed actions: * store message in folder: INBOX Implicit keep: (none) sieve-test(gowen): Info: final result: success $ cat trace2.log ## Started executing script 'test' 3: header test 3: starting `:contains' match with `i;ascii-casemap' comparator: 3: extracting `X-Bogosity' headers from message 3: matching value `AHS Kitchen Makeover Giveaway' 3: with key `Spam,' => 0 3: finishing match with result: not matched 3: jump if result is false 3: jumping to line 6 6: header test 6: starting `:contains' match with `i;ascii-casemap' comparator: 6: extracting `X-Bogosity' headers from message 6: matching value `Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3' 6: with key `Unsure,' => 0 6: finishing match with result: not matched 6: jump if result is false 6: jumping to line 11 11: keep action; store message in default mailbox 12: stop command; end all script execution ## Finished executing script 'test' $ === end session === Now lets remove the "Return-Path" from that message. (Of the original 26 headers, all but the 7 shown in message.2 could be removed and still have this message break the filter). This alters thing enough that sieve starts working as expected (in fact, removing any one of the 6 non-X-Bogosity headers will cause this to start working). === begin message.3 === Subject: =?utf-8?B?QUhTIEtpdGNoZW4gTWFrZW92ZXIgR2l2ZWF3YXkg?= Subject: ÎHS KitÑhen ÎakÐµovÐµr GivÐµaway Subject: =?ISO-8859-1?Q?AHS Kitchen Makeover Giveaway =A0=A0=A0?= Subject: Subject: AHS Kitchen Makeover Giveaway X-Bogosity: Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3 Home Warranty Summer Giveaway === end message.3 === Now when we run test-sieve it works as expected, filing into SPAM.SPAM and showing the correct matching value for line 3: === begin session === $ sieve-test -t trace3.log -Tlevel=matching test.sieve message.3 Performed actions: * store message in folder: SPAM.SPAM Implicit keep: (none) sieve-test(gowen): Info: final result: success $ cat trace3.log ## Started executing script 'test' 3: header test 3: starting `:contains' match with `i;ascii-casemap' comparator: 3: extracting `X-Bogosity' headers from message 3: matching value `Spam, tests=bogofilter, spamicity=1.000000, version=1.2.3' 3: with key `Spam,' => 1 3: finishing match with result: matched 3: jump if result is false 3: not jumping 4: fileinto action 4: store message in mailbox `SPAM.SPAM' 5: stop command; end all script execution ## Finished executing script 'test' $ === end session ===

3 3

LDA randomly failing to write email to disk
by Andy Dills 03 Oct '14

03 Oct '14

We're using 2.2.13 with pigeonhole 0.4.3, in a clustered environment (maildir on netapp, dual dovecot instances where each server is both a proxy and a backend). Every now and then (once a month per user, maybe?), users will see a blank email in their inbox. Investigating further, and we will see that the only information recorded in the maildir file for the message is the Return-Path, the Delivered-To, and the first Received line (the one generated by the local LDA via LMTP). For example, here is what I found in one such email today: Return-Path: <xxx(a)xecu.net> Delivered-To: <yyy(a)xecu.net> Received: from mail5.xecu.net ([10.0.1.85]) by mail2.xecu.net (Dovecot) with LMTP idX86eBjgaI1RdyQAA3SxDBg for <yyy(a)xecu.net>; Wed, 24 Sep 2014 15:24:20 -0400 Everything past that is lost, as if the either the LDA on mail2 exited early or was never sent the information from the dovecot instance on mail5. Here is a little more detail from the header of the same email, different recipient (all other recipients received the email properly, only one failed to receive properly): Return-Path: <xxx(a)xecu.net> Delivered-To: <yyz(a)xecu.net> Received: from mail5.xecu.net ([10.0.1.85]) by mail1.xecu.net (Dovecot) with LMTP id KPh8ItMXI1StBAAA/c3zFg for <yyz(a)xecu.net>; Wed, 24 Sep 2014 15:24:20 -0400 Received: from mail5.xecu.net by mail5.xecu.net (Dovecot) with LMTP id QsUBFoQZI1RjhgAAXyr1JQ ; Wed, 24 Sep 2014 15:24:20 -0400 Received: from mail5.xecu.net (localhost [127.0.0.1]) by mail5.xecu.net (Postfix) with ESMTP id 30AAE323BB18; Wed, 24 Sep 2014 15:24:20 -0400 (EDT) ... This is what I see in the logs of mail2, which is where the failure happened: Sep 24 15:24:20 mail2 dovecot: lmtp(51549, yyy(a)xecu.net) X86eBjgaI1RdyQAA3SxDBg: sieve: msgid=unspecified: stored mail into mailbox 'INBOX' When I look at the logs of mail1, which is where the successful delivery happened: Sep 24 15:24:20 mail1 dovecot: lmtp(1197, yyz(a)xecu.net) KPh8ItMXI1StBAAA/c3zFg: sieve: msgid=<20140924192412.1435.qmail(a)xxx.xecu.net>: stored mail into mailbox 'INBOX' Nothing of note seems to be logged on mail5 (only a message from postfix, nothing from the proxy side of the lmtp). I do notice, when I check for the PID of 51549 in the logs, all of its other transactions seem to register with proper msgids and were delivered fine. Also, I notice plenty of other messages that have the msgid=unspecified error, but which were delivered with no problems and not truncated, so I'm suspecting what may be happening is that somehow the backend instance is not receiving the actual data portion, and only getting the envelope from proxy instance. How do I approach debugging this? It's very infrequent, but yet quite annoying. Seems to have started since we upgraded to 2.2.13 (from an older 2.1 build) earlier this year. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---

3 3

GETMETADATA UTF8 encoding of folder names instead of UTF7
by Bartłomiej Nogaś 03 Oct '14

03 Oct '14

Hello, I have dovecot 2.2.13 instance and I found that (at least for polish language) folder names in GETMETADATA command response are encoded in UTF8. Shouldn't it be converted to UTF7? Both SETMETADATA and GETMETADATA arguments have to be encoded in UTF7, the problem(?) is related only to server response example: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready. . login user pass . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE METADATA QUOTA ACL RIGHTS=texk] Logged in . GETMETADATA "INBOX/&AVs-olaris" (/private/vendor/kolab/folder-type /shared/vendor/kolab/folder-type) * METADATA {14} INBOX/śolaris (/private/vendor/kolab/folder-type NIL /shared/vendor/kolab/folder-type {4} mail) . OK Getmetadata completed. "INBOX/śolaris" - UTF8 Best Regards,

2 1

home from SQL
by Ralf Hildebrandt 03 Oct '14

03 Oct '14

Currently I'm using user_query = SELECT 1000 AS uid, 1000 AS gid, '/srv/vmail/%2.256Hu/%Lu' AS home, ... so I'm hashing based on %u (basically). But in my SQL db I have a "unique_identifier" field, which never changes, even when the user is changing his/her email address (due to marriage or the like). What I'd really like to do is to use %u to find the value of the unique_identifier field, hash THAT value and use "AS home". But how? That way I can rename users without shuffling directories around the filesystem. -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

3 2

[Dovecot] mdbox-files over 2 MB
by Hardy Flor 01 Oct '14

01 Oct '14

Hello, I received an e-mail, this includes 9 forwarding, each with a file. These files are not removed from the e-mail and stored in the attachments-directory, although they are larger than 16 KB. For all files the "Content-Transfer-Encoding: base64" is set. The email is stored in completely in "m.00000034" ------------------------------------- doveconf: # 2.2.12: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 ext4 mail_attachment_dir = /var/mail/attachments mail_attachment_hash = %{md5} mail_attachment_min_size = 16 k mail_location = mdbox:/var/mail/user/%n mdbox_rotate_interval = 1 weeks mdbox_rotate_size = 2 M ------------------------------------- directory: root@xxx:/var/mail/user/xxx/storage# ls -al ... -rw------- 1 vmail mail 14433784 Mär 16 13:51 m.00000034 ...

6 8

Master user and invalid credentials dovecot-2.2.13
by Götz Reinicke - IT Koordinator 30 Sep '14

30 Sep '14

Hi, I followed the docs from the dovecot wiki (http://wiki2.dovecot.org/Authentication/MasterUsers) and still have some problem: 1. uncommented "!include auth-ldap.conf.ext" in 10-auth.conf 2. htpasswd -b -c -s .... to create user/password for a masteruseruser 3. checked auth-master.conf.ext When I try to test the login, I always get an "invalid credentials" in the logs with auth_debug=yes set. dovecot can read /etc/dovecot/master-users The users are in ldap. The example wiki shows the passwords {SHA1}...., in my password file they are {SHA}.... The "master user" posting in september gave some clues what might be wrong, but did not help to solve my problem. Any suggestions are welcome! Thanks and regards. Götz -- Götz Reinicke IT-Koordinator Tel. +49 7141 969 82 420 E-Mail goetz.reinicke(a)filmakademie.de Filmakademie Baden-Württemberg GmbH Akademiehof 10 71638 Ludwigsburg www.filmakademie.de Eintragung Amtsgericht Stuttgart HRB 205016 Vorsitzender des Aufsichtsrats: Jürgen Walter MdL Staatssekretär im Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg Geschäftsführer: Prof. Thomas Schadt

4 4