February 2017 - dovecot

Query about list policy?
by Andy R 01 Mar '17

01 Mar '17

Greetings for a second time today, As a quick query, what is the policy about linking to user-scripts and small addons ? I've used a couple of perl scripts to help me with dspam training from dovecot-antispam and imapsieve triggering for sorting and feeding messages to dspam in the background. They are most likely not the greatest but seem to work at least. I've posted them to github, but just wanted to ask whether it was an issue if i linked them here, or if they would need any review before posting or any other answer? Many thanks, and to Aki Tuomi and the dovecot team also. Andy R.

1 0

Quota usage value shows 140% of actual disk usage
by Umut Erol Kaçar 28 Feb '17

28 Feb '17

Hello everyone, Our server has these installed: dovecot-2.0.21-2.el6.x86_64 dovecot-pigeonhole-2.0.21-2.el6.x86_64 dovecot-mysql-2.0.21-2.el6.x86_64 ...and has been running for quite a long time, with several hundred domains and thousands of accounts on it. My colleagues reported that it's been showing quota usage values that are more than actual disk usage, so I investigated and found that Dovecot shows a quota usage value roughly around 140% of actual disk usage. It's also valid on newly created accounts. My test account for example: doveadm quota get -u test(a)example.local Quota name Type Value Limit % User quota STORAGE 4359 512000 0 User quota MESSAGE 7 - 0 du -sc /home/vmail/example.local/test/Maildir/{*,.[!.]*} 1044 /home/vmail/example.local/test/Maildir/cur 28 /home/vmail/example.local/test/Maildir/dovecot.index.cache 8 /home/vmail/example.local/test/Maildir/dovecot.index.log 4 /home/vmail/example.local/test/Maildir/dovecot.mailbox.log 4 /home/vmail/example.local/test/Maildir/dovecot-uidlist 4 /home/vmail/example.local/test/Maildir/dovecot-uidvalidity 0 /home/vmail/example.local/test/Maildir/dovecot-uidvalidity.56a4dc8e 4 /home/vmail/example.local/test/Maildir/new 4 /home/vmail/example.local/test/Maildir/subscriptions 4 /home/vmail/example.local/test/Maildir/tmp 24 /home/vmail/example.local/test/Maildir/.Junk 1932 /home/vmail/example.local/test/Maildir/.Sent 44 /home/vmail/example.local/test/Maildir/.Trash 3104 total 4359/3104=1,40431701 So it shows roughly around 1,4*actualDiskUsage. The ratio is mostly the same for almost all the other accounts. It can vary between like 1,3-1,6. So, the gap gets insane when more disk space is used, say like with 2GB disk usage, Dovecot thinks 3,5GB quota is used... dovecot quota recalc does not fix the issue, it only sets the same value again (I've checked with tcpdump and saw the query with the same quota usage value). The method is Dictionary quota with SQL. I'm attaching the dovecot -n output with some other config files. I've tried setting messages and bytes value to -1 on the MariaDB database to force recalculation. But as soon as I run doveadm quota recalc, it gets the same wrong value again. What can I do to fix this? Thanks in advance.

3 3

Index queue
by azurit＠pobox.sk 28 Feb '17

28 Feb '17

Hi, can i, somehow, list mailboxes which are to be indexed by indexer-worker (=index queue?) ? How can i know what part of all mailboxes was indexed so far? Are there any statistics about Solr data dir size based on emails amount? For example, we have about 5TB of emails, what should i except about index size in Solr? Thank you. azur

1 0

Re: Solr 6.4.1 config
by Daniel Miller 28 Feb '17

28 Feb '17

Since I was recently asked off-list - thought I'd make this available if it will help anyone else. Attached is my (I hope) working 6.4.1 config. Also some instructions: First, you have to decide if you're running in standalone or cloud mode. Standalone is simpler, but I seem to be having better luck running in cloud mode - even though I'm only using a single server/core/shard/collection/whatever. I've attached my config. Unzip this somewhere convenient - I will assume "/tmp/solr-dovecot". You may want to tweak it - this is adapted from my version 4 config and is set for English. If you look in solrbasedir/example/files/conf/lang or /solrbasedir/server/solr/configsets/basic_configs/conf/lang you'll find alternatives for "stopwords.txt" - I simply copied the "stopwords_en.txt" file directly to "conf/stopwords.txt". Assuming you extracted the Solr distribution to a base folder, such as /opt/solr-6.4.1, change to that folder. Then execute (again, /tmp/solr-dovecot is the path you extracted my config into): bin/solr create_collection -c dovecot -d /tmp/solr-dovecot That should create a minimal cloud instance. Then start it: bin/solr start -c And hopefully...you'll have a running Solr server. Assuming that's the case - since you've now created a "named" collection you have to specify that for Dovecot. So the fts_solr parameter in 90-plugin.conf needs to be something like: fts_solr = break-imap-search url=http://ser.ver.ip:8983/solr/dovecot/ Restart Dovecot...and with luck you're searching now. Hope this helps somebody! Daniel On 2/10/2017 3:15 PM, Daniel Miller wrote: > 2.2.18 > > On 2/9/2017 11:13 PM, Aki Tuomi wrote: >> What dovecot version are you using? >> >> >> On 10.02.2017 09:12, Aki Tuomi wrote: >>> Yeah, thanks. >>> >>> It seems that there indeed is content-type header (there should not >>> be). >>> We'll look into it. >>> >>> Aki >>> >>> On 10.02.2017 01:44, Daniel Miller wrote: >>>> Does this work (pcap attached)? >>>> >>>> Daniel >>>> >>>> On 2/8/2017 10:57 PM, Aki Tuomi wrote: >>>>> On 09.02.2017 07:54, Daniel Miller wrote: >>>>>> I've been running Solr for a while (4.10.3) - wanted to make the >>>>>> jump >>>>>> to the latest & greatest. I installed 6.4.1, copied over my >>>>>> schema.xml - and after a couple false starts where I needed to tweak >>>>>> it work with the new version...it works! I did not copy the >>>>>> database, >>>>>> started from scratch, and executed a "doveadm fts rescan -A". >>>>>> But... >>>>>> >>>>>> Judging solely from at least one client - it's fine. But looking in >>>>>> the logs I see: >>>>>> 1. The first scan of a mailbox dovecot's error log gives: >>>>>> dovecot: imap(dmiller(a)amfes.com): Error: fts_solr: Lookup >>>>>> failed: >>>>>> Bad Request >>>>>> >>>>>> 2. Subsequent scans do not appear to generate any dovecot error >>>>>> logs >>>>>> - but I'm not certain. Each new mailbox/subfolder scanned will each >>>>>> have one error on the initial scan. >>>>>> >>>>>> 3. Solr's log gives me the following - on every search. >>>>>> 2017-02-09 05:50:12.412 ERROR (qtp205125520-15) [ x:dovecot] >>>>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Bad >>>>>> contentType for search handler :text/xml >>>>>> request={q=from:"test"+OR+to:"test"+OR+cc:"test"+OR+subject:"test"+OR+body:"test"&fl=uid,score&sort=uid+asc&fq=%2Bbox:c1af150abfc9df4d7f7a00003bc41c5f+%2Buser:"dmiller(a)amfes.com"&rows=67135} >>>>>> >>>>>> >>>>> Hi! >>>>> >>>>> can you please use tcpdump or wireshark to capture the actual HTTP >>>>> request causing this exception? >>>>> >>>>> Aki

1 0

Archive key for the Dovecot Automatic Debian Package Archive (Xi) updated
by Stephan Bosch 28 Feb '17

28 Feb '17

Hi, I've updated the archive key for the Xi archive. So, when updating, you will initially get a key error. To fix this, either upgrade the debian-dovecot-auto-keyring package (preferred), or update your key manually from the archive.key located in the repository root. Regards, Stephan.

1 0

indexer-worker assert in v2.2.28 (fts-lucene)
by Daniel J. Luke 27 Feb '17

27 Feb '17

I upgraded to 2.2.28 and started seeing this logged: indexer-worker(dluke): Panic: file mailbox-list.c: line 1158 (mailbox_list_try_mkdir_root): assertion failed: (strncmp(root_dir, path, strlen(root_dir)) == 0) indexer-worker(dluke): Error: Raw backtrace: 2 libdovecot.0.dylib 0x000000010ec63d24 default_fatal_finish + 36 -> 3 libdovecot.0.dylib 0x000000010ec64b5b i_internal_fatal_handler + 43 -> 4 libdovecot.0.dylib 0x000000010ec64039 i_panic + 169 -> 5 libdovecot-storage.0.dylib 0x000000010eac0950 mailbox_list_try_mkdir_root + 1248 -> 6 libdovecot-storage.0.dylib 0x000000010eac0a39 mailbox_list_mkdir_root + 25 -> 7 lib21_fts_lucene_plugin.so 0x0000000110adc949 fts_backend_lucene_update_set_build_key + 73 -> 8 lib20_fts_plugin.so 0x000000010ee736cf fts_backend_update_set_build_key + 79 -> 9 lib20_fts_plugin.so 0x000000010ee74317 fts_build_mail + 599 -> 10 lib20_fts_plugin.so 0x000000010ee7981a fts_mail_precache + 794 -> 11 libdovecot-storage.0.dylib 0x000000010eaa71e9 mail_precache + 25 -> 12 indexer-worker 0x000000010ea9e503 master_connection_input + 1523 -> 13 libdovecot.0.dylib 0x000000010ec78b89 io_loop_call_io + 89 -> 14 libdovecot.0.dylib 0x000000010ec7a96d io_loop_handler_run_internal + 269 -> 15 libdovecot.0.dylib 0x000000010ec7907f io_loop_handler_run + 303 -> 16 libdovecot.0.dylib 0x000000010ec78e58 io_loop_run + 88 -> 17 libdovecot.0.dylib 0x000000010ec03458 master_service_run + 24 -> 18 indexer-worker 0x000000010ea9ddb4 main + 340 -> 19 libdyld.dylib 0x00007fffba7df255 start + 1 Anyone else? -- Daniel J. Luke

1 0

Released Pigeonhole v0.4.17 for Dovecot v2.2.28.
by Stephan Bosch 27 Feb '17

27 Feb '17

Hello Dovecot users, Here's the definitive 0.4.17 release. There were no changes since the release candidate. Changelog v0.4.17: - LDA Sieve plugin: Fixed handling of an early explicit keep during multiscript execution. Action side-effects and the message snapshot would be lost at the final stage where the implicit keep is evaluated. This could result in the IMAP flags assigned to the message to be forgotten or that headers modified by the "editheader" extension would revert to their original state. - file script storage: Amended the up-to-date time stamp comparison for on-disk binaries to include nanoseconds. This will fix problems occurring when both binary and script are saved within the same second. This fix is ineffective on older systems that have no support for nanoseconds in stat() timestamps, which should be pretty rare nowadays. - file script storage: Improve saving and listing permission error to include more details. - imapsieve plugin: Make sure "INBOX" is upper case in static mailbox rules. Otherwise, the mailbox name would never match, since matching is performed case-sensitively and Dovecot only returns the upper-cased "INBOX". - imapsieve plugin: Fixed assert failure occurring when used with virtual mailboxes. - doveadm sieve plugin: Fixed crash when setting Sieve script via attribute's string value. The release is available as follows: http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.17.ta… http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.17.ta… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this release and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

1 0

Re: Sieve removeflag Action
by Stephan Bosch 26 Feb '17

26 Feb '17

Op 2/26/2017 om 4:02 PM schreef Thomas Leuxner: >> Am 26.02.2017 um 15:52 schrieb Thomas Leuxner <tlx(a)leuxner.net>: >> >>> So, I cannot reproduce your problem. >>> >>> Could you try this at your end? >> Hi Stephan, >> >> Tried with: >> >> $ /usr/lib/dovecot/dovecot-lda -d tlx(a)leuxner.net -p ./message.eml >> >> From: User <user(a)example.com> >> To: lists(a)example.org >> Subject: Frop! >> List-Id: <debian-security-announce.lists.debian.org> >> >> Frop! >> >> It goes right to the INBOX with no flags: >> > This time *with sieve* enabled in mail_plugins for LDA. (I’m using LMTP). The mail gets processed correctly. Does it make a difference for LMTP from the code? That should not matter normally. I tested this at my end, but I saw no different behavior. I created the following file in ~/message-lmtp.eml (substitute MAIL FROM/RCPT TO with whatever you need): LHLO frop MAIL FROM:<stephan(a)rename-it.nl> RCPT TO:<stephan> DATA From: User <user(a)example.com> To: stephan(a)rename-it.nl Subject: Frop! List-Id: <debian-security-announce.lists.debian.org> Frop! . Then I invoked LMTP through its unix socket (using socat tool): stephan@drieka:~$ cat ~/message-lmtp.eml | socat gopen:/var/run/dovecot/lmtp stdio 220 drieka.tds Dovecot ready. 250-drieka.tds 250-8BITMIME 250-ENHANCEDSTATUSCODES 250 PIPELINING 250 2.1.0 OK 250 2.1.5 OK 354 OK 250 2.0.0 <stephan> 1D/3FPsfs1ge/QAASZWA2w Saved Evaluating the results shows that there is still no problem here, since the last delivery is fine (I initially forgot to enable Sieve for LMTP): stephan@drieka:~$ doveadm -f table fetch "uid flags" mailbox "INBOX" uid flags 14 \Flagged \Recent $MailFlagBit1 15 \Flagged \Recent $MailFlagBit1 16 \Flagged \Recent $MailFlagBit1 17 \Flagged \Recent $MailFlagBit1 18 \Flagged \Recent $MailFlagBit1 19 \Recent 20 \Flagged \Recent $MailFlagBit1 stephan@drieka:~$ doveadm -f table fetch "uid flags" mailbox ":public/Newsletters/Debian/Security" uid flags 1 \Recent 2 \Recent 3 \Recent 4 \Recent 5 \Recent 6 \Recent Regards, Stephan.

2 1

Re: Dict quota calculation errors "remote disconnected"/"broken pipe" on 2.22.
by ygrishin-lists＠mail2.ca 26 Feb '17

26 Feb '17

Solved the problem, reporting back to the community. /etc/dovecot/dovecot-dict-sql-user.conf had been lacking dovecot group permissions. It was 700/root:root. However why it wasn't reported by Dovecot explicitly in the log -- the greatest mystery to me. Now, after deleting dovecot and all its packages via 'apt' and installing afresh makes "lacking permissions for the file" entry to appear in the log. I can confirm that Dict-quota works perfectly well with Dovecot 2.2.22.

1 0

Sieve removeflag Action
by Thomas Leuxner 26 Feb '17

26 Feb '17

Hi, I recently noticed that some of my automatically processed mails do no longer show flags with current Dovecot builds. This used to work before, not sure what broke it or whether I used it in the wrong way. When the removeflag line is dropped *both* messages get flagged in the client, one in INBOX and one in Trash. With removeflag in place, *both* messages are unflagged, the one in INBOX and the one in Trash. This used to work differently before if memory doesn't fail me. #Test if address :is "From" "user(a)example.com" { addflag "\\Flagged $MailFlagBit1"; keep; removeflag "\\Flagged $MailFlagBit1"; fileinto "Trash"; } # 2.2.devel (01867a4): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.devel (3c071a4) # OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.6 Regards Thomas

4 18