February 2017 - dovecot

dovecot-lda crash after upgrade to 2.2.28
by Max Kostikov 01 Mar '17

01 Mar '17

Hi! I posted this problem few days ago in FreeBSD bugtracker https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217364 So, the problem is in crash of dovecot-lda client while local mailbox delivery with this message in log > dovecot: lda(my@email): Panic: file mail-namespace.c: line 709 > (mail_namespace_find): assertion failed: (ns != NULL) My Dovecot config stills the same from previous 2.2.27 installation to I believe that problem in new Dovecot code related namespace. System: > # uname -v > FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC Options: > root@beta:/usr/ports/mail/dovecot2 # make showconfig | grep =on > DOCS=on: Build and/or install documentation > EXAMPLES=on: Build and/or install examples > KQUEUE=on: kqueue(2) support > LIBWRAP=on: TCP wrapper support > LZ4=on: LZ4 compression support > MYSQL=on: MySQL database support > GSSAPI_BASE=on: Use GSSAPI from base Config: # doveconf -n # 2.2.27 (c0f36b0): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: FreeBSD 11.0-RELEASE-p8 amd64 ufs auth_mechanisms = plain login default_login_user = dovecot dict { sqluserquota = mysql:/usr/local/etc/dovecot/dovecot-dict-sql-user.conf } disable_plaintext_auth = no first_valid_gid = 0 first_valid_uid = 25 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_access_sockets = tcpwrap login_greeting = Dovecot ready! login_log_format_elements = user=<%u> method=%m rip=%r lip=%l %c mail_access_groups = mail mail_gid = 6 mail_location = maildir:/var/mail/%d/%n mail_plugins = acl quota trash mail_privileged_group = mail mail_uid = 26 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { location = maildir:/var/mail/%d/.public:INDEXPVT=/var/mail/%d/%n/public prefix = public. separator = . subscriptions = no type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = separator = . type = private } passdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } plugin { acl = vfile::cache_secs=300 antispam_backend = mailtrain antispam_mail_notspam = %u-revoke antispam_mail_sendmail = /usr/local/etc/dovecot/move-cmd.sh antispam_mail_sendmail_args = antispam_mail_spam = %u-report antispam_spam = Junk antispam_trash = Trash quota = dict:user_quota::proxy::sqluserquota quota_grace = 10%% quota_rule2 = Trash:storage=+10%% quota_rule3 = Junk:storage=+10%% quota_warning = storage=100%% quota-exceeded 100 %u quota_warning2 = storage=95%% quota-warning 95 %u quota_warning3 = storage=90%% quota-warning 90 %u quota_warning4 = storage=75%% quota-warning 75 %u sieve = ~/dovecot.sieve sieve_before = /usr/local/etc/dovecot/default.sieve sieve_dir = ~/sieve sieve_global_path = /usr/local/etc/dovecot/default.sieve trash = /usr/local/etc/dovecot/dovecot-trash.conf } protocols = imap pop3 lmtp sieve service auth { unix_listener auth-client { mode = 0660 user = mailnull } unix_listener auth-master { mode = 0660 user = mailnull } user = root } service dict { unix_listener dict { mode = 0660 user = mailnull } } service imap-login { chroot = login client_limit = 64 executable = /usr/local/libexec/dovecot/imap-login inet_listener imap { port = 143 } inet_listener imaps { port = 993 } process_limit = 32 process_min_avail = 8 service_count = 1 user = $default_login_user vsz_limit = 64 M } service imap { executable = /usr/local/libexec/dovecot/imap } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 1 service_count = 1 user = $default_login_user vsz_limit = 64 M } service managesieve { process_limit = 10 } service pop3-login { chroot = login client_limit = 64 executable = /usr/local/libexec/dovecot/pop3-login inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 } process_limit = 32 process_min_avail = 8 service_count = 1 user = $default_login_user vsz_limit = 64 M } service pop3 { executable = /usr/local/libexec/dovecot/pop3 } service quota-warning { executable = script /usr/local/etc/dovecot/quota_warning.sh unix_listener quota-warning { mode = 0660 user = mailnull } } service tcpwrap { unix_listener login/tcpwrap { mode = 0600 user = $default_login_user } } ssl_ca = </usr/local/etc/letsencrypt/live/peek.ru/chain.pem ssl_cert = </usr/local/etc/letsencrypt/live/peek.ru/cert.pem ssl_cipher_list = ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS ssl_dh_parameters_length = 2048 ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { args = /usr/local/etc/dovecot/dovecot-sql.conf driver = sql } protocol sieve { managesieve_implementation_string = dovecot managesieve_max_line_length = 65536 } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_plugins = acl quota trash antispam imap_acl imap_quota } protocol lmtp { mail_plugins = acl quota trash sieve } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master mail_plugins = acl quota trash sieve postmaster_address = postmaster(a)peek.ru sendmail_path = /usr/sbin/sendmail } local_name *.domain.ru { ssl_ca = </usr/local/etc/letsencrypt/live/domain.ru/chain.pem ssl_cert = </usr/local/etc/letsencrypt/live/domain.ru/cert.pem ssl_key = # hidden, use -P to show it } -- With best regards, Max Kostikov BBM: 24CA5DF8 | W: https://kostikov.co

3 11

Replacement for antispam plugin
by Aki Tuomi 01 Mar '17

01 Mar '17

Hi! Since antispam plugin is deprecated and we would really prefer people not to use it, we wrote instructions on how to replace it with IMAPSieve. Comments and suggestions are most welcome. https://wiki.dovecot.org/HowTo/AntispamWithSieve --- Aki Tuomi Dovecot oy

18 42

Query about list policy?
by Andy R 28 Feb '17

28 Feb '17

Greetings for a second time today, As a quick query, what is the policy about linking to user-scripts and small addons ? I've used a couple of perl scripts to help me with dspam training from dovecot-antispam and imapsieve triggering for sorting and feeding messages to dspam in the background. They are most likely not the greatest but seem to work at least. I've posted them to github, but just wanted to ask whether it was an issue if i linked them here, or if they would need any review before posting or any other answer? Many thanks, and to Aki Tuomi and the dovecot team also. Andy R.

1 0

Quota usage value shows 140% of actual disk usage
by Umut Erol Kaçar 28 Feb '17

28 Feb '17

Hello everyone, Our server has these installed: dovecot-2.0.21-2.el6.x86_64 dovecot-pigeonhole-2.0.21-2.el6.x86_64 dovecot-mysql-2.0.21-2.el6.x86_64 ...and has been running for quite a long time, with several hundred domains and thousands of accounts on it. My colleagues reported that it's been showing quota usage values that are more than actual disk usage, so I investigated and found that Dovecot shows a quota usage value roughly around 140% of actual disk usage. It's also valid on newly created accounts. My test account for example: doveadm quota get -u test(a)example.local Quota name Type Value Limit % User quota STORAGE 4359 512000 0 User quota MESSAGE 7 - 0 du -sc /home/vmail/example.local/test/Maildir/{*,.[!.]*} 1044 /home/vmail/example.local/test/Maildir/cur 28 /home/vmail/example.local/test/Maildir/dovecot.index.cache 8 /home/vmail/example.local/test/Maildir/dovecot.index.log 4 /home/vmail/example.local/test/Maildir/dovecot.mailbox.log 4 /home/vmail/example.local/test/Maildir/dovecot-uidlist 4 /home/vmail/example.local/test/Maildir/dovecot-uidvalidity 0 /home/vmail/example.local/test/Maildir/dovecot-uidvalidity.56a4dc8e 4 /home/vmail/example.local/test/Maildir/new 4 /home/vmail/example.local/test/Maildir/subscriptions 4 /home/vmail/example.local/test/Maildir/tmp 24 /home/vmail/example.local/test/Maildir/.Junk 1932 /home/vmail/example.local/test/Maildir/.Sent 44 /home/vmail/example.local/test/Maildir/.Trash 3104 total 4359/3104=1,40431701 So it shows roughly around 1,4*actualDiskUsage. The ratio is mostly the same for almost all the other accounts. It can vary between like 1,3-1,6. So, the gap gets insane when more disk space is used, say like with 2GB disk usage, Dovecot thinks 3,5GB quota is used... dovecot quota recalc does not fix the issue, it only sets the same value again (I've checked with tcpdump and saw the query with the same quota usage value). The method is Dictionary quota with SQL. I'm attaching the dovecot -n output with some other config files. I've tried setting messages and bytes value to -1 on the MariaDB database to force recalculation. But as soon as I run doveadm quota recalc, it gets the same wrong value again. What can I do to fix this? Thanks in advance.

3 3

Index queue
by azurit＠pobox.sk 28 Feb '17

28 Feb '17

Hi, can i, somehow, list mailboxes which are to be indexed by indexer-worker (=index queue?) ? How can i know what part of all mailboxes was indexed so far? Are there any statistics about Solr data dir size based on emails amount? For example, we have about 5TB of emails, what should i except about index size in Solr? Thank you. azur

1 0

Re: Solr 6.4.1 config
by Daniel Miller 28 Feb '17

28 Feb '17

Since I was recently asked off-list - thought I'd make this available if it will help anyone else. Attached is my (I hope) working 6.4.1 config. Also some instructions: First, you have to decide if you're running in standalone or cloud mode. Standalone is simpler, but I seem to be having better luck running in cloud mode - even though I'm only using a single server/core/shard/collection/whatever. I've attached my config. Unzip this somewhere convenient - I will assume "/tmp/solr-dovecot". You may want to tweak it - this is adapted from my version 4 config and is set for English. If you look in solrbasedir/example/files/conf/lang or /solrbasedir/server/solr/configsets/basic_configs/conf/lang you'll find alternatives for "stopwords.txt" - I simply copied the "stopwords_en.txt" file directly to "conf/stopwords.txt". Assuming you extracted the Solr distribution to a base folder, such as /opt/solr-6.4.1, change to that folder. Then execute (again, /tmp/solr-dovecot is the path you extracted my config into): bin/solr create_collection -c dovecot -d /tmp/solr-dovecot That should create a minimal cloud instance. Then start it: bin/solr start -c And hopefully...you'll have a running Solr server. Assuming that's the case - since you've now created a "named" collection you have to specify that for Dovecot. So the fts_solr parameter in 90-plugin.conf needs to be something like: fts_solr = break-imap-search url=http://ser.ver.ip:8983/solr/dovecot/ Restart Dovecot...and with luck you're searching now. Hope this helps somebody! Daniel On 2/10/2017 3:15 PM, Daniel Miller wrote: > 2.2.18 > > On 2/9/2017 11:13 PM, Aki Tuomi wrote: >> What dovecot version are you using? >> >> >> On 10.02.2017 09:12, Aki Tuomi wrote: >>> Yeah, thanks. >>> >>> It seems that there indeed is content-type header (there should not >>> be). >>> We'll look into it. >>> >>> Aki >>> >>> On 10.02.2017 01:44, Daniel Miller wrote: >>>> Does this work (pcap attached)? >>>> >>>> Daniel >>>> >>>> On 2/8/2017 10:57 PM, Aki Tuomi wrote: >>>>> On 09.02.2017 07:54, Daniel Miller wrote: >>>>>> I've been running Solr for a while (4.10.3) - wanted to make the >>>>>> jump >>>>>> to the latest & greatest. I installed 6.4.1, copied over my >>>>>> schema.xml - and after a couple false starts where I needed to tweak >>>>>> it work with the new version...it works! I did not copy the >>>>>> database, >>>>>> started from scratch, and executed a "doveadm fts rescan -A". >>>>>> But... >>>>>> >>>>>> Judging solely from at least one client - it's fine. But looking in >>>>>> the logs I see: >>>>>> 1. The first scan of a mailbox dovecot's error log gives: >>>>>> dovecot: imap(dmiller(a)amfes.com): Error: fts_solr: Lookup >>>>>> failed: >>>>>> Bad Request >>>>>> >>>>>> 2. Subsequent scans do not appear to generate any dovecot error >>>>>> logs >>>>>> - but I'm not certain. Each new mailbox/subfolder scanned will each >>>>>> have one error on the initial scan. >>>>>> >>>>>> 3. Solr's log gives me the following - on every search. >>>>>> 2017-02-09 05:50:12.412 ERROR (qtp205125520-15) [ x:dovecot] >>>>>> o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Bad >>>>>> contentType for search handler :text/xml >>>>>> request={q=from:"test"+OR+to:"test"+OR+cc:"test"+OR+subject:"test"+OR+body:"test"&fl=uid,score&sort=uid+asc&fq=%2Bbox:c1af150abfc9df4d7f7a00003bc41c5f+%2Buser:"dmiller(a)amfes.com"&rows=67135} >>>>>> >>>>>> >>>>> Hi! >>>>> >>>>> can you please use tcpdump or wireshark to capture the actual HTTP >>>>> request causing this exception? >>>>> >>>>> Aki

1 0

Archive key for the Dovecot Automatic Debian Package Archive (Xi) updated
by Stephan Bosch 27 Feb '17

27 Feb '17

Hi, I've updated the archive key for the Xi archive. So, when updating, you will initially get a key error. To fix this, either upgrade the debian-dovecot-auto-keyring package (preferred), or update your key manually from the archive.key located in the repository root. Regards, Stephan.

1 0

indexer-worker assert in v2.2.28 (fts-lucene)
by Daniel J. Luke 27 Feb '17

27 Feb '17

I upgraded to 2.2.28 and started seeing this logged: indexer-worker(dluke): Panic: file mailbox-list.c: line 1158 (mailbox_list_try_mkdir_root): assertion failed: (strncmp(root_dir, path, strlen(root_dir)) == 0) indexer-worker(dluke): Error: Raw backtrace: 2 libdovecot.0.dylib 0x000000010ec63d24 default_fatal_finish + 36 -> 3 libdovecot.0.dylib 0x000000010ec64b5b i_internal_fatal_handler + 43 -> 4 libdovecot.0.dylib 0x000000010ec64039 i_panic + 169 -> 5 libdovecot-storage.0.dylib 0x000000010eac0950 mailbox_list_try_mkdir_root + 1248 -> 6 libdovecot-storage.0.dylib 0x000000010eac0a39 mailbox_list_mkdir_root + 25 -> 7 lib21_fts_lucene_plugin.so 0x0000000110adc949 fts_backend_lucene_update_set_build_key + 73 -> 8 lib20_fts_plugin.so 0x000000010ee736cf fts_backend_update_set_build_key + 79 -> 9 lib20_fts_plugin.so 0x000000010ee74317 fts_build_mail + 599 -> 10 lib20_fts_plugin.so 0x000000010ee7981a fts_mail_precache + 794 -> 11 libdovecot-storage.0.dylib 0x000000010eaa71e9 mail_precache + 25 -> 12 indexer-worker 0x000000010ea9e503 master_connection_input + 1523 -> 13 libdovecot.0.dylib 0x000000010ec78b89 io_loop_call_io + 89 -> 14 libdovecot.0.dylib 0x000000010ec7a96d io_loop_handler_run_internal + 269 -> 15 libdovecot.0.dylib 0x000000010ec7907f io_loop_handler_run + 303 -> 16 libdovecot.0.dylib 0x000000010ec78e58 io_loop_run + 88 -> 17 libdovecot.0.dylib 0x000000010ec03458 master_service_run + 24 -> 18 indexer-worker 0x000000010ea9ddb4 main + 340 -> 19 libdyld.dylib 0x00007fffba7df255 start + 1 Anyone else? -- Daniel J. Luke

1 0

Released Pigeonhole v0.4.17 for Dovecot v2.2.28.
by Stephan Bosch 26 Feb '17

26 Feb '17

Hello Dovecot users, Here's the definitive 0.4.17 release. There were no changes since the release candidate. Changelog v0.4.17: - LDA Sieve plugin: Fixed handling of an early explicit keep during multiscript execution. Action side-effects and the message snapshot would be lost at the final stage where the implicit keep is evaluated. This could result in the IMAP flags assigned to the message to be forgotten or that headers modified by the "editheader" extension would revert to their original state. - file script storage: Amended the up-to-date time stamp comparison for on-disk binaries to include nanoseconds. This will fix problems occurring when both binary and script are saved within the same second. This fix is ineffective on older systems that have no support for nanoseconds in stat() timestamps, which should be pretty rare nowadays. - file script storage: Improve saving and listing permission error to include more details. - imapsieve plugin: Make sure "INBOX" is upper case in static mailbox rules. Otherwise, the mailbox name would never match, since matching is performed case-sensitively and Dovecot only returns the upper-cased "INBOX". - imapsieve plugin: Fixed assert failure occurring when used with virtual mailboxes. - doveadm sieve plugin: Fixed crash when setting Sieve script via attribute's string value. The release is available as follows: http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.17.ta… http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.17.ta… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this release and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

1 0

Re: Sieve removeflag Action
by Stephan Bosch 26 Feb '17

26 Feb '17

Op 2/26/2017 om 4:02 PM schreef Thomas Leuxner: >> Am 26.02.2017 um 15:52 schrieb Thomas Leuxner <tlx(a)leuxner.net>: >> >>> So, I cannot reproduce your problem. >>> >>> Could you try this at your end? >> Hi Stephan, >> >> Tried with: >> >> $ /usr/lib/dovecot/dovecot-lda -d tlx(a)leuxner.net -p ./message.eml >> >> From: User <user(a)example.com> >> To: lists(a)example.org >> Subject: Frop! >> List-Id: <debian-security-announce.lists.debian.org> >> >> Frop! >> >> It goes right to the INBOX with no flags: >> > This time *with sieve* enabled in mail_plugins for LDA. (I’m using LMTP). The mail gets processed correctly. Does it make a difference for LMTP from the code? That should not matter normally. I tested this at my end, but I saw no different behavior. I created the following file in ~/message-lmtp.eml (substitute MAIL FROM/RCPT TO with whatever you need): LHLO frop MAIL FROM:<stephan(a)rename-it.nl> RCPT TO:<stephan> DATA From: User <user(a)example.com> To: stephan(a)rename-it.nl Subject: Frop! List-Id: <debian-security-announce.lists.debian.org> Frop! . Then I invoked LMTP through its unix socket (using socat tool): stephan@drieka:~$ cat ~/message-lmtp.eml | socat gopen:/var/run/dovecot/lmtp stdio 220 drieka.tds Dovecot ready. 250-drieka.tds 250-8BITMIME 250-ENHANCEDSTATUSCODES 250 PIPELINING 250 2.1.0 OK 250 2.1.5 OK 354 OK 250 2.0.0 <stephan> 1D/3FPsfs1ge/QAASZWA2w Saved Evaluating the results shows that there is still no problem here, since the last delivery is fine (I initially forgot to enable Sieve for LMTP): stephan@drieka:~$ doveadm -f table fetch "uid flags" mailbox "INBOX" uid flags 14 \Flagged \Recent $MailFlagBit1 15 \Flagged \Recent $MailFlagBit1 16 \Flagged \Recent $MailFlagBit1 17 \Flagged \Recent $MailFlagBit1 18 \Flagged \Recent $MailFlagBit1 19 \Recent 20 \Flagged \Recent $MailFlagBit1 stephan@drieka:~$ doveadm -f table fetch "uid flags" mailbox ":public/Newsletters/Debian/Security" uid flags 1 \Recent 2 \Recent 3 \Recent 4 \Recent 5 \Recent 6 \Recent Regards, Stephan.

2 1