January 2021 - dovecot

[BUG REPORT] In some cases dovecot sends (huge) binary data to solr for indexing
by Alexey Panov 21 Jan '21

21 Jan '21

In some cases (exact condition still unknown) dovecot sends binary data (attachments) to SOLR for indexing. This reduces index and overall FTS efficiency dramatically. In extreme condition (below an example of 20MB) dovecot’s hardwired timeout of 60s gets triggered during HTTP exchange with SOLR on just a single file. This results in an unfinished index which, by initial indexing, gets restarted over and over. With multiple affected mailboxes even on moderate usage this can cause an IO overload of the whole system. Message example (doveadm fetch text): https://filebin.ca/5oy5Wc1QrBK3/fetch-text.obfuscated.txt <https://filebin.ca/5oy5Wc1QrBK3/fetch-text.obfuscated.txt> Corresponding raw log data: https://filebin.ca/5oy6yqLSCr3H/rawlog.obfuscated.txt <https://filebin.ca/5oy6yqLSCr3H/rawlog.obfuscated.txt> (Both files were processed with perl doveadm-obfuscate.pl <https://www.dovecot.org/tools/doveadm-obfuscate.pl>; the script doesn’t replace non-latin characters so they were replaced with ‘R’ manually) Workaround: there is a useful patch by John Fawcett <https://www.mail-archive.com/dovecot@dovecot.org/msg82296.html> that allows to set the FTS indexing message body maximum size. It works perfectly, but affected messages are getting completely ignored by FTS. This bug report is a summarised result of this discussion <https://www.mail-archive.com/dovecot@dovecot.org/msg82599.html>.

2 1

Deliver to different namespace with address detail
by Tobia Grosselle [ Terralink ] 21 Jan '21

21 Jan '21

Hello, with this settings: Dovecot 2.3.7.2 (3c910f64b) namespace { list = yes type = private inbox = yes prefix = INBOX/ separator = / } namespace Archive { list = yes type = private inbox = no prefix = Archive/ separator = / mailbox Incoming { auto = create } } lmtp_save_to_detail_mailbox = yes recipient_delimiter = + userdb_query = SELECT accounts.username AS user, accounts.home AS home, CONCAT('mdbox:', accounts.home, '/mdbox') AS mail, CONCAT('mdbox:', accounts.home, '/archive') AS 'namespace/Archive/location', 'vmail' AS uid, 'vmail' as gid FROM accounts WHERE accounts.username = '%u' it is possible to deliver an email to Archive namespace instead of inbox with address detail (email+Archive/Incoming@domain)? If i try i see that the lmtp service search the path from the inbox namespace (so INBOX/Archive/Incoming). Thanks

2 2

Sieve fileinto :create is failing
by Ron Garret 21 Jan '21

21 Jan '21

I’m trying to get a sieve script to move messages into a folder, and to create that folder if it doesn’t already exist. I’m following the example code at: https://doc.dovecot.org/configuration_manual/sieve/examples/ and doing this: > require ["fileinto", "mailbox”]; > … > fileinto :create “myfolder”; > … That results in this error in the log file: error: unknown tagged argument ':create' for the fileinto command What am I doing wrong? rg

4 8

Where does sieve store its "vacation" state?
by Ron Garret 21 Jan '21

21 Jan '21

Pigeonhole/sieve has a “vacation” feature that keeps track of which addresses it has sent messages to already so it doesn’t send too many. The state for this feature seems to survive a dovecot restart, so it must be persistent somewhere and not just stored in RAM. But I don’t recall configuring any place where that state would reside. So where is it? rg

2 1

Mail crypt plugin with HSM
by Dima Kovalyov 21 Jan '21

21 Jan '21

Hello Dovecot developers, I'm using a mail-crypt plugin to perform encryption at rest using per user keys encrypted with their password. Great plugin and I really appreciate it's continuous development and improvement. I have a question regarding mail-crypt keys being stored in HSM. Is it possible? I'm almost sure I've seen some discussion about usage of PKCS#11 in mail-crypt but can't find it in the mailing list. Please advise. Thanks! - Dima

1 0

Timeout/Event leak
by Lefteris Tsintjelis 20 Jan '21

20 Jan '21

Hi, any idea what all this is? Jan 19 18:23:00 master: Warning: Killed with signal 15 (by pid=33729 uid=0 code=kill) Jan 19 18:23:03 config: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill) Jan 19 18:23:03 stats: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill) Jan 19 18:23:03 auth: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Timeout leak: 0x1059ce0 (auth-request-handler.c:617) Jan 19 18:23:03 auth: Warning: Event 0x120f0020 leaked (parent=0x120ecc20): auth-request.c:621 Jan 19 18:23:03 auth: Warning: Event 0x120ece20 leaked (parent=0x120ecc20): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x120ecc20 leaked (parent=0x120ca620): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x120eca20 leaked (parent=0x120ec620): auth-request.c:621 Jan 19 18:23:03 auth: Warning: Event 0x120ec820 leaked (parent=0x120ec620): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x120ec620 leaked (parent=0x120b8620): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x120ec220 leaked (parent=0x1207ee20): auth-request.c:667 Jan 19 18:23:03 auth: Warning: Event 0x120ec020 leaked (parent=0x1207ee20): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x1207ee20 leaked (parent=0x12087220): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x1207ec20 leaked (parent=0x1207e820): auth-request.c:667 Jan 19 18:23:03 auth: Warning: Event 0x1207ea20 leaked (parent=0x1207e820): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x1207e820 leaked (parent=0x12087220): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x1207e620 leaked (parent=0x1207e220): auth-request.c:621 Jan 19 18:23:03 auth: Warning: Event 0x1207e420 leaked (parent=0x1207e220): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x1207e220 leaked (parent=0x120ca620): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x1207e020 leaked (parent=0x120ada20): auth-request.c:621 Jan 19 18:23:03 auth: Warning: Event 0x120adc20 leaked (parent=0x120ada20): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x120ada20 leaked (parent=0x120ad820): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x120ad820 leaked (parent=0x12087220): auth-client-connection.c:338 Jan 19 18:23:03 auth: Warning: Event 0x120ad220 leaked (parent=0x120b8a20): auth-request.c:621 Jan 19 18:23:03 auth: Warning: Event 0x120ad020 leaked (parent=0x120b8a20): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x120b8a20 leaked (parent=0x120ca620): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x120b8420 leaked (parent=0x120cac20): auth-request.c:621 Jan 19 18:23:03 auth: Warning: Event 0x120cae20 leaked (parent=0x120cac20): auth-request.c:111 Jan 19 18:23:03 auth: Warning: Event 0x120cac20 leaked (parent=0x120caa20): auth-request.c:110 Jan 19 18:23:03 auth: Warning: Event 0x120caa20 leaked (parent=0x12087220): auth-client-connection.c:338 Thank you

2 2

Dovecot Gmail OAuth2.0 Setting Question
by 福田泰葵 19 Jan '21

19 Jan '21

Dear Sir or Madam Unable to build OAuth2.0 authentication to Gmail using dovecot as proxy. I have a question about how to use dovecot as a proxy to perform OAuth 2.0 authentication to Gmail using a mail client. 1. Is the following all I need to do to authenticate to Gmail using dovecot as a proxy? - passdb passdb { driver = oauth2 mechanisms = oauthbearer xoauth2 args = /etc/dovecot/dovecot-oauth2.token.conf.ext } passdb { driver = oauth2 mechanisms = plain login args = /etc/dovecot/dovecot-oauth2.plain.conf.ext } - create dovecot-oauth2.token.conf.ext and dovecot-oauth2.plain.conf.ext - create gmail service account api 2. grant_url in dovecot-oauth2.token.conf.ext and dovecot-oauth2.plain.conf.ext is URL for obtaining a Google access token for a web server that I have built myself? 3. I use a Gmail service account, so I don’t need a client ID and secret ID, right? 4. Do I set introspection_url to the URL of my own web server with the access token used for authentication to Google as the response? 5. The documentation says “pass_attrs = host=127.0.0.1”, but if you are authenticating to Gmail, I should use “pass_attrs = proxy=y host=%{if;%s;eq;imap;imap.gmail.com;%{if;%s;eq;pop3;smtp .gmail.com;pop.gmail.com}} port=%{if;%s;eq;imap;993;%{if;%s;eq;pop3;587;465}} proxy_mech=xoauth2 pass=%{oauth2:access_token} user=%{oauth2:email oauth2:email}”? 6. What is the difference between dovecot-oauth2.token.conf.ext and dovecot-oauth2.plain.conf.ext ? Do I need to configure both? I used https://doc.dovecot.org/configuration_manual/authentication/oauth2/#proxy as a reference. I would appreciate your reply. Yours faithfully, ------------------------------ e-mail: taiki.fukuda(a)justsystems.com TEL: 03-5324-7900 mobile: 080-6198-7328 ------------------------------

1 0

dovecot 2.3.13 : make check FAILURE : Assert failed: buffer_append_full_file
by J Lumby 19 Jan '21

19 Jan '21

After successfully building dovecot 2.3.13 (linux kernel 5.10.1 , glibc-2.32 , gcc-9.3) make check had one failure as follows : test-buffer-istream.c:54: Assert failed: buffer_append_full_file(result, TEST_FILENAME, SIZE_MAX, &error) == BUFFER_APPEND_READ_ERROR test-buffer-istream.c:56: Assert failed: error != NULL && *error != '\0' buffer_append_full_file .............................................. : FAILED After a brief look at the referenced code in test-buffer-istream.c , I can't find out what this FAIL really means or whether it is serious. The assert is finding that the result of the call to buffer_append_full_file is not BUFFER_APPEND_READ_ERROR, but it does not report what the result code actually was. Maybe errno was actually 0 and call succeeded? I don't know. Nor can I see how to add some code to find out. Can anyone offer any advice on what this means and/or how to investigate? Cheers, John Lumby

2 6

Changing the default delivery mailbox to something other than INBOX
by Ron Garret 18 Jan '21

18 Jan '21

Is there an easy way (i.e. a built-in configuration setting) to change the name of the mailbox that the dovecot LDA delivers mail into? The default is INBOX but I’d like mail to be delivered to some other mailbox. The reason for this is that I want all incoming mail to be invisible to the user by default until it has been screened for viruses and spam. I know I could do this with Sieve, but that is a PITA. More details in case anyone is interested: The goal of this filter is to make it work with very little training. To bootstrap the process, the filter is given access to outgoing mail (via a milter) which it uses as a reliable training corpus for good messages. It then leverages that information to filter incoming messages. For example, messages from senders which have been the recipients of outgoing messages are presumed to be good. There is also a spam honeypot to provide a reliable spam corpus. One of the heuristics I want to use is to look for the same subject line in multiple messages from unknown users received over a period of an hour or so because those are almost invariably spam. But that requires a time delay between when a message is received and when it is filtered, and that in turn requires a place to store messages for a while before they are processed. While they are in that temporary storage, I don’t want them in the user’s face, but I do want them to be accessible if the user wants to see them (e.g. if they know that an important message is coming in which may be stuck in the temp storage). So I’d like to make something like an INCOMING mailbox where all mail is delivered. The messages in INCOMING are scanned by the filter and moved to Junk or INBOX as appropriate. Thanks, rg

3 12

mbox to pst advice
by Marc Roos 18 Jan '21

18 Jan '21

Anyone an idea how to convert mbox ot pst on linux?

3 6