- dovecot - dovecot.org

FETCH BODY vs FETCH BODYSTRUCTURE + FETCH BODY.PEEK
by Alessio Cecchi 15 Nov '18

15 Nov '18

Hello, we are developing a web page, for a ticket system, to show the email messages ordered by the biggest attachments size present in an IMAP folder, with a pagination of 100 messages per page. Each raw represents a message with "DATE, FROM, SUBJECT, the preview (the first 500 chars of the text/html of the messsage) and the list of the attachments (name + size) it contains. The problem is that the page load is VERY slow because we fetch the entire BODY of all 100 messages, in order to get the preview and the attachments name and size. Now, considering that each messages can contain up to 50 MB of attachments, we risk to download 5000MB from IMAP for every page. An option might be to perform two FETCH for each message: - one fetch with just the BODYSTRUCTURE (where we can get the attachments name and size and the text/html parts). - one fetch with the body parts (BODY.PEEK[]) we need to build the preview (text + html). We are wondering if there is a single FETCH command to get the text and html parts (needed to build the preview) with only the name and the size of the attachments without downloading the entire message. Thanks. -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

2 1

Dovecot proxy: per user/domain 'namespace/inbox/prefix' from MySQL
by Adi Pircalabu 15 Nov '18

15 Nov '18

As a way to try and avoid using "prefix = INBOX." ad infinitum for the inbox namespace, I'm looking for ways to move on to "prefix =" for new mail accounts, and grandfather the existing ones. Previously running Courier-IMAP, now Dovecot, I looked at https://wiki.dovecot.org/Namespaces#Backwards_Compatibility:_Courier_IMAP and decided it's too risky to go down that path and use namespace compat, with so many IMAP clients out there the scope of testing is huge and the outcome is uncertain and not worth it. After reading https://wiki.dovecot.org/Namespaces#Per-user_Namespace_Location_From_SQL I thought I might be able to overwrite the server configuration per user returning 'namespace/inbox/prefix' value from SQL. Here's the setup I attempted, briefly: 1. Client connects to the Dovecot proxy, which authenticates the user and proxies to the backend using a query like this in /etc/dovecot/conf.d/dovecot-sql.conf.ext: driver = mysql connect = <connection_string> password_query = SELECT NULL AS password, 'Y' as nopassword, host, 'any-cert' as 'starttls', 'Y' AS proxy FROM mailbox WHERE email = '%u' AND disabled_smtpauth=0 Works a treat. 2. Next, I'm trying to add the prefix lookup in the picture. In the same file I've added: user_query = SELECT ns_inbox_prefix AS 'namespace/inbox/prefix' FROM mailbox WHERE email = '%u' AND disabled_smtpauth=0 3. The mailbox table schema reads: CREATE TABLE `mailbox` ( `id` int(11) unsigned NOT NULL AUTO_INCREMENT, `email` varchar(255) NOT NULL DEFAULT '', `password` varchar(255) NOT NULL DEFAULT '', `clear_password` varchar(255) NOT NULL DEFAULT '', `name` varchar(255) NOT NULL DEFAULT '', `host` varchar(32) DEFAULT NULL, `port` varchar(32) DEFAULT NULL, `ns_inbox_prefix` varchar(255) NOT NULL DEFAULT '', `lastlog_remote_ips` bigint(20) unsigned NOT NULL DEFAULT 0, `curlog_remote_ips` bigint(20) unsigned NOT NULL DEFAULT 0, `disabled_smtpauth` tinyint(1) NOT NULL DEFAULT 0, `last_modified` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), PRIMARY KEY (`id`), UNIQUE KEY `email` (`email`) ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1; The 2 queries above return: MariaDB [postfix]> SELECT NULL AS password, 'Y' as nopassword, host, 'any-cert' as 'starttls', 'Y' AS proxy FROM mailbox WHERE email = 'adi2(a)0aditest.local' AND disabled_smtpauth=0; +----------+------------+----------------+----------+-------+ | password | nopassword | host | starttls | proxy | +----------+------------+----------------+----------+-------+ | NULL | Y | 192.168.123.24 | any-cert | Y | +----------+------------+----------------+----------+-------+ 1 row in set (0.00 sec) MariaDB [postfix]> SELECT ns_inbox_prefix AS 'namespace/inbox/prefix' FROM mailbox WHERE email = 'adi2(a)0aditest.local' AND disabled_smtpauth=0; +------------------------+ | namespace/inbox/prefix | +------------------------+ | | +------------------------+ 1 row in set (0.00 sec) After reloading dovecot service with auth_debug = yes are the maillog for an IMAP session: Nov 15 12:43:48 proxy1 dovecot: auth: Debug: Loading modules from directory: /usr/lib64/dovecot/auth Nov 15 12:43:48 proxy1 dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/lib20_auth_var_expand_crypt.so Nov 15 12:43:48 proxy1 dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_mysql.so Nov 15 12:43:48 proxy1 dovecot: auth: Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_sqlite.so Nov 15 12:43:48 proxy1 dovecot: auth: Debug: Read auth token secret from /var/run/dovecot/auth-token-secret.dat Nov 15 12:43:48 proxy1 dovecot: auth: Debug: auth client connected (pid=7527) Nov 15 12:43:53 proxy1 dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011session=AorrLqp6drgAAAAAAAAAAAAAAAAAAAAB#011lip=::1#011rip=::1#011lport=1143#011rport=47222#011resp=<hidden> Nov 15 12:43:53 proxy1 dovecot: auth: Debug: sql(adi2@0aditest.local,::1,<AorrLqp6drgAAAAAAAAAAAAAAAAAAAAB>): cache miss Nov 15 12:43:53 proxy1 dovecot: auth-worker(7533): Debug: Loading modules from directory: /usr/lib64/dovecot/auth Nov 15 12:43:53 proxy1 dovecot: auth-worker(7533): Debug: Module loaded: /usr/lib64/dovecot/auth/lib20_auth_var_expand_crypt.so Nov 15 12:43:53 proxy1 dovecot: auth-worker(7533): Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_mysql.so Nov 15 12:43:53 proxy1 dovecot: auth-worker(7533): Debug: Module loaded: /usr/lib64/dovecot/auth/libdriver_sqlite.so Nov 15 12:43:53 proxy1 dovecot: auth-worker(7533): Debug: sql(adi2@0aditest.local,::1,<AorrLqp6drgAAAAAAAAAAAAAAAAAAAAB>): query: SELECT NULL AS password, 'Y' as nopassword, host, 'any-cert' as 'starttls', 'Y' AS proxy FROM mailbox WHERE email = 'adi2(a)0aditest.local' AND disabled_smtpauth=0 Nov 15 12:43:53 proxy1 dovecot: auth: Debug: client passdb out: OK#0111#011user=adi2@0aditest.local#011host=192.168.123.24#011starttls=any-cert#011proxy#011pass=<hidden> Nov 15 12:43:53 proxy1 dovecot: imap-login: Invalid certificate: [...] Nov 15 12:43:53 proxy1 dovecot: imap-login: Invalid certificate: [...] Nov 15 12:43:53 proxy1 dovecot: imap-login: Invalid certificate: [...] Nov 15 12:43:53 proxy1 dovecot: imap-login: Invalid certificate: [...] Nov 15 12:43:53 proxy1 dovecot: imap-login: Invalid certificate: [...] Nov 15 12:43:53 proxy1 dovecot: imap-login: proxy(adi2(a)0aditest.local): started proxying to 192.168.123.24:143: user=<adi2(a)0aditest.local>, method=PLAIN, rip=::1, lip=::1, secured, session=<AorrLqp6drgAAAAAAAAAAAAAAAAAAAAB> Looks like user_query isn't executed, why? And here's the corresponding IMAP session: Trying ::1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready. . LOGIN adi2(a)0aditest.local <hidden> . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE NOTIFY SPECIAL-USE QUOTA] Logged in . NAMESPACE * NAMESPACE (("INBOX." ".")) NIL NIL . OK Namespace completed (0.000 + 0.000 secs). . LIST "" * * LIST (\HasChildren) "." INBOX * LIST (\HasNoChildren \Trash) "." INBOX.Trash * LIST (\HasNoChildren) "." INBOX.Templates * LIST (\HasNoChildren \Sent) "." INBOX.Sent * LIST (\HasNoChildren \Drafts) "." INBOX.Drafts * LIST (\HasNoChildren \Archive) "." INBOX.Archives * LIST (\HasNoChildren \UnMarked \Junk) "." INBOX.Spam . OK List completed (0.000 + 0.000 secs). . LSUB "" * * LSUB (\Archive) "." INBOX.Archives * LSUB (\Drafts) "." INBOX.Drafts * LSUB (\Sent) "." INBOX.Sent * LSUB (\Junk) "." INBOX.Spam * LSUB () "." INBOX.Templates * LSUB (\Trash) "." INBOX.Trash . OK Lsub completed (0.000 + 0.000 secs). . LOGOUT * BYE Logging out . OK Logout completed (0.000 + 0.000 secs). Connection closed by foreign host. How do I overwrite 'namespace/inbox/prefix' for an user on the Dovecot proxy? Is user_query working in this context? -- Adi Pircalabu

1 1

different TLS protocols on different ports
by A. Schulze 15 Nov '18

15 Nov '18

Hello, I'm providing IMAP+Starttls on port 143 for users with legacy MUA. So I've to enable TLS1.0 up to TLS1.3 For IMAPS / port 993 I like to enable TLS1.2 and TLS1.3 only. Is this possible with dovecot-2.2.36 / how to setup this? Thanks for suggestions, Andreas

6 9

Re: Disaster Recovery Help
by Giovanni Bisanti 14 Nov '18

14 Nov '18

I just tried, and no, it does not get email addresses. and the user only has like 12 months on the device, anything older is only headers. It's so bad!! Thanks for everybody's help Giovanni Bisanti helpdesk(a)techtoyoullc.com (mailto:helpdesk@techtoyoullc.com) (239) 851-7222 November 12, 2018 10:16 PM, "James Brown" wrote: Don’t know. Have only used it for Messages. James. On 13 Nov 2018, at 10:07 am, Giovanni Bisanti wrote: Yes, I think I have to try this. Have you ever used iMazing? does it export emails? On Tue, 2018-11-13 at 10:05 +1100, James Brown wrote: On 13 Nov 2018, at 9:48 am, Giovanni Bisanti wrote: the Account on the iPhone is IMAP, I'm not sure I understand the process you are describing iMazing might be able to help you get the mail off the iPhone: https://imazing.com (https://imazing.com/) Then transfer to you Dovecot Maildir. (Have never done this - it’s just a suggestion). James.

1 0

autoexpunged in IMAP logout format
by Alessio Cecchi 14 Nov '18

14 Nov '18

Hi, in my imap_logout_format I register %{deleted} %{expunged} %{trashed} but now I saw that is available also %{autoexpunged}. Is necessary to add also %{autoexpunged} in logout format to have the count of all deleted email via IMAP? Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice

1 0

please reply
by lty＠cndns.com 14 Nov '18

14 Nov '18

Is the dovecot question on this mailing list?

2 1

Trying to do antispam with Sieve
by Adi Pircalabu 14 Nov '18

14 Nov '18

Hi, Using https://wiki.dovecot.org/HowTo/AntispamWithSieve I'm trying to execute scripts when moving to/from Spam folder, however nothing's happening. The actions are: 1. Move to Spam: redirect :copy "spamcop_spam(a)domain.local"; 2. Move from Spam: redirect :copy "spamcop_ham(a)domain.local"; Here's the configuration I'm working with: doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1153) # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.14.80-6.el7xen.x86_64 x86_64 CentOS Linux release 7.5.1804 (Core) ext4 auth_mechanisms = plain login digest-md5 cram-md5 apop auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890&.-_@' disable_plaintext_auth = no first_valid_uid = 30 imap_client_workarounds = delay-newmail imap_logout_format = rcvd=%i, sent=%o mail_attribute_dict = file:/var/qmail/mailnames/%Ld/dovecot-attributes mail_fsync = never mail_home = /var/qmail/mailnames/%Ld/%Ln mail_location = maildir:/var/qmail/mailnames/%Ld/%Ln/Maildir mail_log_prefix = "service=%s, user=%u, ip=[%r]. " mail_max_userip_connections = 100 mail_plugins = " quota" mailbox_list_index = yes maildir_very_dirty_syncs = yes managesieve_logout_format = rcvd=%i, sent=%o managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext imapflags notify imapsieve mmap_disable = yes namespace { hidden = no list = children location = maildir:/var/qmail/mailnames/%Ld/%%Ln/Maildir:INDEXPVT=/var/qmail/mailnames/%Ld/%Ln/user/%%u/Maildir prefix = Other Users.%%n. separator = . subscriptions = no type = shared } namespace { list = children location = maildir:/var/qmail/mailnames/%Ld/public/Maildir:INDEXPVT=/var/qmail/mailnames/%Ld/%Ln/public/Maildir prefix = Public. separator = . subscriptions = no type = public } namespace inbox { inbox = yes location = mailbox Archives { auto = subscribe special_use = \Archive } mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Spam { auto = subscribe autoexpunge = 90 days special_use = \Junk } mailbox Templates { auto = subscribe } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX. separator = . type = private } passdb { driver = plesk } plugin { acl = vfile acl_shared_dict = file:/var/qmail/mailnames/%Ld/shared-mailboxes imapsieve_mailbox1_before = file:/usr/lib64/dovecot/sieve/report-spam.sieve imapsieve_mailbox1_causes = COPY imapsieve_mailbox1_name = Spam imapsieve_mailbox2_before = file:/usr/lib64/dovecot/sieve/report-ham.sieve imapsieve_mailbox2_causes = COPY imapsieve_mailbox2_from = Spam imapsieve_mailbox2_name = * quota = maildir:User quota quota_grace = 0 quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=99%% quota-warning 99 %u sieve = ~/.dovecot.sieve sieve_after = /etc/dovecot/sieve/after sieve_dir = ~/sieve sieve_extensions = +notify +imapflags sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment sieve_pipe_bin_dir = /usr/lib64/dovecot/sieve sieve_plugins = sieve_imapsieve sieve_extprograms } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = rcvd=%i, sent=%o, top=%t/%p, retr=%r/%b, del=%d/%m, size=%s protocols = imap pop3 sieve service auth-worker { group = user = } service auth { group = unix_listener auth-userdb { group = popuser mode = 0600 user = popuser } user = } service imap-login { process_limit = 850 service_count = 1 } service imap { process_limit = 700 service_count = 1 } service pop3 { process_limit = 700 service_count = 1 } service quota-warning { executable = script /usr/local/bin/mail-quota-warning.sh group = popuser unix_listener quota-warning { group = popuser user = popuser } user = popuser } ssl_cert = </usr/local/psa/admin/conf/httpsd.pem ssl_cipher_list = HIGH:!aNULL:!MD5 ssl_key = # hidden, use -P to show it ssl_protocols = TLSv1 TLSv1.1 TLSv1.2 userdb { args = uid=popuser gid=popuser driver = static } protocol imap { imap_metadata = yes mail_plugins = " quota imap_quota acl imap_acl imap_sieve" } protocol pop3 { pop3_uidl_format = UID%u-%v } protocol lda { mail_fsync = optimized mail_plugins = " quota sieve" } protocol lmtp { mail_fsync = optimized } /usr/lib64/dovecot/sieve/report-spam.sieve contains: require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; redirect :copy "spamcop_spam(a)domain.local"; /usr/lib64/dovecot/sieve/report-ham.sieve contains: require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; redirect :copy "spamcop_ham(a)domain.local"; I must be missing something obvious. Thanks, -- Adi Pircalabu

2 5

Sync in the 'new' folder on the standby server
by lty＠cndns.com 14 Nov '18

14 Nov '18

DOVEADM SYNC -U USERNAME(a)EXAMPLE.COM TCP:10.10.1.125:1111 Online server mail is in the cur directory, but the backup server mail is in the new directory

1 0

dovecot lmtp thinks that "disk quota exceeded" is "internal error"
by Arkadiusz Miśkiewicz 14 Nov '18

14 Nov '18

2.2.36 (not migrated to 2.3 yet) reports such problem: > Nov 13 15:50:58 mbox dovecot: lmtp(xxx): session=<ACYRCtLk6ltiEQAALZVUYQ>, Error: open(/var/mail/xxx/mailboxes.lock1bf6ad16b7b8b703) failed: Disk quota exceeded > Nov 13 15:50:58 mbox dovecot: lmtp(xxx): session=<ACYRCtLk6ltiEQAALZVUYQ>, Error: Couldn't create mailbox list lock /var/mail/xxx/mailboxes.lock: file_create_locked(/var/mail/xxx/mailboxes.lock) failed: safe_mkstemp(/var > /mail/xxx/mailboxes.lock) failed: Disk quota exceeded > Nov 13 15:50:58 mbox dovecot: lmtp(awypior): session=<ACYRCtLk6ltiEQAALZVUYQ>, msgid=<db663455902e1b3faa53f344dc353870@xxx>: save failed to open mailbox INBOX.Spam: Internal error occurred. Refer to server log for more informat > ion. [2018-11-13 15:50:58] Looks a bug to me since disk exceeded is not a internal error. Shouldn't lmtp return over quota info instead of error? -- Arkadiusz Miśkiewicz, arekm / ( maven.pl | pld-linux.org )

3 5

execute sieve command - fails to notice script has exited
by Lionel Elie Mamane 13 Nov '18

13 Nov '18

Hi, Running dovecot 2.2.27 (c0f36b0) on Debian GNU/Linux stable (version 9.6), I'm trying to use the execute command in a sieve script, with direct execution (not service socket). The script is executed, and returns (the process exits), but the dovecot-lda process seems to not notice, and thinks the script did not terminate. After sieve_execute_exec_timeout has passed, it sends a TERM, a bit later a KILL and then logs a failure. The "ps -Af f" looks like: S 0:00 \_ /usr/sbin/exim4 -Mc 1gMazS-0007IV-RB S 0:00 \_ /usr/lib/dovecot/dovecot-lda -f master(a)blitz.conuropsis.org Z 0:00 | \_ [foofilter] <defunct> S 0:00 \_ /usr/sbin/exim4 -Mc 1gMazS-0007IV-RB The dovecot debug log looks like: lda(user): Debug: Loading modules from directory: /usr/lib/dovecot/modules lda(user): Debug: Module loaded: /usr/lib/dovecot/modules/lib90_sieve_plugin.so lda(user): Debug: Effective uid=1000, gid=1000, home=/home/user lda(user): Debug: maildir++: root=/home/user/Maildir, index=, indexpvt=, control=, inbox=/home/user/Maildir, alt= lda(user): Debug: userdb lookup skipped, username taken from USER environment lda(user): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= lda(user): Debug: Destination address: user(a)blitz.conuropsis.org (source: user@hostname) lda(user): Debug: sieve: Pigeonhole version 0.4.16 (fed8554) initializing lda(user): Debug: sieve: include: sieve_global is not set; it is currently not possible to include `:global' scripts. lda(user): Debug: sieve: Sieve imapsieve plugin for Pigeonhole version 0.4.16 (fed8554) loaded lda(user): Debug: sieve: Sieve Extprograms plugin for Pigeonhole version 0.4.16 (fed8554) loaded lda(user): Debug: sieve: file storage: Using active Sieve script path: /data/home/user/.dovecot.sieve lda(user): Debug: sieve: file storage: Using script storage path: /home/user/sieve lda(user): Debug: sieve: file storage: Relative path to sieve storage in active link: sieve/ lda(user): Debug: sieve: file storage: Using Sieve script path: /data/home/user/.dovecot.sieve lda(user): Debug: sieve: file script: Opened script `test-foofilter' from `/data/home/user/.dovecot.sieve' lda(user): Debug: sieve: Using the following location for user's Sieve script: /data/home/user/.dovecot.sieve lda(user): Debug: Mailbox stdin: Opened mail UID=1 because: header Message-ID (Cache file is unusable) lda(user): Debug: sieve: Opening script 1 of 1 from `/data/home/user/.dovecot.sieve' lda(user): Debug: sieve: Loading script /data/home/user/.dovecot.sieve lda(user): Debug: sieve: Script binary /data/home/user/.dovecot.svbin successfully loaded lda(user): Debug: sieve: binary save: not saving binary /data/home/user/.dovecot.svbin, because it is already stored lda(user): Debug: sieve: Executing script from `/data/home/user/.dovecot.svbin' lda(user): Debug: sieve: action execute: running program: foofilter lda(user): Debug: Mailbox stdin: Opened mail UID=1 because: mail stream lda(user): Debug: waiting for program `/usr/local/lib/dovecot/sieve-execute/foofilter' to finish after 0 msecs lda(user): Debug: program `/usr/local/lib/dovecot/sieve-execute/foofilter'(24669) execution timed out after 2000 milliseconds: sending TERM signal lda(user): Debug: program `/usr/local/lib/dovecot/sieve-execute/foofilter' (24669) did not die after 5000 milliseconds: sending KILL signal and an strace on the dovecot-lda process shows it is in an "epoll_wait(9 ....". The script is very simple: #! /bin/sh cat > /dev/null exit 1 Anyone has a clue? Thanks in advance. Best Regards, Lionel

2 5