September 2020 - dovecot

Incremental purge?
by Jesus Cea 02 Sep '20

02 Sep '20

I am using mdbox storage format. As far as I know, “doveadm purge” does a complete purge. The time and disk activity would be proportional to the number mdbox files on the storage. The purge time could be huge. I wonder if there is some way of doing a kind of incremental purge, stopping/restarting purging, etc. Purging by file range would be nice too. Advice? Thanks Enviado desde mi iPhone

5 7

Dovecot Proxy
by Thoralf Rickert-Wendt 02 Sep '20

02 Sep '20

Hello everyone, it's my first post here on this mailing list and I hope, I make it right. I posted a question on https://serverfault.com/questions/1031441/dovecot-as-proxy-with-submission and nobody was able to answer it. So I decided to push that question here (I'm talking about any new dovecot version and I've tested it with 2.3.4.1 (f79e8e7e4)). I try to run a dovecot proxy in front of a big number of mail servers (serving SMTP-in, submission, IMAP, POP3, Sieve). I need that proxy, because I run out of IPv4 addresses. Of course I use IPv6 too, but many customers still have problems with there providers and they really don't want to share their mails on a "shared-mailserver". I planed to use Dovecot for IMAPS, POP3S, SMTP-submission(465) and postfix for the rest. If I find a solution for sieve, I would try that too, but that is very optional. With the documentation https://wiki1.dovecot.org/HowTo/ImapProxy (which is really old and should be updated) and some other ascii docs (from an Apple mirror somewhere deep in the web) I was able to build a IMAP/POP3 proxy that forwards requests from outside to a specific backend using SSL (993,995). That works - I think.You can find the config on the serverfault page. In general - all known domains in backend are using SSL and the passdb forwards all requests to the backend via SSL. So - I understand:|| |password_query = SELECT NULL AS password, NULL AS destuser, host, 'Y' AS nologin, 'Y' AS nodelay, 'Y' AS nopassword, 'Y' AS proxy, 'any-cert' AS `ssl` FROM proxy_domain WHERE domain = '%d' | But that is only 50% of the show. The rest ist submission (and maybe sieve). Practically the submission implementation in dovecot works too. But because dovecot by default only opens port 587 (starttls), my passdb setting has a problem. When I try to use that port Dovecot tries to use SSL on the backend/587 too - but that is wrong (it should either use 465 or should try to use starttls). So, I have the following options. - find a way to configure dovecot-proxy to listen on 465 with SSL for submission service and hope that it uses the same port - but I didn't find any documentation for that and need help - find a way to configure dovecot-proxy/passdb to return starttls=y when dovecot-submission is used (use a different passdb) - but I didn't find any documentation for that and I'm not sure, if this worls on service/protocol level - find a way to configure the passdb answer based on the used port/protocol. But I only know the parameter %u, %d and %p. - so it would be nice to find a way to also select the protocol (if already developed) - find a way to make a patch in dovecot (which isn't easy for me, because I don't really know the code) Has somebody an idea, how I can configure the dovecot-proxy in that way. bye Thoralf

3 4

about header address parsing
by TACHIBANA Masashi 02 Sep '20

02 Sep '20

Hi, Is this expected or not? From: user1(a)fuga.example.com <user1(a)example.com> To: user2(a)hoge.example.com <user2(a)example.com> ↓ a uid fetch 43055 (envelope) * 1860 FETCH (UID 43055 ENVELOPE ("Thu, 30 Jul 2020 13:52:59 +0900" "test1" ((NIL NIL "user1" "fuga.example.com")) ((NIL NIL "user1" "fuga.example.com")) ((NIL NIL "user1" "fuga.example.com")) ((NIL NIL "user2" "hoge.example.com")) NIL NIL NIL "<WLXvFmpAZFNeQbPPITjRwimDyamnBm1@test>")) From: "user1(a)fuga.example.com" <user1(a)example.com> To: "user2(a)hoge.example.com" <user2(a)example.com> ↓ a uid fetch 43056 (envelope) * 1861 FETCH (UID 43056 ENVELOPE ("Thu, 30 Jul 2020 13:53:59 +0900" "test1" (("user1(a)fuga.example.com" NIL "user1" "example.com")) (("user1(a)fuga.example.com" NIL "user1" "example.com")) (("user1(a)fuga.example.com" NIL "user1" "example.com")) (("user2(a)hoge.example.com" NIL "user2" "example.com")) NIL NIL NIL "<WLXvFmpAZFNeQbPPITjRwimDyamnBm2@test>")) Regards, -- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp 株式会社クオリティア https://www.qualitia.co.jp/

3 3

Bug: Dovecot appending "MISSING_DOMAIN" to fetch envelope responses
by Michael Gratton 02 Sep '20

02 Sep '20

Hi all, I'm seeing Dovecot include the string "MISSING_DOMAIN" in fetch envelope requests when an mailbox's `addr-spec` part does not have a `domain` part. For example: > C: a022 uid fetch 40 (envelope rfc822.header) > S: * 5 FETCH (UID 40 ENVELOPE ("Sat, 15 Aug 2020 12:53:05 +1000" > "test {{name}}" (("Michael Gratton" NIL "mike" "vee.net")) (("Michael > Gratton" NIL "mike" "vee.net")) (("Michael Gratton" NIL "mike" > "vee.net")) (("postgres" NIL "postgres" "MISSING_DOMAIN")) NIL NIL > NIL NIL) RFC822.HEADER {literal/244b}) I understand that RFC 5322 require a mailbox to have a domain part, however it is still possible for mail to contain mailboxes that do not. In these cases, Dovecot returning random string such as the above is harmful, since it not possible for IMAP clients to determine whether host name string is actually present or something that the IMAP server has made up. As a result, an IMAP client has two options: Either display the string as-is, preventing it from providing an appropriately localised explanation or visual indication of the problem in the user interface, or simply avoid ever requesting IMAP envelopes from the server and always download and parse the complete RFC822 headers itself (as I believe Thunderbird does). The former is untenable since "MISSING_DOMAIN" does not mean anything to non-English speakers, the latter is untenable since it unnecessarily inflates the of volume of data that must be transmitted over the wire to work around a server bug. In these cases, or when any other part of a required IMAP address structure is missing, Dovecot should return the empty string, e.g. "". Cheers, //Mike Dovecot details: > *REDACTED*:~$ sudo dovecot -n > # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.5.4 () > # OS: Linux 4.19.0-9-amd64 x86_64 Debian 10.4 ext4 > # Hostname: *REDACTED* > first_valid_uid = *REDACTED* > hostname = *REDACTED* > last_valid_uid = *REDACTED* > lda_original_recipient_header = X-Original-To > lmtp_rcpt_check_quota = yes > lmtp_save_to_detail_mailbox = yes > mail_gid = *REDACTED* > mail_home = *REDACTED* > mail_location = mdbox:*REDACTED*:UTF-8 > mail_privileged_group = mail > mail_server_admin = mailto:*REDACTED* > mail_temp_dir = *REDACTED* > mail_uid = *REDACTED* > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character vacation subaddress comparator-i;ascii-numeric > relational regex imap4flags copy include variables body enotify > environment mailbox date index ihave duplicate mime foreverypart > extracttext imapsieve vnd.dovecot.imapsieve > namespace inbox { > inbox = yes > location = > mailbox Drafts { > auto = subscribe > special_use = \Drafts > } > mailbox Junk { > auto = subscribe > autoexpunge = 30 days > special_use = \Junk > } > mailbox Sent { > auto = subscribe > special_use = \Sent > } > mailbox Trash { > auto = subscribe > autoexpunge = 30 days > special_use = \Trash > } > prefix = > separator = . > type = private > } > passdb { > args = *REDACTED* > driver = ldap > } > plugin { > imapsieve_mailbox1_before = file:*REDACTED* > imapsieve_mailbox1_causes = COPY > imapsieve_mailbox1_name = Junk > imapsieve_mailbox2_before = file:*REDACTED* > imapsieve_mailbox2_causes = COPY > imapsieve_mailbox2_from = Junk > imapsieve_mailbox2_name = * > recipient_delimiter = + > sieve = file://*REDACTED* > sieve_before = /etc/dovecot/sieve-before.d > sieve_global_extensions = +vnd.dovecot.pipe > +vnd.dovecot.environment +vnd.dovecot.report > sieve_pipe_bin_dir = /etc/dovecot/sieve-scripts > sieve_plugins = sieve_imapsieve sieve_extprograms > sieve_quota_max_storage = 50M > } > postmaster_address = *REDACTED* > protocols = " imap lmtp sieve" > service auth-worker { > user = mail > } > service auth { > unix_listener *REDACTED* { > group = postfix > mode = 0666 > user = postfix > } > } > service lmtp { > process_min_avail = 2 > unix_listener *REDACTED* { > group = postfix > mode = 0600 > user = postfix > } > user = mail > } > ssl = required > ssl_cert = <*REDACTED* > ssl_client_ca_dir = *REDACTED* > ssl_dh = # hidden, use -P to show it > ssl_key = # hidden, use -P to show it > submission_host = *REDACTED*:587 > userdb { > args = uid=8 > driver = static > } > protocol lmtp { > mail_plugins = " sieve" > } > protocol imap { > mail_plugins = " imap_sieve" > } -- Michael Gratton. <https://mjog.vee.net>

2 3

using %d as a variable in the ldap search base
by Luke Schierer 01 Sep '20

01 Sep '20

Hi, I'm trying to configure dovecot to use LDAP authentication directly, and I'm having a bit of trouble. I have a somewhat unusual setup, in that I have an LDAP directory that has 3 OUs each of which have their own set of users, some of which overlap. As I was trying to figure things out, I was setting the search base in my /etc/dovecot/dovecot-ldap.conf.ext file which is referenced by /etc/dovecot/conf.d/auth-ldap.conf.ext file to base = ou=%d,dc=thecrazyguys,dc=net however, the resulting searches against the directory endup just dropping the %d, resulting in ou=,dc=thecrazyguys,dc=net which is invalid. On a whim, I tried a search base of base = dc=%d,dc=thecrazyguys,dc=net and found that it did correctly substitute in the variable, which would be correct, except that my ldap tree is set up with OUs and not an extra DC segment. for whatever reason, it will do variable substitution for dc=%d, but not for ou=%d. this is certainly not documented, and seems like wrong behavior, since having an ou in a search base is valid. I'm including configuration information below. Please let me know if I've missed including information that is required. Thanks! Luke luke@schierer@littera001:/etc/dovecot$ lsb_release -rd Description: Ubuntu 18.04.5 LTS Release: 18.04 luke@schierer@littera001:/etc/dovecot$ luke@schierer@littera001:/etc/dovecot$ dpkg -l | grep -i dovecot ii dovecot-core 1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-imapd 1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-ldap 1:2.2.33.2-1ubuntu4.6 amd64 ii dovecot-pop3d 1:2.2.33.2-1ubuntu4.6 amd64 luke@schierer@littera001:/etc/dovecot$ root@littera001:/etc/dovecot# dovecot -n # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: Linux 4.15.0-112-generic x86_64 Ubuntu 18.04.5 LTS auth_verbose = yes first_valid_uid = 1001 imap_hibernate_timeout = 100 secs lock_method = dotlock mail_cache_min_mail_count = 15 mail_fsync = always mail_location = maildir:~/Maildir:LAYOUT=fs:INDEX=MEMORY mail_nfs_index = yes mail_nfs_storage = yes mail_privileged_group = mail mailbox_list_index = yes maildir_broken_filename_sizes = yes maildir_very_dirty_syncs = yes mbox_min_index_size = 10 B mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocols = " imap pop3" ssl_cert = </etc/dovecot/dovecot.pem ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol imap { mail_max_userip_connections = 20 } root@littera001:/etc/dovecot# root@littera001:/etc/dovecot# cat /etc/dovecot/dovecot-ldap.conf.ext | grep -v ^# | uniq | more hosts = censor001.plerumque.thecrazyguys.net tls = yes tls_require_cert = allow debug_level = 4 auth_bind = yes base = ou=%d,dc=thecrazyguys,dc=net scope = subtree user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid user_filter = (&(objectClass=posixAccount)(uid=%n)) blocking = no root@littera001:/etc/dovecot#

2 4

Re: replicator: Panic: data stack: Out of memory
by Thomas Tsiakalakis 01 Sep '20

01 Sep '20

So nobody has any idea why this could happen? Let me know if I should provide more Information Thanks Thomas Tsiakalakis Team Applikationsbetrieb GDV Dienstleistungs-GmbH Tel: +49(40)33449-4318 Fax: E-Mail: Thomas.Tsiakalakis(a)gdv-dl.de<mailto:Thomas.Tsiakalakis@gdv-dl.de> GDV Dienstleistungs-GmbH Glockengießerwall 1 D-20095 Hamburg www.gdv-dl.de<http://www.gdv-dl.de> Niederlassungen: Wilhelmstraße 43 / 43 G 10117 Berlin Frankenstraße 18 20097 Hamburg Sitz und Registergericht: Hamburg HRB 145291 USt.-IdNr : DE 205183123 Geschäftsführer: Dr. Jens Bartenwerfer Fred di Giuseppe Chiachiarella Aufsichtsratsvorsitzender: Werner Schmidt ------------------------------------------------------------------ Diese E-Mail und alle Anhänge enthalten vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der E-Mail ist nicht gestattet. This e-mail and any attached files may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Von: Thomas Tsiakalakis Gesendet: Dienstag, 23. Juni 2020 11:25 An: dovecot(a)dovecot.org Betreff: AW: replicator: Panic: data stack: Out of memory I managed to convince the System that I really want a core dump Von: Thomas Tsiakalakis Gesendet: Donnerstag, 18. Juni 2020 16:26 An: 'dovecot(a)dovecot.org' <dovecot(a)dovecot.org> Betreff: replicator: Panic: data stack: Out of memory Hi, I have set up 2 SLES 15 Hosts with dovecot and replication. Everything seems to work fine but every time a message is delivered, I get an out of memory error in the logs. The Replication itself seems to work fine though. I increased default_vsz_limit to 512M but the only thing that changed was that dovecot was trying to allocate 1073741864 bytes instead of 268435496 As I said, I’m running SLES 15 SP1 and Dovecot 2.3.10 (0da0eff44) (both Hosts have the same version) Each Host currently has 8GB of Memory. # free -h total used free shared buff/cache available Mem: 7.8Gi 278Mi 7.3Gi 47Mi 183Mi 7.2Gi Swap: 4.0Gi 0B 4.0Gi # journalctl -f Jun 18 15:55:48 mail1 postfix/pickup[3457]: 18533C009C8: uid=0 from=<root> Jun 18 15:55:48 mail1 postfix/cleanup[3669]: 18533C009C8: message-id=<20200618135548.18533C009C8@mail1> Jun 18 15:55:48 mail1 postfix/qmgr[3458]: 18533C009C8: from=<root@mail1>, size=431, nrcpt=1 (queue active) Jun 18 15:55:48 mail1 dovecot[1833]: lmtp(3673): Connect from local Jun 18 15:55:48 mail1 dovecot[1833]: replicator: Panic: data stack: Out of memory when allocating 268435496 bytes Jun 18 15:55:48 mail1 dovecot[1833]: replicator: Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7f8346d6d262] -> /usr/lib64/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7f8346d6d37e] -> /usr/lib64/dovecot/libdovecot.so.0(+0xebcee) [0x7f8346d77cee] -> /usr/lib64/dovecot/libdovecot.so.0(+0xebd91) [0x7f8346d77d91] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8346cd1f8b] -> /usr/lib64/dovecot/libdovecot.so.0(t_pop_last_unsafe+0) [0x7f8346d73660] -> /usr/lib64/dovecot/libdovecot.so.0(+0xe7a50) [0x7f8346d73a50] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10eb58) [0x7f8346d9ab58] -> /usr/lib64/dovecot/libdovecot.so.0(+0xe3ac5) [0x7f8346d6fac5] -> /usr/lib64/dovecot/libdovecot.so.0(buffer_write+0xf2) [0x7f8346d6fdb2] -> dovecot/replicator(replicator_queue_push+0xde) [0x5648989a5cde] -> dovecot/replicator(+0x5464) [0x5648989a5464] -> dovecot/replicator(+0x4a0b) [0x5648989a4a0b] -> dovecot/replicator(+0x4c02) [0x5648989a4c02] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7f8346d90c99] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x134) [0x7f8346d92574] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c) [0x7f8346d90d9c] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f8346d90fc8] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f8346d01873] -> dovecot/replicator(main+0x1a0) [0x5648989a3cb0] -> /lib64/libc.so.6(__libc_start_main+0xea) [0x7f83468f534a] -> dovecot/replicator(_start+0x2a) [0x5648989a3d5a] Jun 18 15:55:48 mail1 dovecot[1833]: lmtp(test(a)example.com)<3673><q5sLCGRy615ZDgAATn/NZw>: Warning: replication(test(a)example.com): Sync failure: Jun 18 15:55:48 mail1 dovecot[1833]: lmtp(test(a)example.com)<3673><q5sLCGRy615ZDgAATn/NZw>: Warning: replication(test(a)example.com): Remote sent invalid input: - Jun 18 15:55:48 mail1 dovecot[1833]: lmtp(test(a)example.com)<3673><q5sLCGRy615ZDgAATn/NZw>: msgid=<20200618135548.18533C009C8@mail1>: saved mail to INBOX Jun 18 15:55:48 mail1 dovecot[1833]: lmtp(3673): Disconnect from local: Client has quit the connection (state=READY) Jun 18 15:55:48 mail1 postfix/lmtp[3672]: 18533C009C8: to=<test(a)example.com>, orig_to=<test(a)domain.com>, relay=mail1[private/dovecot-lmtp], delay=0.17, delays=0.02/0.01/0.02/0.13, dsn=2.0.0, status=sent (250 2.0.0 <test(a)example.com> q5sLCGRy615ZDgAATn/NZw Saved) Jun 18 15:55:48 mail1 postfix/qmgr[3458]: 18533C009C8: removed Jun 18 15:55:48 mail1 dovecot[1833]: replicator: Fatal: master: service(replicator): child 2948 killed with signal 6 (core not dumped - https://dovecot.org/bugreport.html#coredumps - set /proc/sys/fs/suid_dumpable to 2)

2 3

shutdown_clients ignored on Director upgrade/restart
by Tom Sommer 01 Sep '20

01 Sep '20

When yum updates a Dovecot Director, it restarts all processes - however a few imap-login and pop3-login processes hang because they are still proxying connections. It's like the now orphaned processes are stalled, and needs a kill -9 to die. This is a problem because those stalled connections are still connected to backends, so when new connections are made from the same account, they do not end up on the same backend and thus index corruption happens (same account, two different backends). shutdown_clients is enabled but it appears Dovecot restarts regardless of verifying all processes/clients are shut down. Just had this happen upon yum update from dovecot-2.3.11.3-3 to dovecot-2.3.11.3-4 -- Tom

1 0