January 2019 - dovecot

download Dovecot emails over rsync (done!), possible alternatives
by Carles Pina i Estany 27 Jan '19

27 Jan '19

Hello, Apologies for the extra long email. It's a situation with a solved problem and I might need to solve it again, I'd like to have some feedback. Two years ago, in a trip around the Antarctic with a very limited communication I deployed a mail system based on Postfix+Dovecot. I need to do something similar again and I used a "hack" that I thought that I might avoid (that time, let's say, everything was last minute... or later than last minute!). I installed Dovecot on the internet server and the server on the ship. On the ship I had a fetchmail downloading "small emails" (up to 100 KB if I remember correctly) and delivering to the local server. For bigger emails fetchmail had a "size limit" and fetchmail was not downloading them. I wrote a Python script that was connecting from the ship, to the internet server, and was listing the emails that were "too big to be downloaded" (this was for each user. I could have done something different with procmailrc I think on the receiving time). The Python script was fetching, if I remember correctly, the email ID, date, subject and size. The Python script was sending an email to the ship server's user saying "there is an email for you with this size, from, date and subject and this is the email ID" (and inserting this into a database to avoid sending the same notification to the users again). Then the scientists sometimes were ignoring the email, were asking to the sender to send a smaller version, or asking me to download it when possible. If they wanted me to download the email: I had a script that was connecting to the internet server, opening /home/$USER/Maildir/dovecot-uidlist, finding the file for that email UUID (this was a simple script on the internet server and I was executing it using ssh and getting the output). Then it was creating a new IMAP folder named "Downloaded" in the ship's server (it was just creating the folder in the file system and adding it in the subscriptions file, all using file system operations from Python). Then using rsync it was downloading the email file from the internet's server to the ship's server temporary file (taking whatever it would take, with many retries, etc.) and then moving it to the Downloaded folder. (note: it was using rsync and not IMAP because the connection was slow and very unstable. With rsync and the correct options it was continuing the download and retrying until success). For the next expedition I'll have more time to prepare for a similar setup. I've considered if I should stop relying on dovecot-uidlist, local file system's, subscriptions, etc. and if I should write a plugin for Dovecot to expose part of this information using... a JSON API? IMAP commands? The local ship's server bit: could I inject a SMTP mail file (downloaded) using Dovecot admin commands? some API? Any advantage? To discover which file to download using rsync: instead of opening the dovecot-uidlist any better method? Any other different approach that I might want to consider? Sorry again for the long message. I thought that this might need some context. Also, thanks to the design of Dovecot I could do what I did in a very simple way. Actually Dovecot design was as I would have designed for this use case (the index file, subscription from the mailing list, etc.). Thank you very much again! and thanks for the great Dovecot as well! -- Carles Pina i Estany Web: http://pinux.info || Blog: http://pintant.cat GPG Key 0x8CD5C157

1 1

Rsync to backup dbox with SIS
by ash-dovecot＠comtek.co.uk 26 Jan '19

26 Jan '19

We will be deploying a replacement Dovecot server soon, and we are planning to use maildir for the primary storage, but with an archive namespace using mdbox (or perhaps sdbox), and SIS. Our backup servers and (luke)warm spare server need to obtain full copies of the mail store. For the maildirs I know I can simply use rsync (we already use it here). I'm a little wary of using rsync with mdbox and SIS though. Obviously we would have to use the -H option of rsync to avoid duplicating the hard linked data, does anybody have any experience with how this scales? I've considered simply rsyncing the /var/mail/dovecot/attach/??/??/hashes folders. Is there a tool that can reconstruct the /var/mail/dovecot/attach/??/??/*-* links for a list of messages in that case? I'm aware of "doveadm backup", and (although it currently throws up a few errors) it seems like it might be a valid solution for our warm spare server. Our backup servers, on the other hand, aren't supposed to be visible to the production machines, with the exception of the backup machine sshing in to do an rsync each night. We can't install dovecot on them. To perform partial email recovery in maildir, we can simply log into a backup server, grep for emails of interest, and then scp them into a maildir new/ subfolder on the mail server to recover them. What about using dbox? I can identify /var/mail/dovecot/archive/ash/dbox/storage/m.* files, and identify necessary attachments under /var/mail/dovecot/attach/??/??/ but once I've copied them over to the mail server is there a safe way to import them? Actually, converting to a maildir style file would be adequate if there is a script anywhere to do this. We'd still rather be backing up the deduplicated mailstore for obvious reasons. Has anybody got any suggestions? I'm aware of "dsync backup questions" ( https://www.dovecot.org/list/dovecot/2013-February/088070.html ) but there didn't really seem to be a solution. Thanks, Ash PS Our current server has 257 users (though perhaps 100 are active), using around 225G in total. PPS Proposed new server configuration: mail_location = maildir:~/Maildir:LAYOUT=fs mail_attachment_dir = /var/mail/dovecot/attach mail_attachment_fs = sis posix mail_attachment_hash = %{sha1} mail_attachment_min_size = 128 k namespace _archive { list = yes location = mdbox:/var/mail/dovecot/archive/%n/dbox prefix = Archive/ separator = / subscriptions = yes type = private } namespace _private { inbox = yes type = private separator = / subscriptions = yes prefix = } --- To opt-in to receiving communications from us or to update your email preferences please go to https://lists.comtek.co.uk/prefs. To be removed from receiving all communications from the Comtek Group, please unsubscribe at https://lists.comtek.co.uk/prefs and accept our apologies for any inconvenience caused. The Comtek Group respects your privacy. Please find our privacy policy at http://www.comtek.co.uk/privacy-policy This email and any attachments are confidential. If you are not the named recipient, please notify the sender immediately. The unauthorised use, disclosure, copying or alteration of this message or its attachments is strictly prohibited. Opinions expressed in this message are those of the individual and not Comtek Network Systems (UK) Ltd unless indicated by an authorised representative independent of this message. Comtek Network Systems does not accept responsibility for any changes to this message after it has been sent. Company Details: Comtek Network Systems (UK) Ltd. Registered Office: Comtek Network Systems (UK) Ltd, Unit 108, Tenth Avenue (Zone 3), Deeside Industrial Park, Deeside, Flintshire, CH5 2UA. Comtek Network Systems (UK) Ltd is a company registered in England and Wales with company number 05929974.

2 1

problem in setting up proxy
by Ted 25 Jan '19

25 Jan '19

Hello, We're having difficulty with our updated cluster of dovecot servers accessing the email storage on the NFS mounts. It seems index files get corrupted when 2 backend mailservers access the same account, and from documentation setting up a director proxy in front of the backup servers. I'm trying to just set up a straight proxy first, which the documents say is the first step, and although I can see the connections coming into the server when I try to login via the proxy, the connection times out and there are no logs from dovecot anywhere saying what happened to the connection. The configs I have set up for this in dovecot are: dovecot.conf # Protocols we want to be serving. protocols = imap pop3 #when re-enabling quota enforcement add quota in below: mail_plugins = $mail_plugins mail_log notify protocol imap { # Space separated list of plugins to load (default is global mail_plugins). #when re-enabling quota enforcement add imap_quota in below: mail_plugins = $mail_plugins } # A comma separated list of IPs or hosts where to listen in for connections. # "*" listens in all IPv4 interfaces, "::" listens in all IPv6 interfaces. # If you want to specify non-default ports or anything more complex, # edit conf.d/master.conf. #listen = *, :: # Base directory where to store runtime data. #base_dir = /var/run/dovecot/ # Name of this instance. In multi-instance setup doveadm and other commands # can use -i <instance_name> to select which instance is used (an alternative # to -c <config_path>). The instance name is also added to Dovecot processes # in ps output. #instance_name = dovecot # Greeting message for clients. login_greeting = Welcome to easyMail. shutdown_clients = yes # Most of the actual configuration gets included below. The filenames are # first sorted by their ASCII value and parsed in that order. The 00-prefixes # in filenames are intended to make it easier to understand the ordering. !include conf.d/*.conf # A config file can also tried to be included without giving an error if # it's not found: !include_try local.conf service auth { unix_listener auth-master { mode = 0600 user = vmail } } conf.d/10-auth.conf ## ## Authentication processes ## # Username formatting before it's looked up from databases. You can use # the standard variables here, eg. %Lu would lowercase the username, %n would # drop away the domain if it was given, or "%n-AT-%d" would change the '@' into # "-AT-". This translation is done after auth_username_translation changes. auth_username_format = %Lu # Space separated list of wanted authentication mechanisms: # plain login digest-md5 cram-md5 ntlm rpa apop anonymous gssapi otp skey # gss-spnego # NOTE: See also disable_plaintext_auth setting. auth_mechanisms = plain login # # Password database is used to verify user's password (and nothing more). # You can have multiple passdbs and userdbs. This is useful if you want to # allow both system users (/etc/passwd) and virtual users to login without # duplicating the system users into virtual database. # # <doc/wiki/PasswordDatabase.txt> # # User database specifies where mails are located and what user/group IDs # own them. For single-UID configuration use "static" userdb. # # <doc/wiki/UserDatabase.txt> #!include auth-deny.conf.ext #!include auth-master.conf.ext #!include auth-system.conf.ext #!include auth-sql.conf.ext #!include auth-ldap.conf.ext #!include auth-passwdfile.conf.ext #!include auth-checkpassword.conf.ext #!include auth-vpopmail.conf.ext !include auth-static.conf.ext conf.d/auth-static.conf.ext # Static passdb. Included from auth.conf. # This can be used for situations where Dovecot doesn't need to verify the # username or the password, or if there is a single password for all users: # # - proxy frontend, where the backend verifies the password # - proxy backend, where the frontend already verified the password # - authentication with SSL certificates # - simple testing passdb static { driver = static args = nopassword=y default_fields = proxy=y host=10.5.10.121 } #passdb { # driver = static # args = password=test #} #userdb { # driver = static # args = uid=vmail gid=vmail home=/home/%u #} conf.d/10-logging.conf ## ## Log destination. ## # Log file to use for error messages. "syslog" logs to syslog, # /dev/stderr logs to stderr. #log_path = syslog # Log file to use for informational messages. Defaults to log_path. #info_log_path = # Log file to use for debug messages. Defaults to info_log_path. #debug_log_path = # Syslog facility to use if you're logging to syslog. Usually if you don't # want to use "mail", you'll use local0..local7. Also other standard # facilities are supported. #syslog_facility = mail ## ## Logging verbosity and debugging. ## # Log unsuccessful authentication attempts and the reasons why they failed. auth_verbose = yes # In case of password mismatches, log the attempted password. Valid values are # no, plain and sha1. sha1 can be useful for detecting brute force password # attempts vs. user simply trying the same password over and over again. # You can also truncate the value to n chars by appending ":n" (e.g. sha1:6). #auth_verbose_passwords = no # Even more verbose logging for debugging purposes. Shows for example SQL # queries. auth_debug = yes # In case of password mismatches, log the passwords and used scheme so the # problem can be debugged. Enabling this also enables auth_debug. #auth_debug_passwords = no # Enable mail process debugging. This can help you figure out why Dovecot # isn't finding your mails. mail_debug = yes # Show protocol level SSL errors. verbose_ssl = yes # mail_log plugin provides more event logging for mail processes. plugin { # Events to log. Also available: flag_change append #mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename # Available fields: uid, box, msgid, from, subject, size, vsize, flags # size and vsize are available only for expunge and copy events. #mail_log_fields = uid box msgid size } I'm basically expecting this to forward the login requests on to 10.5.10.121 when I try to access the email account through the proxy. When I attempt this I am able to see the connections in a tcp dump, but dovecot does not log anything about the attempt. I clearly must be missing something, can you let me know what I need to do or check? Thank you Ted easyDNS Technologies

4 6

Garbled mails in Maildir storage
by Florian Lohoff 25 Jan '19

25 Jan '19

Hi, from time to time i am having mails delivered to my anon maildrop which completely break mailbox listing. I ignored it mostly as this is an anon maildrop for one time emails so loosing spam wasnt an issue. Today i had a mailbox which wouldnt list at all which was caused by 50 consecutive mails beeing broken. (I am showing the last 20 in the web interface) After some debugging i found a bunch of mails like this: Return-Path: <*************************> X-Original-To: test(a)wasteland.rfc822.org Delivered-To: inbox(a)wasteland.rfc822.org Received: from node11.tmdvps.com (unknown [184.154.211.109]) by pax.zz.de (Postfix) with SMTP id 3B73DD00B5 for <test(a)wasteland.rfc822.org>; Thu, 8 Jun 2017 17:09:25 +0200 (CEST) <U+FEFF>Date: Thu, 8 Jun 2017 10:09:24 +0000 (GMT) From: Apple Support <*************************> REPLY-TO: do_not_reply(a)apple.com To: test(a)wasteland.rfc822.org [ ... ] Yes - there is an UTF-8 BOM in the middle of the mail header. Breaks parsing of the mail header of dovecot. No From and To in the listing. And i found even more: Return-Path: <**********************> X-Original-To: test(a)wasteland.rfc822.org Delivered-To: inbox(a)wasteland.rfc822.org Received: from 36.248.164.233 (unknown [36.248.164.233]) by pax.zz.de (Postfix) with SMTP id B2B0F1000FF; Tue, 22 Jan 2019 10:37:35 +0100 (CET) Received: from 6.51.138.64 by ; Tue, 22 Jan 2019 04:36:35 -0500 Message-ID: <B[20 Mails that end with garbage in them middle of the mail header. This is a standard Maildir storage using the dovecot transport via postfix. Does anyone have an explanation for these kind of mailstorage breakage? How could broken mails like this even be processed by postfix, dovecot deliver etc. As the last mail is pretty current i quickly grepped the postfix logs: Jan 22 10:37:35 pax postfix/smtpd[20269]: B2B0F1000FF: client=unknown[36.248.164.233] Jan 22 10:37:37 pax postfix/cleanup[20371]: B2B0F1000FF: message-id=<B[20 Jan 22 10:37:37 pax postfix/qmgr[9371]: B2B0F1000FF: from=<**********************>, size=233, nrcpt=2 (queue active) Jan 22 10:37:37 pax postfix/pipe[19192]: B2B0F1000FF: to=<inbox(a)wasteland.rfc822.org>, orig_to=<test108(a)wasteland.rfc822.org>, relay=wasteland, delay=2.2, delays=1.7/0/0/0.41, dsn=2.0.0, status=sent (delivered via wasteland service) Jan 22 10:37:37 pax postfix/pipe[21064]: B2B0F1000FF: to=<inbox(a)wasteland.rfc822.org>, orig_to=<test(a)wasteland.rfc822.org>, relay=wasteland, delay=2.2, delays=1.7/0.01/0/0.48, dsn=2.0.0, status=sent (delivered via wasteland service) Jan 22 10:37:37 pax postfix/qmgr[9371]: B2B0F1000FF: removed So even postfix found the message id to be truncated. Does dovecot have a "broken maildir mail storage mail dictionary" for automated tests? Flo -- Florian Lohoff f(a)zz.de UTF-8 Test: The 🐈 ran after a 🐁, but the 🐁 ran away

1 0

Discard mail with from date older than xxxx
by Trever L. Adams 25 Jan '19

25 Jan '19

I know that sieve doesn't do math. The file would be created externally. Based on examples in section 4.4 of https://tools.ietf.org/html/rfc5260#section-4 I figured something like this would work. if date :value "le" :originalzone "date" "date" "2018-10-25" { fileinto "SPAM"; stop; } However, I get the following. spam: line 41: error: unknown tagged argument ':value' for the date test (reported only once at first occurrence). spam: error: validation failed. sievec(root): Fatal: failed to compile sieve script 'spam.sieve' Does anyone have any advice on how to get this working? Thank you. Trever

4 7

Shared instance
by Stepan Khlebalov 25 Jan '19

25 Jan '19

Hello. I have inbox instance for incoming messages and second private instance for archiving mails with different maildir location. Sharing for folders in inbox instance works perfectly. But it does not work for folders in archiving instance. I can't create two shared instances, dovecot fails. Can I have two locations in one shared instance?

3 7

jmap
by Maciej Milaszewski IQ PL 25 Jan '19

25 Jan '19

Hi Is there any chance that Dovecot-2.2.36 supports "jmap" ? -- Maciej Miłaszewski IQ PL Sp. z o.o. Starszy Administrator Systemowy Biuro Obsługi Klienta: e-mail: bok(a)iq.pl tel.: +48 58 326 09 90 - 94 fax: +48 58 326 09 99 Dział pomocy: https://www.iq.pl/pomoc Informacja dotycząca przetwarzania danych osobowych: https://www.iq.pl/kontakt Jakość gwarantuje: ISO 9001:2000 IQ PL Sp. z o.o. z siedzibą w Gdańsku (80-298), ul. Geodetów 16, KRS 0000007725, Sąd rejestrowy: Sąd Rejonowy w Gdańsku VII Wydział KRS, kapitał zakładowy: 140.000 PLN, NIP 5832736211, REGON 192478853

2 1

debian10+dovecot-2.2.33.2
by Maciej Milaszewski IQ PL 24 Jan '19

24 Jan '19

Hi I have little problem with debian10 and dovecot 2.2.33.2 ps -ax 21815 ? S 0:00 dovecot/pop3-login director 21816 ? S 0:00 dovecot/pop3-login director 21817 ? S 0:00 dovecot/pop3-login director 21818 ? S 0:00 dovecot/pop3-login director 21819 ? S 0:00 dovecot/pop3-login director 21821 ? S 0:00 dovecot/pop3-login director 21822 ? S 0:00 dovecot/pop3-login director But if I testes via telnet like this: telnet 46.xxx.xxx.113 143 Trying 46.xxx.xxx.113... Connected to 46.xxx.xxx.113. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Imap ready. ^] telnet> q Connection closed. and ps: 21808 ? S 0:00 dovecot/imap-login ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 21826 ? S 0:00 dovecot/imap-login ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? strace: strace -p 21808 strace: Process 21808 attached gettimeofday({tv_sec=1548168193, tv_usec=419420}, NULL) = 0 epoll_wait(14, doveconf: doveconf -n |head -n 4 # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.8 (0c4ae064f307+) # OS: Linux 4.19.0-1-amd64 x86_64 Debian buster/sid auth_cache_negative_ttl = 5 mins Probably dovecot not closed correctly imap-login and this same for imap-pop3. Any idea ? -- Maciej Miłaszewski IQ PL Sp. z o.o. Starszy Administrator Systemowy Biuro Obsługi Klienta: e-mail: bok(a)iq.pl tel.: +48 58 326 09 90 - 94 fax: +48 58 326 09 99 Dział pomocy: https://www.iq.pl/pomoc Informacja dotycząca przetwarzania danych osobowych: https://www.iq.pl/kontakt Jakość gwarantuje: ISO 9001:2000 IQ PL Sp. z o.o. z siedzibą w Gdańsku (80-298), ul. Geodetów 16, KRS 0000007725, Sąd rejestrowy: Sąd Rejonowy w Gdańsku VII Wydział KRS, kapitał zakładowy: 140.000 PLN, NIP 5832736211, REGON 192478853

2 13

[FTS Xapian] RC release
by Joan Moreau 24 Jan '19

24 Jan '19

Hi, FTS Xapian matches my targets for the plugins (replacing deprecated fts-squat in a production environment) https://github.com/grosjo/fts-xapian Please do not hesitate to add "issues" on github, if the case happen Hope it helps JM

3 2

Moving messages between servers with different configurations
by Marc Roos 22 Jan '19

22 Jan '19

I wanted to move messages from a mbox mailbox on server A to mdbox on server B. I thought I could do this by connecting to the remote server with "doveadm move -S x.x.x.x:x -u testuser Archive/2017 mailbox INBOX/test" but I guess this will only allow and move messages internally on server B? Should I use dsync, or is there another way to move the messages?

3 5