July 2017 - dovecot - dovecot.org

weakforced
by Mark Moseley 17 Aug '17

17 Aug '17

I've been playing with weakforced, so it fills in the 'fail2ban across a cluster' niche (not to mention RBLs). It seems to work well, once you've actually read the docs :) I was curious if anyone had played with it and was *very* curious if anyone was using it in high traffic production. Getting things to 'work' versus getting them to work *and* handle a couple hundred dovecot servers is a very wide margin. I realize this is not a weakforced mailing list (there doesn't appear to be one anyway), but the users here are some of the likeliest candidates for having tried it out. Mainly I'm curious if weakforced can handle serious concurrency and whether the cluster really works under load.

4 5

Latest Virtual Plugin changes seem to have broken flag changes: 2.2.devel (bf2fa36)
by Thomas Leuxner 14 Aug '17

14 Aug '17

Hi, I’m using several views like this: # cat Dovecot/dovecot-virtual :public/Archive/Mailing-Lists/Dovecot/* :public/Mailing-Lists/Dovecot all Before the recent changes read mail stayed in sync with the virtual folder. Now regardless whether mail is read in the original Folder or the virtual Folder, the flags are not synchronized anymore. Read mail is flagged unread in the corresponding folder. Regards Thomas

2 6

[Dovecot] [PATCH, RFC] add APOP authentication mechanism
by Andrey Panin 14 Aug '17

14 Aug '17

Hello all, this patch add APOP authentication mechanism to dovecot 1.0-test23. Please take a look. Best regards. -- Andrey Panin | Linux and UNIX system administrator pazke(a)donpac.ru | PGP key: wwwkeys.pgp.net

3 2

Corrupt index files
by Bruce Guenter 02 Aug '17

02 Aug '17

I am running Dovecot IMAP on Linux, on a LizardFS storage cluster with Maildir storage. This has worked well for most of the accounts for several months. However in the last couple of weeks we are seeing increasing errors regarding corrupted index files. Some of the accounts affected are unable to retrieve messages due to timeouts. It appeared the problems were due to the accounts being accessed from multiple servers simultaneously, so I forced them all to access one server, but the errors remained. It looks like it has something to do with file locking, but LizardFS supports advisory file locking and I do have it enabled. Deleting the corrupted indexes fixes the problem for a while, but it eventually returns, particularly for some accounts. Here are some errors I'm seeing (just a random grab). Actual home directories are munged for confidentiality. imap[25157]: (clientes.standby) Error: Failed to fix view for HOME/clientes:standby/dovecot.index: Missing middle file seq=1 (between 1..1, we have seqs 8): File is already open imap[5565]: (stadiumchair) Error: Transaction log file HOME/stadiumchair/.Drafts/dovecot.index.log: marked corrupted imap[5005]: (stadiumchair) Error: Corrupted transaction log file HOME/stadiumchair/.Drafts/dovecot.index.log seq 2: indexid changed 1418941056 -> 1500658549 (sync_offset=0) imap[20243]: (martha) Error: Transaction log HOME/martha/dovecot.index.log: duplicate transaction log sequence (539) imap[4665]: (emsspam) Error: Index file HOME/emsspam/dovecot.index: indexid changed: 1500658479 -> 1297175382 imap[4665]: (emsspam) Error: Corrupted transaction log file HOME/emsspam/dovecot.index.log seq 3: indexid changed: 1500658479 -> 1297175382 (sync_offset=316) imap[22985]: (emsspam) Error: Corrupted transaction log file HOME/emsspam/dovecot.index.log seq 10742: Invalid transaction log size (9296 vs 9296): HOME/emsspam/dovecot.index.log (sync_offset=9296) imap[3267]: (emsspam) Error: Failed to map view for HOME/emsspam/dovecot.index: Failed to map file seq=10742 offset=9052..18446744073709551615 (ret=0): corrupted, indexid=0 imap[3267]: (emsspam) Error: HOME/emsspam/dovecot.index view is inconsistent: uid=3062271 inserted in the middle of mailbox The output of dovecot -n is pasted in below. Note that some of the boxes are running 4.9, some running 4.4, all have the same problems. Also note that I am using a custom authentication front end for our virtual mailboxes, but it just sets up the minimal environment variables and runs imap. Is there anything I can change to eliminate these problems? Are there any other diagnostics I can provide to shed light on this? # 2.2.31 (65cde28): /etc/dovecot/dovecot.conf # OS: Linux 4.4.66 x86_64 Gentoo Base System release 2.3 log_path = /dev/stderr mail_debug = yes mail_fsync = always mail_location = maildir:~/.maildir mail_log_prefix = "%s[%p]: (%u) " mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX separator = type = private } passdb { args = * driver = pam } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename } ssl_cert = </etc/ssl/dovecot/server.pem ssl_key = # hidden, use -P to show it userdb { driver = passwd } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } -- Bruce Guenter <bruce(a)untroubled.org> http://untroubled.org/

5 11

under another kind of attack
by Olaf Hopp 31 Jul '17

31 Jul '17

Hi folks, "somehow" similar to the thread "under some kind oof attack" started by "MJ": I have dovecot shielded by fail2ban which works fine. But since a few days I see many many IPs per day knocking on my doors with wron password and/or users. But the rate at which they are knocking is very very low. So fail2ban will never catch them. For example one IP: Jul 25 14:03:17 irams1 dovecot: auth-worker(2212): pam(eurodisc,101.231.247.210,<gAulHSNVsNZl5/fS>): unknown user Jul 25 15:16:36 irams1 dovecot: auth-worker(11047): pam(gergei,101.231.247.210,<dPzYIyRVtOpl5/fS>): pam_authenticate() failed: Authentication failure (password mismatch?) Jul 25 16:08:51 irams1 dovecot: auth-worker(3379): pam(icpe,101.231.247.210,<Ws6t3iRVkOhl5/fS>): unknown user Jul 25 16:10:47 irams1 dovecot: auth-worker(4250): pam(endsulei,101.231.247.210,<dceL5SRVGZVl5/fS>): unknown user Note the timestamps. If I look the other way round (tries to one account) I'll get Jul 25 01:30:48 irams1 dovecot: auth-worker(11276): pam(endsulei,60.166.12.117,<slp6mhhViI48pgx1>): unknown user Jul 25 01:31:26 irams1 dovecot: auth-worker(11276): pam(endsulei,222.243.211.200,<s0+6nBhVabHe89PI>): unknown user Jul 25 13:29:22 irams1 dovecot: auth-worker(4745): pam(endsulei,60.2.50.114,<4elhpCJVtcw8AjJy>): unknown user Jul 25 13:30:27 irams1 dovecot: auth-worker(4747): pam(endsulei,222.84.118.83,<kaE1qCJVn7neVHZT>): unknown user Jul 25 16:10:47 irams1 dovecot: auth-worker(4250): pam(endsulei,101.231.247.210,<dceL5SRVGZVl5/fS>): unknown user Jul 25 16:11:45 irams1 dovecot: auth-worker(5933): pam(endsulei,206.214.0.120,<R5H56CRVdJfO1gB4>): unknown user Also note the timestamps! And I see many many distinct IPs per day (a few hundred) trying many many existing and non-existings accounts. As you see in the timestamps in my examples, this can not be handled by fail2ban without affecting regular users with typos. Is anybody observing something similar ? Anybody an idea against this ? Many of these observed IPs are chinese mobile IPs, if this matters. But we have also chinese students and researchers all abroad. Regards, Olaf -- Karlsruher Institut für Technologie (KIT) ATIS - Abt. Technische Infrastruktur, Fakultät für Informatik Dipl.-Geophys. Olaf Hopp - Leitung IT-Dienste - Am Fasanengarten 5, Gebäude 50.34, Raum 009 76131 Karlsruhe Telefon: +49 721 608-43973 Fax: +49 721 608-46699 E-Mail: Olaf.Hopp(a)kit.edu atis.informatik.kit.edu www.kit.edu KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft Das KIT ist seit 2010 als familiengerechte Hochschule zertifiziert.

11 16

Re: under another kind of attack
by Davide Marchi 31 Jul '17

31 Jul '17

> From: Olaf Hopp <Olaf.Hopp(a)kit.edu> > Davide, > yours is all postfix and thus has got no overlap with dovecot. > So no interference. > Olaf Yes, I know, but I preferred not to give anything for granted ;-) Many Thanks Olaf!

1 0

Large Scale Cluster using dsync and tagged director
by Matt Bryant 31 Jul '17

31 Jul '17

Just been reading up on some of dovecot features and was wondering if anyone had played around with large scale systems made up from mutliple small dsync clusters and using tagged director to split users between nodes of a sync'd cluster (rather than nfs). I know traditionally its been shared storage and multiple nodes and dovecots own doco suggest dsync for several thousands rather than millions but given the cloud seems to be all about replication what about many smaller nodes using dsync and scaling horizontally ? The only issue is how does director scale when there are serveral(or rather lots of) 'tags' ... in fact what are the limitations with director ? Matt -- <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campai…>

1 0

dovecot auth errors for a new user
by Ruben Safir 31 Jul '17

31 Jul '17

I've been running dovecott without trouble for quite a why and now when I added a new user, it is not accepting the user and I can not track the problem. It says find more information in the server log, but it is not in /var/log/messages or /var/log/mail.err and nothing with lsof dovecot|grep log show anything to tail www:~ # dovecot -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.16.7-53-pae i686 openSUSE 13.2 (i586) base_dir = /var/run/dovecot/ managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = pop3 ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.key userdb { driver = passwd } www:/etc/dovecot # dovecot --version 2.2.13 Sending of password for user facebook did not succeed. Mail server mrbrklyn.com responded: Internal error occurred. Refer to server log for more information. 2017-07-30T15:41:58.803006-04:00 www dovecot: pop3-login: Login: user=<facebook>, method=PLAIN, rip=10.0.0.62, lip=96.57.23.82, mpid=25269, TLS, session=<D5J/G45VQgAKAAA+> 2017-07-30T15:41:58.812827-04:00 www dovecot: pop3(facebook): Error: user facebook: Initialization failed: Namespace '': Mail storage autodetection failed with home=/home/facebook 2017-07-30T15:41:58.816903-04:00 www dovecot: pop3(facebook): Error: Invalid user settings. Refer to server log for more information. -- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002 http://www.nylxs.com - Leadership Development in Free Software http://www2.mrbrklyn.com/resources - Unpublished Archive http://www.coinhangout.com - coins! http://www.brooklyn-living.com Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013

2 4

Re: under another kind of attack
by Davide Marchi 30 Jul '17

30 Jul '17

Hi to all, @Olaf Hopp I've this filter enabled for fail2ban, my question is: could my filters overlap or interfere with those suggested by you? this is my filter: Contents of /etc/fail2ban/jail.conf: [postfix] # Ban for 10 minutes if it fails 6 times within 10 minutes enabled = true port = smtp,ssmtp filter = postfix logpath = /var/log/mail.log maxretry = 6 bantime = 600 findtime = 600 Contents of /etc/fail2ban/filter.d/postfix.conf: # Fail2Ban configuration file # Author: Cyril Jaquier # $Revision$ [Definition] # Option: failregex # Notes.: regex to match the password failures messages in the logfile. The # host must be matched by a group named "host". The tag "<HOST>" can # be used for standard IP/hostname matching and is only an alias for # (?:::f{4,6}:)?(?P<host>[\w\-.^_]+) # Values: TEXT # # Jul 11 02:35:08 mail postfix/smtpd[16299]: lost connection after AUTH from unknown[196.12.178.73] failregex = lost connection after AUTH from unknown\[<HOST>\] # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. # Values: TEXT # ignoreregex = Many thanks!

2 1

Re: under another kind of attack
by Joseph Tam 28 Jul '17

28 Jul '17

(I think I am testing other readers' patience, so if you want to follow-up, you can Email me directly.) > but how often do you have to type your username ? Not often, but I'm not talking the typical case. The larger the population you serve, the more circumstances you'll have to cover. > Only on the initial config of your mailer. After that you are done. Mail reader setups by users is often an error prone process, judging from the number of times I have to correct a setup. This is especially true for an educational institution that typically has a large turnover of accounts. If a user gets it initially wrong, then fixes their mistake, they can't get it to work despite trying all sorts of config variations, not realizing it can't be resolved anymore. Result: trouble call. If someone blows it setting on a multi-user workstation, other users with a working setups can't log in. Result: trouble call. If a student flubs their credentials, all the roommates behind their residential NAT gateway suffer. Result: trouble call. If a user screws up using an external service that slurps their mail (e.g. Gmail, Yahoo, uniboxapp, etc.), or worse, someone malicious does it on puropse, all other users of this service will be DoS'd. Result: trouble call(s). If a user acquires a DHCP address in a polluted network ..., well you get the idea. Not to mention ex-users who forget to remove their mail accounts from their smartphones, leaving a trail of blacklist entries in their wake as they travel from coffee shops to other public WiFi hotspots. > So this is why I decided to use two distinct jails with > different policies. It seems to work reasonable well. Until it doesn't. If it works for you, more power to you. The cost/benefit of a hair-trigger blacklist policy is that it saves you a few log entries showing futile attempts at finding weak passwords (because you have strong passwords, don't you?) at the risk of dealing with any of the above situations. Joseph Tam <jtam.home(a)gmail.com>

1 0