- dovecot - dovecot.org

[Dovecot] Fishing attempt locking up dovecot
by Patrick Milvich 29 Dec '07

29 Dec '07

I've mentioned this before but only heard from one other person who has experienced this, but it's becoming a pretty serious issue. The situation: A spammer sets a bot on a fishing attempt to gain email addresses, causing numerous login processes to spawn and suck up all available resources. The problem: Obviously this can act like a dos attack, but the real issue is after the spammer stops (by virtue of being added to our firewall blacklist, being caught and shut down by their isp, or otherwise), dovecot doesn't seem to relinquish the resources, causing "too many files open" errors for normal usage. The master process usually hangs around 40-50 files open at any given time, with about 10,000 logins a day (I use: lsof -p `cat /var/run/ dovecot/master.pid ` | wc -l), after the attempt is over I always see the files open shoot up to near 3,000 without it able to go down until a dovecot restart. This usually happens about once a month, though we can get unlucky and have it happen a few days apart. I have some log excerpts below. Wondering if this is happening through my own fault or at least within my ability to alleviate the issue. Is there a way to limit the number of connections from an ip address? Has anyone used login_executable to first hit their own solution to keep track of, and implement connection restrictions similar to what I'm aiming for? Thanks for any insight Patrick dovecot -n # 1.0.8: /usr/local/mail/dovecot/etc/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot protocols: imap imaps pop3 pop3s ssl_cert_file: /usr/local/mail/ssl/certs/dovecot.pem ssl_key_file: /usr/local/mail/ssl/private/dovecot.pem ssl_cipher_list: ALL:!LOW disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/local/mail/dovecot/libexec/dovecot/ imap-login login_executable(imap): /usr/local/mail/dovecot/libexec/dovecot/imap- login login_executable(pop3): /usr/local/mail/dovecot/libexec/dovecot/pop3- login first_valid_uid: 500 mail_location: maildir:/Volumes/data/mail/%Ld/%Ln mail_executable(default): /usr/local/mail/dovecot/libexec/dovecot/imap- before mail_executable(imap): /usr/local/mail/dovecot/libexec/dovecot/imap- before mail_executable(pop3): /usr/local/mail/dovecot/libexec/dovecot/pop3- before mail_plugin_dir(default): /usr/local/mail/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/mail/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/mail/dovecot/lib/dovecot/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: mechanisms: plain digest-md5 login passdb: driver: sql args: /usr/local/mail/dovecot/etc/dovecot-sql.conf userdb: driver: static args: uid=exim gid=exim plugin: quota: maildir From the logs, first entries showing the problem: dovecot: Dec 10 16:17:02 Info: pop3-login: Disconnected: rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:02 Info: pop3-login: Disconnected: rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:03 Info: pop3-login: Disconnected: rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:03 Info: pop3-login: Disconnected: rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:03 Info: pop3-login: Disconnected: rip=207.245.39.90, lip=<lip> A little later on: dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<pwitest>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<tsinternetuser>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<bill>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<web>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<barbara>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<www>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<user>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<nathan>, method=PLAIN, rip=207.245.39.90, lip=<lip> dovecot: Dec 10 16:17:12 Info: pop3-login: Aborted login: user=<webmaster>, method=PLAIN, rip=207.245.39.90, lip=<lip> etc. etc.

3 2

[Dovecot] v1.1.beta11 released
by Timo Sirainen 29 Dec '07

29 Dec '07

http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta11.tar.gz http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta11.tar.gz.sig This one should be the last beta release before the first v1.1 release candidate. I'll try to stay away from this list and Dovecot in general for the next 1,5 weeks. I've several exams coming up and I should have started studying for them days ago already.. :) The main things left in my TODO are: - Squat indexes should handle expunges - Lucene indexes are broken - See if \recent flag handling could be fixed for maildir/mbox (it works for dbox though) - Some other minor things which I'll do if I have time, but won't block v1.1.0 release. Changes since beta10: - One major performance issue found and fixed. This was the main reason why we weren't yet in release candidate stage. - configure --with-package now gives an error if it can't find the necessary libs/headers for the package. Use --with-package=auto for the old behavior. - Memory leak fixes - Convert plugin fixes. Conversion is now done to namespace with INBOX. - Changes to handling group adding/removing to processes. Hopefully didn't break anything. - Message flag/keyword searches are now faster - Fixes/optimizations to handling SORT. After building initial indexes, my 260k mail mailbox sorts in about 0.01 seconds. :) - Fixes to searching mails with unknown charsets/invalid data - deliver_log_format setting

3 8

[Dovecot] 1.1.beta10 pop3 process hangs with 100% CPU
by arvids 29 Dec '07

29 Dec '07

Hello, we have observed pop3 process which got stuck consuming all available CPU. It seems that it happened because of some kind of abnormal POP3 connection termination. Here is strace info for this process: 13:36:05.866190 writev(1, [{"508qWWH96If+uVXeH2Zxl/hkn+plVwmI"..., 3975}, {"HP1oxt+np0o4Xtz27VQBtxx0zWfGuA3r"..., 193}], 2) = -1 EPIPE (Broken pipe) 13:36:05.866250 --- SIGPIPE (Broken pipe) @ 0 (0) --- 13:36:05.866288 epoll_ctl(6, EPOLL_CTL_DEL, 1, {0, {u32=135259776, u64=135259776}}) = 0 13:36:05.866335 setsockopt(1, SOL_TCP, TCP_CORK, [1], 4) = 0 13:36:05.866372 writev(1, [{"508qWWH96If+uVXeH2Zxl/hkn+plVwmI"..., 3975}, {"HP1oxt+np0o4Xtz27VQBtxx0zWfGuA3r"..., 193}], 2) = -1 EPIPE (Broken pipe) 13:36:05.866441 --- SIGPIPE (Broken pipe) @ 0 (0) --- 13:36:05.866478 writev(1, [{"508qWWH96If+uVXeH2Zxl/hkn+plVwmI"..., 3975}, {"HP1oxt+np0o4Xtz27VQBtxx0zWfGuA3r"..., 193}], 2) = -1 EPIPE (Broken pipe) 13:36:05.866539 --- SIGPIPE (Broken pipe) @ 0 (0) --- 13:36:05.866575 epoll_ctl(6, EPOLL_CTL_ADD, 1, {EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=135259776, u64=135259776}}) = 0 13:36:05.866623 setsockopt(1, SOL_TCP, TCP_CORK, [0], 4) = 0 13:36:05.866660 gettimeofday({1197372965, 866675}, NULL) = 0 13:36:05.866698 epoll_wait(6, {{EPOLLHUP, {u32=135259776, u64=135259776}}}, 4, 6986) = 1 13:36:05.866737 gettimeofday({1197372965, 866752}, {4294967176, 0}) = 0 Regards, Arvids

2 1

Re: [Dovecot] namespace problem / 1.1beta11
by ja nein 29 Dec '07

29 Dec '07

Hi Timo, well, I don't know how I managed it, either :) Here's the output of the log file with mail_debug=yes. Interestingly it prints out the correct namespace in the log file (I did a synchronization of some folders in this period): Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): Loading modules from directory: /opt/dovecot-1.1-beta12/lib/dovecot/imap Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): Module loaded: /opt/dovecot-1.1-beta12/lib/dovecot/imap/lib20_fts_plugin.so Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): Module loaded: /opt/dovecot-1.1-beta12/lib/dovecot/imap/lib21_fts_squat_plugin.so Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): Effective uid=40000, gid=40000, home=/home/dovecot/mail/smcc.net/merk@smcc.net Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): Namespace: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): maildir: data=/home/dovecot/mail/smcc.net/merk@smcc.net:INDEX=/home/dovecot/index/sm… Dec 28 23:24:33 51 dovecot: IMAP(merk(a)smcc.net): maildir++: root=/home/dovecot/mail/smcc.net/merk@smcc.net, index=/home/dovecot/index/smcc.net/merk@smcc.net, control=, inbox=/home/dovecot/mail/smcc.net/merk@smcc.net Dec 28 23:24:33 51 dovecot: imap-login: Login: user=<merk(a)smcc.net>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Dec 28 23:24:52 51 dovecot: IMAP(merk(a)smcc.net): Disconnected: Logged out bytes=17572/71038 It seems that the output to the client doesn't correspond to the settings dovecot uses internally?! Regards, Sebastian ----- Ursprüngliche Mail ---- Von: Timo Sirainen <tss(a)iki.fi> An: ja nein <reg9009(a)yahoo.de> CC: Dovecot Mailing List <dovecot(a)dovecot.org> Gesendet: Freitag, den 28. Dezember 2007, 18:45:56 Uhr Betreff: Re: [Dovecot] namespace problem / 1.1beta11 On Sat, 2007-12-22 at 13:48 +0000, ja nein wrote: > I think I've hit a separator bug: > > This is namespace query of dovecot and source server: > > * NAMESPACE (("INBOX/" ".")) NIL NIL Yes, that does look like a bug, but I've no idea how you managed to cause it. I just tested your namespace configuration and it works fine with me. Also there shouldn't even be any way to cause this configuration for a list=yes namespace: imap(tss)(pid=31754): namespace configuration error: list=yes requires prefix=INBOX/ to end with separator What do logs show with mail_debug=yes? __________________________________ Ihr erstes Baby? Holen Sie sich Tipps von anderen Eltern. www.yahoo.de/clever

2 1

[Dovecot] auth-ldap not resetting connection state after failed bind
by Brendan Braybrook 29 Dec '07

29 Dec '07

timo, i think i found a small problem with the ldap implementation: when using auth_bind (this might be in all conditions, not just that one - i haven't tested further), if the server lost connection to the ldap server (ie, the ldap server died) dovecot-auth would never reconnect to the ldap server and all subsequent auth attempts would fail. after a little more digging, i discovered that if the ldap server went down and came back up before the next dovecot request then everything would be fine. however, if a request came in while the ldap server was down then dovecot-auth would "cache" that the server was unavailable and never recheck it. i believe i tracked it down to a couple lines in db_ldap_bind and fixed it - dovecot-auth is reconnecting to ldap in the condition where it was not previously: --- dovecot-1.0.3/src/auth/db-ldap.c 2007-10-15 18:26:55.983349000 +0000 +++ dovecot-1.0.3/src/auth/db-ldap.c.new 2007-10-15 18:28:03.124136000 +0000 @@ -446,7 +446,10 @@ msgid = ldap_bind(conn->ld, conn->set.dn, conn->set.dnpass, LDAP_AUTH_SIMPLE); if (msgid == -1) { - db_ldap_connect_finish(conn, ldap_get_errno(conn)); + if (db_ldap_connect_finish(conn, ldap_get_errno(conn)) < 0) { + /* lost connection, close it */ + ldap_conn_close(conn, TRUE); + } i_free(ldap_request); return -1; }

3 3

[Dovecot] sieve variables
by Pavel Volkovitskiy 29 Dec '07

29 Dec '07

Hello! I wonder if that possible to add support for sieve variables extension in upcoming dovecot 1.1? ( ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-ietf-sieve-variable… ) -- Pavel

3 2

[Dovecot] dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server
by Rolf E. Sonneveld 28 Dec '07

28 Dec '07

Hi, I'm new to the list; excuse me if this has been asked recently/before. running: # dovecot --version 1.0.8 Output of dovecot -n: base_dir: /var/run/dovecot protocols: none ssl_disable: yes login_dir: /var/run/dovecot/login login_executable: /usr/local/dovecot/libexec/dovecot/imap-login auth default: verbose: yes debug: yes debug_passwords: yes passdb: driver: ldap args: /usr/local/dovecot/etc/dovecot-ldap.conf userdb: driver: ldap args: /usr/local/dovecot/etc/dovecot-ldap.conf socket: type: listen client: path: /usr/local/postfix/spool/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot-auth-master mode: 438 Using dovecot only for the purpose of providing SMTP AUTH service for Postfix, using Active Directory (AD) as authentication source. As you can see from the config, connectivity between Dovecot and AD is implemented using LDAP. Contents of /usr/local/dovecot/etc/dovecot-ldap.conf (some site-specific info masqueraded): auth_bind = yes pass_filter = (sAMAccountName=%u) hosts = AD1.domain.com:3268 AD2.domain.com:3268 dn = CN=Service Account Postfix,OU=Services,DC=domain,DC=com dnpass = secret ldap_version = 3 base = DC=domain,DC=com scope = subtree Now, this works fine and authentication via SMTP AUTH + Dovecot works perfect. Yet, every 15 minutes I get the following warning in the syslog logfile: Dec 24 07:20:00 hostname dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server Dec 24 07:35:01 hostname dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server Dec 24 07:50:01 hostname dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server Dec 24 08:05:02 hostname dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server Dec 24 08:20:02 hostname dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server Dec 24 08:35:03 hostname dovecot: auth(default): LDAP: ldap_result() failed: Can't contact LDAP server I've looked through the archives and it seems that this problem is caused by the fact that Dovecot (using the OpenLDAP client libraries?) keeps the LDAP connection open; after (in our case) 15 minutes Active Directory closes the connection and Dovecot signals this in the syslog (and presumably automatically will create a new connection to AD). I'm pretty sure that the OpenLDAP client libraries provide options to use a client-side timeout for LDAP connections. My questions are: * is there a reason that Dovecot wants to keep the LDAP connection open? * Will the new V1.1 version have a config parameter to set the LDAP client timeout or a default timeout value to close the connection? * If there's no 'fix' foreseen for V1.1, I'd like to file a request to add such a parameter. How can I file such a request? The warning is annoying; the sysadmin will create a filter, to filter the warning out from the syslog, but I'm afraid that if there's a real problem with authentication, that will be filtered too and the real problem is not noticed, before users start to complain. /rolf

2 3

Re: [Dovecot] namespace problem / 1.1beta11
by ja nein 28 Dec '07

28 Dec '07

Hi Timo, I think I've hit a separator bug: This is namespace query of dovecot and source server: ---snip--- * OK Dovecot ready. 1 login ... 1 OK Logged in. 2 NAMESPACE * NAMESPACE (("INBOX/" ".")) NIL NIL 2 OK Namespace completed. 3 logout * BYE Logging out 3 OK Logout completed. Connection closed by foreign host. Escape character is '^]'. * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS AUTH=CRAM-MD5 AUTH=DIGEST-MD5 AUTH=NTLM AUTH=SRP AUTH=OTP AUTH=PASSDSS-3DES-1 SASL-IR] Cyrus IMAP4 v2.3.8 server ready 1 login ... 1 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID LOGINDISABLED ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH] User logged in 2 NAMESPACE * NAMESPACE (("INBOX/" "/")) (("user/" "/")) (("" "/")) 2 OK Completed 3 logout * BYE LOGOUT received ---snip--- So Dovecot is stating Namespace separator is ".". But my dovecot -n namespace section shows: namespace: type: private separator: / prefix: INBOX/ inbox: yes list: yes subscriptions: yes According to this I've set separator to "/". And this is the problem. I think problem arised somewhere between beta9 and beta11, as it worked before... The separator in imapsync was for both "/" (--sep1 /, --sep2 /), I just played with all possibilities... Well, never mind if v1.2 or v2.0, as long as LMTP makes it in ;) Anyway, we can provide some good stress testing in such an environment... Regards and for all Merry Christmas and a happy new year! Sebastian ----- Ursprüngliche Mail ---- Von: Timo Sirainen <tss(a)iki.fi> An: ja nein <reg9009(a)yahoo.de> CC: dovecot(a)dovecot.org Gesendet: Freitag, den 21. Dezember 2007, 18:19:31 Uhr Betreff: Re: [Dovecot] namespace problem / 1.1beta11 On Wed, 2007-12-12 at 07:16 +0000, ja nein wrote: > From Folder [INBOX/Trash] .. > /home/imapsync/imapsync --host1 ip.of.host.1 --user1 user1 --password1 password --host2 127.0.0.1 --user2 user2 --password2 password --authmech2 LOGIN --regextrans2 s/\./_/g --delete2 --sep1 . --sep2 . --justfolders Shouldn't those be -sep1 / -sep2 /. Or at least the one that's on Dovecot's side. > Btw., as we work in a load balanced environment, LMTP delivery would be a great feature for upcoming releases, as it would enable people to use multiple frontends of SMTP servers without the need of connecting them to a somewhat shared file system. Forwarding all mail to a SMTP server on the machine running dovecot and doing delivery there isn't the favoured option (at least for us). Yes, LMTP would be nice, but it pretty much requires v2.0 framework. I'm still not sure if the next version will be v1.2 or v2.0.. Jetzt Mails schnell in einem Vorschaufenster überfliegen. Dies und viel mehr bietet das neue Yahoo! Mail - www.yahoo.de/mail

3 2

[Dovecot] assertion on deliver (1.1.beta10)
by Nuno Lopes 28 Dec '07

28 Dec '07

Hi, I'm testing version 1.1.beta10 and getting these assertions on deliver: file index-transaction.c: line 54 (index_transaction_finish_rollback): assertion failed: (t->mail_ref_count == 0) This was after an upgrade from a 1.0 release candidate. If you need I can try and reproduce this in a more controled environment to pinpoint the problem. Thanks, -- Nuno Lopes

2 4

[Dovecot] Dovecot + Cygwin the 2nd
by Ronny 28 Dec '07

28 Dec '07

Hi, i still try to get dovecot to run under cygwin. After a post on the cygwin list i am able tu run dovecot. Under Cygwin you have to run dovecot by inetd. Starting the binary "dovecot.exe" does not work. Well the server is running but answers with a fatal error. Account: 'localhost', Server: 'localhost', Protokoll: POP3, Serveranswer: 'Fatal: EOF while reading environment from master', Port: 110, Secure (SSL): No, error on server: 0x800CCC90, Fehlernummer: 0x800CCC90 This error is from Outlook, Thunderbird just says there is an error without any error message. So maybe you can help me to bring my story to a hapy end ;-) Cheers Ronny

3 11