November 2009 - dovecot

[Dovecot] PostgreSQL connection bug
by Daniel Howard 10 Dec '09

10 Dec '09

Hello, There seems to be a bug which can affect systems using PostgreSQL as a user or password database. Usually there is just one postgres connection, but if the query fails for any reason, the failed connection is left open and dovecot will keep opening new connections for each auth attempt. I discovered this with a setup using PAM, AND a database to authenticate. Most users login with an account number (integer) which is looked up in the postgres database. Some users have a unix system account, and use their system username to log in (string). Dovecot was set up to check the database first, then try PAM. When a system user logged in, the SQL query failed because it couldn't coerce the system user's username into an integer. In these cases, Dovecot would still fall back to PAM authentication, and the user would log in, but the postgres connection would remain stuck. Gradually the number of connections would increase until postgres failed. I worked around this by converting the account numbers in the database from integers to strings. Now the query never fails and there's only ever 1 dovecot connection. Hopefully this will help someone. dovecot --version 1.0.15 dovecot -n # 1.0.15: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login mail_privileged_group: mail mail_location: maildir:~/Maildir auth default: passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf passdb: driver: pam userdb: driver: passwd userdb: driver: prefetch cat dovecot-sql.conf | egrep -v "^#" driver = pgsql connect = host=localhost dbname=rapidgroup user=dovecot password=********* default_pass_scheme = PLAIN password_query = SELECT userid as user, password, userdb_mail, userdb_uid, userdb_gid FROM dovecotview WHERE userid='%u' Cheers, Dan

3 6

[Dovecot] Virtual mailboxes not always up to date?
by Patrick Nagel 10 Dec '09

10 Dec '09

Hi, I have been experimenting with the Virtual mailboxes plugin [1] recently. This is my setup: the following dovecot-virtual file exists in a virtual mailbox called "todo": ------ dovecot-virtual ------ * -Trash -Trash/* -Spam -Spam/* OR KEYWORD $TODO KEYWORD todo ----------------------------- so if any mail in any mailbox except Trash+Spam has the keyword "$TODO" (that's what KMail sets when you mark a message as "action item") or "todo" (that's what Thunderbird sets when you put the Todo tag on a message), it should be shown in that 'todo' virtual mailbox. This works for the first time after I edit the dovecot-virtual file and then access the virtual mailbox (all mails I expect to be shown in the todo mailbox are there) - but further accesses always show the same content, even though I removed the keyword from one of the mails (and I can confirm that the keyword has actually been removed by dovecot, since the filename doesn't contain the letter corresponding to the keyword anymore). It seems that changing the dovecot-virtual file triggers the execution of the "search program" - but shouldn't simply accessing the virtual mailbox also trigger it? Or is there a (undocumented?) timeout between search program executions? Patrick. [1]: http://wiki.dovecot.org/Plugins/Virtual # dovecot -n # 1.2.6: /etc/dovecot/dovecot.conf # OS: Linux 2.6.22-vs2.2.0.7-gentoo x86_64 Gentoo Base System release 2.0.1 protocols: imaps managesieve listen: x.x.x.x ssl: required ssl_ca_file: /etc/ssl/dovecot/sub.class1.server.ca.crt ssl_cert_file: /etc/ssl/dovecot/ssl.crt ssl_key_file: /etc/ssl/dovecot/ssl.key login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login mail_max_userip_connections(default): 20 mail_max_userip_connections(imap): 20 mail_max_userip_connections(managesieve): 10 mail_location: maildir:~/.maildir mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_process_size: 2048 mail_plugins(default): virtual fts fts_squat mail_plugins(imap): virtual fts fts_squat mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve imap_client_workarounds(default): tb-extra-mailbox-sep imap_client_workarounds(imap): tb-extra-mailbox-sep imap_client_workarounds(managesieve): namespace: type: private separator: . inbox: yes list: yes subscriptions: yes namespace: type: private separator: . prefix: virtual. location: virtual:~/.maildir/virtual list: yes subscriptions: yes lda: postmaster_address: postmaster@x mail_plugins: sieve virtual auth default: passdb: driver: pam args: * userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix plugin: fts: squat fts_squat: partial=4 full=10 -- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779 PGP key: E883A005 https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005

3 7

[Dovecot] 2.0.alpha3: imap-login segfault in hex_to_binary
by Mike Abbott 10 Dec '09

10 Dec '09

Dovecot-2.0.alpha3's imap-login segfaulted with this backtrace: 0 hex_to_binary 1 master_send_request 2 anvil_input 3 io_loop_handler_run 4 io_loop_run 5 master_service_run 6 main 7 start Looks to me like auth_client_get_cookie() returned NULL in master_send_request().

2 1

[Dovecot] Bug report
by Bryan Seitz 10 Dec '09

10 Dec '09

One of my users is causing dovecot to exit with signal 6 and generate a core file. Seems to be in the mbox code. Dovecot configuration: # 1.2.4: /usr/local/etc/dovecot.conf # OS: FreeBSD 7.2-STABLE amd64 protocols: imap imaps pop3s listen(default): localhost:143 listen(imap): localhost:143 listen(pop3): * ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /etc/ssl/certs/mail.bsd-unix.net.pem ssl_key_file: /etc/ssl/certs/mail.bsd-unix.net.pem login_dir: /var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 100 mail_privileged_group: mail mail_location: mbox:~/mail/:INBOX=/var/mail/%u mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_process_size: 1024 mail_plugins(default): fts fts_squat mail_plugins(imap): fts fts_squat mail_plugins(pop3): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep imap_client_workarounds(pop3): pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh lda: postmaster_address: postmaster(a)example.com sendmail_path: /usr/sbin/sendmail auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: fts: squat FreeBSD triton.bsd-unix.net 7.2-STABLE FreeBSD 7.2-STABLE #17: Sat Oct 3 15:47:44 EDT 2009 root@triton.bsd-unix.net:/usr/obj/usr/src/sys/TRITON amd64 dovecot-1.2.4_1 Secure and compact IMAP and POP3 servers CPU Arch: x86_64 Filesystem: UFS (gdb) bt full #0 0x0000000040b2989c in kill () from /lib/libc.so.7 No symbol table info available. #1 0x0000000040b286bb in abort () from /lib/libc.so.7 No symbol table info available. #2 0x00000000004d9165 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:160 backtrace = 0x40d4d048 "" #3 0x00000000004d9f7e in i_internal_fatal_handler (type=Could not find the frame base for "i_internal_fatal_handler". ) at failures.c:440 No locals. #4 0x00000000004d944a in i_panic (format=Could not find the frame base for "i_panic". ) at failures.c:207 args = Could not find the frame base for "i_panic". (gdb) bt #0 0x0000000040b2989c in kill () from /lib/libc.so.7 #1 0x0000000040b286bb in abort () from /lib/libc.so.7 #2 0x00000000004d9165 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:160 #3 0x00000000004d9f7e in i_internal_fatal_handler (type=Could not find the frame base for "i_internal_fatal_handler". ) at failures.c:440 #4 0x00000000004d944a in i_panic (format=Could not find the frame base for "i_panic". ) at failures.c:207 #5 0x000000000045aea8 in istream_raw_mbox_get_body_size (stream=0x40d76070, expected_body_size=18446744073709551615) at istream-raw-mbox.c:578 #6 0x00000000004622f3 in mbox_sync_read_next_mail (sync_ctx=0x7fffffffe290, mail_ctx=0x7fffffffe0b0) at mbox-sync.c:151 #7 0x00000000004649f6 in mbox_sync_loop (sync_ctx=0x7fffffffe290, mail_ctx=0x7fffffffe0b0, partial=false) at mbox-sync.c:1004 #8 0x0000000000465c00 in mbox_sync_do (sync_ctx=0x7fffffffe290, flags=MBOX_SYNC_UNDIRTY) at mbox-sync.c:1535 #9 0x00000000004667b8 in mbox_sync_int (mbox=0x40d23848, flags=MBOX_SYNC_UNDIRTY, lock_id=0x7fffffffe4b8) at mbox-sync.c:1842 #10 0x0000000000466a26 in mbox_sync (mbox=0x40d23848, flags=MBOX_SYNC_UNDIRTY) at mbox-sync.c:1894 #11 0x0000000000466b8a in mbox_storage_sync_init (box=0x40d23848, flags=65) at mbox-sync.c:1936 #12 0x0000000000488d34 in mailbox_sync_init (box=0x40d23848, flags=65) at mail-storage.c:562 #13 0x0000000000488dfa in mailbox_sync (box=0x40d23848, flags=65, status_items=239, status_r=0x7fffffffe580) at mail-storage.c:593 #14 0x000000000041f77a in select_open (ctx=0x40d4d148, mailbox=0x40d52170 "Drafts", readonly=false) at cmd-select.c:273 #15 0x000000000041fba0 in cmd_select_full (cmd=0x40d4d048, readonly=false) at cmd-select.c:381 #16 0x000000000041fbda in cmd_select (cmd=0x40d4d048) at cmd-select.c:388 #17 0x0000000000423164 in client_command_input (cmd=0x40d4d048) at client.c:611 #18 0x00000000004233a1 in client_command_input (cmd=0x40d4d048) at client.c:660 #19 0x00000000004234e3 in client_handle_next_command (client=0x40d45030, remove_io_r=0x7fffffffe73d) at client.c:701 #20 0x0000000000423568 in client_handle_input (client=0x40d45030) at client.c:713 #21 0x00000000004236d4 in client_input (client=0x40d45030) at client.c:752 #22 0x00000000004e5da9 in io_loop_handler_run (ioloop=0x40d1f0c0) at ioloop-kqueue.c:149 #23 0x00000000004e4dfb in io_loop_run (ioloop=0x40d1f0c0) at ioloop.c:335 #24 0x000000000042f016 in main (argc=3, argv=0x7fffffffe8a0, envp=0x7fffffffe8c0) at main.c:327 Log messages: Nov 11 23:11:20 triton dovecot: IMAP(sam): Panic: file istream-raw-mbox.c: line 578 (istream_raw_mbox_get_body_size): assertion failed: (rstream->body_offset != (uoff_t)-1) Nov 11 23:11:20 triton kernel: pid 35530 (imap), uid 3033: exited on signal 6 (core dumped) Any help would be appreciated! -- Bryan G. Seitz

2 1

[Dovecot] expire_dict - Folder names with spaces
by michael＠euroinfissi.eu 10 Dec '09

10 Dec '09

Hi! How to deal with IMAP foldernames that include spaces when setting up the expire tool for them? I have the folder Elementy usuni&ARk-te but expire: Wiadomo&AVs-ci-&AVo-mieci 7 Elementy?usuni&ARk-te 7 INBOX 365 does not work... and expire: Wiadomo&AVs-ci-&AVo-mieci 7 Elementy usuni&ARk-te 7 INBOX 365 produces the following: Stopping Dovecot Imap: [ OK ] Starting Dovecot Imap: ☺ILoading modules from directory: /usr/lib64/dovecot/imap ☺IModule loaded: /usr/lib64/dovecot/imap/lib10_quota_plugin.so ☺IModule loaded: /usr/lib64/dovecot/imap/lib11_imap_quota_plugin.so ☺IModule loaded: /usr/lib64/dovecot/imap/lib20_autocreate_plugin.so ☺IModule loaded: /usr/lib64/dovecot/imap/lib20_expire_plugin.so ☺IEffective uid=99, gid=99, home=/tmp ☺IQuota root: name=user backend=dict args=:proxy::quota ☺IQuota rule: root=user mailbox=* bytes=1073741824 messages=0 ☺IQuota warning: bytes=1020054732 (95%) messages=0 command=/usr/local/bin/quota-warning.sh 95 ☺IQuota warning: bytes=858993459 (80%) messages=0 command=/usr/local/bin/quota-warning.sh 80 ☺Iexpire: pattern=Wiadomo&AVs-ci-&AVo-mieci type=expunge secs=604800 ☺Iexpire: pattern=Elementy type=expunge secs=0 ☺Iexpire: pattern=7 type=expunge secs=0 ☺Fexpire: Missing expire days for mailbox '365' Error: imap dump-capability process returned 89 Fatal: Invalid configuration in /etc/dovecot.conf [FAILED] My conf: # OS: Linux 2.6.30.9-96.fc11.x86_64 x86_64 Fedora release 11 (Leonidas) info_log_path: /var/log/maillog protocols: imaps ssl_listen: * ssl: required ssl_cert_file: /etc/pki/certs/dovecot/dovecot.pem ssl_key_file: /etc/pki/certs/dovecot/dovecot_key.pem ssl_cipher_list: ALL:!ADH!LOW:!SSLv2:!EXP:+HIGH:+MEDIUM disable_plaintext_auth: yes verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login mail_privileged_group: dovecot_mail mail_debug: yes mail_plugins: autocreate quota imap_quota expire lda: postmaster_address: root@localhost auth default: passdb: driver: sql args: /etc/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot-sql.conf plugin: quota: dict:user::proxy::quota quota_rule: *:storage=1048576 quota_warning: storage=95%% /usr/local/bin/quota-warning.sh 95 quota_warning2: storage=80%% /usr/local/bin/quota-warning.sh 80 expire: Wiadomo&AVs-ci-&AVo-mieci 7 Elementy?usuni&ARk-te 7 INBOX 365 expire_dict: proxy::expire dict: quota: mysql:/etc/dovecot-dict-quota.conf expire: mysql:/etc/dovecot-dict-expire.conf Thanks! Michael Lattari

3 4

[Dovecot] Patch: event port-based ioloop and notify
by Emanuele Pucciarelli 10 Dec '09

10 Dec '09

Greetings, thanks to all of you who work on Dovecot! I have prepared a small patch to support Solaris 10 and Opensolaris' event port mechanism for both the ioloop and the notify subsystems. It seems to work fine for me, but I haven't conducted any extensive testing. It would be great if someone could review and/or test it (and if it could eventually enter the code base). I have uploaded a copy at http://cr.opensolaris.org/~e.p/dovecot/dovecot-01-fen.diff . Thanks! -- Emanuele

2 3

[Dovecot] v2.0.alpha3 released
by Timo Sirainen 09 Dec '09

09 Dec '09

http://dovecot.org/releases/2.0/alpha/dovecot-2.0.alpha3.tar.gz http://dovecot.org/releases/2.0/alpha/dovecot-2.0.alpha3.tar.gz.sig Nothing horribly exciting since alpha2, but since I did other releases I thought I might as well release this one too. - Most of the stuff in master.conf is now built into binaries so the file itself is small. dovecot -n also doesn't show the defaults anymore. - Settings now have size and time types. So instead of always wondering if the units are bytes or kilobytes or megabytes, you can just use e.g. 100k, 100M values. And the same for 100 secs, 100 mins, 100 hours, etc. - verbose_proctitle=yes was pretty useless in alpha2 after the pre/post-login changes. I've now enabled Linux proctitle hack that allows changing process title (in ps). It now shows e.g. the current command being executed by imap process. There is also some hope of getting a proper setproctitle() function to Linux, the patch just got to -mm code tree today! - anvil process now stays alive across SIGHUPs. So "doveadm who" no longer forgets users after SIGHUP. - remote_ip IP {} and local_ip IP {} changed to remote host|IP {} and local host|IP {}, i.e. if it's not an IP Dovecot does host name lookup. - Using host names is also important to get TLS SNI working. With TLS SNI support if client supports it, you can serve multiple certificates with a single IP address. - LMTP server adds now Received: header - Fixes to LMTP server - Fixes to dsync Some of the larger changes left: - plugin API probably needs to be changed, so that it's possible to handle multiple users by same process but with different plugins - config process is too slow - rawlog no longer works because of how pre/post-login works now. it probably needs to be changed into a service and configured something like: service imap-rawlog { executable = rawlog imap-real /home/%u unix_listener imap { } } service imap { unix_listener imap-real { } } The point would be that when imap-login connects to imap, it actually connects to rawlog which then connects to imap-real and starts proxying the connection. A few problems: 1) userdb lookup is done by imap process, so rawlog doesn't know user's home directory. Perhaps more code could be added so that rawlog requests imap process to send it back, but maybe that's too much trouble. That's why I thought just adding a 3rd /home/%u parameter, where rawlog expands %u to username since that's at least known. 2) Currently it's not possible to remove default internal unix_listeners, so the above config wouldn't actually work anyway. I guess there needs to be some new syntax like "!unix_listener imap" or something that removes it.. Then again, a nice thing about this is that the rawlog will work with all types of services, even imap-logins, although that's going to be almost useless for SSL connections.

3 9

Re: [Dovecot] fts squat non-english search for 2 words
by vuser1＠test123.ru 09 Dec '09

09 Dec '09

Timo Sirainen <tss(a)iki.fi>: > On Sun, 2009-11-22 at 20:35 +0700, vuser1(a)test123.ru wrote: >> Timo, thank you for answer. Meanwhile I was trying to setup >> horde+dovecot+search. Next step was dovecot 1.2.4 + solr 1.4. It >> works! Now it can find 2 non-latin words. >> 1) I cannot search by substrings - neither "plane" nor "plane*" does >> find "planet" > > Try if attached patch helps? > Quick answer is "no" 8)). Now the story. I debugged and realized that patched plugin generates - for search "xxx yyy": q=body:"XXX YYY*" It should be: q=body:XXX* +body:YYY* (not q=body:"XXX*" +body:"YYY*" - quotation does matter) But this does not work as expected. Prefix searches (with asterisk) are case-sensitive. I googled around and found this post - http://michaelkimsal.com/blog/solr-case-sensitivty/comment-page-1/#comment-… . It is old - 2007, but it looks SOLR is still case-sensitive for *. Because of dovecot capitalizes query (and this is right, I think), the search will never find a thing. I played with Solr admin for 3 evenings and I have to say - its behaviour is strange. For example, if I send several emails with body of 2 words: "xxx yyy", "yyy xxx", "xxX yYY" etc. - different case and different word order - it does not find "xxx" in all emails. Maybe Solr 1.4 is not production-ready yet and 1.3 is better. But it is enough for me, maybe I will return to it in next year. Now I will try to apply your changeset (fts-squat: Fixed searching multi-byte characters) to dovecot 1.2.4 (debian stable). If you think 1.2.8 is better, I will follow your recommendation.

2 1

[Dovecot] postfix+sasl+dovecot script help!
by Tudod Ki 09 Dec '09

09 Dec '09

postfix+sasl+dovecot script help! Hi everybody! I am trying to get the solution for days now, and I don't know what to do really :S I just want to write a script, what does automatically install postfix, dovecot, and it configures sasl authentication for the clients. I'm here right now [the script!!]: #################### http://pastebin.ca/1687824 [I have written the reference howto in the script.] #################### I run it after a fresh Lenny netinstall, on VirtualBox. If someone knows, what am I missing, please help me :\ I can see with "netstat -tulpn", that the server is listeing on port 993,955,25. #################### 1.) But: when I want to get the e-mails through imap, with Thunderbird, on another machine, it just waits, and waits, and waits...[and finally it gives this: "Connection to server debian.lan timed out."] :\ logs: mail.info log: http://pastebin.com/f6c486374 mail.log: http://pastebin.com/f6e60f9b other logs are empty #################### 2.) When I want to send a message from a testuser to a testuser, with Thunderbird, on the client, it just keeps asking for the password, when I want click send. :S logs: mail.info: http://pastebin.com/f1bca774f mail.log: http://pastebin.com/f5c0be27c mail.warn: http://pastebin.com/f46806d2f other logs are empty #################### Info about the os & softwares [e.g.: "dpkg -l | grep postfix"]: http://pastebin.com/f1d0cefd2 192.168.56.4 is the client 192.168.56.5 is the server [VirtualBox, Host-Only networking, they can ping each other] #################### Later I want to use it with Squirrel Mail [plus spam filtering + antivirus], but first, I just want to get it work :( Thank you for any help :S [_good_ docs, howtos, solution, or anything please! :S :( ]

2 1

[Dovecot] LDA: handling non-existent mail dirs
by Edgar Fuß 09 Dec '09

09 Dec '09

We have users existing in LDAP but not supposed to receive mail. We used to handle that (somewhat ungracefully) by simply not creating the top mail directory for those accounts. Since switching to Dovecots LDA, these non-existing dirs are considered temporary failures and so the junk mails pile up in postfix' queue Would it be possible for deliver to optionally treat this as a permanent error? I know the clean solution woud be checking for an LDAP attribute in postfix before even accepting the mail.

3 2