April 2009 - dovecot - dovecot.org

[Dovecot] Dovecot, LVS and the issues I have with it.
by neil 08 Apr '09

08 Apr '09

We run around 5 dovecot (debian etch 1.0.rc15) POP/IMAP 'nodes' using the LVS load balancer and an NFS based SAN. it works pretty well. I love the robustness of load balancing POP/IMAP. We do a reasonable amount of throughput through these especially at peak times pushing our SAN to around 1.5k IOP/s We currently have two issues with this setup. One of which is NFS index corruption issues we get due to NFS/dovecot locking. Basically the UUID list or a .index gets corrupt. This causes a full re-indexing of the mailbox / broken mailbox until i delete the indexes. In the UUID lists case the symptom tends to effect use who use POP rather than IMAP and insist on keeping messages on the server. Because it's corrupt it gets rebuilt one way or the other and the users email client proceeds to redownload the entire mailbox again until he remarks them to be saved. This tends to annoy the user a lot. After a bit of testing we do however expect this to be fixed by version 1.1. However if anyone has any comments on this I would certainly be interested. The other issue is a little tricky to describe or even log effectively. Occasionally a node basically receives more connections than it is able to handle. The connection count goes through the roof for that node. It will go up way beyond 150 dovecot authentication threads and 100 or so active POP/IMAP dovecot threads. The IO_wait/cpu and memory usage also tends to be spiking at this point. The server gets to the tipping point where it can no longer serve it's POP/IMAP requests fast enough compared to the number of connections it's getting. I'd be fine with this but this creates some less than desirable symptoms: 1. We obviously reach the auth thread cap eventually so any new auth requests basically get refused by the server. To the user this is a seen as an outlook/mail client re-auth pop-up request. This annoys them. Ideally if the server stops accepting auth requests it should fall off our load balancer until it can consistently accept them again. Since the LVS detects a node fail by whether the tcp port is still open this doesn't happen, since dovecot keeps the port open. This is obviously more an LVS issue and not for this mailing list I expect unless anyone has any config tweaking tips here? 2. Now here's my real gripe. Dovecot does not handle running out resources very gracefully at all in our setup. It does start killing threads after a while. I get multiple *"dovecot: child 17989 (login) killed with signal 9". *I'm not exactly sure what's going on here because after this all I can see is the machine totally out of memory and the kernel starts killing absolutely everything. All services are killed (including ssh etc..) and I plug a monitor into the server and find the last few lines of the console listing init and other rather important things having just been killed. At this point it is a case of power cycling the server and all is back to normal again. I imagine there's not a huge amount of people using dovecot in this way. But anyone got any recommendations here? I really like using dovecot in this setup it handles it pretty well and the redundancy and functionality options it provides have been invaluable. Neil

4 3

[Dovecot] Global Sieve File
by James Butler 08 Apr '09

08 Apr '09

Is anyone here using a global Sieve file that handles messages for an entire server with many users? I understand and use local Sieve files, but I would like to learn more about how to set up Sieve to filter ALL incoming messages using a single file. I would love to read about how you managed to get it working. I'm running Fedora 10 with Postfix 2.5.5 and Dovecot 1.2.rc2. Thanks for any insight! James

4 6

[Dovecot] dovecot doesnt see any emails
by alexus 08 Apr '09

08 Apr '09

$ dovecot --version 1.1.13 $ grep ^mail_location /usr/local/etc/dovecot.conf mail_location = maildir:~vpopmail/domains/%d/%u/Maildir/ $ i use SquirrelMail as a client to IMAP in, but when I login, I dont see any emails in my INBOX I even tried pop3 same story $ telnet localhost pop3 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. +OK Dovecot ready. USER XXXXXX(a)XXX.XXX +OK PASS XXXXXX +OK Logged in. UIDL +OK . QUIT +OK Logging out. Connection closed by foreign host. $ -- http://alexus.org/

4 8

[Dovecot] Compiling v1.3 on different OSes
by Timo Sirainen 08 Apr '09

08 Apr '09

I was hoping to finally get shared libdovecot.so and libdovecot-storage.so libraries for v1.3, so Sieve (and maybe others) could link against them. But I'm running into trouble getting it to compile in Solaris 10. Could you non-Linux users test if this compiles with you? http://dovecot.org/tmp/dovecot-1.3.UNSTABLE.tar.gz Or if anyone has ideas why this happens, I'd like to know: libtool: link: gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -I/home/cras/include -o .libs/rawlog rawlog.o ../lib-dovecot/.libs/libdovecot.so -lrt -lsocket -lsendfile -Wl,-rpath -Wl,/usr/local/lib/dovecot ../lib-dovecot/.libs/libdovecot.so: undefined reference to `dlerror(a)SUNW_1.22' ../lib-dovecot/.libs/libdovecot.so: undefined reference to `dlopen(a)SUNW_1.22' ../lib-dovecot/.libs/libdovecot.so: undefined reference to `dlsym(a)SUNW_1.22' ../lib-dovecot/.libs/libdovecot.so: undefined reference to `dlclose(a)SUNW_1.22' If I disable plugins (so dl*() functions aren't used) it compiles, so it's only those functions that are problematic. pvs -s /lib/libc.so lists the dl* functions under SUNW_1.22 so I'd think that should have worked..

9 20

[Dovecot] Segfault in dovecot 1.2rc2 with quota + public namespace enabled
by Markus Werner 07 Apr '09

07 Apr '09

Hi, I have configured a public namespace ("#Shared"). Since dovecot 1.2rc2 (1.1.11 worked fine) I get a segfault, if I try to access a folder under this namespace when quota plugin is enabled. When the quota plugin is disabled, accessing those folders works. $ dovecot -n # 1.2.rc2: /opt/dovecot/etc/dovecot.conf # OS: Linux 2.6.26-1-686 i686 Debian 5.0 base_dir: /var/run/dovecot protocols: imap pop3 listen: 127.0.0.1,192.168.128.151 ssl_listen: 192.168.128.151 ssl_ca_file: /opt/dovecot/etc/certs/cacert.pem ssl_cert_file: /opt/dovecot/etc/certs/asterixCert.pem ssl_key_file: /opt/dovecot/etc/certs/asterixKey.pem ssl_parameters_regenerate: 0 ssl_cipher_list: ALL:!LOW:!SSLv2 disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_user: dovecotl login_process_per_connection: no login_processes_count: 1 login_max_processes_count: 3 max_mail_processes: 100 mail_max_userip_connections: 8 first_valid_uid: 2000 last_valid_uid: 2000 first_valid_gid: 2000 last_valid_gid: 2000 mail_access_groups: dovecot mail_uid: 2000 mail_gid: 2000 mail_location: maildir:~/Maildir mail_debug(default): yes mail_debug(imap): yes mail_debug(pop3): no mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota acl imap_acl fts fts_squat mail_plugins(imap): quota imap_quota acl imap_acl fts fts_squat mail_plugins(pop3): mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 pop3_no_flag_updates(default): no pop3_no_flag_updates(imap): no pop3_no_flag_updates(pop3): yes namespace: type: private separator: / prefix: INBOX/ inbox: yes list: yes subscriptions: yes namespace: type: public separator: / prefix: #Shared/ location: maildir:/var/spool/dovecot/sharedfolders:INDEX=~/Maildir/index/public:CONTROL=~/Maildir/control/public list: yes subscriptions: yes namespace: type: shared separator: / prefix: #User/%%u/ location: maildir:%%h/Maildir:INDEX=~/Maildir/index/shared/%%u list: yes auth default: mechanisms: plain login user: dovecota worker_max_count: 3 passdb: driver: passwd-file args: /opt/dovecot/etc/passwd userdb: driver: passwd-file args: /opt/dovecot/etc/passwd socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 432 user: dovecota group: dovecot plugin: sieve: ~/.dovecot/.dovecot.sieve acl: vfile:/opt/dovecot/etc/dovecot-acls acl_shared_dict: file:/var/lib/dovecot/shared-mailboxes.db acl_anyone: allow quota: dict:user::file:%h/Maildir/dovecot-quota quota_rule: *:storage=10G:messages=500000 fts: squat fts_squat: partial=4 full=10 $ gdb /opt/dovecot/libexec/dovecot/imap /var/spool/dovecot/users/markus/core GNU gdb 6.8-debian Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu"... warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/i686/cmov/libdl.so.2...done. Loaded symbols for /lib/i686/cmov/libdl.so.2 Reading symbols from /lib/i686/cmov/libc.so.6...done. Loaded symbols for /lib/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /opt/dovecot/lib/dovecot/imap/lib01_acl_plugin.so...done. Loaded symbols for /opt/dovecot/lib/dovecot/imap/lib01_acl_plugin.so Reading symbols from /opt/dovecot/lib/dovecot/imap/lib02_imap_acl_plugin.so...done. Loaded symbols for /opt/dovecot/lib/dovecot/imap/lib02_imap_acl_plugin.so Reading symbols from /opt/dovecot/lib/dovecot/imap/lib10_quota_plugin.so...done. Loaded symbols for /opt/dovecot/lib/dovecot/imap/lib10_quota_plugin.so Reading symbols from /opt/dovecot/lib/dovecot/imap/lib11_imap_quota_plugin.so...done. Loaded symbols for /opt/dovecot/lib/dovecot/imap/lib11_imap_quota_plugin.so Reading symbols from /opt/dovecot/lib/dovecot/imap/lib20_fts_plugin.so...done. Loaded symbols for /opt/dovecot/lib/dovecot/imap/lib20_fts_plugin.so Reading symbols from /opt/dovecot/lib/dovecot/imap/lib21_fts_squat_plugin.so...done. Loaded symbols for /opt/dovecot/lib/dovecot/imap/lib21_fts_squat_plugin.so Reading symbols from /lib/i686/cmov/libnss_compat.so.2...done. Loaded symbols for /lib/i686/cmov/libnss_compat.so.2 Reading symbols from /lib/i686/cmov/libnsl.so.1...done. Loaded symbols for /lib/i686/cmov/libnsl.so.1 Reading symbols from /lib/i686/cmov/libnss_nis.so.2...done. Loaded symbols for /lib/i686/cmov/libnss_nis.so.2 Reading symbols from /lib/i686/cmov/libnss_files.so.2...done. Loaded symbols for /lib/i686/cmov/libnss_files.so.2 Core was generated by `imap'. Program terminated with signal 11, Segmentation fault. [New process 31957] #0 0x080ec1e2 in array_idx_modifiable_i (array=0x18, idx=0) at array.c:10 10 pos = idx * array->element_size; (gdb) bt full #0 0x080ec1e2 in array_idx_modifiable_i (array=0x18, idx=0) at array.c:10 pos = 0 #1 0xb7dbf88a in quota_get_mail_user_quota (user=0x0) at quota-storage.c:482 No locals. #2 0xb7db98d8 in quota_transaction_begin (box=0x9771440) at quota.c:739 user = (struct mail_user *) 0x0 #3 0xb7dbfdee in quota_mailbox_transaction_begin (box=0x9771440, flags=MAILBOX_TRANSACTION_FLAG_HIDE) at quota-storage.c:87 t = (struct mailbox_transaction_context *) 0x977bac0 qt = <value optimized out> #4 0xb7daf55e in fts_transaction_begin (box=0x9771440, flags=MAILBOX_TRANSACTION_FLAG_HIDE) at fts-storage.c:937 fbox = (struct fts_mailbox *) 0x978ac80 t = <value optimized out> #5 0x080b38ae in mailbox_transaction_begin (box=0x9771440, flags=MAILBOX_TRANSACTION_FLAG_HIDE) at mail-storage.c:787 trans = (struct mailbox_transaction_context *) 0x18 #6 0x08069f2b in imap_fetch_begin (ctx=0x9769d18) at imap-fetch.c:349 No locals. #7 0x08061bb0 in cmd_fetch (cmd=0x9769bd8) at cmd-fetch.c:226 client = <value optimized out> ctx = (struct imap_fetch_context *) 0x9769d18 args = (const struct imap_arg *) 0x976ec58 next_arg = (const struct imap_arg *) 0x18 search_args = (struct mail_search_args *) 0x9773ba8 messageset = <value optimized out> ret = <value optimized out> #8 0x0806700c in client_command_input (cmd=0x9769bd8) at client.c:603 client = (struct client *) 0x9769948 command = <value optimized out> __PRETTY_FUNCTION__ = "client_command_input" #9 0x080670a9 in client_command_input (cmd=0x9769bd8) at client.c:652 client = (struct client *) 0x9769948 command = <value optimized out> __PRETTY_FUNCTION__ = "client_command_input" #10 0x080676ed in client_handle_input (client=0x9769948) at client.c:693 _data_stack_cur_id = 3 ret = <value optimized out> remove_io = <value optimized out> handled_commands = false #11 0x08067ba3 in client_input (client=0x9769948) at client.c:748 cmd = <value optimized out> output = (struct ostream *) 0x9769afc bytes = <value optimized out> __PRETTY_FUNCTION__ = "client_input" #12 0x080f73f0 in io_loop_handler_run (ioloop=0x9762ae8) at ioloop-epoll.c:208 ctx = (struct ioloop_handler_context *) 0x9762bf0 event = (const struct epoll_event *) 0x9762c30 list = (struct io_list *) 0x9769b80 io = (struct io_file *) 0x9769b58 tv = {tv_sec = 1, tv_usec = 729686} t_id = 2 msecs = <value optimized out> ret = 1 i = 0 j = 0 call = <value optimized out> #13 0x080f6880 in io_loop_run (ioloop=0x9762ae8) at ioloop.c:338 No locals. #14 0x080704c5 in main (argc=Cannot access memory at address 0x0 ) at main.c:320 No locals. -- Markus

2 4

Re: [Dovecot] Compiling v1.3 on different OSes
by Alan Ferrency 07 Apr '09

07 Apr '09

> On Mon, 2009-04-06 at 14:08 -0400, Timo Sirainen wrote: > > http://dovecot.org/tmp/dovecot-1.3.UNSTABLE.tar.gz > > OK, updated the URL once more. Now iconv problems should be gone? And > everyone else is happy with it too? :) It worked fine in FreeBSD 6.2. Thanks, Alan Ferrency pair Networks, Inc.

1 0

[Dovecot] v1.3 plugin locations
by Timo Sirainen 07 Apr '09

07 Apr '09

It's getting more and more annoying to have separate plugin directories for lda, pop3 and imap. Currently all the plugins are installed into $lib/dovecot/ and then symlinked to the lda, pop3, imap directories. The original reason was that v0.99 just loaded everything in those directories since there was no mail_plugins setting. But then there are things like expire-tool and convert-tool (and the upcoming doveadm) that don't load plugins at all, even though they probably should. Adding just more directories to them seems to be pointless. Also SQL plugins are shared by dovecot-auth and dict and those would need symlinking.. And in future you might want to load SQL plugins to e.g. imap process. So, I was planning on just putting everything into $lib/dovecot/. expire-tool etc. would use mail_plugins setting that wasn't inside protocol {}. So for example: mail_plugins = quota protocol lda { mail_plugins = quota sieve } protocol imap { mail_plugins = quota imap_quota } protocol pop3 { # defaults to mail_plugins=quota above, no need to set it explicitly } That leaves auth and dict.. I was considering adding new auth_plugins and dict_plugins settings, but that might make it more difficult to start using distro-compiled Dovecot where sql, ldap, etc. are in plugins. So perhaps the auth/ and dict/ directories can stay and contain symlinks. But the $lib/dovecot/sql/ directory could go away and the sql plugins be stored just in $lib/dovecot/ among everything else. Any thoughts?

2 1

[Dovecot] Getting Error 24 device timeout
by Jones, Keven 07 Apr '09

07 Apr '09

Hello, Does anyone know what causes this error and how to fix this Begin 'Pop52-Bounceback'. Connect: 172.20.23.52 110 +OK USER bounceback +OK PASS *#*#*#*#*#* +OK Logged in. STAT ERROR! FUNCTION: MainLoop:Pop3Stat POP Account: Pop52-Bounceback Error: 24 Device Time-Out

2 5

[Dovecot] Compile and configure Solr plugin
by Rui Carneiro 07 Apr '09

07 Apr '09

Hi all, I'm having some kind of troubles on Solr's integration. I configured Dovecot with the Solr argument (--with-solr) and everything proceeded just fine. But, when I started Dovecot I got this error: Plugin fts_solr not found from directory /usr/lib64/dovecot/modules/imap Error: imap dump-capability process returned 89 Fatal: Invalid configuration in /etc/dovecot/dovecot.conf In fact, there is no Solr plugin in that directory. So I tried to make a symbolic link from where the compiled Solr plugin was to this directory: ln -s /usr/local/lib/dovecot/lib21_fts_solr_plugin.so And now I have the following error: dlopen(/usr/lib64/dovecot/modules/imap/lib21_fts_solr_plugin.so) failed: /usr/lib64/dovecot/modules/imap/lib21_fts_solr_plugin.so: undefined symbol: mailbox_get_virtual_box_patterns Couldn't load required plugins Error: imap dump-capability process returned 89 Fatal: Invalid configuration in /etc/dovecot/dovecot.conf I'm missing some steps in this installation for sure, do anyone have any clue? Thanks, Rui Carneiro

1 1

[Dovecot] Modified websieve for use with dovecot
by Wolfgang.Friebel＠desy.de 07 Apr '09

07 Apr '09

In order to let users construct a sieve filter using a GUI I have taken websieve.pl from Alain Turbide (for use with Cyrus) and modified it to work with dovecot. In order to get it working quickly I have removed all the code I did not need (e.g. ldap support, password authentication) and coded Kerberos5 auth (via GSSAPI or SASL) in. I understand that this makes the work mostly useless to others. As however the dovecot adaptations were done by removing internally used perl packages by ones on CPAN such as Mail::IMAPClient and Net::Managesieve that part of the code should be of a broader interest. If someone is interested to have a look or even wants to help make it again a more universal tool I have put the code on ftp://ftp-zeuthen.desy.de/pub/unix/mail/websieve.pl Current status: I have tested the creation and modification of sieve rules, inspecting and setting ACL's (dovecot 1.2), displaying quota and creating folders. All the other code (e.g. the part that deals with an administrative user) is untested and probably not compatible yet with the new perl modules used. -- Wolfgang Friebel Deutsches Elektronen-Synchrotron DESY Phone/Fax: +49 33762 77372/216 Platanenallee 6 Mail: Wolfgang.Friebel AT desy.de D-15738 Zeuthen Germany

1 0