July 2014 - dovecot - dovecot.org

OOM in Dovecot 2.2.13 imap
by Bernhard Schmidt 14 Jul '14

14 Jul '14

Hi, we run Dovecot 2.2.13 on Debian Wheezy with a couple thousand mailboxes. We have two users that repeatedly trigger an OOM condition with IMAP. Jun 23 12:53:21 lxmhs74 dovecot: imap(USER): Fatal: pool_system_realloc(268435456): Out of memory Jun 23 12:53:21 lxmhs74 dovecot: imap(USER): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x6c15f) [0x7f11766cc15f] -> /usr/lib/dovecot/libdovecot.so.0(+0x6c1be) [0x7f11766cc1be] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f1176685568] -> /usr/lib/dovecot/libdovecot.so.0(+0x81e80) [0x7f11766e1e80] -> /usr/lib/dovecot/libdovecot.so.0(+0x86cda) [0x7f11766e6cda] -> /usr/lib/dovecot/libdovecot.so.0(+0x86f96) [0x7f11766e6f96] -> /usr/lib/dovecot/libdovecot.so.0(+0x87b48) [0x7f11766e7b48] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0xcd) [0x7f11766e60cd] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f11766e615a] -> /usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so(+0x5849) [0x7f1175692849] -> /usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so(+0x5982) [0x7f1175692982] -> /usr/lib/dovecot/modules/lib30_imap_zlib_plugin.so(+0x5b62) [0x7f1175692b62] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_flush+0x4d) [0x7f11766e5d6d] -> /usr/lib/dovecot/libdovecot.so.0(+0x85e2c) [0x7f11766e5e2c] -> dovecot/imap [USER IP UID fetch](client_output+0xe9) [0x7f1176e8d269] -> /usr/lib/dovecot/libdovecot.so.0(+0x879d5) [0x7f11766e79d5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4e) [0x7f11766dcfbe] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xd7) [0x7f11766ddfb7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) [0x7f11766dd049] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f11766dd0c8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f117668a7b3] -> dovecot/imap [USER IP UID fetch](main+0x2ae) [0x7f1176e8152e] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7f11762f3ead] -> dovecot/imap [USER IP UID fetch](+0xd69d) [0x7f1176e8169d] Jun 23 12:53:21 lxmhs74 dovecot: imap(USER): Fatal: master: service(imap): child 33659 killed with signal 6 (core dumped) The gdb backtrace looks like this: gdb) bt full #0 0x00007f1176307475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007f117630a6f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007f11766cc155 in default_fatal_finish (type=<optimized out>, status=status@entry=83) at failures.c:193 backtrace = 0x7f11789d4088 "/usr/lib/dovecot/libdovecot.so.0(+0x6c15f) [0x7f11766cc15f] -> /usr/lib/dovecot/libdovecot.so.0(+0x6c1be) [0x7f11766cc1be] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f1176685568] -> /usr/lib/d"... #3 0x00007f11766cc1be in i_internal_fatal_handler (ctx=0x7fff5ffedfb0, format=<optimized out>, args=<optimized out>) at failures.c:657 status = 83 #4 0x00007f1176685568 in i_fatal_status (status=status@entry=83, format=format@entry=0x7f1176702ba8 "pool_system_realloc(%lu): Out of memory") at failures.c:295 ctx = {type = LOG_TYPE_FATAL, exit_status = 83, timestamp = 0x0} args = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7fff5ffee0a0, reg_save_area = 0x7fff5ffedfe0}} #5 0x00007f11766e1e80 in pool_system_realloc (pool=<optimized out>, mem=0x0, old_size=134217728, new_size=268435456) at mempool-system.c:134 __FUNCTION__ = "pool_system_realloc" #6 0x00007f11766e6cda in o_stream_grow_buffer (fstream=fstream@entry=0x7f11789f41c0, bytes=<optimized out>) at ostream-file.c:440 size = 268435456 new_size = <optimized out> end_size = <optimized out> #7 0x00007f11766e6f96 in o_stream_add (fstream=fstream@entry=0x7f11789f41c0, data=0x7f11789fe549, size=12113) at ostream-file.c:501 unused = <optimized out> sent = <optimized out> i = <optimized out> #8 0x00007f11766e7b48 in o_stream_file_sendv (stream=0x7f11789f41c0, iov=<optimized out>, iov_count=<optimized out>) at ostream-file.c:588 fstream = 0x7f11789f41c0 size = <optimized out> total_size = <optimized out> added = <optimized out> optimal_size = <optimized out> i = <optimized out> ret = 0 __FUNCTION__ = "o_stream_file_sendv" #9 0x00007f11766e60cd in o_stream_sendv (stream=0x7f11789f4250, iov=iov@entry=0x7fff5ffee1a0, iov_count=iov_count@entry=1) at ostream.c:239 _stream = 0x7f11789f41c0 i = <optimized out> total_size = 12113 ret = <optimized out> __FUNCTION__ = "o_stream_sendv" #10 0x00007f11766e615a in o_stream_send (stream=<optimized out>, data=<optimized out>, size=size@entry=12113) at ostream.c:217 iov = {iov_base = 0x7f11789fe549, iov_len = 12113} #11 0x00007f1175692849 in o_stream_zlib_send_outbuf (zstream=0x7f11789f9340) at ostream-zlib.c:94 ret = <optimized out> size = 12113 #12 0x00007f1175692982 in o_stream_zlib_send_flush (zstream=zstream@entry=0x7f11789f9340, final=final@entry=true) at ostream-zlib.c:189 zs = 0x7f11789f9420 len = <optimized out> done = false ret = <optimized out> flush = <optimized out> __FUNCTION__ = "o_stream_zlib_send_flush" #13 0x00007f1175692b62 in o_stream_zlib_flush (stream=0x7f11789f9340) at ostream-zlib.c:235 zstream = 0x7f11789f9340 ret = <optimized out> #14 0x00007f11766e5d6d in o_stream_flush (stream=stream@entry=0x7f11789f93d0) at ostream.c:157 _stream = 0x7f11789f9340 ret = 1 __FUNCTION__ = "o_stream_flush" #15 0x00007f11766e5e2c in o_stream_default_cork (_stream=0x7f11789f9340, set=<optimized out>) at ostream.c:429 No locals. #16 o_stream_default_cork (_stream=0x7f11789f9340, set=<optimized out>) at ostream.c:422 No locals. #17 0x00007f1176e8d269 in client_output (client=0x7f11789f4470) at imap-client.c:1019 ret = 0 __FUNCTION__ = "client_output" #18 0x00007f11766e79d5 in stream_send_io (fstream=0x7f11789f41c0) at ostream-file.c:468 ostream = 0x7f11789f4250 ret = <optimized out> #19 0x00007f11766dcfbe in io_loop_call_io (io=0x7f1178a5bbe0) at ioloop.c:441 ioloop = 0x7f11789dc750 t_id = 2 __FUNCTION__ = "io_loop_call_io" #20 0x00007f11766ddfb7 in io_loop_handler_run_internal (ioloop=ioloop@entry=0x7f11789dc750) at ioloop-epoll.c:220 ctx = 0x7f11789dd3e0 events = 0x0 event = 0x7f11789de250 list = 0x7f11789dee40 io = <optimized out> tv = {tv_sec = 297, tv_usec = 43980} events_count = <optimized out> msecs = <optimized out> ret = 1 i = <optimized out> call = <optimized out> __FUNCTION__ = "io_loop_handler_run_internal" #21 0x00007f11766dd049 in io_loop_handler_run (ioloop=ioloop@entry=0x7f11789dc750) at ioloop.c:488 No locals. #22 0x00007f11766dd0c8 in io_loop_run (ioloop=0x7f11789dc750) at ioloop.c:465 __FUNCTION__ = "io_loop_run" #23 0x00007f117668a7b3 in master_service_run (service=0x7f11789dc5e0, callback=callback@entry=0x7f1176e96a00 <client_connected>) at master-service.c:566 No locals. #24 0x00007f1176e8152e in main (argc=1, argv=0x7f11789dc390) at main.c:400 set_roots = {0x7f11770a4960, 0x0} login_set = {auth_socket_path = 0x7f11789d4048 "\001", postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = 0x7f1176e96880 <login_client_connected>, failure_callback = 0x7f1176e96580 <login_client_failed>, request_auth_token = 1} service_flags = <optimized out> storage_service_flags = <optimized out> username = 0x0 c = <optimized out> I already deleted the index, but the issue persists. Bernhard

4 8

protocol specific AuthDatabase Queries
by claus 14 Jul '14

14 Jul '14

Hi, i have to configure different sql-queries for protocol lmtp, another for protocol smtp and one query for other protocols. I have configured in "/etc/dovecot/conf.d/auth-sql.conf.ext" one entry for all protocols and 2 protocol specific entries: protocol lmtp { passdb { driver = sql args = /etc/dovecot/dovecot-lmtp-sql.conf.ext } } protocol smtp { passdb { driver = sql args = /etc/dovecot/dovecot-smtp-sql.conf.ext } } passdb { driver = sql args = /etc/dovecot/dovecot-sql.conf.ext } is this correct or do i have to configure all possible protocols seperately? First i had a working configuration for all protocols except for lmtp (but now i need additionally a special query for smtp): protocol !lmtp { .. } protocol lmtp { .. } Can you tell me whats the prefered configuration? Thanks Claus

1 0

doveadm dump ignores .pvt
by Thomas Herrmann 14 Jul '14

14 Jul '14

While debugging a different problem, I used "doveadm dump" to show information about a maildir folder; it happily reported values from dovecot.index and dovecot.index.log (confirmed by strace): [...] messages count ........... = 978 seen messages count ...... = 967 [...] However, mail clients reported different values, I guess because the Seen flags were actually stored in the dovecot.index.pvt.log files. Is this a known problem or a result of my (maybe unsupported?) setup? My public maildir folder contains only links to mailboxes of the private maildir of the mail admin, i.e.: /home/vmail/public/.Info -> ../adwin.de/mail-admin/Maildir/.Info namespace { list = children location = maildir:/home/vmail/public:INDEXPVT=/home/vmail/adwin.de/%n/public prefix = ADwin/ separator = / subscriptions = no type = public } My setups leads to .pvt-files in the public namespace, could this be a problem? http://wiki2.dovecot.org/SharedMailboxes/Symlinks explicitly states that symlinking _should_ work. Using version 2.2.13 (debian packages). Regards, Thomas

1 0

Flags in public folders disappear when more than 25 flags are used
by Thomas Herrmann 14 Jul '14

14 Jul '14

Hello, I have been using flags in public folders for quite a while with no problems. Once the flags were added to all clients (Thunderbird), they were visible and synchronized properly. Now I have added some new flags, and in the dovecot-keywords files I see that I am now using more than 25 flags (in which case they are not stored using an additional letter in the filename). Now I see that newly added flags are only visible in the client for a short time and then suddenly disappear. How can I debug this, or what may be a reason for this? Regards Thomas Herrmann root@mailsrv2:~# dovecot --version 2.1.7 root@mailsrv2:~# dovecot -n # 2.1.7: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.5 xfs disable_plaintext_auth = no log_timestamp = "%Y-%m-%d %H:%M:%S " mail_fsync = never mail_location = maildir:/home/vmail/adwin.de/%n/Maildir mail_privileged_group = mail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave imapflags namespace { inbox = yes location = prefix = separator = / subscriptions = yes type = private } namespace { list = children location = maildir:/home/vmail/public:INDEX=/home/vmail/adwin.de/%n/public prefix = ADwin/ separator = / subscriptions = no type = public } passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { acl = vfile sieve = /home/vmail/adwin.de/%n/dovecot.sieve sieve_dir = /home/vmail/adwin.de/%n/sieve sieve_extensions = +imapflags +imap4flags } protocols = imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { mode = 0600 user = vmail } user = root } service imap-login { inet_listener imap { address = *,:: port = 56143 } inet_listener imaps { address = *,:: port = 56993 ssl = yes } service_count = 0 } service imap { vsz_limit = 512 M } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service managesieve-login { executable = /usr/lib/dovecot/managesieve-login service_count = 0 } service managesieve { executable = /usr/lib/dovecot/managesieve } ssl_ca = </etc/dovecot/ca-bundle.crt ssl_cert = </etc/dovecot/XXXXXXXXXXXXXX.crt ssl_key = </etc/dovecot/XXXXXXXXXXXXXXX.key userdb { driver = passwd } userdb { args = uid=5000 gid=5000 home=/home/vmail/%d/%n allow_all_users=yes driver = static } protocol imap { mail_max_userip_connections = 100 mail_plugins = acl imap_acl } protocol imaps { mail_max_userip_connections = 100 mail_plugins = acl imap_acl } protocol lda { auth_socket_path = /var/run/dovecot/auth-master log_path = /home/vmail/dovecot-deliver.log mail_fsync = optimized mail_plugins = sieve acl postmaster_address = XXXXXXXX(a)XXXXX.de } protocol lmtp { mail_plugins = sieve acl }

2 4

Panic: file mail-index.c: line 650 (mail_index_close): assertion failed: (index->open_count > 0)
by Hardy Flor 14 Jul '14

14 Jul '14

Hello, I'm use the virtual plugin for list all unread messages in the public namespace. Every time the connection is terminated with this error message when I select the virtual/projekte mailbox. Do not show any emails. The version is build today from source 2.2.13.auto+102 Hardy *** dovecot-virtual *** !INBOX Projekte/* unseen ********************** *** part of dovecot -n *** # 2.2.13 (fd0616d553b0): /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.5 ext4 namespace { list = children location = mdbox:/var/mail/projekte/:INDEXPVT=/var/mail/user/%n/projekte prefix = Projekte/ separator = / subscriptions = no type = public } ************************* GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/lib/dovecot/imap...(no debugging symbols found)...done. Attaching to program: /usr/lib/dovecot/imap, process 6839 ptrace: Kein passender Prozess gefunden. [New LWP 6839] warning: Can't read pathname for load map: Eingabe-/Ausgabefehler. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `dovecot/imap imap-postlogin'. Program terminated with signal 6, Aborted. #0 0x00007f38fc37c475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt full #0 0x00007f38fc37c475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007f38fc37f6f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007f38fc742395 in ?? () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #3 0x00007f38fc7423fe in ?? () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #4 0x00007f38fc6fa96e in i_panic () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #5 0x00007f38fca6a1e1 in mail_index_close () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #6 0x00007f38fca5c46c in index_storage_mailbox_close () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #7 0x00007f38fca2fb8d in mailbox_close () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #8 0x00007f38fb0f600d in ?? () from /usr/lib/dovecot/modules/lib20_virtual_plugin.so No symbol table info available. #9 0x00007f38fb0f61b0 in virtual_backend_box_open () from /usr/lib/dovecot/modules/lib20_virtual_plugin.so No symbol table info available. #10 0x00007f38fb0f7860 in virtual_storage_sync_init () from /usr/lib/dovecot/modules/lib20_virtual_plugin.so No symbol table info available. #11 0x00007f38fca30703 in mailbox_sync_init () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #12 0x00007f38fca30827 in mailbox_sync () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #13 0x00007f38fca5afd0 in index_storage_get_status () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #14 0x00007f38fb0f5458 in ?? () from /usr/lib/dovecot/modules/lib20_virtual_plugin.so No symbol table info available. #15 0x00007f38fbb1b794 in ?? () from /usr/lib/dovecot/modules/lib01_acl_plugin.so No symbol table info available. #16 0x00007f38fca31a30 in mailbox_get_status () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #17 0x000000000041eb4b in imap_status_get () No symbol table info available. #18 0x0000000000414329 in cmd_status () No symbol table info available. #19 0x000000000041849c in command_exec () No symbol table info available. #20 0x00000000004174e0 in ?? () No symbol table info available. #21 0x0000000000417574 in ?? () No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #22 0x0000000000417855 in client_handle_input () No symbol table info available. #23 0x0000000000417c02 in client_input () No symbol table info available. #24 0x00007f38fc7538be in io_loop_call_io () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #25 0x00007f38fc7548cb in io_loop_handler_run_internal () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #26 0x00007f38fc753949 in io_loop_handler_run () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #27 0x00007f38fc7539c8 in io_loop_run () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #28 0x00007f38fc6ffe13 in master_service_run () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #29 0x000000000040c238 in main () No symbol table info available. (gdb)

1 0

High user number tune
by Edwardo Garcia 13 Jul '14

13 Jul '14

Halo, This might seem like another indirect thread. I Monday started new job, first task must replace courier. The job use POP3. and IMAP only for webmail, but must server support 4K POP3 at one times. I see from similar thread current settings not adequate , we will use TLS and if wiki read right Iprotocol So am close in think that need ? service pop3-login { process_limit = 4096 } service pop3 { process_limit = 4096 } service imap-login { process_limit = 2048 } service imap { process_limit = 2048 } then we have service managesieve { process_limit = 1024 } is that all we need change? I see in other thread someone says service anvil but I can not find out about it, does need change as well ? Why is it so complicated to change to let more user on, most other change one setting, dovecot want dozen :(

2 1

Pigeonhole sieve misfile
by Patrick Domack 13 Jul '14

13 Jul '14

I'm having an issue with sieve putting some emails into the spam folder, that should not be matching the spam rule. I tried looking but can't seem to find a debug option. Is there any way I can get a log of what the sieve rule path that matches so I can locate the misfiring rule? or maybe something else that is going on. I already checked the dovecot sieve log file, and there is nothing in there.

2 1

macros
by Arkadiusz Miśkiewicz 12 Jul '14

12 Jul '14

Does dovecot support any form of macros? I would like to share configuration file between several servers. Configs are different only in tiny aspects. Somethine like: dovecot-server.conf (different on each server): %define ID 55 %define SOMETHING SELECT FROM * WHERE something %define MECHANISMS digest-md5 dovecot-main.conf (common, shared config) !include dovecot-server.conf user_query = ${SOMETHING} AND id=${ID} auth_mechanisms = plain login ${MECHANISMS} etc -- Arkadiusz Miśkiewicz, arekm / maven.pl

2 1

imap(hardy): Panic: file istream-tee.c: line 138 (i_stream_tee_read): assertion failed: (last_high_offset == input->v_offset + size)
by Hardy Flor 12 Jul '14

12 Jul '14

Hello, I'm copy with imapsync (on the new server with dovecot) with a single user all mails from a old server to the new server. The error-message ist after 20-30 minutes in the log, and I must restart the imapsync. The next mail is normal store on the destination box. Hardy

2 4

Replication on dovecot 2.0
by Tom Barber(Alabs) 11 Jul '14

11 Jul '14

Hi Guys I'm running an Ubuntu 12.04 LTS server and not being particually inclined to upgrade I'm wondering what options I have for replication as it appears to be running dovecot 2.0.19. I saw the wiki page about replication and whilst it says you should use 2.2+ it doesn't say what the minimum requirements are, so I tried doing what it says but I get: Fatal: Plugin 'replication' not found from directory /usr/lib/dovecot/modules Any ideas or tips. Thanks Tom

2 2