- dovecot - dovecot.org

indexer-worker crashes handling mails with big attachments (dovecot 2.2.16/2.2.18 + FTS Apache Solr + Tika)
by Robinson Bomze 03 Jun '15

03 Jun '15

Hi, yesterday i tried to setup Dovecot with Solr (3.6.2) + Tika (1.8) for FTS. i used a fresh Debian 8.0 system in the beginning with Dovecot 2.2.13 from the Debian repository. After i got some issues with Tika/Dovecot and i read on the mailinglist that these problems where fixed in 2.2.14+, so i tried 2.2.18. With 2.2.18 i get panics with big (ok... huge) attachments. Most mailboxes (and their attachments) get index fine, but on some i got panics from the indexer-worker. i was able to isolate the problem. It seems that when Tika (which works flawless) sends a big reply to Dovecot and Dovecot sends this data to Solr, communication crashes between Dovecot and Solr. Eg. indexing an email with a 200k char wordfile results in a panic of the indexer-worker: Jun 02 23:50:57 indexer-worker(username): Warning: I/O leak: 0x7ff65f39f540 (line 120, fd 20) Jun 02 23:50:57 indexer-worker(username): Warning: Timeout leak: 0x7ff65f39f2e0 (line 325) Jun 02 23:50:57 indexer: Error: Indexer worker disconnected, discarding 1 requests for username Jun 02 23:50:57 imap(username): Error: indexer failed to index mailbox INBOX.username Jun 02 23:50:57 indexer-worker(username): Fatal: master: service(indexer-worker): child 11429 killed with signal 11 (core dumped) I got similar issues results with 2.2.16: Jun 02 23:21:12 indexer-worker(username): Warning: I/O leak: 0x7ffff7811cc0 (line 127, fd 20) Jun 02 23:21:12 indexer-worker(username): Panic: file ioloop.c: line 39 (io_add_file): assertion failed: (callback != NULL) Jun 02 23:21:12 indexer-worker(username): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0(+0x77130) [0x7ffff7842130] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x7 Jun 02 23:21:12 indexer: Error: Indexer worker disconnected, discarding 1 requests for username Jun 02 23:21:12 imap(username): Error: indexer failed to index mailbox INBOX.username Jun 02 23:21:12 indexer-worker(username): Fatal: master: service(indexer-worker): child 7909 killed with signal 6 (core dumps disabled) The problem was already posted: http://dovecot.org/pipermail/dovecot/2015-May/100901.html I could trigger the same panic running the indexer via 'doveadm index -u username MAILBOX'. Here is a backtrace (bt) of the 2.2.18-crash (on line #8 you see a fragement of the text sent to solr): #0 array_count_i (array=0x8) at array.h:155 #1 array_get_modifiable_i (count_r=<synthetic pointer>, array=0x8) at array.h:228 #2 priorityq_remove_idx (pq=0x0, idx=0) at priorityq.c:121 #3 0x00007ff65f3ef5eb in priorityq_remove (pq=<optimized out>, item=item@entry=0xa26920) at priorityq.c:138 #4 0x00007ff65f3e1e70 in timeout_remove (_timeout=<optimized out>) at ioloop.c:288 #5 0x00007ff65f3e2781 in io_loop_move_timeout (_timeout=_timeout@entry=0xa27f98) at ioloop.c:861 #6 0x00007ff65f39ff37 in http_client_connection_switch_ioloop (conn=conn@entry=0xa27ea0) at http-client-connection.c:1357 #7 0x00007ff65f3a3d68 in http_client_switch_ioloop (client=client@entry=0xa0bf20) at http-client.c:211 #8 0x00007ff65f39c005 in http_client_request_continue_payload (_req=_req@entry=0xa0ee88, data=0xa42fa0 "k for evidence of fluid spill.\nIf the device is mounted on a stand, examine the condition of the mount.\nIf the device moves on casters, check the condition of the casters. Check operation of brakes, i"..., size=55453) at http-client-request.c:566 #9 0x00007ff65f39c22a in http_client_request_send_payload (_req=_req@entry=0xa0ee88, data=<optimized out>, size=<optimized out>) at http-client-request.c:625 #10 0x00007ff65e972429 in solr_connection_post_more (post=0xa0ee80, data=<optimized out>, size=size@entry=55453) at solr-connection.c:504 #11 0x00007ff65e96ea09 in fts_backed_solr_build_commit (ctx=0xa1a880) at fts-backend-solr.c:341 #12 0x00007ff65e96eaad in fts_backend_solr_update_set_mailbox (_ctx=0xa1a880, box=0x0) at fts-backend-solr.c:407 #13 0x00007ff65eb7cfac in fts_backend_set_cur_mailbox (ctx=ctx@entry=0xa1a880) at fts-api.c:129 #14 0x00007ff65eb7cfe3 in fts_backend_update_deinit (_ctx=<optimized out>) at fts-api.c:143 #15 0x00007ff65eb8303c in fts_transaction_end (t=t@entry=0xa11ed0) at fts-storage.c:550 #16 0x00007ff65eb83e91 in fts_transaction_commit (t=0xa11ed0, changes_r=0x7ffdcdca5e30) at fts-storage.c:615 #17 0x00007ff65f688a82 in mailbox_transaction_commit_get_changes (_t=_t@entry=0x7ffdcdca5ee0, changes_r=changes_r@entry=0x7ffdcdca5e30) at mail-storage.c:1837 #18 0x00007ff65f688b2e in mailbox_transaction_commit (t=t@entry=0x7ffdcdca5ee0) at mail-storage.c:1818 "bt full" looks like this: #0 array_count_i (array=0x8) at array.h:155 No locals. #1 array_get_modifiable_i (count_r=<synthetic pointer>, array=0x8) at array.h:228 No locals. #2 priorityq_remove_idx (pq=0x0, idx=0) at priorityq.c:121 count = <optimized out> #3 0x00007ff65f3ef5eb in priorityq_remove (pq=<optimized out>, item=item@entry=0xa26920) at priorityq.c:138 No locals. #4 0x00007ff65f3e1e70 in timeout_remove (_timeout=<optimized out>) at ioloop.c:288 timeout = 0xa26920 #5 0x00007ff65f3e2781 in io_loop_move_timeout (_timeout=_timeout@entry=0xa27f98) at ioloop.c:861 new_to = 0xa1adf0 old_to = <optimized out> #6 0x00007ff65f39ff37 in http_client_connection_switch_ioloop (conn=conn@entry=0xa27ea0) at http-client-connection.c:1357 No locals. #7 0x00007ff65f3a3d68 in http_client_switch_ioloop (client=client@entry=0xa0bf20) at http-client.c:211 conn = 0xa27ea0 _conn = 0xa27ea0 host = <optimized out> peer = <optimized out> #8 0x00007ff65f39c005 in http_client_request_continue_payload (_req=_req@entry=0xa0ee88, data=0xa42fa0 "k for evidence of fluid spill.\nIf the device is mounted on a stand, examine the condition of the mount.\nIf the device moves on casters, check the condition of the casters. Check operation of brakes, i"..., size=55453) at http-client-request.c:566 prev_ioloop = 0x9f4730 req = 0xa36970 conn = 0xa27ea0 client = 0xa0bf20 ret = <optimized out> __FUNCTION__ = "http_client_request_continue_payload" #9 0x00007ff65f39c22a in http_client_request_send_payload (_req=_req@entry=0xa0ee88, data=<optimized out>, size=<optimized out>) at http-client-request.c:625 __FUNCTION__ = "http_client_request_send_payload" #10 0x00007ff65e972429 in solr_connection_post_more (post=0xa0ee80, data=<optimized out>, size=size@entry=55453) at solr-connection.c:504 conn = 0xa0be50 __FUNCTION__ = "solr_connection_post_more" Hope anyone fixes the code... i need this feature :) Thanks a lot in advance!

2 1

multi sync (>2 servers) + selective sync + trigger
by b-dovecot.org＠grmbl.net 02 Jun '15

02 Jun '15

Hi all, I've been researching ways to replicate mail across multiple mailstores and have a few questions. Synching 2 mailstores (M1 & M2) via dsync works fine. I want to add a 3rd and 4th (M3 & M4) server to also be synced. Multi-sync (>2 servers): 1) How do I tell M1 and/or M2 to also sync to M3 and/or M4? Selective sync: 2) How do I sync specific domains to specific servers and not to others? Trigger: 3) Does a normal sync check only check the mailbox being changed or does it get triggered on mailbox changes to check ALL indexes and sync all changed mailboxes accordingly? 4) Can I use an email to trigger a sync for any other mailboxes which might be out of sync? TIA B

3 3

Outlook Express with XP...
by SH Development 02 Jun '15

02 Jun '15

Dovecot 2.0.9 I am able to connect successfully with Thunderbird, Win 8, Apple Mail, iPhone, iPad. Outlook Express on Win XP receives just fine, but will not send. I get a "Client host rejected: Access denied” message. I believe this is simply Postfix telling me the connection isn’t properly authenticated, as setting up the exact same credentials on another client (non OE) works. It is only with OE on XP that I am unable to send/authenticate. I have literally tried every combination of settings in OE and nothing works. What else might I need to enable to get this working? Using SSL, ports 993, 587 PLAINTEXT. dovecot -n as follows: # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-504.16.2.el6.x86_64 x86_64 CentOS release 6.6 (Final) ext4 log_timestamp = "%Y-%m-%d %H:%M:%S " mail_location = maildir:/home/vmail/%d/%n/Maildir mail_max_userip_connections = 50 namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocols = imap pop3 service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { mode = 0600 user = vmail } user = root } ssl_ca = </etc/pki/dovecot/certs/intermediate.pem ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = uid=5000 gid=5000 home=/home/vmail/%d/%n allow_all_users=yes driver = static } protocol lda { auth_socket_path = /var/run/dovecot/auth-master log_path = /home/vmail/dovecot-deliver.log postmaster_address = postmaster(a)starionhost.net } protocol pop3 { pop3_uidl_format = %08Xu%08Xv }

7 11

Re: Outlook Express with XP...
by Joseph Tam 02 Jun '15

02 Jun '15

SH Development writes: > I have literally tried every combination of settings in OE and nothing > works. What else might I need to enable to get this working? > > Using SSL, ports 993, 587 PLAINTEXT. As others have noted, your MTA handles the connection. I believe Windows XP needs to be updated to the latest service patch to handle SSLv3 disabled sites. Try checking to make sure your Postfix SSL security settings and your WinXP SSL capabilities are compatible. Joseph Tam <jtam.home(a)gmail.com>

1 0

Pop3 service stops responding
by Mike Eriksson 02 Jun '15

02 Jun '15

Hi! I have problem with the pop3 service stops responding at random intervals on my server. We started to notice this problem as we're running icinga to check the imap & pop3 service. When icinga reports that the service is down. We try to telnet to the server on port 110 but it refuses the connection. I've checked the netstat during this time and it's listens on port 110. My first thought we were hitting the max connections and similar. Changed login_max_connections from 128 to 256. Didnt solve the problem. Started the debug on the dovecot server but could not see anything in the logs. Also the error log or the syslog dosnt report anything that I can see. Best thing would of course be to start and upgrade the software and os(Debian 5) but I'm not comfortable upgrading it at the moment as my knowledge about dovecot / postfix isn't great. Do you have any idéas how to move forward? dovecot -n # 1.0.15: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot-error.log info_log_path: /var/log/dovecot-info.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: pop3 imap imaps pop3s ssl_cipher_list: ALL:!LOW disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_max_processes_count: 256 first_valid_uid: 1090 last_valid_uid: 1090 first_valid_gid: 1090 last_valid_gid: 1090 mail_privileged_group: mail mail_location: maildir:~/Maildir mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): quota mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@% username_translation: %@ username_format: %Lu passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: quota: maildir sieve_global_path: /etc/dovecot/sieve_scripts/spam-sort.sieve login_max_processes_count: 256 login_max_connections: 256 max_mail_processes: 1024 Best Regards Mike

2 1

Failed running extprograms execute via socket - fatal recv(MSG_PEEK) failed disconnected
by E.B. 01 Jun '15

01 Jun '15

I switched from running my extprograms execute script directly to running with dovecot socket. Log shows only this dovecot: lmtp(test(a)example.com): Debug: wdi0Tb5VPlGfPnEAM/SpMA: sieve: action execute: running program: test dovecot: lmtp(test(a)example.com): Debug: Namespace : Using permissions from /vmail/example.com/test: mode=0770 gid=default dovecot: script: Fatal: recv(MSG_PEEK) failed: disconnected For testing I opened up the script and socket with permissions 777 but the error seems to indicate less about permissions more about some kind of protocol problem i guessing. Sieve script calls using this: execute "test"; Plugin config: plugin { sieve_plugins = sieve_extprograms sieve_global_extensions = +vnd.dovecot.execute sieve_execute_socket_dir = sieve-execute sieve_before = /usr/local/etc/dovecot/sieve } service test { executable = script /usr/local/etc/dovecot/sieve_globals/test.sh unix_listener sieve-execute/test { mode = 0660 group = vmail } } FYI I have quota-warning sockets configured identical to this and they work good. Dovecot 2.2.16 Pigeonhole 0.4.7 Help appreciate a lot.

2 14

Sieve extprograms ?not exexuting?
by E.B. 01 Jun '15

01 Jun '15

Hello, I was testing the extprograms plugin. I think I had it working in the past, but many things have changed since then, so no use trying to figure out where it broke - starting over again... Debug-enabled log give me: Apr 27 04:11:36 mail dovecot: lmtp(test(a)example.com): Debug: qOGyA0DePHVaOyHEM/SpMA: sieve: action execute: running program: test.sh Apr 27 04:11:36 mail dovecot: lmtp(test(a)example.com): Debug: waiting for program `/usr/local/etc/dovecot/sieve_globals/test.sh' to finish after 0 seconds So I guess it thinks it is running my script? But simple test script does nothing. Here it is: #!/bin/sh read INPUT INPUT="Hello world: $INPUT" echo "$INPUT" >> /tmp/hello echo "---------------------------" >> /tmp/hello Permissions on this script file for now are rwxrwxrwx But nothing goes to /tmp/hello at all. Script works when I run it manually. I also tried without the "read" but I think that's required isn't it? Anyway, what else can I do to debug this? Dovecot 2.2.16 Pigeonhole 0.4.7

4 7

OT: Central sieve management
by list＠airstreamcomm.net 29 May '15

29 May '15

A bit off topic, but I was wondering if anyone here has a solution for centrally managing sieve for multiple users from a custom web application? We would like to implement pigeonhole sieve on our dovecot cluster, however we need to be able to access user’s sieve configurations from a central location for troubleshooting and support purposes.

3 3

Enabling Master User for migration
by Charles Marcus 29 May '15

29 May '15

Hi all, Ok, just need a quick-n-dirty way to enable a master user for a migration... Reading the docs, I'm not sure what the bare minimum is I need to do this... At: http://wiki2.dovecot.org/Authentication/MasterUsers The Example config shows: auth_master_user_separator = * passdb { driver = passwd-file args = /etc/dovecot/passwd.masterusers master = yes pass = yes } passdb { driver = shadow } userdb { driver = passwd } But, if I'm using a non-real user (just need a master user that can log in as all other users for the migration), do I need all three of these? Or just the first two? Or, just the first one? I think I need the first two and not the last one, but am unsure. This is on a production system so don't want to break things, and I'm unfamiliar with using multiple passdb's... Thanks, Charles

2 6

Shared inbox?
by Chris Ross 29 May '15

29 May '15

I'm running dovecot 2.2.16 on my FreeBSD mail server. I've read information on the wiki about setting up shared mailboxes, but I want to do something that isn't really coved by the instructions I was reading there. My son (now 7 years old) has an account on the system, but doesn't use it directly. But, for things he's interested in like Minecraft, and/or the local zoo, we have given out his email address in a small number of places. What I would like to set up, both for now while he's not actually ever reading his email himself, and perhaps even for the future when we teach and/or allow for that, is for my wife and I to be able to "view" his inbox from our accounts. The instructions for setting up shared folders all are written so that they're secondary folders to all accounts. Is there a way to either (a) configure sharing someone elses Inbox by other accounts, or (b) setting up a separately configured shared folder to _act_ as the inbox for a single account? Thanks. Any suggestions to achieve the above described end goal would be appreciated. - Chris

3 6