October 2015 - dovecot

Storage Design (Regligious War, Whatever)
by Kelsey Cummings 13 Oct '15

13 Oct '15

We've been running dovecot director setup using MAILDIR++ spools on NFS served by 4 (aging) Netapp clustered filers and are considering modernizing our storage. There's nothing particularly wrong with the existing storage except the age, density and power consumption don't compare well when against new systems. Refreshing the Netapps is an option but they come at a substantial cost. The last time we were looking one of the recommended hot DIY setups was Linux NFS backed by XFS on N mirrors but this predated the availability of stable ZFS in FreeBSD or Linux as well as products like FreeNAS. A FreeBSD NFS/ZFS filer on commodity hardware with zil/l2arc on a PCIe SSD seems like an attractive, affordable and easily scaled out solution which also would allow us to leverage compression at the filesystem layer. Does anyone have any experience running ZFS spool storage? If so, how do you handle DR/HA for spool storage? Thoughts on how this might compare to using DRBD? Any pitfalls to watch out for or general pointers? Suggestions on pool configuration? -K

1 0

Unknown cache fields?
by Larry Rosenman 13 Oct '15

13 Oct '15

I assume this is expected, but just checking. I do NOT use POP3. Oct 5 09:41:20 thebighonker dovecot: indexer-worker(mrm): Debug: Ignoring unknown cache field: pop3.order Oct 5 09:41:20 thebighonker dovecot: indexer-worker(mrm): Debug: Ignoring unknown cache field: binary.parts Oct 5 09:41:20 thebighonker dovecot: indexer-worker(mrm): Debug: Ignoring unknown cache field: body.snippet -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961

2 1

doveadm index assertion failed
by Nick Rosier 13 Oct '15

13 Oct '15

Hi, one of my mailboxes returns following error when I run doveadm index on it: Panic: file charset-iconv.c: line 85 (charset_to_utf8_try): assertion failed: (srcleft <= CHARSET_MAX_PENDING_BUF_SIZE) OS: FreeBSD 10.2 Dovecot: 2.1.19 Tika: 1.10 SOLR: 5.3.1 Running doveadm -D index does not show any more information indicating what causes this error (mail/folder/...) How I can find what is causing this error? Rgds, N.

2 1

Re: bug report: dovecot-imapd core dump
by Tamas Papp 13 Oct '15

13 Oct '15

Hi, could not find it -- back to the list, maybe someone can suggest a fix/workaround. Best, Tamas On Wed, Oct 07 2015, Dominik Breu wrote: > Hello, > > yeah this one looks familiar to me can you search the list back in > september there was a somewhat similar bug with thunderbird imho timo > posted a notice about it. > > greetings, > > dominik > > Am Mittwoch, den 07.10.2015, 15:53 +0200 schrieb Tamas Papp: >> Hi Dominik, >> >> Thanks for the help, the backtrace is here (I did not sent it to the >> list since I am not sure if it contains password or message >> information, >> sorry but I am unfamiliar with these things). >> >> Core was generated by `dovecot/imap'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 0x00007f404e58e3b6 in mail_index_keywords_unref >> #(_keywords=_keywords@entry=0x7ffecb8adeb8) at mail-index.c:383 >> 383 mail-index.c: No such file or directory. >> (gdb) bt full >> #0 0x00007f404e58e3b6 in mail_index_keywords_unref >> #(_keywords=_keywords@entry=0x7ffecb8adeb8) at mail-index.c:383 >> keywords = 0x7f404e27066d <buffer_alloc+45> >> __FUNCTION__ = "mail_index_keywords_unref" >> #1 0x00007f404e559975 in mailbox_keywords_unref >> (keywords=keywords@entry=0x7ffecb8adeb8) at mailbox-keywords.c:94 >> No locals. >> #2 0x00007f404ea254ad in cmd_append_handle_args >> (nonsync_r=0x7ffecb8ade8f, >> #args=<optimized out>, cmd=0x7f4050bc3250) at cmd-append.c:602 >> ctx = 0x7f4050bc3348 >> flags_list = 0x7f4050bcc840 >> flags = 9 >> keywords_list = 0x7f4050bc33f0 >> internal_date_str = 0x7f4050bcca00 "17-Sep-2015 14:19:23 >> +0200" >> internal_date = -1 >> timezone_offset = 0 >> keywords = 0x0 >> ret = <optimized out> >> client = 0x7f4050bc2670 >> cat_list = 0x0 >> input = <optimized out> >> valid = <optimized out> >> #3 cmd_append_parse_new_msg (cmd=cmd@entry=0x7f4050bc3250) at >> cmd-append.c:770 >> client = <optimized out> >> ctx = 0x7f4050bc3348 >> args = 0x7f4050bcc488 >> msg = <optimized out> >> arg_min_count = <optimized out> >> fatal = 255 >> nonsync = true >> last_literal = <optimized out> >> ret = <optimized out> >> __FUNCTION__ = "cmd_append_parse_new_msg" >> #4 0x00007f404ea25a17 in cmd_append (cmd=0x7f4050bc3250) at >> cmd-append.c:932 >> client = 0x7f4050bc2670 >> ctx = 0x7f4050bc3348 >> mailbox = 0x7f4050bafb78 "Cemex" >> #5 0x00007f404ea30e4c in command_exec (cmd=cmd@entry=0x7f4050bc3250) >> at >> imap-commands.c:158 >> hook = 0x7f4050bad0d0 >> ret = <optimized out> >> #6 0x00007f404ea2fd62 in client_command_input (cmd=0x7f4050bc3250) >> at >> imap-client.c:780 >> client = 0x7f4050bc2670 >> command = <optimized out> >> __FUNCTION__ = "client_command_input" >> #7 0x00007f404ea2fe16 in client_command_input (cmd=0x7f4050bc3250) >> at >> imap-client.c:841 >> client = 0x7f4050bc2670 >> command = <optimized out> >> __FUNCTION__ = "client_command_input" >> #8 0x00007f404ea3014d in client_handle_next_command >> (remove_io_r=<synthetic >> pointer>, client=0x7f4050bc2670) at imap-client.c:879 >> ---Type <return> to continue, or q <return> to quit--- >> No locals. >> #9 client_handle_input (client=0x7f4050bc2670) at imap-client.c:891 >> _data_stack_cur_id = 3 >> remove_io = false >> handled_commands = false >> client = 0x7f4050bc2670 >> #10 0x00007f404ea304f5 in client_input (client=0x7f4050bc2670) at >> imap-client.c:933 >> cmd = 0xf013c >> output = 0x7f4050bc30a0 >> bytes = 135 >> __FUNCTION__ = "client_input" >> #11 0x00007f404e285247 in io_loop_call_io (io=0x7f4050bc3170) at >> ioloop.c:388 >> ioloop = 0x7f4050bac720 >> t_id = 2 >> #12 0x00007f404e286079 in io_loop_handler_run >> (ioloop=ioloop@entry=0x7f4050bac720) at ioloop-epoll.c:220 >> ctx = 0x7f4050bad3b0 >> io = <optimized out> >> tv = {tv_sec = 1799, tv_usec = 999282} >> events_count = <optimized out> >> msecs = <optimized out> >> ret = 1 >> i = 0 >> j = <optimized out> >> call = <optimized out> >> __FUNCTION__ = "io_loop_handler_run" >> #13 0x00007f404e284d88 in io_loop_run (ioloop=0x7f4050bac720) at >> ioloop.c:412 >> __FUNCTION__ = "io_loop_run" >> #14 0x00007f404e231de3 in master_service_run (service=0x7f4050bac5b0, >> callback=<optimized out>) at master-service.c:566 >> No locals. >> #15 0x00007f404ea2424e in main (argc=1, argv=0x7f4050bac390) at >> main.c:400 >> set_roots = {0x7f404ec489c0 <imap_setting_parser_info>, 0x0} >> login_set = {auth_socket_path = 0x7f4050ba4048 "\001", >> postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = >> 0x7f404ea39f40 <login_client_connected>, >> failure_callback = 0x7f404ea39c20 <login_client_failed>, >> request_auth_token = 1} >> service_flags = <optimized out> >> storage_service_flags = <optimized out> >> username = 0x0 >> c = <optimized out> >> (gdb) quit >> >> Best, >> >> Tamas >> >> On Wed, Oct 07 2015, Dominik Breu wrote: >> >> > Hello >> > >> > Tamas it would be a better backtace when you have installed the >> > dovecot >> > debug symboly mostly located under dovecote-dbg in apt. Further >> > debug >> > help can you find in the wiki. >> > >> > greetings >> > >> > dominik >> > >> > Am Mittwoch, den 07.10.2015, 15:09 +0200 schrieb Tamas Papp: >> > > Typical log message: >> > > >> > > Oct 7 11:39:36 hostname dovecot: imap(username): Fatal: master: >> > > service(imap): child 14864 killed with signal 11 (core dumped) >> > > >> > > What triggers the problem: >> > > >> > > User moving multiple messages to a subfolder via IMAP (from >> > > another >> > > account, on another server, client: Thunderbird). Moving single >> > > messages >> > > either does not trigger the problem, or with much lower >> > > probability >> > > (could not reproduce). >> > > >> > > Version: >> > > $ dpkg -l 'dovecot*' >> > > Desired=Unknown/Install/Remove/Purge/Hold >> > > > Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig >> > > > -aWait/Trig-pend >> > > > / Err?=(none)/Reinst-required (Status,Err: uppercase=bad) >> > > > > / Name Version >> > > > > Architecture Description >> > > +++-=====================================-======================= >> > > -======================= >> > > -================================================================ >> > > ==== >> > > ============ >> > > un dovecot-common <none> >> > > <none> (no description available) >> > > ii dovecot-core 1:2.2.9-1ubuntu5 >> > > amd64 secure POP3/IMAP server - core files >> > > un dovecot-gssapi <none> >> > > <none> (no description available) >> > > ii dovecot-imapd 1:2.2.9-1ubuntu5 >> > > amd64 secure POP3/IMAP server - IMAP daemon >> > > un dovecot-ldap <none> >> > > <none> (no description available) >> > > un dovecot-lmtpd <none> >> > > <none> (no description available) >> > > un dovecot-managesieved <none> >> > > <none> (no description available) >> > > un dovecot-mysql <none> >> > > <none> (no description available) >> > > un dovecot-pgsql <none> >> > > <none> (no description available) >> > > un dovecot-pop3d <none> >> > > <none> (no description available) >> > > un dovecot-sieve <none> >> > > <none> (no description available) >> > > un dovecot-solr <none> >> > > <none> (no description available) >> > > un dovecot-sqlite <none> >> > > <none> (no description available) >> > > tamas@szob:~$ lsb_release -a >> > > No LSB modules are available. >> > > Distributor ID: Ubuntu >> > > Description: Ubuntu 15.04 >> > > Release: 15.04 >> > > Codename: vivid >> > > >> > > Also tried 2.2.18 from Ubuntu wiley, same results. >> > > >> > > Backtrace (I am not sure this is useful, extraced core from >> > > Ubuntu's >> > > Apport): >> > > >> > > (gdb) bt full >> > > #0 0x00007f404e58e3b6 in mail_index_keywords_unref () from >> > > /usr/lib/dovecot/libdovecot-storage.so.0 >> > > No symbol table info available. >> > > #1 0x00007f404ea254ad in _start () >> > > No symbol table info available. >> > > >> > > Any help would be appreciated, I am happy to provide more info if >> > > necessary. >> > > >> > > Tamas

2 2

dovecot as proxy and verification of the backends certificate
by Heiko Schlittermann 13 Oct '15

13 Oct '15

Hello, I'm using a dovecot as proxy, connecting to one or more backends. The backends use X.509 certificates. The proxy's passdb returns extra fields: user=foo proxy host=backend1.<domain> ssl=yes nopassword=y Thus the proxy connects to the backend but can't verify the backends certificate. The following comment suggests using ssl_client_ca_file for that. # Directory and/or file for trusted SSL CA certificates. These are used only # when Dovecot needs to act as an SSL client (e.g. imapc backend). The # directory is usually /etc/ssl/certs in Debian-based systems and the file is # /etc/pki/tls/cert.pem in RedHat-based systems. #ssl_client_ca_dir = #ssl_client_ca_file = ssl_client_ca_file = /tmp/certs/ca-local.pem But that does not work! Instead I've to use ssl_ca # PEM encoded trusted certificate authority. Set this only if you intend to use # ssl_verify_client_cert=yes. The file should contain the CA certificate(s) # followed by the matching CRL(s). (e.g. ssl_ca = </etc/ssl/certs/ca.pem) # ssl_ca = ssl_ca = </tmp/certs/ca-local.pem Bug or feature? Mainly I'm asking because the comments do not indicate that I should have used ssl_ca for this type of operation (dovecot as a SSL client) Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Dovecot auth-ldap ignores tls_* settings when using ldaps://
by Heiko Schlittermann 13 Oct '15

13 Oct '15

Hi, I'm using dovecot 2.2.9 (but after checking src/auth/db-ldap.c in 2.2.13 there seems to be the same bug/feature). The userdb and passdb use LDAP. All further configuration is done in auth-ldap.conf.ext. uri = ldaps://<host>/ # tls = tls_cert_file = /etc/ssl/certs/client-cert.pem tls_key_file = /etc/ssl/certs/client-key.file Dovecot ignores the tls_* options. If I use an ldap:// URI and switch on TLS using tls=yes it works as expected. But I do not see any reason why LDAPS should not read the tls_* settings. This small patch solved it for me --- dovecot-2.2.9/src/auth/db-ldap.c 2013-11-24 14:37:39.000000000 +0100 +++ dovecot-2.2.9.hs12/src/auth/db-ldap.c 2015-10-08 21:24:47.051446465 +0200 @@ -1043,7 +1043,7 @@ static void db_ldap_set_tls_options(struct ldap_connection *conn) { - if (!conn->set.tls) + if (!(conn->set.tls || strncmp(conn->set.uris, "ldaps:", 6) == 0)) return; #ifdef OPENLDAP_TLS_OPTIONS It would be great, if somebody can confirm this and if this or some equivalent patch could make it upstream. Best regards from Dresden/Germany Viele Grüße aus Dresden Heiko Schlittermann -- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} - gnupg encrypted messages are welcome --------------- key ID: F69376CE - ! key id 7CBF764A and 972EAC9F are revoked since 2015-01 ------------ -

2 2

Multiple core solr support
by Daniel Miller 13 Oct '15

13 Oct '15

Is it possible to use variables, such as domain or username, in the plugin url declaration? This would allow using multiple cores to separate the stored indexes. So when my own giant mailbox gets corrupt...I don't have to rebuild everybody's... Something like url=http://mysolr.server.com:8983/solr/%u -- Daniel

2 1

fts solr ignores new mailboxes until fts rescan
by Giorgio Paolucci 13 Oct '15

13 Oct '15

Hello, on my debian wheezy installation, I implemented fts plugin with solr as indexing system. I noticed that if a user creates a new folder, that folder it is ignored by dovecot indexing until a doveadm fts rescan is performed for that user. From that time on everything works ok on that folder. As a test I set fts_autoindex=yes and put in that folder a brand new mail never indexed before. in my logs indexer-worker says it indexed 0 messages Obviously this is a major issue as users often create new folders, but a search always returns empty results. My system is a debian wheezy with dovecot packages from backports dovecot 2.2.13-11~bpo70+1. Am I missing something? Any clue would be appreciated. Thank you all in advance. Bests Giorgio Paolucci ============================= doveconf -n # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.8 auth_default_realm = xxx disable_plaintext_auth = no log_path = /var/log/dovecot/dovecot.log mail_fsync = always mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = quota listescape mail_log notify fts fts_solr managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mdbox_rotate_interval = 1 days mdbox_rotate_size = 10 M mmap_disable = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / } passdb { args = /etc/dovecot/conf.d/dovecot-sql.conf.ext driver = sql } passdb { args = /etc/dovecot/conf.d/dovecot-sqlSSO.conf.ext driver = sql } plugin { fts = solr fts_solr = url=http://%n.%1n.solr.csia:8983/solr/ mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size pop3_migration_mailbox = INBOX quota = dict:User quota::file:/SYSTEM/USERS/homes/%d/%1u/%u/%u.quota quota_rule = *:storage=7000M quota_rule2 = Trash:storage=+100M quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /SYSTEM/USERS/mailbox/%d/%1n/%u/.dovecotactive.sieve sieve_default = /etc/dovecot/default.sieve sieve_dir = /SYSTEM/USERS/mailbox/%d/%1n/%u/sieve } pop3c_host = 147.162.10.68 protocols = imap pop3 lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service imap-login { inet_listener imaps { ssl = no } process_min_avail = 20 service_count = 1 } service imap { executable = imap postlogin process_limit = 4096 } service lmtp { executable = lmtp inet_listener lmtp { port = 2223 } process_min_avail = 15 } service pop3-login { inet_listener pop3s { ssl = no } } service pop3 { process_limit = 256 } service postlogin { executable = script-login -d rawlog user = vmail } service quota-warning { executable = script /SYSTEM/DOVECOT/scripts/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = no userdb { driver = prefetch } userdb { args = /etc/dovecot/conf.d/dovecot-sql.conf.ext driver = sql } userdb { driver = prefetch } userdb { args = /etc/dovecot/conf.d/dovecot-sqlSSO.conf.ext driver = sql } protocol lmtp { info_log_path = /tmp/dovecot-lmtp-info.log log_path = /tmp/dovecot-lmtp.log mail_plugins = quota listescape mail_log notify sieve } protocol lda { mail_plugins = quota listescape mail_log notify } protocol imap { imap_max_line_length = 128 k mail_max_userip_connections = 20 mail_plugins = quota listescape mail_log notify fts fts_solr imap_quota } protocol doveadm { mail_plugins = quota listescape mail_log notify fts fts_solr pop3_migration } protocol pop3 { pop3_no_flag_updates = yes pop3_uidl_format = %10v.%09u

2 3

Problems sync with mail client MAC and dovecot-2.2.10-4.el7_0.1.el7_0.1.x86_64
by grupo correo 13 Oct '15

13 Oct '15

Hello, i have installed Centos 7.1.1503 with Dovecot 2.2.10-4.el7_0.1. With Thunderbird all works fine, but when i configure the email client of Mac (Yosemite) with IMAP i can see a error in the log and the client not see the emails: Oct 9 14:31:34 localhost dovecot: imap(xxxxxxx@xxxxxxx): Fatal: master: service(imap): child 18639 killed with signal 11 (core dumps disabled) I have read the instructions of http://www.dovecot.org/bugreport.html, but core file is not writing. Does anyone have this issues? I think that can be a problem in the configuration Best regards

2 1

iterate users with passwd-file passdb?
by Eric Abrahamsen 12 Oct '15

12 Oct '15

Apologies for the basic question, but I've been googling and can't figure out if this is possible or not. Simply: Is it possible to iterate over users if I'm using the passwd-file passdb driver? Do I need a SQL-based driver if I want to iterate? Here's what I've got right now. I don't have many users, and would prefer to retain the simplicity of the passwd-file if possible. Thanks! passdb { driver = passwd-file args = username_format=%u scheme=ssha512 /etc/dovecot/passwd.db deny = no master = no pass = no skip = never result_failure = continue result_internalfail = continue result_success = return-ok } userdb { driver = static args = uid=5000 gid=5000 home=/var/mail/vmail/%d/%n }

1 0