January 2015 - dovecot

mdbox GUID
by Nagy, Attila 04 Jan '15

04 Jan '15

Hi, The dbox page says: "Each message has a 128 bit globally unique identifier (GUID)" I guess this is what guid_128_generate() generates. Is this believed to be really globally unique? As far as I can tell, in the current architecture they are -given there are enough time precision-, because all dovecot daemons are single threaded, so they have concurrency but doesn't have parallelism (they are not multi threaded). I would like to list all of the messages, with this GUID as the unique identifier, so a collision would be fatal here...

1 0

ntlm_auth random failures with dovecot
by Jason Gunthorpe 04 Jan '15

04 Jan '15

I'm still a bit fuzzy on exactly what has blown up here since my 1.2 install (or maybe it was broken then and I never noticed), but it looks like the way dovecot is calling out to ntlm_auth is violating the --helper-protocol=squid-2.5-ntlmssp scheme. The issue is how it handles simultaneous clients connecting - for instance launching thunderbird with NTLM auth creates multiple imapds that all have to be auth'd. Since dovecot doesn't (and apparrently didn't in 1.2?) serialize this it ends up sending a jumble to ntlm_auth. Strace sayth, as example: read(0, "YR xxxxxxx=\n", 4096) = 48 read(0, "YR xxxxxxx=\n", 4096) = 48 read(0, "KK xxxxxxx=\n",4096) = 176 read(0, "KK xxxxxxx=\n",4096) = 176 That is two clients connecting at once, and the sequence has become jumbled. Fiddling around with ntlm_auth manually I can get it to give me this: YR xxx # 1 TT xxx # 1 YR xxx # 2 TT xxx # 2 KK xxx # 2 AF jgg # 2 KK xxx # 1 Called NTLMSSP after state machine was 'done' GENSEC login failed: NT_STATUS_INVALID_PARAMETER NA NT_STATUS_INVALID_PARAMETER Ie, reordering the sequence (# 1 and # 2) causes it to tell you that, no, the sequence cannot be reordered. To me this says the samba folks expect that the YY/TT/KK/AF sequence is *NOT* reordered. The implication is that the mech-winbind in dovecot must seralize everything, and it doesn't! So, this is fairly broken, I can hit these failure causes with a high probability when using thunderbird. Any thoughts on how to repair this? The simplest answer would be to pool and assign a ntlm_auth process to each incoming auth context, or to actually serialize auth. But it can't treat ntlm_auth as a stateless helper. Jason

1 0

[PATCH] LAYOUT=imapdir is broken in v2.2
by Jason Gunthorpe 04 Jan '15

04 Jan '15

The next thing I noticed in my v1.2 -> 2.2 upgrade is that mail_location = maildir:[..]:LAYOUT=imapdir is broken, the symptom is dovecot returning this to the client when requesting any mailbox beyond INBOX: Character not allowed in mailbox name: ' Which is actually trying to say "Character not allowed in mailbox name: '\0'", but since the %c is not escaped it ends up with the truncated string. This patch fixes it: diff --git a/src/lib-storage/list/mailbox-list-maildir.c b/src/lib-storage/list/mailbox-list-maildir.c index c99a2900a6d6..ae5f35d955ac 100644 --- a/src/lib-storage/list/mailbox-list-maildir.c +++ b/src/lib-storage/list/mailbox-list-maildir.c @@ -46,6 +46,7 @@ static struct mailbox_list *imapdir_list_alloc(void) list = p_new(pool, struct maildir_mailbox_list, 1); list->list = imapdir_mailbox_list; list->list.pool = pool; + list->sep = '.'; list->global_temp_prefix = IMAPDIR_GLOBAL_TEMP_PREFIX; list->temp_prefix = p_strconcat(pool, list->global_temp_prefix, Analysis: I noticed this while upgrading a dovecot install from 1.2.15 (squeeze) to 2.2.13 (jessie). This upstream commit author Timo Sirainen <tss(a)iki.fi> Thu Jan 20 20:59:07 2011 +0200 (2011-01-20) changeset 12586 a2780b694b2d parent 12585 b748c622e896 child 12587 c3a258ee96c4 lib-storage: mailbox_alloc() now takes a virtual mailbox name and other related API changes. All storage_name <-> vname conversions now go through the same two mailbox_list methods. This has many benefits, such as: * listescape plugin is now much simpler and bugfree * allows changing lib-storage API to use UTF-8 mailbox names in future * allows creation of "mailbox aliases" plugin Restructed the _alloc functions to move the hierarchy_sep from the initializer into the _alloc call itself: @@ -29,6 +30,7 @@ static struct mailbox_list *maildir_list_alloc(void) list = p_new(pool, struct maildir_mailbox_list, 1); list->list = maildir_mailbox_list; list->list.pool = pool; + list->sep = '.'; list->global_temp_prefix = MAILDIR_GLOBAL_TEMP_PREFIX; list->temp_prefix = p_strconcat(pool, list->global_temp_prefix, [..] struct mailbox_list maildir_mailbox_list = { .name = MAILBOX_LIST_NAME_MAILDIRPLUSPLUS, - .hierarchy_sep = '.', .props = MAILBOX_LIST_PROP_NO_MAILDIR_NAME | MAILBOX_LIST_PROP_NO_ALT_DIR | MAILBOX_LIST_PROP_NO_NOSELECT, [..] struct mailbox_list imapdir_mailbox_list = { .name = MAILBOX_LIST_NAME_IMAPDIR, - .hierarchy_sep = '.', .props = MAILBOX_LIST_PROP_NO_MAILDIR_NAME | MAILBOX_LIST_PROP_NO_ALT_DIR | MAILBOX_LIST_PROP_NO_NOSELECT, Noting that heierarchy_sep was removed from maildir_mailbox_list and imapdir_mailbox_list but only added to maildir_list_alloc(), and not imapdir_list_alloc(). This ultimately results in mailbox_list_get_hierarchy_sep() returning '\0' and mailbox_verify_name() failing everything (all strings contain '\0' according to strchr). This ended up as debian bug #774533 Regards, Jason

1 0

dovecot and ntlm_auth through winbind
by Jason Gunthorpe 04 Jan '15

04 Jan '15

Hi all, I am upgrading a dovecot 1.2 installation to a 2.2 installation and have found and fixed a number of problems.. I've seen several postings in the archive about ntlm_auth not working, and it is true, there are several regressions in dovecot here. The first and simplest is that the enablement instructions in the wikki are wrong. ntlm_auth must be called as root, which is what 1.2 did. 2.2 defaults to calling it as the auth user which subtly doesn't work: Dec 30 20:30:21 quartz dovecot[8439]: auth: Error: Login for user []\[jgg]@[wakko] failed due to [Reading winbind reply failed!] Dec 30 20:30:21 quartz dovecot[8439]: auth: Error: ../auth/ntlmssp/ntlmssp_server.c:454: Checking NTLMSSP password for \jgg failed: NT_STATUS_UNSUCCESSFUL Dec 30 20:30:21 quartz dovecot[8439]: auth: Error: GENSEC login failed: NT_STATUS_UNSUCCESSFUL Dec 30 20:30:21 quartz dovecot[8439]: auth: Error: winbind: ntlm_auth exited with exit code 0 The fix is simple, run auth as root: service auth { user = root } This ended up as debian bug #774263 which has a few more details. Regards, Jason

1 0

dovecot-lda segfaults with replication
by Stefan Le Breton 03 Jan '15

03 Jan '15

Hi, I have set up replication, but dovecot-lda now fails with a segmentation fault. Problem seems to be in replication/replication-plugin.c line 253: struct replication_user *ruser = REPLICATION_USER_CONTEXT(ctx->ns->user); resulting in ruser = 0x0 # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 (3e924b1b6c5c+) # OS: FreeBSD 10.0-RELEASE amd64 gdb Backtrace + doveconf -n below. Regards Stefan gdb backtrace: #0 0x0000000802001bc4 in replication_mail_transaction_commit (txn=0x801c811b0, changes=0x7fffffffd3d0) at replication-plugin.c:258 ctx = (struct replication_mail_txn_context *) 0x801c811b0 ruser = (struct replication_user *) 0x0 priority = REPLICATION_PRIORITY_NONE #1 0x0000000801490749 in notify_contexts_mail_transaction_commit (t=0x801c24600, changes=0x7fffffffd3d0) at notify-plugin.c:132 ctx = (struct notify_context *) 0x801c8d020 mail_txn = (struct notify_mail_txn *) 0x801c282b0 #2 0x0000000801491a53 in notify_transaction_commit (t=0x801c24600, changes_r=0x7fffffffd3d0) at notify-storage.c:182 lt = (struct notify_transaction_context *) 0x0 lbox = (union mailbox_module_context *) 0x801cade50 #3 0x0000000800ae1ff9 in mailbox_transaction_commit_get_changes (_t=0x801ce01b0, changes_r=0x7fffffffd3d0) at mail-storage.c:1916 _data_stack_cur_id = 3 t = (struct mailbox_transaction_context *) 0x801c24600 box = (struct mailbox *) 0x801cad840 save_count = 1 ret = -128 #4 0x0000000800ae1f49 in mailbox_transaction_commit (t=0x801ce01b0) at mail-storage.c:1897 changes = {pool = 0x801cd5420, uid_validity = 1234722882, saved_uids = { arr = {buffer = 0x801cd5440, element_size = 8}, v = 0x801cd5440, v_modifiable = 0x801cd5440}, ignored_modseq_changes = 0, changed = true, no_read_perm = false} ret = 8 #5 0x000000080244bfb2 in sieve_action_opr_optional_read () from /usr/local/lib/dovecot-2.2-pigeonhole/libdovecot-sieve.so.0 No symbol table info available. #6 0x0000000802443924 in sieve_result_implicit_keep () from /usr/local/lib/dovecot-2.2-pigeonhole/libdovecot-sieve.so.0 No symbol table info available. #7 0x0000000802443dcd in sieve_result_execute () from /usr/local/lib/dovecot-2.2-pigeonhole/libdovecot-sieve.so.0 No symbol table info available. #8 0x0000000802452f1c in sieve_execute () from /usr/local/lib/dovecot-2.2-pigeonhole/libdovecot-sieve.so.0 No symbol table info available. #9 0x0000000802205912 in sieve_plugin_init () from /usr/local/lib/dovecot/lib90_sieve_plugin.so No symbol table info available. #10 0x0000000800829104 in mail_deliver (ctx=0x7fffffffda08, storage_r=0x7fffffffd960) at mail-deliver.c:400 ret = 8 #11 0x0000000000402f2e in main (argc=7, argv=0x7fffffffdb08) at main.c:437 set_roots = {0x604970, 0x0} ctx = {pool = 0x801c29420, set = 0x801c67988, session = 0x801c29440, dup_ctx = 0x801c237c0, session_id = 0x0, src_mail = 0x801c48840, src_envelope_sender = 0x0, dest_user = 0x801c87040, dest_addr = 0x7fffffffddaf "mail(a)maserver.de", final_dest_addr = 0x7fffffffddaf "mail(a)maserver.de", dest_mailbox_name = 0x403d68 "INBOX", dest_mail = 0x0, var_expand_table = 0x0, tempfail_error = 0x0, tried_default_save = false, saved_mail = false, save_dest_mail = false, mailbox_full = false, dsn = false} service_flags = 1027 user = 0x7fffffffdd9e "mail(a)maserver.de" errstr = 0x37f <Error reading address 0x37f: Bad address> path = 0x7fffffffddc0 "/tmp/mail" lda_set = (struct lda_settings *) 0x801c67988 storage_service = (struct mail_storage_service_ctx *) 0x801c46840 service_user = (struct mail_storage_service_user *) 0x801c67190 service_input = {module = 0x403d55 "lda", service = 0x403d55 "lda", username = 0x7fffffffdd9e "mail(a)maserver.de", session_id = 0x0, local_ip = { family = 0, u = {ip6 = {__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family = 0, u = {ip6 = {__u6_addr = { __u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, local_port = 0, remote_port = 0, userdb_fields = 0x0, flags_override_add = 0, flags_override_remove = 0, no_userdb_lookup = 0} storage = (struct mail_storage *) 0x0 user_source = 0x404090 "" destaddr_source = 0x403d73 "-a parameter" process_euid = 0 stderr_rejection = false ret = 1 c = -1 error = MAIL_ERROR_NONE # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 (3e924b1b6c5c+) # OS: FreeBSD 10.0-RELEASE amd64 auth_mechanisms = plain login digest-md5 doveadm_password = welcomemyguest first_valid_gid = 0 first_valid_uid = 125 listen = * login_greeting = McImap mail_location = maildir:~/.maildir mail_log_prefix = "%Us(%u): " mail_plugins = " notify replication" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate passdb { args = dovecot driver = pam } passdb { args = /usr/local/etc/dovecot/dovecot-mysql.conf driver = sql } plugin { antispam_backend = mailtrain antispam_mail_notspam = some@where antispam_mail_sendmail = /usr/sbin/sendmail antispam_mail_spam = some@elsewhere antispam_spam_pattern = INBOX.spam* plugin_debug = yes } protocols = imap sieve service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0666 user = vmail } unix_listener auth-userdb { group = vmail mode = 0660 user = vmail } user = root } service doveadm { inet_listener { port = 31337 ssl = yes } } service managesieve-login { inet_listener sieve { port = 4190 } } service replicator { unix_listener replicator-doveadm { group = vmail mode = 0600 user = vmail } } ssl_cert = </a/cert.pem ssl_key = </a/key.pem userdb { driver = passwd } userdb { args = /usr/local/etc/dovecot/dovecot-mysql.conf driver = sql } verbose_proctitle = yes protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep mail_plugins = " notify replication antispam" } protocol lda { mail_plugins = " notify replication sieve" } protocol lmtp { mail_plugins = " notify replication sieve" }

1 0

Help with installing Apache Solr with Dovecot 2.2
by Kev Lau 03 Jan '15

03 Jan '15

Hi, I am relatively new to Apache Solr and I am trying to get it to integrate with Dovecot 2.2 running on Ubuntu 14.04 and needed a little help with configuration. The following steps I have already configured. Appreciate if someone could help, so please help me! 1. Apache Solr installed, up and running 6406 [main] INFO org.eclipse.jetty.server.AbstractConnector ? Started SocketConnector@0.0.0.0:8983 I can then ssh into it and see the running solr admin. ssh -t -L 8983:localhost:8983 user(a)mydomain.com 2. Schema file added startTime: 3 minutes ago instanceDir: /opt/solr/solr/collection1/ dataDir: /opt/solr/solr/collection1/data/ In my collection1 core selector I have the following files:- 1.solrconfig.xml 2.schema.xml The output of schema.xml is shown below:- <?xml version="1.0" encoding="UTF-8" ?>  <schema name="dovecot" version="1.5"> <field name="_version_" type="long" indexed="true" stored="true"/> <field name="_root_" type="string" indexed="true" stored="false"/> <types>  <fieldType name="string" class="solr.StrField" omitNorms="true"/> <fieldType name="long" class="solr.LongField" omitNorms="true"/> <fieldType name="slong" class="solr.SortableLongField" omitNorms="true"/> <fieldType name="float" class="solr.FloatField" omitNorms="true"/> <fieldType name="boolean" class="solr.BoolField" omitNorms="true"/> <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/>  <filter class="solr.SnowballPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.NGramFilterFactory" minGramSize="3" maxGramSize="15"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0"/> <filter class="solr.LowerCaseFilterFactory"/>  <filter class="solr.SnowballPorterFilterFactory" protected="protwords.txt"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> </fieldType> </types> <fields> <field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="uid" type="slong" indexed="true" stored="true" required="true" /> <field name="box" type="string" indexed="true" stored="true" required="true" /> <field name="user" type="string" indexed="true" stored="true" required="true" /> <field name="hdr" type="text" indexed="true" stored="false" /> <field name="body" type="text" indexed="true" stored="false" /> <field name="from" type="text" indexed="true" stored="false" /> <field name="to" type="text" indexed="true" stored="false" /> <field name="cc" type="text" indexed="true" stored="false" /> <field name="bcc" type="text" indexed="true" stored="false" /> <field name="subject" type="text" indexed="true" stored="false" /> </fields> <uniqueKey>id</uniqueKey> <defaultSearchField>body</defaultSearchField> <solrQueryParser defaultOperator="AND" /> </schema> 3. Dovecot package dovecot-solr installed 4. Dovecot configured with following files to run on solr-fts. In 10-mail.conf:- # Space separated list of plugins to load for all services. Plugins specific to # IMAP, LDA, etc. are added to this list in their own .conf files. #mail_plugins = mail_plugins = fts fts_solr In 90-plugin.conf:- plugin { fts = solr fts_solr = break-imap-search url=http://localhost:8983/solr/ } I would really appreciate if someone could please help me here. Request for someone to guide me here. Thanks a lot Kevin

2 4

Designing an Object Storage Plugin for a specific platform
by Listas＠Adminlinux 02 Jan '15

02 Jan '15

Hi! I'm planning to develop a OSP (Object Storage Plugin) to support a specific platform of my company. Is there any API documentation or sample code of a simple plugin that can be used as a basis for an implementation? Thanks! -- Thiago Henrique

1 1

Disabling compressed storage for one public namespace - is it possible?
by Marc Stürmer 02 Jan '15

02 Jan '15

Greetings, I've got Dovecot 2.2.X running with gzip compressed storage enabled by default. My question though is: is it possible to disable this compressed storage for one specific, public namespace only? I haven't found quite much information about it on the wiki. Thanks in advance.

2 1

dovecot 2 low TCP speed (fetching big mail)
by Wojciech Puchar 01 Jan '15

01 Jan '15

i use dovecot 2 under FreeBSD (dovecot2-2.2.15 compiled from ports). tried both with kqueue enabled or not. everything works very fast, EXCEPT fetching big mail. tried multiple clients (thunderbird on windows, alpine on the same server, alpine on other unix server connected by 1Gb/s LAN) and fetching peaks at 1MB/s. NO SSL! could you please point me where to search for a problem. hints: writing over IMAP (eg. writing to Sent folder) works at full speed. Doesn't seem like a problem with unix server, every other service goes at speed 70-100MB/s both directions (like samba, ftp). tried to turn off "kqueue" compile time option, no difference. Please give me a hint at least what to check. Does dovecot use other way to sent data over TCP socket than other programs? thank you.

2 2

Postfix and Dovecot SASL - wiki update
by Dan Langille 01 Jan '15

01 Jan '15

I wanted to mention I updated the Postfix and Dovecot SASL wiki entry yesterday. http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL The entry: submission inet n - - - - smtpd was incorrect. It should be: submission inet n - n - - smtpd By specifying n for chroot, it ensures that DNS lookups can occur. Otherwise, if you are using reject_unknown_recipient_domain the following error will occur: Recipient address rejected: Domain not found Hope that helps. — Dan Langille http://langille.org/

2 2