November 2013 - dovecot

[Dovecot] Replication via sneakernet
by David Bishop 07 Dec '13

07 Dec '13

Is there a way to get a running transaction log from a Dovecot instance to synchronize (and I use that word very, very loosely) multiple instances that don't share a network? Why I would want to do such a thing: I have a couple way stations along a long, winding road through very hilly country, and at each of these stations, I have a few people (and many, many more at one end of the road). Which people are where varies from hour to hour (but they're all members of a known, larger, group of people with accounts). It would be helpful to get communication (however limited) along this road (and establishing direct communication, either through radio or landline, is not feasible within the budget constraints I have), and it is possible to store a complete copy of the mail spool to a small box at each station (the total mail spool over the period I care about is in the tens of megs). There are trams shuttling back and forth along this road (stopping at each station), and adding a small box (such as a weatherproofed Raspberry Pi with a wifi dongle) to transport files up and down the road is pretty simple. Doing incremental copies of the mail spool is also simple, keeping track of files that get moved or deleted is a bit harder (but still not that bad), but I have no idea how hard it is to track what messages have been seen, especially for the shared folders (which we use instead of mailing lists). I realize this is not a "normal" use of Dovecot, but given this, does anyone who knows the software (and how it keeps track of messages) better have suggestions on how to approach the problem?

4 6

[Dovecot] Full text search improvements
by Timo Sirainen 05 Dec '13

05 Dec '13

FTS indexing is something I hear quite often nowadays. I’ve added some hacks to make it work better for some installations, but it’s about time to think about the whole design and how it could be improved for everyone in future. Here are some of my initial thoughts. Currently Dovecot supports 3 full text search engines: Solr, CLucene and Dovecot Squat. CLucene plugin has various features built in, which should have been built in a generic way to work with all the engines (although Solr has most of those already built-in). Squat was abandoned a few years ago in favor of Solr/CLucene, but perhaps it could be brought back to life, since it looks like its index sizes could be smaller than Lucene's. Here's a list of things that should be added to generic Dovecot FTS code to improve all the backends: 1. Support for multiple languages. Use textcat while indexing to guess the language of the indexed data. (Perhaps run it separately for each paragraph to handle multi-language mails? Or at least many emails begin/end with different language than the text in the middle, e.g. "Foo Bar wrote:" is often in various languages.) Index the data using the detected language's stemming and other features. Keep track of which languages have been used in the index, and when searching stem the search words to all the used languages. Since each added language requires additional searches and there's the possibility of wrong detection, the list of allowed languages could be configurable. See also http://ntextcat.codeplex.com/ or at least change textcat to use UTF8. 2. Word stemming. This can be done for many languages with Snowball library. Solr has also implemented several other languages, perhaps its code can be somehow automatically translated to C(++) for use with Dovecot? 3. Don't index language-specific stopwords. We can get the word lists from e.g. Solr. 4. Try to detect compound words and index each part separately for languages that use them. http://wiki.apache.org/solr/LanguageAnalysis#Decompounding suggests two possible ways to do it. 5. Normalize words (e.g. drop diacritics). libicu can be used for this. 6. Drop (Unicode) characters that don't belong to the language? Or especially don't index most of the weird Unicode characters. This would avoid filling the index with unnecessary garbage. 7. Don't index non-text data? For example if there is large block of base64 data or something else that definitely doesn't look like text, it's pretty useless to index it. Then again, we do want to index all kinds of IDs that someone might want to search. This could be a bit difficult to implement well. 8. Index attachments separately, so it would be possible to search only attachments. (Should "SEARCH BODY word1 BODY word2" return matches if word1 and word2 are in different attachments?) 9. Attachments can be translated to indexable UTF-8 text already with fts_decoder setting by doing it via a conversion script. This could also support Apache Tika server directly. 10. It should be configurable which fields are indexed. Body and header would always be separately indexed. Optionally there could be also at least: attachments, From, To, Cc, Bcc and Subject. The From/To/Cc/Bcc could also be indexed together in one "addresses" field. The more fields there are, the larger the index, but better/faster search results. 11. Each indexed mail should have metadata: Mailbox GUID, mail UID and the language the mail was indexed with. For attachments there should also be the MIME part number. When matching results, drop results if returned language doesn't match the query language. Squat ----- Currently Squat index consists of a trie containing all the words and pointer to a file listing all the message UIDs that contain them. Each node in the trie has a pointer to the UIDs, so e.g. with "abc" the "a" node will contain UIDs of all mails that contain the "a" letter (e.g. 1,3-5,10). "ab" node will contain mails that have the "ab" substring. Since the "ab" is a subset of "a", the "ab" won't contain UIDs directly but instead it contains indexes to the "a" list to get a better compression (e.g. UID 3-5,10 -> 2-4 indexes in the "1,3-5,10" list). The "abc" node then similarly refers to the "ab" node's indexes. It's configurable how long words Squat will index. Also substring matching is configurable. By default both are 4 letters, so words longer than 4 letters will be split to 4 letter pieces which are indexed (e.g. "dovecot" -> "dove", "ovec", "veco", "ecot"). When searching these pieces are looked up and the results are merged. It's pretty pointless to do a search for 1-2 letter substrings. Most likely the user wants to find 1-2 letter word instead. Perhaps this is true also for 3 letters? The Squat index could be changed to only add results for the first 1-2 (or 1-3?) letters only for full words, not to word prefixes. This of course would mean that the "ab" referring to "a" UID list would no longer work for the first nodes. Substring searching likely wouldn't work very nicely for stemmed words. So Squat should probably index the full stemmed word and then also index the unstemmed word in the small 4 letter pieces. It should be possible to also disable substring searching entirely. Squat already attempts to reduce disk space by encoding the common characters with less bits than other characters. This is hardcoded for English language though. Each index compression could analyze the most common letters and dynamically use those. Perhaps different languages could even use separate Squat index files to get the most advantage of this? Although if there are a lot of languages it's a bit annoying to have many different index files. Squat currently indexes each Unicode character as one letter, so there can be quite a huge number of different 4 letter words. This substring indexing should probably be disabled for words that contain letters not used by the current language. The UID lists are not compressed very efficiently. See http://www2009.eprints.org/41/1/p401.pdf for alternatives. There is currently one Squat index per mailbox. There should be one per user. This requires adding a separate ID -> { mailbox GUID, UID, language, mime part } mapping file. Squat indexes should be updated in two parts. For old mails there are the large trie+uidlist base files in optimized format. For new mails there are small trie+uidlist files in a format where they are just appended to. When the trie or uidlist becomes large enough, it's first internally sorted/optimized, and then merged with the base files by recreating new optimized base files. This fixes the scalability problem with current large Squat indexes where with large unsorted indexes the sorting could have taken forever. Since substring searches in Squat produce only "maybe matches" UID lists, the mails still need to be opened and searched through. This step could probably also do the same language detection + stemming as is done during indexing to improve the search results. (And this feature could be enabled even if no FTS index is enabled.)

6 9

[Dovecot] BUG: Authentication client sent unknown handshake command
by Emmanuel Dreyfus 04 Dec '13

04 Dec '13

Hi After upgrading the kernel, everything is fine, except dovecot authentication. I get this trange thing (data after REQUEST? changed just in case it contains anything sensitive): Nov 29 16:56:01 volanges dovecot: auth: Error: BUG: Authentication client sent unknown handshake command: REQUEST?6970356762?616?6?235264ef69dbd1665538af54d12fdaea?session_pid=453?req... Nov 29 16:56:01 volanges dovecot: imap: Error: Authentication server didn't send valid SPID as expected: MECH PLAIN plaintext Nov 29 16:56:01 volanges dovecot: imap: Error: Disconnected from auth server, aborting (client-pid=161 client-id=1) Nov 29 16:56:01 volanges dovecot: imap-login: Internal login failure (pid=161 id=1) (internal failure, 1 successful auths): user=<jdoe>, method=PLAIN, rip=192.0.2.251, lip=192.0.2.10, mpid=453, TLS, TLSv1 with cipher AES128-SHA (128/128 bits) Reverting to the previous kernel fixed the problem, but I have not been able to spot what the problem was. Any idea? -- Emmanuel Dreyfus manu(a)netbsd.org

3 5

[Dovecot] Gettings mails recently moved to a folder
by Florian Lindner 03 Dec '13

03 Dec '13

Hello, some spam training tools learn the same message over and over again. Since I want to use a daily cronjob to train messages and I do not want to have the same messages trained multiple times, I need to way to get new messages in a folder. Is there a (consolish) way to get a list of messages that were recently moved to a folder? dovecot does not modify mtime, according to RFC 3501. Thanks, Florian

5 9

[Dovecot] proxy, userdb and passdb
by Jogi Hofmüller 03 Dec '13

03 Dec '13

Dear all, We are getting closer to the migration of our mailsystem. Now I have a special question. We are successfully using passdb { driver = pam } and that is good. Now, how would I tell dovecot to proxy certain users (the ones not yet migrated) to the old server? My attempts to configure an additional userdb failed since this seems to override the passdb setting. Grateful for any hints! Cheers, -- j.hofmüller Optimism doesn't alter the laws of physics. - Subcommander T'Pol

4 5

[Dovecot] dovecot-antispam plugin problem with multiple messages
by Richard Platel 29 Nov '13

29 Nov '13

Hi With dovecot 2.2.5, and dovecot-antispam built from a recent HEAD pull, when copying multiple messages to or from a Spam folder, the plugin sends multiple copies of the first message to the backend. I've tried this with the pipe and spool2dir backends. For example with the spool2dir backend, via IMAP doing A COPY 1:3 Spam yields 3 copies of message id 1 in the dir: dev:imap-8.1 rplatel@imap01:/var/run/dovecot_as_tmp$ sudo -u mail md5sum * 28ad0a215eb7ecbd3a814a8a334d85bf 00000000001385586164-rplatel(a)ff-dev.com-00001s 28ad0a215eb7ecbd3a814a8a334d85bf 00000000001385586164-rplatel(a)ff-dev.com-00002s 28ad0a215eb7ecbd3a814a8a334d85bf 00000000001385586164-rplatel(a)ff-dev.com-00003s I see the same behaviour with the pipe backend, the pipe program is invoked 3 times, but with the same message content.

1 1

[Dovecot] proxy_maybe & director incompatibility
by Trent Lloyd 29 Nov '13

29 Nov '13

Hi Folks, I spent quite some time yesterday understanding how proxy works along with the director. I came to the conclusion that proxy_maybe and director cannot be used together, but this isn’t a true incompatibility so much as caused by the way things are handled and the order they are processed in. The way proxy_maybe works is that it is processed by the auth provider once it gets the response from the passdb, it checks for proxy_maybe and then checks for the ‘host’ parameter and compares it to the local IP (this is always null at that stage, because director won’t add host until later). proxy_maybe is deleted and then if the IPs do not match (i.e. the connection should be proxied) it sets proxy. This result is returned from the auth provider and then piped into director, which adds the relevant ‘host’ parameter if ‘proxy’ is set. The problem here is that because proxy_maybe is processed before director, it is not possible to conditionally proxy when using director — only if host is also returned from passdb. The secondary problem is that director only adds host= if proxy is set (and the auth code generally assumes proxy/proxy_maybe/proxy_always are exclusive settings) — this logic would also need to change. You would also need some logic to add host only if host doesn’t already exist, to handle situations where proxies might come from both passdb and/or director. I am seeking to understand if there is any significant reason proxy_maybe is handled during the auth section, it would seem better to simply always set ‘proxy=yes’, and then optionally have proxy_maybe passed all the way through to the connection stage and then do the local host check there. This would solve my use case, and I cannot imagine what else it would break — but I am no expert on dovecot or other people’s use cases, so I am hoping for feedback from others on this and what else would need to be considered or why this would not work before I spent time trying to implement the change. Thanks, Trent Lloyd w: www.webinabox.net.au

2 1

[Dovecot] Dovecot's brittle configuration syntax
by Kai Hendry 29 Nov '13

29 Nov '13

Hi there, Whilst trying to come up with a minimal configuration for Dovecot: http://dabase.com/blog/Minimal_Dovecot/ I noticed the configuration syntax is a bit admin unfriendly. It's easy to get an infamous Error code 89. Is there any back story to the grammar or language this configuration is in? Kind regards,

2 1

Re: [Dovecot] Dovecot's brittle configuration syntax
by Joseph Tam 29 Nov '13

29 Nov '13

Kai Hendry <hendry(a)dabase.com> wrote: > Whilst trying to come up with a minimal configuration for Dovecot: > http://dabase.com/blog/Minimal_Dovecot/ I think your characterization of the complexity of dovecot configuration by simply counting configuration lines to get it working is off base. It's conceivable to have all the default values work for you and only specify those that differ from defaults (or configure your system to conform to defaults if you want to place the proverbial cart before the horse), but that doesn't make it less complex -- it just hides it. (See doveconf -a if you want to see what you're shorthanding). If you want minimal configuration fuss, you can try uw-imapd. Not a lot of knobs to twist on that thing, but you'll have to accept all its limitations as well. A well chosen set of defaults that works for most people is a good design decision, but I've seen plenty of different and oddball layouts (including the sneakernet post preceding this one) that requires a lot of flexibility, which necessarily requires a lot of configuration to express them. The sample configuration is verbose because it doubles as documentation: you can use it as a starting point and remove the parts you don't need, or start from the documentation and move forward. I agree, though, that the documentation needs an editorial overhaul -- it's a little bewildering to find what you need along with the contextual information. > I noticed the configuration syntax is a bit admin unfriendly. It's easy > to get an infamous Error code 89. > > Is there any back story to the grammar or language this configuration is > in? It appears to me to have grown organically, and maybe will need pruning some day. Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: [Dovecot] Replication via sneakernet
by David Bishop 29 Nov '13

29 Nov '13

D'oh. Replied to Timo instead of to the list. Apologies! On Thu, Nov 28, 2013 at 1:07 PM, David Bishop <dovecot(a)dpe.lusars.net>wrote: > On Thu, Nov 28, 2013 at 1:02 AM, Timo Sirainen <tss(a)iki.fi> wrote: > >> On 28.11.2013, at 5.17, David Bishop <dovecot(a)dpe.lusars.net> wrote: >> >> > There are trams shuttling back and forth along this road (stopping at >> each >> > station), and adding a small box (such as a weatherproofed Raspberry Pi >> > with a wifi dongle) to transport files up and down the road is pretty >> > simple. >> >> But if you do it this way and you can keep a full copy of the shared mail >> storage on your Raspberry, that would be possible already with dsync I >> think. dsync supports quick incremental updates by keeping track of the >> previous state between the servers. This state is saved in a file, so you >> could keep a different state for each different dsynced server. >> > > Bravo for a solution that doesn't require a code change and doesn't seem > to require directly touching the spools! :) > > Reading the man page, it looks like only mirroring for people who happen > to be checking email at a given place (as well as people who have received > mail and the shared mailboxes) seems like a pretty simple thing to do. > Hurrah! > > Thank you, this is wonderful. > > And more questions... > > If I'm running a virtual mail domain, the user/pass I give is for the > virtual mail user, correct? And, in a virtual mail setup, do I specify > "username" or "username@domain"? > > Is there a window beyond which synchronization becomes more difficult? For > instance, if messages (or metadata updates, like "I deleted this message") > get parked overnight without updating, and in the morning (9 hours later), > the trams (one at a time) pull into range of wifi, there won't be > confusion, right? > >

2 1