November 2007 - dovecot

[Dovecot] New full text search indexer
by Timo Sirainen 14 Nov '07

14 Nov '07

As described earlier (http://dovecot.org/list/dovecot/2006-December/018055.html) Dovecot nowadays has full text search indexing support in CVS HEAD. Currently there are two backends: Lucene and Squat. Lucene's problem is that standard IMAP SEARCH command can't be used with it without breaking IMAP RFC. So Lucene can be used only with non-standard X-BODY-FAST and X-TEXT-FAST search commands. Squat's code then is quite complex and it's pretty slow. It works with 4 character blocks, so it can answer: - Search string < 4 characters: Can't answer anything - Search string = 4 characters: Definite yes/no - Search string > 4 characters: Probable yes / definite no Since search strings are usually more than 4 characters, Dovecot uses the index only to filter out messages which definitely won't match and then does regular searching for the non-filtered messages. It works ok, but it could work better. The Squat indexes take maybe 30-50% of the mailbox's size. Most of the space goes to UID lists (list of messages matching each 4 character block). For example one UID list could be "2,4-10,20-30". This is already stored in a compressed format by storing the UID numbers relative to previous UID, which allows storing them usually with a single byte per UID, even if the UIDs are large. So, this morning I had an idea how this could maybe be improved with two changes: 1. Don't use 4 character blocks. Instead allow variable sized paths, and keep track of the UIDs for each node in the path. So for example adding "abc" to UID 1 will place the UID 1 to node "a", "ab" and "abc". 2. Use UIDs relative to parent's UIDs. For example if node "a" has UIDs 1,3,5 and "ab" is added for UID 5, the UID is changed to 2 (3th in 1,3,5 list - 1). This can help the compression a lot. The best case scenario for the above example is if "ab" contains also 1,3,5 UIDs. Then it can be written with a single 0-2 range. With these changes, the indexer can be configured to perform in different ways. Text can be indexed in two ways: full words, and substrings within those words (as required by IMAP). I think a useful combination of these would be to index all 4 character substrings and all full words up to 10 characters or so. This kind of an index can answer: - Search string <= 4 characters: Definite yes/no - For X-BODY-FAST / X-TEXT-FAST: - Search string 5-10 characters: Definite yes/no - Search string 11+ characters: Probable yes / definite no - For standard IMAP SEARCH: - Search string 5-10 characters: Definite yes if message contains the key as non-substring, otherwise same as below. This helps only if user does a search that matches a lot of messages. - Search string 11+ characters: Probable yes / definite no UTF-8 text is indexed as bytes, but the path depth can be counted as characters instead of as bytes. I've a test program finished, and unless there are some bugs left I think it shows that this can work pretty well: Squat-like 4 byte substrings (but can answer 1-3 char queries also): Message count: 7495 Index time: 4.19 CPU seconds (7.39MB/CPUs) Node memory: 3014072 bytes (2943 kB) in 90550 nodes UID memory: 8277212 bytes (8083 kB) in 352778 lists Total: 11291284 / 32474899 bytes = 34.77% Indexing the same mailbox file with squat-test gives: - Index time: 9.55 CPU seconds, 9.63 real seconds (3.24MB/CPUs) - 4145192 bytes in 102208 nodes (12.76%) - 7286058 bytes in 305501 uid_lists (22.44%) - 11431250 bytes total of 32474899 (35.20%) So the new indexer is over twice as fast and uses slightly less space. Indexing 4 char substrings and words up to 10 chars: Index time: 5.13 CPU seconds (6.04MB/CPUs) Node memory: 6002804 bytes (5862 kB) in 615501 nodes UID memory: 9044610 bytes (8832 kB) in 519488 lists Total: 15047414 / 32474899 bytes = 46.34% So it takes somewhat more space, but definitely less than having both Squat + Lucene. No substring indexing, words up to 10 chars (Lucene replacement): Index time: 1.39 CPU seconds (22.28MB/CPUs) Node memory: 3297663 bytes (3220 kB) in 548540 nodes UID memory: 1803735 bytes (1761 kB) in 213554 lists Total: 5101398 / 32474899 bytes = 15.71% Is there a need for Lucene anymore with this kind of a speed and disk space usage? :) The above tests are run with everything being indexed except space and control chars (ie. practically space + tab). If only alphanumeric characters are indexed, the total lines are (in same order as above): Total: 7048664 / 32474899 bytes = 21.70% (-13.07%) Total: 9610543 / 32474899 bytes = 29.59% (-16.75%) Total: 3963335 / 32474899 bytes = 12.20% (-3.51%) There are probably still a few optimizations that can be done to get the disk space usage even lower. I did spent quite a lot of time making it use less CPU though. Even tried a bit different coding style variations in the critical functions to get gcc to create faster code.. One interesting thing I noticed was that binary searching a character from a char array was slower than just sequentially going through the characters. I tried adding checks to do it only if the array was big, but it was always slower. Maybe something to do with CPU cache. If you want to try yourself, the test program is in http://dovecot.org/tmp/new-indexer.c

7 10

Re: [Dovecot] SSL/TLS with Outlook client
by Nikolay Shopik 14 Nov '07

14 Nov '07

Agree with Hugo most root CA have intermidate certificates which should supplied with your server certificate. Otherwise chain won't work and any client don't trust it. - original message - Subject: Re: [Dovecot] SSL/TLS with Outlook client From: Hugo Monteiro <hugo.monteiro(a)fct.unl.pt> Date: 14/11/2007 00:14 Eli Sand wrote: > Hugo Monteiro wrote: > >> Ah ... wildcard certs .. from what i recall, certs issued like >> *.example.com were not very well accepted by M$ clients. You should >> test against non wildcard certs and see how it behaves. >> > > Already have and no luck :( My domain is elisand.com and I have tried > *.elisand.com, mx1.elisand.com (I believe that's what my MX record is... if > not, whatever it is is what I tried) and mail.elisand.com which is the > smtp/imap server name I use in Outlook. All three yield the same result :( > > Eli. > > > I have taken the liberty to connect to your server, using openssl, i've seen the following: $ openssl s_client -CApath /usr/share/ca-certificates/cacert.org/ -connect mail.elisand.com:993 CONNECTED(00000003) depth=1 /O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing Authority/emailAddress=support(a)cacert.org verify return:1 depth=0 /CN=*.elisand.com verify return:1 --- Certificate chain 0 s:/CN=*.elisand.com i:/O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing Authority/emailAddress=support(a)cacert.org --- i believe you should change two things. If the name you wish to use on your clients is mail.alisand.com, then the certificate should read CN=mail.elisand.com. Furthermore, it's always a good idea to provide the chaining certificate path on dovecots side. Try using the ssl_ca_file directive on dovecot's configuration. Regards, Hugo Monteiro. -- ci.fct.unl.pt:~# cat .signature Hugo Monteiro Email : hugo.monteiro(a)fct.unl.pt Telefone : +351 212948300 Ext.15307 Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt apoio(a)fct.unl.pt ci.fct.unl.pt:~# _

1 0

[Dovecot] Using Dovecot as Asterisk PBX voicemail server
by Kenneth Porter 14 Nov '07

14 Nov '07

I'm reading the Asterisk book, 2nd edition, and it describes how one can set up voicemail to be delivered by IMAP to a voicemail folder. Asterisk can monitor flags on the folder so that the "message available" light on one's phone tracks the state of the "read" flag in the folder. One can either dial in for one's voicemail or listen to it from one's favorite IMAP client. (Dovecot is mentioned in the book.) I'm wondering how one would set up auth for this? Asterisk needs to log in to the account to deliver the voicemail and to check its state, but I wouldn't want to put everyone's mail password in plaintext in Asterisk's voicemail.conf file. Would I set up a separate IMAP account for each user to hold their voicemail, and use simple passwords (such as the PIN they use for phone access to voicemail) in the Asterisk config? Or is there a way to set up Dovecot to allow a second user to login to a specific IMAP folder in all accounts, in the same way the local delivery agent has write access to all users' mailbox? Google's page for the book: <http://books.google.com/books?id=vtQxJ3oSm64C&dq=asterisk+book> Dovecot mentions: <http://books.google.com/books?id=vtQxJ3oSm64C&dq=asterisk+book&q=dovecot#se…>

3 3

[Dovecot] namespace public folders
by Nikolay Shopik 13 Nov '07

13 Nov '07

Hi list, Here is thing im trying to accomplish. I would like to setup public folders which resposible for delivering mails for some speical addresses like info(a)example.com. And it will be perfect if "seen" status for this folder will be saved per user. I mean if one user read message and set it as read other user should see it still as unread. As I far i know there some limitation in 1.0 tree but not in lates 1.1. My question is can I do such setup with dovecot+postfix. Im really appreciate for simple examples how can i accomplish this kind setup if it possible :)

3 3

[Dovecot] Caching?
by Bjørn T Johansen 13 Nov '07

13 Nov '07

Any way to minimize the number of sql statements that dovecot do? (and I tried adding userdb prefetch { } but number of sql calls are still the same) Am I doing something wrong or? Regards, BTJ -- ----------------------------------------------------------------------------------------------- Bjørn T Johansen btj(a)havleik.no ----------------------------------------------------------------------------------------------- Someone wrote: "I understand that if you play a Windows CD backwards you hear strange Satanic messages" To which someone replied: "It's even worse than that; play it forwards and it installs Windows" -----------------------------------------------------------------------------------------------

2 2

[Dovecot] v1.1.beta8 released
by Timo Sirainen 13 Nov '07

13 Nov '07

http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta8.tar.gz http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta8.tar.gz.sig Changes since beta7: - Added a new "raw" mail storage backend which allows opening files/streams as single mail mailboxes. deliver uses this now instead of opening the incoming mail as a mbox. So deliver should be now faster and it doesn't anymore remove Dovecot's private mbox headers if you're not using mbox. - Cache file tracks message expunges again - All kinds of other fixes - Some optimizations I think we're getting closer to v1.1 RCs.

9 17

[Dovecot] Dovecot does not fetch mails from Maildir
by Aras Koktas 13 Nov '07

13 Nov '07

Hello I have the following configuration. I am trying to login through pop3 and fetch e-mails but it does not fetch anything. I see that there is a mail there. Here is what i have List from my homedir. Set correct on MySQL as (home) -- [root@axis info]# ls -al new total 12 drwx------ 2 exim exim 4096 Nov 12 21:11 . drwxrwx--- 6 exim exim 4096 Nov 12 21:11 .. -rw------- 1 exim exim 1996 Nov 12 21:11 1194919868.H288251P7967.axis.phidot.com -- -- FROM sql File-- password_query = SELECT userid as user, password FROM mailacct WHERE userid = '%u' user_query = SELECT home, uid, gid FROM mailacct WHERE userid = '%' -- DOVECOT CONFIG -- # 1.0.7: /usr/local/etc/dovecot.conf base_dir: /var/run/dovecot/ protocols: imap imaps pop3 pop3s ssl_cert_file: /usr/share/ssl/certs/dovecot.pem ssl_key_file: /usr/share/ssl/private/dovecot.pem disable_plaintext_auth: no login_dir: /var/run/dovecot//login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login first_valid_uid: 93 mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: mechanisms: PLAIN passdb: driver: sql args: /etc/dovecot-sql.conf userdb: driver: sql args: /etc/dovecot-sql.conf

2 4

[Dovecot] Missing deliver
by Sean Smitz 13 Nov '07

13 Nov '07

I've installed Red Hat Enterprise Linux 5 on a new server. I'm configuring sendmail and dovecot; however, /usr/local/libexec/dovecot was not created when I installed the dovecot rpm (rpm -Uvh dovecot-1.0-1.2.rc15.el5.x86_64.rpm). The problem is I can't find the deliver executable which is supposed to be in this directory. Any help would be appreciated.

2 1

[Dovecot] login_process_size too small on x86_64 Fedora/RHEL
by Tomas Janousek 12 Nov '07

12 Nov '07

Hello, as described in [253363], login_process_size of 32M is not enough on 64-bit versions of Fedora and RHEL (and possibly other distributions as well). [253363] https://bugzilla.redhat.com/show_bug.cgi?id=253363 As I understand it, there's an intersegment gap between read-only/executable and writable segments of shared libs. That's probably a security feature or something. I found a similar issue here[1], if anyone wants a comment from someone who understands it. [1] http://gcc.gnu.org/ml/libstdc++/2006-12/msg00033.html Shall we just increase login_process_size or make it arch-dependent or something? Regards, -- Tomas Janousek, SW Engineer, Red Hat, Inc.

2 2

[Dovecot] Secure authentication?
by Bjørn T Johansen 12 Nov '07

12 Nov '07

I have enabled SSL support for my dovecot installation but if I enable secure authentication in my MUA, I get an error from dovecot telling me that this is not supported.. Is this because dovecot does not support this or am I missing some config? Regards, BTJ -- ----------------------------------------------------------------------------------------------- Bjørn T Johansen btj(a)havleik.no ----------------------------------------------------------------------------------------------- Someone wrote: "I understand that if you play a Windows CD backwards you hear strange Satanic messages" To which someone replied: "It's even worse than that; play it forwards and it installs Windows" -----------------------------------------------------------------------------------------------

7 9