November 2007 - dovecot

[Dovecot] Squat full text search indexer redesign
by Timo Sirainen 26 Nov '07

26 Nov '07

I was writing this to beta9 announcement, but it grew so huge that I guess a separate mail is better. :) I've spent quite a lot of time this last week redesigning/rewriting Squat indexer. I think the redesign is good enough now that I'll just replace the existing squat with this new one in next release. I don't think many people are using the old squat, and there are also some bugs in it that cause some searches not to find anything (haven't tried to fix it). The old Squat indexes about 3MB/s on my machine while the new one indexes 8MB/s. Or it's actually 10MB/s, but at the end of indexing it does a compression step that merges all fragmented UID lists. That drops the index size something like 5-10%, but it's pretty slow because it has to go through all the fragmented lists. It could probably be optimized to be a lot faster, but I think I'll forget about it for now. Besides being faster there are other improvements in it: - Old squat could answer only 4 character long substring searches. New squat can answer 1..n characters long substring searches, where n can be configurable. The larger the n, the larger the index size. For example 4..10 index sizes are about 35% -> 46% -> 53% -> 59% -> 63% -> 67% -> 70% of the mailbox size (with 32MB of Dovecot mailing list archives). With both old and new squat if the search keyword is longer than n, Dovecot looks up the list of all possible n char long combinations of the word, gets a list of their common UIDs and then reads those mails to verify that the word is really found from them. - New squat can also be configured to index longer words for non-substring searches. This allows it to give a definite list of UIDs where the word is found, so Dovecot doesn't have to open those mails to verify it. However it can't give a definite list of UIDs where the substring isn't found. So it still has to do the n-char-combination lookup described above, and for those returned UIDs which aren't also in the definite-UIDs list Dovecot still has to read them to check if the substring is found or not. This could be skipped for non-standard X-FAST-TEXT and X-FAST-BODY searches which don't try to search substrings. Adding 255 char long non-substring searches to 4 char long substring searches grows the index about 7% (35%->42% for 32MB, 28%->36% for 200MB, 27%->33% for 1,4GB). Unfortunately it also grows memory usage pretty much. For a 200MB mailbox heap usage grows from 34MB to 163MB. For 1,4GB lkml mailbox heap usage grows from 80MB to 700MB. So I'm not sure if this should be enabled by default until something can be done about the memory usage. - The current list of indexed characters are: A-Z, 0-9, @.-+#$%_& and all 8bit chars. When trying to search for non-indexed characters Dovecot has to split the search word. For example "a,b" would search "a" and "b", combine the results and then open those mails to see where it really exists. The same thing happens with "a b". If the search word doesn't contain any indexable characters, Dovecot has to read all mails. This indexable character list could be changed runtime, so if a user keeps searching for some non-indexed character, it could be added to indexed characters and the index be rebuilt. - To be able to give definite replies for BODY searches squat has to keep track of whether a word was found from body or header. This also optimizes HEADER searches because then Dovecot has to search only those mails that have the word in their headers, instead of the mails that have it in either header or body. Unfortunately this makes the indexes somewhat larger again (22%->28% for 200MB), but I think this should be done by default. The header+body is stored by storing UID 1=message 1 header, UID 2=message 1 body, UID 3=message 2 header, etc. I haven't yet tried how well the UID lists could be compressed if could support only-header or only-body ranges. For example now if word is found from all headers but from no bodies, it stores the UID list as 1,3,5,7,etc, while it could be stored as simple "only bodies, 1:7". I guess the UID list compression could use all kinds of other algorithms as well. Currently it supports only the whole list being either a bitmask or UID range. A long UID list might compress better as a bitmask +range combination. UID lists take most of the Squat space, so even small improvements can give huge benefits. If you're interested in trying, you can download the current test code and modify uidlist_write_array() for writing and node_uidlist_get_at_offset() for reading. http://dovecot.org/tmp/test.c http://dovecot.org/tmp/test-uidlist.c Place to src/plugins/fts-squat/, run with ./test some-mbox-file Another somewhat different idea to substring indexing (suggested to me by someone else) would be to index the messages' words normally like most indexers do, and then use Squat indexes for those words. So for example when searching for "ord" it first looks up all words that contain "ord" substring in them. Then it looks up message UIDs for those words, and the resulting UID list is a definite list of messages that has the substring. I'm not sure how well the above would work in practise. If the list of words would be mostly static it would work great. Unfortunately every mail has at least two unique strings: Message-ID and Received header's ESMTP ID. Dovecot must be able to search those as well. Also with some languages, like Finnish, a single word can be written in tens of different ways and combined with other words. A quick preliminary testing for 32MB Dovecot mailing list shows: - 142265 unique words takes 2,0MB, one per line - indexing those words creates 6,7MB Squat indexes So not including extra overhead for the unique word index, these use a total of 8,8MB. 4-char-substring Squat indexes for the same mailbox take 8,5MB and 4-char-substring+255-char-full-word Squat indexes take 11MB. So this doesn't look very promising.

1 0

[Dovecot] Configuration Error
by Mike McGonagle 26 Nov '07

26 Nov '07

Hello, I am a newbie when it comes to Dovecot, and in following this ( http://workaround.org/articles/ispmail-etch/ ) document in setting these things up, I have run into an error that I am not quite certain what to do... I am running Ubuntu Gutsy Gibbon, and using 1.0.5 for Dovecot. The message I am getting when I try to 'restart' Dovecot (using '/etc/init.d/dovecot restart') is: Error: Error in configuration file /etc/dovecot/dovecot.conf line 505: Protocol section not allowed here (section changed at line 505) Fatal: Invalid configuration in /etc/dovecot/dovecot.conf ...fail! Does this mean that I need to move this section of the configuration file to a different spot in the file? All I did was uncomment these things, and make the changes described in the link above, I would think that because this is just an edited version of the configuration file, that something like this shouldn't happen. Any help would be appreciated. Thank you, Mike -- Peace may sound simple—one beautiful word— but it requires everything we have, every quality, every strength, every dream, every high ideal. —Yehudi Menuhin (1916–1999), musician

1 1

[Dovecot] Migrating from 0.99 to 1.0 - questions before upgrading
by Erick Perez 25 Nov '07

25 Nov '07

I order to investigate a locking problem, people in this forum suggested to upgrade from 0.99 to 1.0. Which is obviously a good response. However I have some special setup in dovecot.conf and I want to know if I will break something if I upgrade. So, here is my dovecot.conf 0.99 Note: Here are the only changes I made, the rest of the parameters were not touched and are #commented protocols = imap pop3 imaps pop3s imap_listen = * pop3_listen = * imaps_listen = * pop3s_listen = * ssl_disable = no log_path = /var/log/dovecot.log login_dir = /var/run/dovecot-login ## ## IMAP login process ## login = imap login_process_per_connection = no ## ## POP3 login process ## login = pop3 mail_extra_groups = mail postfix 1000 default_mail_env = mbox:/var/mail/virtualmail/%d/%n client_workarounds = oe6-fetch-no-newmail outlook-idle outlook-pop3-no-nuls mbox_locks = fcntl # Authentication process name. auth = default auth_mechanisms = plain auth_userdb = static uid=1000 gid=1000 home=/var/mail/virtualmail/%d/%u auth_passdb = passwd-file /etc/postfix/dovecotpasswd auth_user = root auth_verbose = yes Thanks in advance for your help Erick Perez

2 1

Re: [Dovecot] Problems reading mbox mail boxes.
by James 24 Nov '07

24 Nov '07

Weird, The problem seems to have gone away, I can read all mail boxes now James

2 2

[Dovecot] dspam integration
by Eugene Prokopiev 24 Nov '07

24 Nov '07

Hi, I see http://johannes.sipsolutions.net/Projects/dovecot-antispam and http://blog.cynapses.org/2007/09/13/dovecot-dspam-plugin/ solutions. Second git repo is not answer, first one is cloned, but I see some drawbacks with it. Dspam backend is more native solution but libdspam using instead of dspam external binary looks like preffered way to integrate dspam. Another problem is waiting for dspam child process exit, so IMAP session is blocked. Why not to fork and forget dspam child process? Email sender backend is non-blocking, but using MTA is great overhead. Calling external program with some args and mail body via pipe can be more suitable way, so external program may be dspam binary directly or shell/perl/python filter to any antispam engine via sendmail or something else. Has anybody plans to implement this features or it will be better to try it myself?

5 8

[Dovecot] Weirdness and Questions...
by James 24 Nov '07

24 Nov '07

version 1.0.7 OK, so here is my weirdness. I sent mail to the server from the command line. mail username@localhost < /etc/passwd When I open /var/spool/mail/username I can see 2 messages. The "system" message at the top and the the message produced from my command line email. If I HUP the process that runs xinet.d and re-open the users mail spool the 2nd email disappears which I wouldn't expect. Has anyone seen this before. My SMTP server could be EXIM or Sendmail. My co- admin is in the process from switching over from to EXIM. I am not sure if the SMTP server has anything to do with it or not. I can see the contents of all the email in ~/mail which is rsynced from my old server so that seems to be working properly. I have mail_locatiion in my config file, do I need anything else? mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u In my list of mailboxes that are in ~/mail I see 2 folders that I'd rather not see. There .customeflags and .INBOX Anyway to hide these, as they are not needed in my opinion. Thanks, James

1 0

[Dovecot] Problems reading mbox mail boxes.
by James 24 Nov '07

24 Nov '07

I just installed dovecot 1.0.7 Set up the pem files for SSL, set up the xinet.d/imaps files and it logs into the server gets a list of all the mbox files in the ~/mail directory and creates representations of them in Apple mail. The problem I am having is when I click on one any of them I see no content. Execpt one. For some reason one of them shows the 2 emails that are inside of it. I am very confused by it. Help? James

1 0

[Dovecot] deliver and append brocken in hg head ("tip"?)
by Jakob Hirsch 23 Nov '07

23 Nov '07

Hi, > Nov 22 18:48:15 seba deliver(jh): file maildir-uidlist.c: line 844 (maildir_uidlist_set_ext): assertion failed: (rec != NULL) > Nov 22 18:48:15 seba deliver(jh): Raw backtrace: /usr/local/libexec/dovecot/deliver(i_syslog_fatal_handler+0x2b) [0x80be37b] -> /usr/local/libexec/dovecot/deliver [0x80be18c] -> /usr/local/libexec/dovecot/deliver [0x8066e56] -> /usr/local/libexec/dovecot/deliver [0x8069f00] -> /usr/local/libexec/dovecot/deliver [0x806a041] -> /usr/local/libexec/dovecot/deliver(maildir_save_finish+0xec) [0x806a77c] -> /usr/local/libexec/dovecot/deliver(mail_storage_copy+0xfa) [0x80ad59a] -> /usr/local/libexec/dovecot/deliver(maildir_copy+0x52) [0x8067f42] -> /usr/local/libexec/dovecot/deliver(deliver_save+0xcf) [0x805945f] -> /usr/local/libexec/dovecot/deliver(main+0x12e5) [0x805a955] -> /lib/libc.so.6(__libc_start_main+0xe0) [0x948f70] -> /usr/local/libexec/dovecot/deliver [0x8058771]

2 1

Re: [Dovecot] dovecot loading during boot
by Karsten Bräckelmann 22 Nov '07

22 Nov '07

[ adding the list back to Cc ] On Thu, 2007-11-22 at 14:28 +0100, Marcus Rueckert wrote: > On 2007-11-22 13:31:59 +0100, Karsten Bräckelmann wrote: > > And impossible for SuSE out-of-the-box, given their > > braindead [1] init scripts. > > what is so braindead about it? See these posts, the second one in particular. Also, my original Shorewall rules and documentation might be interesting. http://www.mail-archive.com/shorewall-users@lists.sourceforge.net/msg03986.… http://www.mail-archive.com/shorewall-users@lists.sourceforge.net/msg03985.… Please note that the initial reason for the above pinning down NFS ports is firewall-friendly behavior and sane rules. With NFS, most involved services use random ports by default, particularly statd, lockd, mountd, rquotad. Which leads to somewhat unsatisfying rules as shown in [1]. The init script shipped by SuSE offers no way whatsoever to pass rpc.statd options, even though it does for rpc.mountd -- and thus no way to pin down the port out-of-the-box short of hacking the init script. Marcus, please feel free to keep me posted on this issue and a fix. I'll happily forward updates to the Shorewall lists. guenther [1] http://shorewall.net/ports.htm#NFS -- char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4"; main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1: (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

2 2

[Dovecot] Broken system.
by Marcelo Iturbe 22 Nov '07

22 Nov '07

Hello, I had an Inbox with 975,604 emails on ext3 filesystem. I keept getting dropped on the comand "copy 1:2000 inbox2", the syslog gave me an error message indicating "out of memory" problems. This despite the fact that the server has 800MB of ram free. Anyways, I asked if I could manuanlly move the files at the filesystem level or if that would mess up dovecot (break the GUIDs etc). I was informed that dovecot does not use GUIDs and that I could go ahead and move the emails at the shell level. So I did that, now the /cur folder only has 40,000 emails, but when I connect to the account via telnet, I still get the following: telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK Dovecot ready. 110 login somelameuser somelamepass 110 OK Logged in. 111 select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 975604 EXISTS * 0 RECENT * OK [UNSEEN 1] First unseen. * OK [UIDVALIDITY 1188426666] UIDs valid * OK [UIDNEXT 979500] Predicted next UID 111 OK [READ-WRITE] Select completed. So even though there are only 40,000 emails in /cur, dovecot still thinks that there are 9756304 emails. How can I fix this? Thanks Marcelo

4 4