May 2007 - dovecot - dovecot.org

[Dovecot] New full text search indexer
by Timo Sirainen 14 Nov '07

14 Nov '07

As described earlier (http://dovecot.org/list/dovecot/2006-December/018055.html) Dovecot nowadays has full text search indexing support in CVS HEAD. Currently there are two backends: Lucene and Squat. Lucene's problem is that standard IMAP SEARCH command can't be used with it without breaking IMAP RFC. So Lucene can be used only with non-standard X-BODY-FAST and X-TEXT-FAST search commands. Squat's code then is quite complex and it's pretty slow. It works with 4 character blocks, so it can answer: - Search string < 4 characters: Can't answer anything - Search string = 4 characters: Definite yes/no - Search string > 4 characters: Probable yes / definite no Since search strings are usually more than 4 characters, Dovecot uses the index only to filter out messages which definitely won't match and then does regular searching for the non-filtered messages. It works ok, but it could work better. The Squat indexes take maybe 30-50% of the mailbox's size. Most of the space goes to UID lists (list of messages matching each 4 character block). For example one UID list could be "2,4-10,20-30". This is already stored in a compressed format by storing the UID numbers relative to previous UID, which allows storing them usually with a single byte per UID, even if the UIDs are large. So, this morning I had an idea how this could maybe be improved with two changes: 1. Don't use 4 character blocks. Instead allow variable sized paths, and keep track of the UIDs for each node in the path. So for example adding "abc" to UID 1 will place the UID 1 to node "a", "ab" and "abc". 2. Use UIDs relative to parent's UIDs. For example if node "a" has UIDs 1,3,5 and "ab" is added for UID 5, the UID is changed to 2 (3th in 1,3,5 list - 1). This can help the compression a lot. The best case scenario for the above example is if "ab" contains also 1,3,5 UIDs. Then it can be written with a single 0-2 range. With these changes, the indexer can be configured to perform in different ways. Text can be indexed in two ways: full words, and substrings within those words (as required by IMAP). I think a useful combination of these would be to index all 4 character substrings and all full words up to 10 characters or so. This kind of an index can answer: - Search string <= 4 characters: Definite yes/no - For X-BODY-FAST / X-TEXT-FAST: - Search string 5-10 characters: Definite yes/no - Search string 11+ characters: Probable yes / definite no - For standard IMAP SEARCH: - Search string 5-10 characters: Definite yes if message contains the key as non-substring, otherwise same as below. This helps only if user does a search that matches a lot of messages. - Search string 11+ characters: Probable yes / definite no UTF-8 text is indexed as bytes, but the path depth can be counted as characters instead of as bytes. I've a test program finished, and unless there are some bugs left I think it shows that this can work pretty well: Squat-like 4 byte substrings (but can answer 1-3 char queries also): Message count: 7495 Index time: 4.19 CPU seconds (7.39MB/CPUs) Node memory: 3014072 bytes (2943 kB) in 90550 nodes UID memory: 8277212 bytes (8083 kB) in 352778 lists Total: 11291284 / 32474899 bytes = 34.77% Indexing the same mailbox file with squat-test gives: - Index time: 9.55 CPU seconds, 9.63 real seconds (3.24MB/CPUs) - 4145192 bytes in 102208 nodes (12.76%) - 7286058 bytes in 305501 uid_lists (22.44%) - 11431250 bytes total of 32474899 (35.20%) So the new indexer is over twice as fast and uses slightly less space. Indexing 4 char substrings and words up to 10 chars: Index time: 5.13 CPU seconds (6.04MB/CPUs) Node memory: 6002804 bytes (5862 kB) in 615501 nodes UID memory: 9044610 bytes (8832 kB) in 519488 lists Total: 15047414 / 32474899 bytes = 46.34% So it takes somewhat more space, but definitely less than having both Squat + Lucene. No substring indexing, words up to 10 chars (Lucene replacement): Index time: 1.39 CPU seconds (22.28MB/CPUs) Node memory: 3297663 bytes (3220 kB) in 548540 nodes UID memory: 1803735 bytes (1761 kB) in 213554 lists Total: 5101398 / 32474899 bytes = 15.71% Is there a need for Lucene anymore with this kind of a speed and disk space usage? :) The above tests are run with everything being indexed except space and control chars (ie. practically space + tab). If only alphanumeric characters are indexed, the total lines are (in same order as above): Total: 7048664 / 32474899 bytes = 21.70% (-13.07%) Total: 9610543 / 32474899 bytes = 29.59% (-16.75%) Total: 3963335 / 32474899 bytes = 12.20% (-3.51%) There are probably still a few optimizations that can be done to get the disk space usage even lower. I did spent quite a lot of time making it use less CPU though. Even tried a bit different coding style variations in the critical functions to get gcc to create faster code.. One interesting thing I noticed was that binary searching a character from a char array was slower than just sequentially going through the characters. I tried adding checks to do it only if the array was big, but it was always slower. Maybe something to do with CPU cache. If you want to try yourself, the test program is in http://dovecot.org/tmp/new-indexer.c

7 10

[Dovecot] Possible Caching Bug showing up as a MIME Boundary Issue
by Kevin A. McGrail 27 Oct '07

27 Oct '07

Possible Caching Bug showing up as a MIME Boundary Issue I'm using Dovecot version 1.0.0. I was using Dovecot version 1.0.0 beta3 or alpha4. I upgraded to Dovecot 1.0.0 to make sure that was not the issue. Over the past few weeks on a server running a stable dovecot, I have seem a few emails arriving where the MIME document structure dividers are visible. I've included a screenshot showing this. Using OE6 as client. Have tried adding oe6-fetch-no-newmail per clients wiki but this seems to stop the ability to login at all. I am using two different machines with Outlook Express to access the same email. The same problem will show up on both. Usually, if I delete the .imap cache for the folder, the email comes down correctly. My dovecot -n output is: # /etc/dovecot.conf listen: *:144 ssl_listen: *:994 ssl_cert_file: /usr/local/ssl/certs/stunnel.pem ssl_key_file: /usr/local/ssl/certs/stunnel.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login max_mail_processes: 50 verbose_proctitle: yes default_mail_env: mbox:~/:INBOX=/var/spool/mail/%u mail_location: mbox:~/:INBOX=/var/spool/mail/%u imap_client_workarounds: outlook-idle delay-newmail auth default: passdb: driver: pam userdb: driver: passwd userdb: driver: prefetch The server is on a heavily modified RedHat 7.2 on x86 box with ext3 Thanks in Advance, KAM

2 3

[Dovecot] BUG REPORT - Stops logging after sleep
by Bruce Bodger 09 Sep '07

09 Sep '07

Version: dovecot --version: 1.0.0 OS: OS X - 10.4.9 Architecture: G5 Dual/1.8Ghz Had a fairly normal time correction last night: May 9 20:56:56 G518X2 ntpd[219]: time set -1.212733 s Which dovecot duly noted: May 9 20:56:56 G518X2 dovecot: auth(default): Time just moved backwards by 1 seconds. I'll sleep now until we're back in present. This morning all of the dovecot processes are running fine but dovecot has stopped logging. (I have modified nothing in dovecot.conf relative to logging: dovecot logs to syslog using the "mail" syslog_facility.) ps -aux | grep dovecot root 53 0.0 -0.0 27772 544 ?? Ss 4:23PM 0:04.69 /usr/local/sbin/dovecot -F root 79 0.0 -0.0 28176 556 ?? S 4:23PM 0:03.21 dovecot-auth dovecot 9548 0.0 -0.0 27780 496 ?? S 7:12AM 0:00.05 imap-login dovecot 9631 0.0 -0.0 27780 500 ?? S 7:17AM 0:00.03 imap-login dovecot 9632 0.0 -0.0 27780 500 ?? S 7:17AM 0:00.03 imap-login I am starting dovecot via OS X's launchdaemon facility thus the "-F" command line switch requirement. Sending a "kill -HUP" to the process did not get it to start logging. Sending a "Kill -9" to the process forced a restart and logging began again. ps -aux | grep dovecot root 9738 0.0 -0.0 27772 540 ?? Ss 7:26AM 0:00.03 /usr/local/sbin/dovecot -F root 9739 0.0 -0.0 28176 544 ?? S 7:26AM 0:00.02 dovecot-auth dovecot 9740 0.0 -0.0 27780 500 ?? S 7:26AM 0:00.02 imap-login dovecot 9751 0.0 -0.0 27780 500 ?? S 7:27AM 0:00.01 imap-login dovecot 9762 0.0 -0.0 27780 500 ?? S 7:27AM 0:00.01 imap-login Before posting this bug I compared v1.0.0's /src/lib/ioloop.c to that in the nightly snapshot and saw no adjustments in, what I presumed to be, the relevant portion of the code. Output of "dovecot -n" protocols: imap ssl_disable: yes disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login first_valid_uid: 490 mail_extra_groups: mail mail_location: mbox:~/Library/Mail/IMAP:INBOX=/var/mail/%u:INDEX=%h/ Library/Mail/DOVECOT.indexes dotlock_use_excl: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd Thank you.

4 14

[Dovecot] UW-IMAP style mbox snarfing from /var/mail/ to ~/mbox
by Timo Sirainen 09 Aug '07

09 Aug '07

http://dovecot.org/patches/mbox-snarf-plugin.c Also committed to CVS HEAD. Usage: mail_location = mbox:~/mail:INBOX=/var/mail/%u protocol imap { mail_plugins = mbox_snarf } protocol pop3 { mail_plugins = mbox_snarf } plugin { mbox_snarf = ~/mbox } If ~/mbox doesn't exist, nothing special is done. If ~/mbox exists, it's treated as INBOX and mails are moved from /var/mail/%u whenever the mailbox is synchronized. If all the messages couldn't be copied, for example because of out of quota, then none of the messages are copied. I suppose that could be a problem. It might be possible to fix that by changing: if (mailbox_copy(dest_trans, mail, 0, NULL, NULL) < 0) { if (!mail->expunged) { ret = -1; break; } } to just: if (mailbox_copy(dest_trans, mail, 0, NULL, NULL) < 0) { if (!mail->expunged) break; } But I'm not sure about that. Try and tell me. :)

2 18

[Dovecot] [PATCH] Split sql drivers from lib-sql to plugins
by Tomas Janousek 07 Aug '07

07 Aug '07

Hi, in order to solve rhbz#170960 [1], I split the sql drivers to plugins so that they are loaded dynamically and can be split to separate packages. [1] https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=170960 The patches are here: [2] http://people.redhat.com/tjanouse/dovecot/145241/dovecot-1.0-split.patch [3] http://people.redhat.com/tjanouse/dovecot/145241/dovecot-trunk-split.patch (mv src/lib-sql/driver-mysql.c src/plugins/sql-mysql/, etc., afterwards) I'd like to ask for your comments and whether it's ok for inclusion or at least ok to use in the Fedora package. Regards, -- Tomas Janousek, SW Engineer, Red Hat, Inc.

2 6

[Dovecot] mbox bug in 1.0.0
by bhayden＠umn.edu 01 Aug '07

01 Aug '07

Hi folks. We're moving users from rc24 and 27 to 1.0.0. On four mailboxes so far we've seen these errors after the move (quick and dirty anonymizing follows): Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.info] imap-login: Login: user=<USER>, pid=<9046>, method=PLAIN, rip=***.***.***.***, lip=***.***.***.***, TLS Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.error] [9073] IMAP(USER): mbox /var/mail/user: seq=1 uid=8808 uid_broken=0 originally needed 10 bytes, now needs 23 bytes Apr 23 13:22:41 server.umn.edu dovecot: [ID 107833 local6.error] [9073] IMAP(USER): Raw backtrace: 0x7b800 -> 0x43434 -> 0x3e018 -> 0x3fc34 -> 0x40080 -> 0x6d3cc -> 0x29a88 -> 0x21324 -> 0x214c0 -> 0x22f48 -> 0x22ec0 -> 0x230c0 -> 0x81c18 -> 0x814e4 -> 0x2b5cc -> 0x1dc58 Points of interest: 1. I can't reproduce this. It still happens after the user's .imap directories are removed. If I make a copy of the mailbox on another machine running the same version of Dovecot with the same config, the problem doesn't occur. Weird. 2. So far, have only seen it on quite large mailboxes (close to or over 100MB). Most large boxes have not had the problem, but the ones that have are all large. Any thoughts? Dovecot -n output below. -Brian Hayden OIT Internet Services U of Minnesota # dovecot -n # /etc/opt/dovecot/dovecot.conf syslog_facility: local6 protocols: imap imaps pop3 pop3s listen(default): *:143 listen(imap): *:143 listen(pop3): *:110 ssl_listen(default): *:993 ssl_listen(imap): *:993 ssl_listen(pop3): *:995 ssl_cert_file: /etc/opt/openssl/certs/dovecot.pem ssl_key_file: /etc/opt/openssl/private/dovecot.pem shutdown_clients: no login_dir: /var/opt/dovecot/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_log_format_elements: user=<%u> pid=<%p> method=%m rip=%r lip=%l %c login_processes_count: 32 login_max_processes_count: 4096 max_mail_processes: 8192 first_valid_uid: 100 default_mail_env: mbox:~/:INBOX=/var/mail/%u mail_location: mbox:~/:INBOX=/var/mail/%u mail_full_filesystem_access: yes mbox_lazy_writes: no mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 mail_log_prefix: [%p] %Us(%u): imap_client_workarounds(default): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(imap): delay-newmail outlook-idle tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv namespace: type: private separator: / location: mbox:~/:INBOX=/var/mail/%u inbox: yes hidden: yes auth default: username_format: %Lu verbose: yes debug: yes passdb: driver: pam args: session=yes * userdb: driver: passwd

2 4

[Dovecot] quota warning - take 2
by Nicolas Boullis 16 Jul '07

16 Jul '07

Hi, About one month ago, I sent a patch to this list to implement warnings when a user is close to reach his quota. I tried to follow Timo's advice to improve my code, and here (at last) is a new patch. Things I did not fix: - This patch is still for dovecot 1.0, as I will need warnings with dovecot 1.0. I may rewrite it for CVS HEAD in the future but can't promise. - I'm still using system() rather than fork()+exec()+waitpid() since I don't know a good time to run waitpid(). Comments are welcome. Timo, will you consider to add this to your unofficial patches? Cheers, Nicolas Boullis

3 5

[Dovecot] PHP implementation of dovecotpw passwords
by Frank Church 09 Jul '07

09 Jul '07

I am looking for PHP functions that implement passwords that much the dovecotpw implementation. I downloaded one from PEAR, Crypt_HMAC, but the passwords it generates look nothing like the dovecotpw passwords, which could be my fault because I know nothing about the field. Is there a guide or some library that implements them to match dovecotpw?

4 4

[Dovecot] Semi-static userdb...?
by John Robinson 03 Jul '07

03 Jul '07

I'm trying to add "virtual" mailboxes to a system. Real users with different uids own domains. Each domain has a passwd-file passdb. I don't want to use this passwd-file for the userdb, because I want to fix the home, mail and uid/gid settings. Can I use the static userdb in a less static manner, e.g. userdb static { args = uid=%{owner of /vmail/%d} gid=%{gid of /vmail/%d} home=/vmail/%d/users/%u mail=mbox:/vmail/%d/users/%u/mail:INBOX=/vmail/%d/users/%u/INBOX } Or is it just a bad idea to let relatively untrusted domain owners manage their own passwd-file at all...? Cheers, John.

2 4

[Dovecot] 1.0.rc28 released
by Timo Sirainen 27 Jun '07

27 Jun '07

http://dovecot.org/releases/dovecot-1.0.rc28.tar.gz http://dovecot.org/releases/dovecot-1.0.rc28.tar.gz.sig Still a bit more fixes. My coding TODO list is again empty. Unless something special happens in the next few weeks, I'll still make rc29 with the documentation included and v1.0 will be released April 13. * deliver + userdb static: Verify the user's existence from passdb, unless allow_all_users=yes * dovecot --exec-mail: Log to configured log files instead of stderr * Added "-example" part to doc/dovecot-sql-example.conf and doc/dovecot-ldap-example.conf. They are now also installed to $sysconfdir with "make install". + When copying/syncing a lot of mails, send "* OK Hang in there" replies to client every 15 seconds so it doesn't just timeout the connection. + Added idxview and logview utilities to examine Dovecot's index files + passdb passwd and shadow support blocking=yes setting now also + mbox: If mbox file changes unexpectedly while we're writing to it, log an error. + deliver: Ignore -m "" parameter to make calling it easier. + deliver: Added new -n parameter to disable autocreating mailboxes. It affects both -m parameter and Sieve plugin's fileinto action - mbox: Using ~/ in the mail root directory caused a ~ directory to be created (instead of expanding it to home directory) - auth cache: If unknown user was found from cache, we didn't properly return "unknown user" status, which could have caused problems in deliver. - mbox: Fixed "UID inserted in the middle of mailbox" in some conditions with broken X-UID headers - Index view syncing fixes - rc27 didn't compile with some non-GCC compilers - vpopmail support didn't compile in rc27 - NFS check with chrooting broke home direcotry for the first login - deliver: If user lookup returned "unknown user", it logged "BUG: Unexpected input" - convert plugin didn't convert INBOX

8 18