September 2007 - dovecot

[Dovecot] alert function, V2
by Stewart Dean 25 Apr '08

25 Apr '08

I guess the alert function is not part of the spec, because it appears to be unimplemented in Dovecot, even though most clients implement it. Timo, I would hope this would be easy to implement, and it's functionality that will be valuable to every installation. Here's why. America recently had the tragedy of a shooting rampage on a large university campus (yes, America should do something about the violence and free access to guns in our culture/society). What was worse was that, due to the lack of an emergency notification process, some 20+ people died needlessly. Now all college and university administration are scrambling to set up emergency notification processes. Our first take on this is a massive all-hands mailing list, but a) best case, it would take 15 minutes (and as much as an hour) to get to everybody's mailbox. b) users would have to read through their mail and stumble on the message What's needed is an "in-your-face" unavoidable alert, and the UWIMAP alert functionality provides this. As I recall, you touch just about anything in your IMAP client and there's a popup message in the middle of the client IMAP window...and it happens repeatedly. We tried it back 3-4 years ago and found it too annoying to use. For this purpose, it's perfect. The wy UWIMAP did it, the presence of the file /etc/imapd.alert triggers it, and the text in that file is transmitted to the client. The message has to be short, a one-liner for a small popup window, but this could be a pointer to a shared system message folder. Of course, DC could sense the activation of the alert in some other fashion that the presence of a file. Ideally, whatever activates the alert function should be able to be turned on remotely; I had in mind to do it using a mail aliases that is a pipe to a executable... What say you? -- ==== Stewart Dean, Unix System Admin, Henderson Computer Resources Center of Bard College, Annandale-on-Hudson, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

6 9

[Dovecot] Allowing tilde at start of mailbox names
by pod 11 Mar '08

11 Mar '08

In lib-storage/index/maildir/maildir-storage.c maildir_is_valid_create_name() and maildir_is_valid_existing_name() the following sequence of tests appear but I don't really understand why if ((storage->flags & MAIL_STORAGE_FLAG_FULL_FS_ACCESS) != 0) return TRUE; if (*name == '~' || strchr(name, '/') != NULL) return FALSE; If MAIL_STORAGE_FLAG_FULL_FS_ACCESS is set then the validator funcs return TRUE immediately and code elsewhere will do appropriate home_exapand for mailbox names. However if MAIL_STORAGE_FLAG_FULL_FS_ACCESS is not set the (*name == '~') test prevents mailbox names that begin with '~'. Why is it not possible to have mailbox names that begin with '~' but without '~' carrying any special meaning? I have been working on a patch (appended but incomplete; the option only works against maildir atm) that adds a new option mail_allow_tilde and my admittedly limited testing seems to indicate that it does the right thing but maybe I am barking up the wrong tree. Am I missing something here? ---------------------------------------- # HG changeset patch # User <pod(a)sysdev.oucs.ox.ac.uk> # Date 1185368923 -3600 # Node ID ab9cf3790ea9b5486afa6498f60c6bf6fb2e5f1e # Parent a23be695672737967a10613d7dca2a6f53b5d96d create new option mail_allow_tilde diff -r a23be6956727 -r ab9cf3790ea9 src/lib-storage/index/maildir/maildir-storage.c --- a/src/lib-storage/index/maildir/maildir-storage.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/lib-storage/index/maildir/maildir-storage.c Wed Jul 25 14:08:43 2007 +0100 @@ -240,7 +240,9 @@ static bool maildir_is_valid_create_name if ((storage->flags & MAIL_STORAGE_FLAG_FULL_FS_ACCESS) != 0) return TRUE; - if (*name == '~' || strchr(name, '/') != NULL) + if ((*name == '~' && + (storage->flags & MAIL_STORAGE_FLAG_ALLOW_TILDE) == 0) || + strchr(name, '/') != NULL) return FALSE; if (name[0] == '.' && (name[1] == '\0' || @@ -264,7 +266,9 @@ static bool maildir_is_valid_existing_na if ((storage->flags & MAIL_STORAGE_FLAG_FULL_FS_ACCESS) != 0) return TRUE; - if (*name == '~' || strchr(name, '/') != NULL) + if ((*name == '~' && + (storage->flags & MAIL_STORAGE_FLAG_ALLOW_TILDE) == 0) || + strchr(name, '/') != NULL) return FALSE; if (name[0] == '.' && (name[1] == '\0' || diff -r a23be6956727 -r ab9cf3790ea9 src/lib-storage/mail-storage.c --- a/src/lib-storage/mail-storage.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/lib-storage/mail-storage.c Wed Jul 25 14:08:43 2007 +0100 @@ -66,6 +66,8 @@ void mail_storage_parse_env(enum mail_st *flags_r = 0; if (getenv("FULL_FILESYSTEM_ACCESS") != NULL) *flags_r |= MAIL_STORAGE_FLAG_FULL_FS_ACCESS; + if (getenv("ALLOW_TILDE") != NULL) + *flags_r |= MAIL_STORAGE_FLAG_ALLOW_TILDE; if (getenv("DEBUG") != NULL) *flags_r |= MAIL_STORAGE_FLAG_DEBUG; if (getenv("MMAP_DISABLE") != NULL) diff -r a23be6956727 -r ab9cf3790ea9 src/lib-storage/mail-storage.h --- a/src/lib-storage/mail-storage.h Tue Jul 24 05:48:03 2007 +0300 +++ b/src/lib-storage/mail-storage.h Wed Jul 25 14:08:43 2007 +0100 @@ -34,7 +34,9 @@ enum mail_storage_flags { fail to create the storage. */ MAIL_STORAGE_FLAG_NO_AUTOCREATE = 0x200, /* Rely on O_EXCL when creating dotlocks */ - MAIL_STORAGE_FLAG_DOTLOCK_USE_EXCL = 0x400 + MAIL_STORAGE_FLAG_DOTLOCK_USE_EXCL = 0x400, + /* Allow folder names to begin with '~' TILDE */ + MAIL_STORAGE_FLAG_ALLOW_TILDE = 0x800 }; enum mail_storage_lock_method { diff -r a23be6956727 -r ab9cf3790ea9 src/master/mail-process.c --- a/src/master/mail-process.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/mail-process.c Wed Jul 25 14:08:43 2007 +0100 @@ -254,6 +254,8 @@ mail_process_set_environment(struct sett env_put("DEBUG=1"); if (set->mail_full_filesystem_access) env_put("FULL_FILESYSTEM_ACCESS=1"); + if (set->mail_allow_tilde) + env_put("ALLOW_TILDE=1"); if (set->pop3_no_flag_updates) env_put("POP3_NO_FLAG_UPDATES=1"); if (set->pop3_reuse_xuidl) diff -r a23be6956727 -r ab9cf3790ea9 src/master/master-settings-defs.c --- a/src/master/master-settings-defs.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/master-settings-defs.c Wed Jul 25 14:08:43 2007 +0100 @@ -67,6 +67,7 @@ static struct setting_def setting_defs[] DEF(SET_INT, mailbox_idle_check_interval), DEF(SET_BOOL, mail_debug), DEF(SET_BOOL, mail_full_filesystem_access), + DEF(SET_BOOL, mail_allow_tilde), DEF(SET_INT, mail_max_keyword_length), DEF(SET_BOOL, mail_save_crlf), DEF(SET_BOOL, mail_read_mmaped), diff -r a23be6956727 -r ab9cf3790ea9 src/master/master-settings.c --- a/src/master/master-settings.c Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/master-settings.c Wed Jul 25 14:08:43 2007 +0100 @@ -216,6 +216,7 @@ struct settings default_settings = { MEMBER(mailbox_idle_check_interval) 30, MEMBER(mail_debug) FALSE, MEMBER(mail_full_filesystem_access) FALSE, + MEMBER(mail_allow_tilde) FALSE, MEMBER(mail_max_keyword_length) 50, MEMBER(mail_save_crlf) FALSE, MEMBER(mail_read_mmaped) FALSE, diff -r a23be6956727 -r ab9cf3790ea9 src/master/master-settings.h --- a/src/master/master-settings.h Tue Jul 24 05:48:03 2007 +0300 +++ b/src/master/master-settings.h Wed Jul 25 14:08:43 2007 +0100 @@ -75,6 +75,7 @@ struct settings { unsigned int mailbox_idle_check_interval; bool mail_debug; bool mail_full_filesystem_access; + bool mail_allow_tilde; unsigned int mail_max_keyword_length; bool mail_save_crlf; bool mail_read_mmaped;

3 12

[Dovecot] Dovecot, LDAP and something akin to Postfix' "table search order" from virtual deliver.
by Pawel Sawicki 01 Feb '08

01 Feb '08

Hi. /* The short version. */ Is there a way to mimic Postfix' "table search order" in Dovecot's LDAP configuration of pass_filter/user_filter? /* Here goes the detailed description :) */ Just to quote the http://www.postfix.org/virtual.8.html: --- TABLE SEARCH ORDER (...) The search order is as follows. The search stops upon the first successful lookup. (...) o The user(a)domain.tld address, without address exten- sion, is looked up next. o Finally, the recipient @domain is looked up. --- Basicaly this allows to have separate users assigned addresses like j.doe(a)example.com (first account), j.smith(a)example.com (second account) and at the same time to have a "catch all" anything(a)example.com mailbox (for the administrator e.g.). What do I need it for? To have dovecot-lda working. Why? I want Sieve :) The test data may look as follows: account #1: + mail: j.doe(a)example.com + mailAlternateAddress: j.c.doe(a)example.com account #2: + mail: j.smith(a)example.com + mailAlternateAddress: j.t.smith(a)example.com account #3 (administrator): + mail: bofh(a)example.com + mailAlternateAddress: @example.com (means - anything possible) The filter I used is (more or less - for both pass_filter and user_filter): (|(mail=%u)(mailAlternateAddress=%u)(mailAlternateAddress=@%d)) Just for testing purposes I've stripped off some irrelevant data (like checking if the account is active), since that made the filter about 140 characters long ;) The trouble is when there're both a "catch all" and a true account that are linked to different accounts. As an example, if I try to send mail to "j.doe(a)example.com" I receive: dovecot: auth(default): ldap(j.doe(a)example.com): Multiple replies found for user Which is of course true, since the filter returns "account #2" and "account #3". What I'd like to get? I'd like to set the filter to: (|(mail=%u)(mailAlternateAddress=%u)) Where "%u" is expanded to "j.doe(a)example.com" during the first run Then, if dovecot can't find any user that would match this literaly it should retry the search operation, but now with "@example.com" as the "%u". I've tried to fiddle with the filter itself, but I see any reasonable way of doing it. I don't even know if it is possible at all. Could anyone give me a hint? Or maybe it's already on someone's whish list? Some final info. * I've followed the examples on dovecot.org website: http://wiki.dovecot.org/LDA#head-dacb9b9a1f19c3ea86bb6f8caa3d64e3ddad9ef8 * Here's the dovecot's master.conf entry: dovecot unix - n n - - pipe flags=DRhu user=mail:mail argv=/usr/libexec/dovecot/deliver -d ${recipient} Best regards! Pawel Sawicki

2 2

[Dovecot] Too many open files
by Andrew Hutchings 10 Jan '08

10 Jan '08

Hi Guys, Just got this on one of my test72 dovecot servers: - Jun 10 12:40:28 svr21 dovecot: auth(default): net_connect_unix(/var/run/dovecot/auth-worker.18818) failed: Too many open files Jun 10 12:40:28 svr21 dovecot: pop3-login: Disconnected: method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=xxx.xxx.xxx.xxx Jun 10 12:40:28 svr21 dovecot: pop3-login: Disconnected: method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=xxx.xxx.xxx.xxx Jun 10 12:40:28 svr21 dovecot: child 18818 (auth) returned error 89 Jun 10 12:40:29 svr21 dovecot: Auth process died too early - shutting down Jun 10 12:40:29 svr21 dovecot: auth(default): mysql: Connected to localhost (maildb) Jun 10 12:40:29 svr21 dovecot: auth(default): net_connect_unix(/var/run/dovecot/auth-worker.13784) failed: No such file or directory Jun 10 12:40:29 svr21 dovecot: child 13784 (auth) returned error 89 - Is dovecot leaving files open? Or is it a large mailbox? Regards Andrew -- Andrew Hutchings (A-Wing) Linux Guru - Netserve Consultants Ltd. - http://www.domaincity.co.uk/ Admin - North Wales Linux User Group - http://www.nwlug.org.uk/ BOFH excuse 122: because Bill Gates is a Jehovah's witness and so nothing can work on St. Swithin's day.

5 5

[Dovecot] What would you tell the CIO in an "ABCs of Email" overview?
by Esther Schindler 07 Jan '08

07 Jan '08

One of the key points that came up when I researched and wrote the "Five Things CIOs Should Know about Fighting Spam" article (http:// www.cio.com/article/28830) was that they should know the basics of how email works. Otherwise, said plenty of techies, the CIO won't have the first idea of what the email admin is complaining about. So I'm going to do my part. I'm going to write an "ABCs of Email" article (to accompany the many other ABCs articles we have on CIO.com, at http://www.cio.com/article/40242 ). I'd like your input on the topics that should be included, keeping in mind the fact that the target reader is a CIO, IT manager, or someone who wants to understand the basics, *not* actively get involved in email management. You don't need to write an essay for me or inundate me with links (though hey, if you want to make my life easier I shall not complain). What I'm looking for, primarily, are the categories of information that I should cover. In other words, if your CIO had an email ephiphany and asked you to give a half-hour presentation, what would you include? This won't be an "Expert says..." article nor will it be "Geek on the street says...." I intend to compile and research the "least you need to know" for the not-necessarily-techie bosses out there. And hopefully the end result will be that you have one less dumb question to deal with in your life. So: any suggestions? (You can reply privately if you prefer.) Esther Schindler senior online editor, CIO.com her blog: http://advice.cio.com/taxonomy/term/34

2 3

[Dovecot] about quota_warning behavior
by naoki 21 Dec '07

21 Dec '07

Hi, We've been using v1.0.5 +quota_warning patch, however quota_warning does't seem to be working properly. For example, in order to enable 80% over for quota warning, and we configured like following: ----------------------------------------------------------------------- plugin { quota_warning = storage=80%% /usr/local/path/to/plugin } ----------------------------------------------------------------------- But it does not work. However we set quota_warning value to 20%, it seems to be working. ----------------------------------------------------------------------- plugin { quota_warning =storage=20%% /usr/local/path/to/plugin } ----------------------------------------------------------------------- If this is the case, do we always have to set quota value to what we have after subtracting the required quota warning value from 100. for example, if we want quota warning value to be 70%, we actually have to write 30% in configuration?? Will there be another patch to fix this? Thanks -- Naoki Takazawa Marino Ltd., Yokohama, JAPAN

2 1

[Dovecot] MANAGESIEVE patch v7 for dovecot 1.0.3 and TLS
by Andreas Schneider 09 Dec '07

09 Dec '07

Hi, I allow only TLS/SSL connection to dovecot. The managesieve doesn't work with TLS but it works without it. gladiac@magrathea:~> gnutls-cli -p 2000 -s --insecure -V milliways.cynapses.org Resolving 'milliways.cynapses.org'... Connecting to '213.239.217.124:2000'... - Simple Client Mode: - Received[194]: "IMPLEMENTATION" "dovecot" "SASL" "PLAIN" "SIEVE" "fileinto reject envelope vacation imapflags notify subaddress relational comparator-i;ascii-numeric regex" "STARTTLS" OK "Dovecot ready." STARTTLS - Sent: 9 bytes - Received[33]: OK "Begin TLS negotiation now." Let me know if you need more information. Thanks, -- andreas -- http://www.cynapses.org/ - cybernetic synapses

3 4

[Dovecot] imap process consuming 100% CPU (Dovecot 1.0.3)
by Robert Tomanek 20 Nov '07

20 Nov '07

Hi, I have yet another problem with Dovecot: sometimes (rarely, maybe once every few days) one of the imap processes will 'hang', consuming all available CPU time. It does not seem to 'finish' in any reasonable amount of time (in one instance I waited a few days). This process will not even exit gracefully, it needs to be killed with 'kill -9 <PID>'. It has just happened again. The output from gdb is: > # gdb attach 29401 > GNU gdb 6.3 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i586-suse-linux"...attach: No such file or directory. > > Attaching to process 29401 > Reading symbols from /usr/local/libexec/dovecot/imap...done. > Using host libthread_db library "/lib/tls/libthread_db.so.1". > Reading symbols from /lib/libdl.so.2...done. > Loaded symbols for /lib/libdl.so.2 > Reading symbols from /lib/tls/libc.so.6...done. > Loaded symbols for /lib/tls/libc.so.6 > Reading symbols from /lib/ld-linux.so.2...done. > Loaded symbols for /lib/ld-linux.so.2 > 0x0806049d in imap_sync_more (ctx=0x80d9770) at imap-sync.c:104 > 104 if (ctx->seq == 0) { > (gdb) bt full > #0 0x0806049d in imap_sync_more (ctx=0x80d9770) at imap-sync.c:104 > flags = 1075042320 > keywords = <value optimized out> > str = (string_t *) 0x80c31e0 > ret = 1 > #1 0x08060797 in cmd_sync_continue (cmd=0x80ccdbc) at imap-sync.c:207 > ctx = (struct cmd_sync_context *) 0x80cf000 > ret = <value optimized out> > #2 0x0805ad5a in _client_output (context=0x80ccd78) at client.c:465 > cmd = (struct client_command_context *) 0x80ccdbc > ret = 0 > #3 0x080af0bd in stream_send_io (context=0x80cceb0) at ostream-file.c:356 > ostream = (struct ostream *) 0x80ccee0 > ret = <value optimized out> > #4 0x080aa441 in io_loop_handler_run (ioloop=0x80cb9b0) at ioloop-epoll.c:203 > ctx = (struct ioloop_handler_context *) 0x80cb9d8 > list = (struct io_list *) 0x80d7780 > io = (struct io *) 0x80dc430 > tv = {tv_sec = 2, tv_usec = 540048} > events_count = 4 > t_id = 2 > msecs = <value optimized out> > ret = 1 > i = 0 > j = 1 > call = <value optimized out> > #5 0x080a9889 in io_loop_run (ioloop=0x80cb9b0) at ioloop.c:329 > No locals. > #6 0x08062087 in main (argc=1, argv=0xbffffaf4, envp=0xbffffafc) at main.c:290 > plugin_dir = <value optimized out> Mail log does not reveal anything immediately interesting. Thoughts? Server config ('dovecot -n -c /etc/dovecot/dovecot.conf'): > # 1.0.3: /etc/dovecot/dovecot.conf > base_dir: /var/run/dovecot/ > protocols: imap imaps pop3 pop3s > ssl_ca_file: /etc/ssl/current/cacert.pem > ssl_cert_file: /etc/ssl/current/server.pem > ssl_key_file: /etc/ssl/current/private.pem > disable_plaintext_auth: no > login_dir: /var/run/dovecot//login > login_executable(default): /usr/local/libexec/dovecot/imap-login > login_executable(imap): /usr/local/libexec/dovecot/imap-login > login_executable(pop3): /usr/local/libexec/dovecot/pop3-login > first_valid_uid: 50 > mail_debug: yes > mail_drop_priv_before_exec: yes > mail_executable(default): /usr/local/libexec/dovecot/imap > mail_executable(imap): /usr/local/libexec/dovecot/imap > mail_executable(pop3): /usr/local/libexec/dovecot/pop3 > mail_plugin_dir(default): /usr/local/lib/dovecot/imap > mail_plugin_dir(imap): /usr/local/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 > mail_log_prefix: %Us(%u)(%p): > pop3_uidl_format(default): > pop3_uidl_format(imap): > pop3_uidl_format(pop3): %08Xu%08Xv > auth default: > debug_passwords: yes > passdb: > driver: passwd-file > args: /etc/dovecot/passwd > userdb: > driver: passwd-file > args: /etc/dovecot/passwd > socket: > type: listen > client: > path: /var/spool/postfix/private/auth > mode: 432 > user: postfix > group: postfix -- Best regards, Robert Tomanek mailto:rth23@poczta.onet.pl

4 14

[Dovecot] New full text search indexer
by Timo Sirainen 14 Nov '07

14 Nov '07

As described earlier (http://dovecot.org/list/dovecot/2006-December/018055.html), Dovecot nowadays has full text search indexing support in CVS HEAD. Currently there are two backends: Lucene and Squat. Lucene's problem is that standard IMAP SEARCH command can't be used with it without breaking IMAP RFC. So Lucene can be used only with non-standard X-BODY-FAST and X-TEXT-FAST search commands. Squat's code then is quite complex and it's pretty slow. It works with 4 character blocks, so it can answer: - Search string < 4 characters: Can't answer anything - Search string = 4 characters: Definite yes/no - Search string > 4 characters: Probable yes / definite no Since search strings are usually more than 4 characters, Dovecot uses the index only to filter out messages which definitely won't match and then does regular searching for the non-filtered messages. It works ok, but it could work better. The Squat indexes take maybe 30-50% of the mailbox's size. Most of the space goes to UID lists (list of messages matching each 4 character block). For example one UID list could be "2,4-10,20-30". This is already stored in a compressed format by storing the UID numbers relative to previous UID, which allows storing them usually with a single byte per UID, even if the UIDs are large. So, this morning I had an idea how this could maybe be improved with two changes: 1. Don't use 4 character blocks. Instead allow variable sized paths, and keep track of the UIDs for each node in the path. So for example adding "abc" to UID 1 will place the UID 1 to node "a", "ab" and "abc". 2. Use UIDs relative to parent's UIDs. For example if node "a" has UIDs 1,3,5 and "ab" is added for UID 5, the UID is changed to 2 (3th in 1,3,5 list - 1). This can help the compression a lot. The best case scenario for the above example is if "ab" contains also 1,3,5 UIDs. Then it can be written with a single 0-2 range. With these changes, the indexer can be configured to perform in different ways. Text can be indexed in two ways: full words, and substrings within those words (as required by IMAP). I think a useful combination of these would be to index all 4 character substrings and all full words up to 10 characters or so. This kind of an index can answer: - Search string <= 4 characters: Definite yes/no - For X-BODY-FAST / X-TEXT-FAST: - Search string 5-10 characters: Definite yes/no - Search string 11+ characters: Probable yes / definite no - For standard IMAP SEARCH: - Search string 5-10 characters: Definite yes if message contains the key as non-substring, otherwise same as below. This helps only if user does a search that matches a lot of messages. - Search string 11+ characters: Probable yes / definite no UTF-8 text is indexed as bytes, but the path depth can be counted as characters instead of as bytes. I've a test program finished, and unless there are some bugs left I think it shows that this can work pretty well: Squat-like 4 byte substrings (but can answer 1-3 char queries also): Message count: 7495 Index time: 4.19 CPU seconds (7.39MB/CPUs) Node memory: 3014072 bytes (2943 kB) in 90550 nodes UID memory: 8277212 bytes (8083 kB) in 352778 lists Total: 11291284 / 32474899 bytes = 34.77% Indexing the same mailbox file with squat-test gives: - Index time: 9.55 CPU seconds, 9.63 real seconds (3.24MB/CPUs) - 4145192 bytes in 102208 nodes (12.76%) - 7286058 bytes in 305501 uid_lists (22.44%) - 11431250 bytes total of 32474899 (35.20%) So the new indexer is over twice as fast and uses slightly less space. Indexing 4 char substrings and words up to 10 chars: Index time: 5.13 CPU seconds (6.04MB/CPUs) Node memory: 6002804 bytes (5862 kB) in 615501 nodes UID memory: 9044610 bytes (8832 kB) in 519488 lists Total: 15047414 / 32474899 bytes = 46.34% So it takes somewhat more space, but definitely less than having both Squat + Lucene. No substring indexing, words up to 10 chars (Lucene replacement): Index time: 1.39 CPU seconds (22.28MB/CPUs) Node memory: 3297663 bytes (3220 kB) in 548540 nodes UID memory: 1803735 bytes (1761 kB) in 213554 lists Total: 5101398 / 32474899 bytes = 15.71% Is there a need for Lucene anymore with this kind of a speed and disk space usage? :) The above tests are run with everything being indexed except space and control chars (ie. practically space + tab). If only alphanumeric characters are indexed, the total lines are (in same order as above): Total: 7048664 / 32474899 bytes = 21.70% (-13.07%) Total: 9610543 / 32474899 bytes = 29.59% (-16.75%) Total: 3963335 / 32474899 bytes = 12.20% (-3.51%) There are probably still a few optimizations that can be done to get the disk space usage even lower. I did spent quite a lot of time making it use less CPU though. Even tried a bit different coding style variations in the critical functions to get gcc to create faster code.. One interesting thing I noticed was that binary searching a character from a char array was slower than just sequentially going through the characters. I tried adding checks to do it only if the array was big, but it was always slower. Maybe something to do with CPU cache. If you want to try yourself, the test program is in http://dovecot.org/tmp/new-indexer.c

7 10

[Dovecot] dovecot-1.0.3 & apacheds ldap
by sergey ivanov 31 Oct '07

31 Oct '07

I have problem with dovecot-1.0.3 and apacheds ldap server. If I change just uris in dovecot-ldap.conf to point to fedora-ds server, everything works O.K. I've tried with apacheds ldap server versions 1.0.2 and 1.5.0 command line search with parameters taken from dovecot.debug log gives me all needed attributes. Comments and help welcome. Here is my data: --- # /opt/dovecot/sbin/dovecot -n # 1.0.3: /opt/dovecot/etc/dovecot.conf log_path: /tmp/dovecot.log info_log_path: /tmp/dovecot.debug protocols: pop3 pop3s imap imaps ssl_cert_file: /etc/ssl/parkheights.cert ssl_key_file: /etc/ssl/parkheights.key login_dir: /opt/dovecot/var/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login mail_extra_groups: mail mail_location: maildir:/var/spool/imap/%n/.imap mail_debug: yes mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv auth default: verbose: yes debug: yes debug_passwords: yes passdb: driver: ldap args: /opt/dovecot/etc/dovecot-ldap.conf userdb: driver: ldap args: /opt/dovecot/etc/dovecot-ldap.conf userdb: driver: prefetch socket: type: listen master: path: /opt/dovecot/var/run/dovecot/auth-master mode: 384 user: vmail group: vmail --- # grep -v '#' /opt/dovecot/etc/dovecot-ldap.conf |grep -v '^\s*$' hosts = 192.168.10.43:389 dn = uid=admin,ou=system dnpass = Ahma3zoc sasl_bind = no auth_bind = no ldap_version = 3 base = ou=people,dc=parkheights,dc=dyndns,dc=org user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid user_filter = (&(objectClass=posixAccount)(uid=%Ln)) pass_attrs = uid=user,gidNumber=userdb_gid,userPassword=password,homeDirectory=userdb_home,uidNumber=userdb_uid default_pass_scheme = MD5 --- # tail -5 /tmp/dovecot.debug dovecot: Aug 03 08:07:10 Info: auth(default): client in: AUTH 1 PLAIN service=IMAP secured lip=127.0.0.1 rip=127.0.0.1 resp=AHNlc(here_also_skipped_something)3N1Pg== dovecot: Aug 03 08:07:10 Info: auth(default): ldap(seriv,127.0.0.1): pass search: base=ou=people,dc=parkheights,dc=dyndns,dc=org scope=subtree filter=(&(objectClass=posixAccount)(uid=seriv)) fields=uid,gidNumber,userPassword,homeDirectory,uidNumber dovecot: Aug 03 08:07:10 Info: auth(default): ldap(seriv,127.0.0.1): result: uid(user)=seriv gidnumber(?unknown?)= userpassword(?unknown?)= homedirectory(?unknown?)= uidnumber(?unknown?)= dovecot: Aug 03 08:07:12 Info: auth(default): client out: FAIL 1 user=seriv temp dovecot: Aug 03 08:10:10 Info: imap-login: Disconnected: Inactivity: user=<seriv>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured --- # tail -3 /tmp/dovecot.log dovecot: Aug 03 08:06:53 Warning: auth(default): Killed with signal 15 dovecot: Aug 03 08:06:53 Warning: Killed with signal 15 dovecot: Aug 03 08:07:10 Error: auth(default): ldap(seriv,127.0.0.1): No password in reply --- here is the snippet from apacheds debug log: --- message Id : 2 Search Request Base Object : 'ou=people,dc=parkheights,dc=dyndns,dc=org' Scope : whole subtree Deref Aliases : never Deref Aliases Size Limit : no limit Time Limit : no limit Types Only : false Filter : '(&(objectClass=posixAccount)(uid=seriv))' Attributes : gidNumber, uid, userPassword, homeDirectory, uidNumber baseDn : 'ou=people,dc=parkheights,dc=dyndns,dc=org' filter : '(& (objectClass=posixAccount) (uid=seriv) ) ' scope : whole subtree typesOnly : false no limit Time Limit : no limit Deref Aliases : never Deref Aliases attributes : 'gidNumber', 'uid', 'userPassword', 'homeDirectory', 'uidNumber' message Id : 2 Search Result Entry Object Name : 'uid=seriv,ou=people,dc=parkheights,dc=dyndns,dc=org' Attributes Attributes Attribute id : 'uid', Values : ['seriv'] Attribute id : 'gidnumber', Values : ['1927'] Attribute id : 'userpassword', Values : [0x7B 0x4D ... (skipped some bytes here) ... 0x3D 0x3D ] Attribute id : 'homedirectory', Values : ['/var/spool/imap/seriv'] Attribute id : 'uidnumber', Values : ['1001']

4 11