October 2007 - dovecot

[Dovecot] MANAGESIEVE patch v7 for dovecot 1.0.3 and TLS
by Andreas Schneider 09 Dec '07

09 Dec '07

Hi, I allow only TLS/SSL connection to dovecot. The managesieve doesn't work with TLS but it works without it. gladiac@magrathea:~> gnutls-cli -p 2000 -s --insecure -V milliways.cynapses.org Resolving 'milliways.cynapses.org'... Connecting to '213.239.217.124:2000'... - Simple Client Mode: - Received[194]: "IMPLEMENTATION" "dovecot" "SASL" "PLAIN" "SIEVE" "fileinto reject envelope vacation imapflags notify subaddress relational comparator-i;ascii-numeric regex" "STARTTLS" OK "Dovecot ready." STARTTLS - Sent: 9 bytes - Received[33]: OK "Begin TLS negotiation now." Let me know if you need more information. Thanks, -- andreas -- http://www.cynapses.org/ - cybernetic synapses

3 4

[Dovecot] imap process consuming 100% CPU (Dovecot 1.0.3)
by Robert Tomanek 20 Nov '07

20 Nov '07

Hi, I have yet another problem with Dovecot: sometimes (rarely, maybe once every few days) one of the imap processes will 'hang', consuming all available CPU time. It does not seem to 'finish' in any reasonable amount of time (in one instance I waited a few days). This process will not even exit gracefully, it needs to be killed with 'kill -9 <PID>'. It has just happened again. The output from gdb is: > # gdb attach 29401 > GNU gdb 6.3 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i586-suse-linux"...attach: No such file or directory. > > Attaching to process 29401 > Reading symbols from /usr/local/libexec/dovecot/imap...done. > Using host libthread_db library "/lib/tls/libthread_db.so.1". > Reading symbols from /lib/libdl.so.2...done. > Loaded symbols for /lib/libdl.so.2 > Reading symbols from /lib/tls/libc.so.6...done. > Loaded symbols for /lib/tls/libc.so.6 > Reading symbols from /lib/ld-linux.so.2...done. > Loaded symbols for /lib/ld-linux.so.2 > 0x0806049d in imap_sync_more (ctx=0x80d9770) at imap-sync.c:104 > 104 if (ctx->seq == 0) { > (gdb) bt full > #0 0x0806049d in imap_sync_more (ctx=0x80d9770) at imap-sync.c:104 > flags = 1075042320 > keywords = <value optimized out> > str = (string_t *) 0x80c31e0 > ret = 1 > #1 0x08060797 in cmd_sync_continue (cmd=0x80ccdbc) at imap-sync.c:207 > ctx = (struct cmd_sync_context *) 0x80cf000 > ret = <value optimized out> > #2 0x0805ad5a in _client_output (context=0x80ccd78) at client.c:465 > cmd = (struct client_command_context *) 0x80ccdbc > ret = 0 > #3 0x080af0bd in stream_send_io (context=0x80cceb0) at ostream-file.c:356 > ostream = (struct ostream *) 0x80ccee0 > ret = <value optimized out> > #4 0x080aa441 in io_loop_handler_run (ioloop=0x80cb9b0) at ioloop-epoll.c:203 > ctx = (struct ioloop_handler_context *) 0x80cb9d8 > list = (struct io_list *) 0x80d7780 > io = (struct io *) 0x80dc430 > tv = {tv_sec = 2, tv_usec = 540048} > events_count = 4 > t_id = 2 > msecs = <value optimized out> > ret = 1 > i = 0 > j = 1 > call = <value optimized out> > #5 0x080a9889 in io_loop_run (ioloop=0x80cb9b0) at ioloop.c:329 > No locals. > #6 0x08062087 in main (argc=1, argv=0xbffffaf4, envp=0xbffffafc) at main.c:290 > plugin_dir = <value optimized out> Mail log does not reveal anything immediately interesting. Thoughts? Server config ('dovecot -n -c /etc/dovecot/dovecot.conf'): > # 1.0.3: /etc/dovecot/dovecot.conf > base_dir: /var/run/dovecot/ > protocols: imap imaps pop3 pop3s > ssl_ca_file: /etc/ssl/current/cacert.pem > ssl_cert_file: /etc/ssl/current/server.pem > ssl_key_file: /etc/ssl/current/private.pem > disable_plaintext_auth: no > login_dir: /var/run/dovecot//login > login_executable(default): /usr/local/libexec/dovecot/imap-login > login_executable(imap): /usr/local/libexec/dovecot/imap-login > login_executable(pop3): /usr/local/libexec/dovecot/pop3-login > first_valid_uid: 50 > mail_debug: yes > mail_drop_priv_before_exec: yes > mail_executable(default): /usr/local/libexec/dovecot/imap > mail_executable(imap): /usr/local/libexec/dovecot/imap > mail_executable(pop3): /usr/local/libexec/dovecot/pop3 > mail_plugin_dir(default): /usr/local/lib/dovecot/imap > mail_plugin_dir(imap): /usr/local/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 > mail_log_prefix: %Us(%u)(%p): > pop3_uidl_format(default): > pop3_uidl_format(imap): > pop3_uidl_format(pop3): %08Xu%08Xv > auth default: > debug_passwords: yes > passdb: > driver: passwd-file > args: /etc/dovecot/passwd > userdb: > driver: passwd-file > args: /etc/dovecot/passwd > socket: > type: listen > client: > path: /var/spool/postfix/private/auth > mode: 432 > user: postfix > group: postfix -- Best regards, Robert Tomanek mailto:rth23@poczta.onet.pl

4 14

[Dovecot] New full text search indexer
by Timo Sirainen 14 Nov '07

14 Nov '07

As described earlier (http://dovecot.org/list/dovecot/2006-December/018055.html), Dovecot nowadays has full text search indexing support in CVS HEAD. Currently there are two backends: Lucene and Squat. Lucene's problem is that standard IMAP SEARCH command can't be used with it without breaking IMAP RFC. So Lucene can be used only with non-standard X-BODY-FAST and X-TEXT-FAST search commands. Squat's code then is quite complex and it's pretty slow. It works with 4 character blocks, so it can answer: - Search string < 4 characters: Can't answer anything - Search string = 4 characters: Definite yes/no - Search string > 4 characters: Probable yes / definite no Since search strings are usually more than 4 characters, Dovecot uses the index only to filter out messages which definitely won't match and then does regular searching for the non-filtered messages. It works ok, but it could work better. The Squat indexes take maybe 30-50% of the mailbox's size. Most of the space goes to UID lists (list of messages matching each 4 character block). For example one UID list could be "2,4-10,20-30". This is already stored in a compressed format by storing the UID numbers relative to previous UID, which allows storing them usually with a single byte per UID, even if the UIDs are large. So, this morning I had an idea how this could maybe be improved with two changes: 1. Don't use 4 character blocks. Instead allow variable sized paths, and keep track of the UIDs for each node in the path. So for example adding "abc" to UID 1 will place the UID 1 to node "a", "ab" and "abc". 2. Use UIDs relative to parent's UIDs. For example if node "a" has UIDs 1,3,5 and "ab" is added for UID 5, the UID is changed to 2 (3th in 1,3,5 list - 1). This can help the compression a lot. The best case scenario for the above example is if "ab" contains also 1,3,5 UIDs. Then it can be written with a single 0-2 range. With these changes, the indexer can be configured to perform in different ways. Text can be indexed in two ways: full words, and substrings within those words (as required by IMAP). I think a useful combination of these would be to index all 4 character substrings and all full words up to 10 characters or so. This kind of an index can answer: - Search string <= 4 characters: Definite yes/no - For X-BODY-FAST / X-TEXT-FAST: - Search string 5-10 characters: Definite yes/no - Search string 11+ characters: Probable yes / definite no - For standard IMAP SEARCH: - Search string 5-10 characters: Definite yes if message contains the key as non-substring, otherwise same as below. This helps only if user does a search that matches a lot of messages. - Search string 11+ characters: Probable yes / definite no UTF-8 text is indexed as bytes, but the path depth can be counted as characters instead of as bytes. I've a test program finished, and unless there are some bugs left I think it shows that this can work pretty well: Squat-like 4 byte substrings (but can answer 1-3 char queries also): Message count: 7495 Index time: 4.19 CPU seconds (7.39MB/CPUs) Node memory: 3014072 bytes (2943 kB) in 90550 nodes UID memory: 8277212 bytes (8083 kB) in 352778 lists Total: 11291284 / 32474899 bytes = 34.77% Indexing the same mailbox file with squat-test gives: - Index time: 9.55 CPU seconds, 9.63 real seconds (3.24MB/CPUs) - 4145192 bytes in 102208 nodes (12.76%) - 7286058 bytes in 305501 uid_lists (22.44%) - 11431250 bytes total of 32474899 (35.20%) So the new indexer is over twice as fast and uses slightly less space. Indexing 4 char substrings and words up to 10 chars: Index time: 5.13 CPU seconds (6.04MB/CPUs) Node memory: 6002804 bytes (5862 kB) in 615501 nodes UID memory: 9044610 bytes (8832 kB) in 519488 lists Total: 15047414 / 32474899 bytes = 46.34% So it takes somewhat more space, but definitely less than having both Squat + Lucene. No substring indexing, words up to 10 chars (Lucene replacement): Index time: 1.39 CPU seconds (22.28MB/CPUs) Node memory: 3297663 bytes (3220 kB) in 548540 nodes UID memory: 1803735 bytes (1761 kB) in 213554 lists Total: 5101398 / 32474899 bytes = 15.71% Is there a need for Lucene anymore with this kind of a speed and disk space usage? :) The above tests are run with everything being indexed except space and control chars (ie. practically space + tab). If only alphanumeric characters are indexed, the total lines are (in same order as above): Total: 7048664 / 32474899 bytes = 21.70% (-13.07%) Total: 9610543 / 32474899 bytes = 29.59% (-16.75%) Total: 3963335 / 32474899 bytes = 12.20% (-3.51%) There are probably still a few optimizations that can be done to get the disk space usage even lower. I did spent quite a lot of time making it use less CPU though. Even tried a bit different coding style variations in the critical functions to get gcc to create faster code.. One interesting thing I noticed was that binary searching a character from a char array was slower than just sequentially going through the characters. I tried adding checks to do it only if the array was big, but it was always slower. Maybe something to do with CPU cache. If you want to try yourself, the test program is in http://dovecot.org/tmp/new-indexer.c

7 10

[Dovecot] dovecot: pipe() failed: Too many open files
by Ralf Hildebrandt 12 Nov '07

12 Nov '07

I'm encountering lots of Oct 31 11:14:25 postamt dovecot: pipe() failed: Too many open files errors around midday ever since the latest upgrade to 1.0.7. The errors appear in pairs like this one: Oct 31 11:20:01 postamt dovecot: pipe() failed: Too many open files Oct 31 11:20:01 postamt dovecot: imap-login: Internal login failure: user=<username>, method=PLAIN, rip=193.175.174.239, lip=141.42.4.250, TLS What gives? -- Ralf Hildebrandt (Ralf.Hildebrandt(a)charite.de) plonk(a)charite.de Postfix - Einrichtung, Betrieb und Wartung Tel. +49 (0)30-450 570-155 http://www.arschkrebs.de We're using Postfix - it generally seems to take the approach that the message that arrives is the message that gets delivered, without any extra bits being added. -- Chris Stratford

2 3

[Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
by Adam McDougall 11 Nov '07

11 Nov '07

I noticed this in an earlier beta and took a note, and I still see it sometimes in 1.1b4, but it is only occasional so I have not tracked down yet what dovecot is doing. Just a FYI for the list incase others see it and to say that its an issue I tend to track down that I did not see in 1.0, and its been low on my list until now. Sometimes when I login to dovecot and read a message from an imap maildir folder, the message contents do not load for at least 10 seconds, sometimes closer to a minute, but after that first message loads, the rest seem fine in that mailbox. The time seems to vary some, usually longer rather than shorter.

2 9

[Dovecot] Outlook can't see namespace folders
by James 10 Nov '07

10 Nov '07

Hi guys i understand that you can't subscribe to the Prefix names of namespace folders right now with dovecot. But for instance if i user outlook express and it can't see the prefix name then it can't see the subfolders of that "prefix folder" either. So even though it can detect and see the subfolders it won't display them cause it can't see the "parent" Is there any fix for this? it's also kind of annoying in thunderbird but at least you can still see the subfolders. TIA James Here's my config # /etc/dovecot/dovecot.conf protocols = imap imaps pop3 pop3s log_timestamp = "%Y-%m-%d %H:%M:%S " first_valid_uid = 100 login_user=vmail mail_extra_groups = mail mail_debug = yes pop3_uidl_format = %08Xu%08Xv mail_location = maildir:/home/vmail/domains/%n/Maildir #default_mail_env = maildir:/home/vmail/domains/%n/Maildir auth default { mechanisms = plain passdb ldap { args = /etc/dovecot/dovecot-ldap.conf } userdb ldap { args = /etc/dovecot/dovecot-ldap.conf } } # maildir namespace private { separator = / prefix = location = maildir:/home/vmail/domains/%n/Maildir inbox = yes hidden = no } # mbox namespace private { separator = / prefix = "Mbox/" location = mbox:/home/vmail/domains/%n/Mbox inbox = no hidden = no } #public spam folder namespace public { separator = / prefix = /Spam/ location = maildir:/home/vmail/domains/Spam/Maildir/ hidden = no } ssl_disable = no ssl_cert_file = /etc/ssl/certs/dovecot.pem ssl_key_file = /etc/ssl/private/dovecot.pem ssl_verify_client_cert = no ssl_cipher_list = ALL:!LOW:HIGH:MEDIUM:+SSLv2:RSA

3 10

[Dovecot] MANAGESIEVE patch v8 for dovecot 1.0.7
by Stephan Bosch 03 Nov '07

03 Nov '07

Hello dovecot users, I have updated the MANAGESIEVE patch to the latest stable version of dovecot (1.0.7). This new version also adds extra configuration settings to explicitly specify the location of the sieve storage and the active sieve script. Additionally, the README now more clearly explains the configuration of the daemon. It currently won't compile with 1.1 due to significant changes in the master code. However, because 1.1 now approaches a stable release, the next version is going to be updated to version 1.1. Change Log V8 ------------- + Fixed a few incompatibilities with 1.0.7 version. For instance, the "Logged in" message is now sent by the -login process and not by the managesieve daemon anymore. This caused a segfault every once in a while. + Probably fixed the settings problem reported by Steffen Kaiser regarding login_dir. 'dovecot -n' now reports correct results, but testing will show whether the whole problem is solved. + The managesieve daemon now accepts the sieve_storage and sieve configuration settings, so it is now possible to explicitly configure the location of the sieve storage and the active script respectively. The daemon still falls back to using the mail_location (MAIL) settings if nothing else is specified. + The cyrus timsieved does not use the + character in string literals and many clients have adopted to this behaviour. The latest managesieve (08) advises to accept a missing + from clients. The server should not send any + characters as well. This behavior is now implemented on the server. + Cleaned up sieve-storage.c: split up the sieve_storage_create function in various sub-functions for obtaining the various paths and directories. + Forced manual intervention if rescueing a non-symlink file at the active script path fails somehow. Previously, this presented the admin with a log message that it had just eaten the script, which is not very nice. + Restructured the README.managesieve file and added some more explanation with regard to the configuration of the daemon. This patch still includes (yet another) instance of the CMU Sieve source, as explained in one of my previous e-mails (http://dovecot.org/list/dovecot/2006-July/015016.html). The patch can be downloaded at: http://sinas.rename-it.nl/~sirius/dovecot-1.0.7-MANAGESIEVE-v8.diff.gz The README.managesieve is located in the root of the dovecot source tree after applying the patch. Have fun testing the patch. Don't hesitate to notify me when there are problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl IRC: Freenode, #dovecot, S[r]us

4 6

[Dovecot] Listening only on localhost
by tobias-mail＠gmx.net 02 Nov '07

02 Nov '07

Hi! I've got dovecot 1.0.rc15. I want dovecot to listen only on localhost for imap, and on any interfaces for imaps. So I put the following into my config: protocol imap { listen = 127.0.0.1:143 ssl_listen = *:10993 .. } dovecot -n correctly reports: listen: 127.0.0.1:143 ssl_listen: *:10993 But I can still do a 'telnet MY_HOST 143' from the outside, and 'netstat -pltn | grep dovecot' reports: [ip changed ;)] tcp 0 0 77.my.ip.253:10993 0.0.0.0:* LISTEN 4847/dovecot tcp 0 0 77.my.ip.253:143 0.0.0.0:* LISTEN 4847/dovecot Am I doing something wrong? Or is there a bug? -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail

3 2

[Dovecot] Errors in Dovecot 1.0.5
by Jeff Tucker 02 Nov '07

02 Nov '07

Hello, I am running Dovecot 1.0.5 and seem to have lots of errors in my Dovecot logs. The worst errors are things like: Oct 30 16:11:42 delta2 dovecot: IMAP(x): Corrupted transaction log file /home/spamcop-net2/deputies/dovecot.index.log: end_offset (392) > current sync_offset (388) Oct 30 16:11:42 delta2 dovecot: IMAP(x): file mail-index-sync-update.c: line 841 (mail_index_sync_update_index): assertion failed: (view->hdr.messages_count == map->hdr.messages_count) Oct 30 16:11:42 delta2 dovecot: IMAP(x): Raw backtrace: imap [0x462a7e] -> imap [0x46234c] -> imap(mail_index_sync_update_index+0x7b2) [0x44a982] -> imap(mail_index_sync_commit+0xe7) [0x447f87] -> imap(maildir_sync_index_finish+0xd2) [0x420b32] -> imap [0x4225af] -> imap(maildir_storage_sync_init+0x63) [0x4227b3] -> imap(imap_sync_nonselected+0xf) [0x41be9f] -> imap(_cmd_select_full+0xc5) [0x413f95] -> imap(cmd_select+0xb) [0x41411b] -> imap [0x415a87] -> imap [0x415b2c] -> imap(_client_input+0x75) [0x415cd5] -> imap(io_loop_handler_run+0xee) [0x4684ce] -> imap(io_loop_run+0x2d) [0x46797d] -> imap(main+0x3e3) [0x41d7b3] -> /lib/libc.so.6(__libc_start_main+0xdb) [0x2aaaaace049b] -> imap [0x410cca] When this happens, my users just get timeouts trying to do stuff. I have my messages (maildirs) and indices all stored on NFS. For locking, I have mmap_disable = yes and lock_method = dotlock Do you have any idea what is going on here? In general, I get tons of errors about corrupted transaction log files for all users. Is this normal? Thanks Jeff

4 5

[Dovecot] Migration from courier pop3 to dovecot pop3
by Shailesh Singh 02 Nov '07

02 Nov '07

Hello, In mail client users have settings leave message on server. I dont want user download those messages again. i tried with courier-dovecot-migrate.plscript after running this if i am doing send recive all messages downloaded again please help me in this. Thanks Shailesh Singh

2 1