July 2007 - dovecot - dovecot.org

[Dovecot] Too many open files
by Andrew Hutchings 10 Jan '08

10 Jan '08

Hi Guys, Just got this on one of my test72 dovecot servers: - Jun 10 12:40:28 svr21 dovecot: auth(default): net_connect_unix(/var/run/dovecot/auth-worker.18818) failed: Too many open files Jun 10 12:40:28 svr21 dovecot: pop3-login: Disconnected: method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=xxx.xxx.xxx.xxx Jun 10 12:40:28 svr21 dovecot: pop3-login: Disconnected: method=PLAIN, rip=xxx.xxx.xxx.xxx, lip=xxx.xxx.xxx.xxx Jun 10 12:40:28 svr21 dovecot: child 18818 (auth) returned error 89 Jun 10 12:40:29 svr21 dovecot: Auth process died too early - shutting down Jun 10 12:40:29 svr21 dovecot: auth(default): mysql: Connected to localhost (maildb) Jun 10 12:40:29 svr21 dovecot: auth(default): net_connect_unix(/var/run/dovecot/auth-worker.13784) failed: No such file or directory Jun 10 12:40:29 svr21 dovecot: child 13784 (auth) returned error 89 - Is dovecot leaving files open? Or is it a large mailbox? Regards Andrew -- Andrew Hutchings (A-Wing) Linux Guru - Netserve Consultants Ltd. - http://www.domaincity.co.uk/ Admin - North Wales Linux User Group - http://www.nwlug.org.uk/ BOFH excuse 122: because Bill Gates is a Jehovah's witness and so nothing can work on St. Swithin's day.

5 5

[Dovecot] What would you tell the CIO in an "ABCs of Email" overview?
by Esther Schindler 07 Jan '08

07 Jan '08

One of the key points that came up when I researched and wrote the "Five Things CIOs Should Know about Fighting Spam" article (http:// www.cio.com/article/28830) was that they should know the basics of how email works. Otherwise, said plenty of techies, the CIO won't have the first idea of what the email admin is complaining about. So I'm going to do my part. I'm going to write an "ABCs of Email" article (to accompany the many other ABCs articles we have on CIO.com, at http://www.cio.com/article/40242 ). I'd like your input on the topics that should be included, keeping in mind the fact that the target reader is a CIO, IT manager, or someone who wants to understand the basics, *not* actively get involved in email management. You don't need to write an essay for me or inundate me with links (though hey, if you want to make my life easier I shall not complain). What I'm looking for, primarily, are the categories of information that I should cover. In other words, if your CIO had an email ephiphany and asked you to give a half-hour presentation, what would you include? This won't be an "Expert says..." article nor will it be "Geek on the street says...." I intend to compile and research the "least you need to know" for the not-necessarily-techie bosses out there. And hopefully the end result will be that you have one less dumb question to deal with in your life. So: any suggestions? (You can reply privately if you prefer.) Esther Schindler senior online editor, CIO.com her blog: http://advice.cio.com/taxonomy/term/34

2 3

[Dovecot] New full text search indexer
by Timo Sirainen 14 Nov '07

14 Nov '07

As described earlier (http://dovecot.org/list/dovecot/2006-December/018055.html) Dovecot nowadays has full text search indexing support in CVS HEAD. Currently there are two backends: Lucene and Squat. Lucene's problem is that standard IMAP SEARCH command can't be used with it without breaking IMAP RFC. So Lucene can be used only with non-standard X-BODY-FAST and X-TEXT-FAST search commands. Squat's code then is quite complex and it's pretty slow. It works with 4 character blocks, so it can answer: - Search string < 4 characters: Can't answer anything - Search string = 4 characters: Definite yes/no - Search string > 4 characters: Probable yes / definite no Since search strings are usually more than 4 characters, Dovecot uses the index only to filter out messages which definitely won't match and then does regular searching for the non-filtered messages. It works ok, but it could work better. The Squat indexes take maybe 30-50% of the mailbox's size. Most of the space goes to UID lists (list of messages matching each 4 character block). For example one UID list could be "2,4-10,20-30". This is already stored in a compressed format by storing the UID numbers relative to previous UID, which allows storing them usually with a single byte per UID, even if the UIDs are large. So, this morning I had an idea how this could maybe be improved with two changes: 1. Don't use 4 character blocks. Instead allow variable sized paths, and keep track of the UIDs for each node in the path. So for example adding "abc" to UID 1 will place the UID 1 to node "a", "ab" and "abc". 2. Use UIDs relative to parent's UIDs. For example if node "a" has UIDs 1,3,5 and "ab" is added for UID 5, the UID is changed to 2 (3th in 1,3,5 list - 1). This can help the compression a lot. The best case scenario for the above example is if "ab" contains also 1,3,5 UIDs. Then it can be written with a single 0-2 range. With these changes, the indexer can be configured to perform in different ways. Text can be indexed in two ways: full words, and substrings within those words (as required by IMAP). I think a useful combination of these would be to index all 4 character substrings and all full words up to 10 characters or so. This kind of an index can answer: - Search string <= 4 characters: Definite yes/no - For X-BODY-FAST / X-TEXT-FAST: - Search string 5-10 characters: Definite yes/no - Search string 11+ characters: Probable yes / definite no - For standard IMAP SEARCH: - Search string 5-10 characters: Definite yes if message contains the key as non-substring, otherwise same as below. This helps only if user does a search that matches a lot of messages. - Search string 11+ characters: Probable yes / definite no UTF-8 text is indexed as bytes, but the path depth can be counted as characters instead of as bytes. I've a test program finished, and unless there are some bugs left I think it shows that this can work pretty well: Squat-like 4 byte substrings (but can answer 1-3 char queries also): Message count: 7495 Index time: 4.19 CPU seconds (7.39MB/CPUs) Node memory: 3014072 bytes (2943 kB) in 90550 nodes UID memory: 8277212 bytes (8083 kB) in 352778 lists Total: 11291284 / 32474899 bytes = 34.77% Indexing the same mailbox file with squat-test gives: - Index time: 9.55 CPU seconds, 9.63 real seconds (3.24MB/CPUs) - 4145192 bytes in 102208 nodes (12.76%) - 7286058 bytes in 305501 uid_lists (22.44%) - 11431250 bytes total of 32474899 (35.20%) So the new indexer is over twice as fast and uses slightly less space. Indexing 4 char substrings and words up to 10 chars: Index time: 5.13 CPU seconds (6.04MB/CPUs) Node memory: 6002804 bytes (5862 kB) in 615501 nodes UID memory: 9044610 bytes (8832 kB) in 519488 lists Total: 15047414 / 32474899 bytes = 46.34% So it takes somewhat more space, but definitely less than having both Squat + Lucene. No substring indexing, words up to 10 chars (Lucene replacement): Index time: 1.39 CPU seconds (22.28MB/CPUs) Node memory: 3297663 bytes (3220 kB) in 548540 nodes UID memory: 1803735 bytes (1761 kB) in 213554 lists Total: 5101398 / 32474899 bytes = 15.71% Is there a need for Lucene anymore with this kind of a speed and disk space usage? :) The above tests are run with everything being indexed except space and control chars (ie. practically space + tab). If only alphanumeric characters are indexed, the total lines are (in same order as above): Total: 7048664 / 32474899 bytes = 21.70% (-13.07%) Total: 9610543 / 32474899 bytes = 29.59% (-16.75%) Total: 3963335 / 32474899 bytes = 12.20% (-3.51%) There are probably still a few optimizations that can be done to get the disk space usage even lower. I did spent quite a lot of time making it use less CPU though. Even tried a bit different coding style variations in the critical functions to get gcc to create faster code.. One interesting thing I noticed was that binary searching a character from a char array was slower than just sequentially going through the characters. I tried adding checks to do it only if the array was big, but it was always slower. Maybe something to do with CPU cache. If you want to try yourself, the test program is in http://dovecot.org/tmp/new-indexer.c

7 10

[Dovecot] Possible Caching Bug showing up as a MIME Boundary Issue
by Kevin A. McGrail 27 Oct '07

27 Oct '07

Possible Caching Bug showing up as a MIME Boundary Issue I'm using Dovecot version 1.0.0. I was using Dovecot version 1.0.0 beta3 or alpha4. I upgraded to Dovecot 1.0.0 to make sure that was not the issue. Over the past few weeks on a server running a stable dovecot, I have seem a few emails arriving where the MIME document structure dividers are visible. I've included a screenshot showing this. Using OE6 as client. Have tried adding oe6-fetch-no-newmail per clients wiki but this seems to stop the ability to login at all. I am using two different machines with Outlook Express to access the same email. The same problem will show up on both. Usually, if I delete the .imap cache for the folder, the email comes down correctly. My dovecot -n output is: # /etc/dovecot.conf listen: *:144 ssl_listen: *:994 ssl_cert_file: /usr/local/ssl/certs/stunnel.pem ssl_key_file: /usr/local/ssl/certs/stunnel.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login max_mail_processes: 50 verbose_proctitle: yes default_mail_env: mbox:~/:INBOX=/var/spool/mail/%u mail_location: mbox:~/:INBOX=/var/spool/mail/%u imap_client_workarounds: outlook-idle delay-newmail auth default: passdb: driver: pam userdb: driver: passwd userdb: driver: prefetch The server is on a heavily modified RedHat 7.2 on x86 box with ext3 Thanks in Advance, KAM

2 3

[Dovecot] dspam plugin
by Goran Mekić 30 Sep '07

30 Sep '07

When I drag n drop spam mail to SPAM folder, I get this error: Jun 11 17:58:46 sparrow dspam[28452]: Unable to open file for reading: /var/spool/dspam/data/local/mymail/mymail.sig/466d652f315451306914567.sig: No such file or directory mymail is user that runs dovecot transport (postfix config). The .sig file exist in /var/spool/dspam/data/<domain>/<user>/<user>.sig/466d652f315451306914567.sig. I've tried to google this problem, but no luck. Any advices? Thanx!

2 1

[Dovecot] dovecot-dspam plugin & virtual users
by Wicher Minnaard 30 Sep '07

30 Sep '07

Hi list, hi Johannes, I'm trying to use the dovecot-dspam plugin ( http://johannes.sipsolutions.net/Projects/dovecot-dspam-integration ) with vpopmail's virtual users. All seems well, except for the fact that dspam should be called with '--user currentuser(a)currentdomain.com' at line 157 here: http://johannes.sipsolutions.net/Projects/dovecot-dspam-integration/beta7-c… For my needs, 'currentuser@currentdomain' could be taken as the login name I use for logging in to Dovecot. As I'm not a C-programmer by a long shot, I was wondering if one of you could help me use the bincimap-API to import the login name into a variable so I can use that variable to construct a suitable dspam-command. Hope this is possible! Regards, Wicher Minnaard

2 2

[Dovecot] Binary packagers: BSD license issues
by Timo Sirainen 23 Sep '07

23 Sep '07

Adding this SHA256 code made me read the BSD license once again. It says: * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. Then there are a few files from Cyrus as well which contain: * 4. Redistributions of any form whatsoever must retain the following * acknowledgment: * "This product includes software developed by Computing Services * at Carnegie Mellon University (http://www.cmu.edu/computing/)" And something similar in utc-mktime.c for Berkeley university as well. I think I'd really like to get rid of those base64.c and utc-mktime.c exceptions.. There's probably also an easier/faster way to implement utc_mktime(). Anyway.. Currently these copyrights or acknowledgments aren't listed anywhere else than in the source files. I don't know if binary packagers have added those, but somehow I doubt it. So I think I should add these to COPYING file somehow. Any suggestions? Perhaps appending something like this: src/lib/sha1.c, sha2.c : BSD license * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * Copyright (C) 2005, 2007 Olivier Gay <olivier.gay(a)a3.epfl.ch> * All rights reserved. * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. src/lib/base64.c, utc-mktime.c : * Copyright (c) 2000 Carnegie Mellon University. All rights reserved. * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * 3. The name "Carnegie Mellon University" must not be used to * endorse or promote products derived from this software without * prior written permission. For permission or any other legal * details, please contact * Office of Technology Transfer * Carnegie Mellon University * 5000 Forbes Avenue * Pittsburgh, PA 15213-3890 * (412) 268-4387, fax: (412) 268-7395 * tech-transfer(a)andrew.cmu.edu * * 4. Redistributions of any form whatsoever must retain the following * acknowledgment: * "This product includes software developed by Computing Services * at Carnegie Mellon University (http://www.cmu.edu/computing/)" * * CARNEGIE MELLON UNIVERSITY DISCLAIMS ALL WARRANTIES WITH REGARD TO * THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY * AND FITNESS, IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY BE LIABLE * FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN * AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING * OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. * Copyright (c) 1987, 1989, 1993 * The Regents of the University of California. All rights reserved. * This code is derived from software contributed to Berkeley by * Arthur David Olson of the National Cancer Institute. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE.

2 2

[Dovecot] BUG REPORT - Stops logging after sleep
by Bruce Bodger 09 Sep '07

09 Sep '07

Version: dovecot --version: 1.0.0 OS: OS X - 10.4.9 Architecture: G5 Dual/1.8Ghz Had a fairly normal time correction last night: May 9 20:56:56 G518X2 ntpd[219]: time set -1.212733 s Which dovecot duly noted: May 9 20:56:56 G518X2 dovecot: auth(default): Time just moved backwards by 1 seconds. I'll sleep now until we're back in present. This morning all of the dovecot processes are running fine but dovecot has stopped logging. (I have modified nothing in dovecot.conf relative to logging: dovecot logs to syslog using the "mail" syslog_facility.) ps -aux | grep dovecot root 53 0.0 -0.0 27772 544 ?? Ss 4:23PM 0:04.69 /usr/local/sbin/dovecot -F root 79 0.0 -0.0 28176 556 ?? S 4:23PM 0:03.21 dovecot-auth dovecot 9548 0.0 -0.0 27780 496 ?? S 7:12AM 0:00.05 imap-login dovecot 9631 0.0 -0.0 27780 500 ?? S 7:17AM 0:00.03 imap-login dovecot 9632 0.0 -0.0 27780 500 ?? S 7:17AM 0:00.03 imap-login I am starting dovecot via OS X's launchdaemon facility thus the "-F" command line switch requirement. Sending a "kill -HUP" to the process did not get it to start logging. Sending a "Kill -9" to the process forced a restart and logging began again. ps -aux | grep dovecot root 9738 0.0 -0.0 27772 540 ?? Ss 7:26AM 0:00.03 /usr/local/sbin/dovecot -F root 9739 0.0 -0.0 28176 544 ?? S 7:26AM 0:00.02 dovecot-auth dovecot 9740 0.0 -0.0 27780 500 ?? S 7:26AM 0:00.02 imap-login dovecot 9751 0.0 -0.0 27780 500 ?? S 7:27AM 0:00.01 imap-login dovecot 9762 0.0 -0.0 27780 500 ?? S 7:27AM 0:00.01 imap-login Before posting this bug I compared v1.0.0's /src/lib/ioloop.c to that in the nightly snapshot and saw no adjustments in, what I presumed to be, the relevant portion of the code. Output of "dovecot -n" protocols: imap ssl_disable: yes disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login first_valid_uid: 490 mail_extra_groups: mail mail_location: mbox:~/Library/Mail/IMAP:INBOX=/var/mail/%u:INDEX=%h/ Library/Mail/DOVECOT.indexes dotlock_use_excl: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd Thank you.

4 14

[Dovecot] ssh tunneling and maildir
by Joel Becker 17 Aug '07

17 Aug '07

Hello everyone, please keep my CC'd, I'm not on the list. I've been using dovecot happily via an ssh tunnel. My mutt is configured to use IMAP, and the tunnel is as follows in my muttrc: set tunnel="ssh -q mymailhost 'MBOX_LOCKS=fcntl MAIL=/home/jlbec/mail /usr/libexec/dovecot/imap'" It should be pretty obvious that this logs into mymailhost as me (via SSH keyauth) and runs /usr/libexec/dovecot/imap directly as a foreground process. Setting MAIL, it points dovecot imap to my mbox folders. dovecot imap speaks imap over stdin/out, this goes over the ssh to mutt. It works great. Really great. I don't need root access at all, the server isn't running dovecot globally, and I get IMAP through firewalls if I can ssh through a firewall/vpn. I'm using mbox and fcntl, so my MTA has no trouble working in concert. However, this doesn't appear to be documented anywhere. A friend showed it to me, and I don't even remember who that was. I'm now considering changing to maildir folders. I figure dovecot imap might figure out a folder is maildir when stat says S_IFDIR, but without documentation, I can't figure out how to tell dovecot imap to use maildir when I create a folder. I'm also unsure how this will interact with my MTA. For the latter, I'm thinking I just need to configure my MTA to use dovecot's deliver program. For the former, is there some magic environment variable I need to add to my tunnel command line? Thanks Joel -- "The first requisite of a good citizen in this republic of ours is that he shall be able and willing to pull his weight." - Theodore Roosevelt http://www.jlbec.org/ jlbec(a)evilplan.org

4 8

[Dovecot] MANAGESIEVE patch v5 for dovecot 1.0.2
by Stephan Bosch 15 Aug '07

15 Aug '07

Hello dovecot users, I have finally updated the MANAGESIEVE patch to fix the currently known small problems with the protocol implementation. Also, I included a proxy implementation based on imap-proxy.c. This patch is designed for dovecot release 1.0.2 and it will not apply cleanly to the 1.1 versions yet. Change Log V5 ------------- - Applied patch by Uldis Pakuls to fix master_dump_settings bug - Added some compilation/installation info to the README - Moved README to source tree's root as README.managesieve - Fixed minor error handling bug in sieve_storage.c with respect to a missing root directory. - Now sieve capabilities are reported as they are specified by the implementing library and not in forced upper case. The sieve RFC now explicitly states that sieve capability identifiers are case-sensitive. This broke compatibility with SquirrelMail/Avelsieve. - Disabled ANONYMOUS login entirely until proper support is implemented. V4 claimed to do so as well, but in fact it only stopped announcing it. - Implemented managesieve-proxy. It is not so much a clean copy of imap-proxy, since the managesieve greeting is much more complex and requires parsing. Configuration is identical to imap-proxy. This seems to be a little under-documented however (http://wiki.dovecot.org/PasswordDatabase/ExtraFields) This patch still includes (yet another) instance of the CMU Sieve source, as explained in one of my previous e-mails (http://dovecot.org/list/dovecot/2006-July/015016.html) Current plans include merging src/lib-managesieve with src/lib-imap, updating the patch for dovecot-1.1 and fixing any problems you might find. The patch can be downloaded at: http://sinas.rename-it.nl/~sirius/dovecot-1.0.2-MANAGESIEVE-v5.diff.gz The README.managesieve is located in the root of the dovecot source tree after applying the patch. Have fun testing the patch. Notify me when there are problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl IRC: Freenode, #dovecot, S[r]us

5 13