- dovecot - dovecot.org

Re: [Dovecot] Search for (any of) multiple terms slow
by Patrick Nagel 14 Aug '08

14 Aug '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, 13 Aug 2008 17:18:37 -0400, Timo Sirainen <tss(a)iki.fi> wrote: > On Aug 13, 2008, at 3:25 AM, Patrick Nagel wrote: > >> When searching for multiple terms at once ("any of") with >> Thunderbird/Dovecot >> (using FTS Squat indexes), it takes much longer (not double the >> time, or a bit >> more than that, but really much, much longer) than when I search for >> only one term. > > Yea, OR isn't supported by FTS plugin yet.. I had hoped no-one would > notice it ;) I guess I should do something about it. Heh, I see ;) As it is, the FTS plugin is quite useless to us though :( most of the time people are doing a (real world) search, they don't remember the exact word they're looking for. For example they only remember that the mail they're looking for contained "English" and "automobile" - or was it "Englisch" and "Automobil", because the mail was in German? So to get a list of all possible mails, they want to make use of the OR connection of search terms... > Requires a lot > more complexity to the code though.. Couldn't Dovecot just search the index with each single word (as it does now without OR), and then merge the results, filtering out double UIDs? This is probably not the most efficient way to solve this, but it would be definitely *much* faster than not using the indexes at all - and doesn't sound too complex to me. Are there any other not-yet-implemented SEARCH features that don't work with the FTS plugin? Thanks, Patrick. - -- STAR Software (Shanghai) Co., Ltd. http://www.star-group.net/ Phone: +86 (21) 3462 7688 x 826 Fax: +86 (21) 3462 7779 PGP key: https://stshacom1.star-china.net/keys/patrick_nagel.asc Fingerprint: E09A D65E 855F B334 E5C3 5386 EF23 20FC E883 A005 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: http://getfiregpg.org iEYEARECAAYFAkijga4ACgkQ7yMg/OiDoAUnFwCfXDUDHR+UdlMKkqwn2/nkB61I X3MAn1nK8KVW8RbtzQ6qQGipGb3uMXRa =I8Eh -----END PGP SIGNATURE-----

2 1

[Dovecot] Can't purge folders in Trash with listescape loaded in 1.1.2
by Adam McDougall 14 Aug '08

14 Aug '08

I made sure I didn't have the Trash folder open by restarting dovecot, then going to File -> Empty Trash in thunderbird. rawlog in, listescape loaded: 3 select "Trash" 4 list "" "Trash/*" 5 delete "Trash/test" 6 IDLE rawlog out, listescape loaded: * OK [RAWLOG TIMESTAMP] 2008-08-08 10:54:14 * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5 \*)] Flags permitted. * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1194838495] UIDs valid * OK [UIDNEXT 55158] Predicted next UID 3 OK [READ-WRITE] Select completed. * LIST (\HasNoChildren) "/" "Trash/test" 4 OK List completed. 5 NO Unknown internal list error + idling rawlog in, listescape loaded: 3 select "Trash" 4 list "" "Trash/*" 5 delete "Trash/test" 6 unsubscribe "Trash/test" 7 IDLE rawlog in, listescape loaded: * OK [RAWLOG TIMESTAMP] 2008-08-08 10:56:34 * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5 \*)] Flags permitted. * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1194838495] UIDs valid * OK [UIDNEXT 55158] Predicted next UID 3 OK [READ-WRITE] Select completed. * LIST (\HasNoChildren) "/" "Trash/test" 4 OK List completed. 5 OK Delete completed. 6 OK Unsubscribe completed. + idling dovecot -n: # 1.1.2: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache22/ssl/mail.pem ssl_key_file: /usr/local/etc/apache22/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_max_processes_count: 512 max_mail_processes: 1024 mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_access_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_process_size: 1024 mail_plugins: acl fts fts_squat listescape quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs:noenforcing:inode_per_mail acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat

2 2

[Dovecot] mbox empty messages in Sent folder
by Diego Liziero 14 Aug '08

14 Aug '08

I'm talking about mbox Sent folder, where some mailers append through imap server a copy of each message they send. Sometimes, just three header lines got appended instead of the whole mail message, such as: >From xxxxxxxx(a)xxxxxx.xxxxxx.xxxxx.xx.xx Fri May 23 12:30:14 2008 X-UID: 2852 Status: RO This happened in the past (dovecot-1.1-beta/rc with Evolution and with Thunderbird), and happened last week with dovecot-1.1-rc5 (+ last week hg patches). Anyone had a similar issue? Regards, Diego.

2 17

[Dovecot] 1.1.x problems
by Andre Hübner 14 Aug '08

14 Aug '08

Hi, since upgrading to 1.1.x i still have this bugs "Next message unexpectedly lost " I read this Maillist some months ago and i believe that im not the only one... The Panics and crashes in 1.1.1 are gone by upgrading to 1.1.2 but these "Next message unexpectedly lost " bugs are really annoying if users can't download complete mails. is there something new to this issue. all recommend changes did not help. The only thing i could do is downgrade to 1.0.15 and all works fine. i can not keep the 1.1.x versions in my repo untill this case is solved. :( Thanks Andre

3 4

[Dovecot] [PATCH] Support GSS-SPNEGO natively
by Jason Gunthorpe 13 Aug '08

13 Aug '08

I cooked this up while trying to figure out why thunderbird on Windows w/ SSPI was not working, but it turned out thunderbird does not use it, so I haven't been able to test it yet. I'm presenting it for discussion only, unless someone else can try it :) Modern versions of MIT kerberos support GSS-SPNEGO natively, but are only willing to negotiate for kerberos tickets and not NTLM messages. This is how the SPNEGO works in libapache-mod-auth-kerb-5.3 which simply passes SPNEGO packets directly to gssapi if the library is new enough. There is even a configure feature test for the gssapi library in that packages configure script. Note that Debian etch's standard kerb libaries (1.4) are not good enough for this. Having this work means the other gssapi policy knobs in dovecot, like auth_gssapi_hostname, work properly. Instead of 'whatever it is that samba does'. I guess it fixes the concerns about winbind accesses blocking (although don't the gssapi calls block??) In light of this it is really only useful to use winbind if you want to support NTLM as kerberos will do the necessary exchanges with the kdc for an Active Directory domain. In truth a proper and complete design would somehow layer the built in NTLM module under the SPNEGO negotiation and only pass NTLM messages off to samba but I doubt anyone cares with NTLM being rather obsolete. --- dovecot-1.0.13/src/auth/mech-gssapi.c 2007-12-11 11:52:08.000000000 -0700 +++ dovecot-1.0.13-jgg/src/auth/mech-gssapi.c 2008-08-11 23:52:15.000000000 -0600 @@ -417,4 +424,21 @@ mech_gssapi_auth_free }; +/* MTI Kerberos > 1.5 supports SPNEGO for Kerberos tickets internally. + Nothing else needs to be done here. Note however that this does not + support SPNEGO when the only available credential is NTLM.. */ +const struct mech_module mech_gssapi_spnego = { + "GSS-SPNEGO", + + MEMBER(flags) 0, + + MEMBER(passdb_need_plain) FALSE, + MEMBER(passdb_need_credentials) FALSE, + + mech_gssapi_auth_new, + mech_gssapi_auth_initial, + mech_gssapi_auth_continue, + mech_gssapi_auth_free +}; + #endif --- dovecot-1.0.13/src/auth/mech.c 2007-12-11 11:52:08.000000000 -0700 +++ dovecot-1.0.13-jgg/src/auth/mech.c 2008-08-11 21:30:56.000000000 -0600 @@ -72,6 +72,7 @@ extern struct mech_module mech_anonymous; #ifdef HAVE_GSSAPI extern struct mech_module mech_gssapi; +extern struct mech_module mech_gssapi_spnego; #endif void mech_init(void) @@ -86,6 +87,7 @@ mech_register_module(&mech_anonymous); #ifdef HAVE_GSSAPI mech_register_module(&mech_gssapi); + mech_register_module(&mech_gssapi_spnego); #endif } @@ -101,5 +103,6 @@ mech_unregister_module(&mech_anonymous); #ifdef HAVE_GSSAPI mech_unregister_module(&mech_gssapi); + mech_unregister_module(&mech_gssapi_spnego); #endif }

3 9

[Dovecot] [PATCH] Allow GSSAPI to work with multihomed hosts
by Jason Gunthorpe 13 Aug '08

13 Aug '08

I saw some past chatter on this in the list archives, but here is another stab and another rational. This patch follows a similar patch to openssh in that it allows any key in the specified keytab to match the incoming host key. This is necessary for multihomed hosts. See: https://bugzilla.mindrot.org/show_bug.cgi?id=928 IMAP/POP seem to be a strong candidate to be multihomed because they are very likely to have an internal to a firewall and external to a firewall name. Due to the way MIT kerberos handles host name resolution via reverse IP lookup this can often result in multiple principle names and there is simply nothing to be done about it. This is my situation.. When you add windows into the mix, which uses a totally different set of rules to determine the principle all hope seems to be lost to have a single service principle name for an imap server :( I was able to make things work for my environment using only the auth_gssapi_hostname feature if only MIT kerberos was used, but as soon as windows SSPI was involved it choose a different hostname. Thus we want to have GSSAPI to match any service principle in the keytab. As far as security concerns go, the admin can configure the keytab for dovecot to be seperate from the system key tab and contain only valid imap SPNs, but in truth it probably doesn't matter. I choose to just use the magic configurable: auth_gssapi_hostname = $ALL rather than introduce more configurables FWIW, after applying this I now am happy to say I have an Active Directory KDC, with dovecot providing gssapi auth to thunderbird on both linux (mit kerb 1.6) and windows (sspi).. --- dovecot-1.0.13/src/auth/mech-gssapi.c 2007-12-11 11:52:08.000000000 -0700 +++ dovecot-1.0.13-jgg/src/auth/mech-gssapi.c 2008-08-11 23:52:15.000000000 -0600 @@ -101,6 +101,13 @@ gss_name_t gss_principal; const char *service_name; + if (strcmp(request->auth->gssapi_hostname,"$ALL") == 0) { + auth_request_log_info(request, "gssapi", + "Using all keytab entires"); + *ret = GSS_C_NO_CREDENTIAL; + return GSS_S_COMPLETE; + } + if (strcasecmp(request->service, "POP3") == 0) { /* The standard POP3 service name with GSSAPI is called just "pop". */

2 2

[Dovecot] dovecot sieve sends vacation messages with null envelope sender
by Josef Liška 13 Aug '08

13 Aug '08

I am using this simple sieve script to do "vacation": require ["fileinto", "vacation"]; if exists "X-Spam-Flag" { stop; } vacation :days 1 :subject "Out of office reply" :addresses ["<email address hidden>", "<email address hidden>" ] "I am on vacation until 2008-08-07. If you have something urgent, please contact: Jan Novak <email address hidden> "; I use sql lookups for users and passwords. For some reason the default location ~/.dovecot-sieve did not work for some reason. However I prefer to put sieve scripts to /var/lib/sieve and when I added additional field to lookup with this location it started to work. But it sends vacation messages with null envelope sender, which is in my opinion undesirable, because many vacation messages are caught in spam filters. I think there could be same sender on envelope. In dovecot-sieve/src/sieve-cmu.c around line 380 is a call to smtp_client_open(src->addr, NULL, &f); NULL could be replaced with src->fromaddr Maybe some more patching may be necesary. I found similar patch for cyrus-imapd at http://oss.digirati.com.br/mail/cyrus.html <http://oss.digirati.com.br/mail/cyrus.html> My current system: 1) Ubuntu 8.04.1 jl@mail:~$ dovecot --version 1.0.15 It is a backported package from ubuntu intrepid ibex. However the same applies for dovecot 1.0.10 which is in ubuntu 8.04. Summary: I expect messages sent by vacation recipe should have envelope from same as user's e-mail address. This is then writen to Return-Path: header by receiving MTA NULL envelope from is used, resulting (in my case) to Return-Path: <> rewriten by postfix to Return-Path: <MAILER-DAEMON> by receiving MTA

7 6

[Dovecot] RfE: use HOSTNAME environment variable in hostpid_init()
by Heiko Schlichting 13 Aug '08

13 Aug '08

Hi, All parts of dovecot except deliver uses the result of hostpid_init() in src/lib/hostpid.c as a hostname which only asks gethostname(). deliver honours environment variable HOSTNAME in src/deliver/deliver.c: getenv("HOSTNAME"); and uses the hostname of hostpid_init() as a fallback. Wouldn't it be consequent to evaluate the environment variable HOSTNAME in hostpid_init() with fallback to gethostname() and remove the special behavior from deliver? The dovecot master process should export $HOSTNAME to all childs in this case. Motivation: Dovecot uses the hostname as part of Maildir file names and gethostname() is not fully qualified on all operating systems (it is on IRIX but not on Linux). But unqualified hostnames are not unique and it is frequent that dovecot hosts are named "mail" oder "imap" as hostname. To distinguish between mail.domain1.com and mail.domain2.net would be easier if dovecot evaluate $HOSTNAME in all case not just deliver as program imap can create Maildir files as well. Heiko Heiko Schlichting Freie Universität Berlin heiko(a)FU-Berlin.DE Zentraleinrichtung für Datenverarbeitung (ZEDAT) Telefon +49 30 838-54327 Fabeckstraße 32 Telefax +49 30 838454327 14195 Berlin

2 1

[Dovecot] syslog with PID
by Heiko Schlichting 13 Aug '08

13 Aug '08

Hi, LOG_NDELAY is the only option for openlog() in dovecot 1.1.2. Wouldn't be LOG_NDELAY|LOG_PID as option parameter much more useful? Without logging the pid, it is impossible to match 'Disconnected' log entries and the corresponding session start/login. Therefore I suggest to use LOG_NDELAY|LOG_PID in the options of i_set_failure_syslog() at all 6 occurrences which passes it to openlog(). I know that it is possible to - use logging to file - patch it myself but I can't see good reasons in not using LOG_PID here. As far as I know, most other programs specify LOG_PID. Without LOG_PID: Jul 31 10:21:08 name dovecot: IMAP(hsg): Disconnected: Logged out bytes=28/242 With LOG_PID: Jul 31 10:21:08 name dovecot[17016]: IMAP(hsg): Disconnected: Logged out bytes=28/242 Heiko Heiko Schlichting Freie Universität Berlin heiko(a)FU-Berlin.DE Zentraleinrichtung für Datenverarbeitung (ZEDAT) Telefon +49 30 838-54327 Fabeckstraße 32 Telefax +49 30 838454327 14195 Berlin

3 5

[Dovecot] sieve - Sendmail process terminated abnormally, exit status 70
by Thomas Harold 13 Aug '08

13 Aug '08

How do we start troubleshooting this? deliver(user(a)example.com): Aug 12 18:27:19 Error: Sendmail process terminated abnormally, exit status 70 deliver(user(a)example.com): Aug 12 18:27:19 Info: sieve runtime error: Vacation: Error sending mail deliver(user(a)example.com): Aug 12 18:27:19 Info: msgid=<48A20E3C.7070307(a)example.com>: saved mail to INBOX deliver(user(a)example.com): Aug 12 18:27:19 Error: sieve_execute_bytecode(/var/vmail/example.com/user//Home/.dovecot.sievec) failed It seems to only choke on the vacation portion. It creates entries in the ".dovecot.lda-dupes" file. But then dies while sending the reply e-mail. The contents of the user's .dovecot.sieve file is: ------------------------------------- require ["fileinto", "include", "vacation"]; # Move spam to spam folder if exists "X-Spam-Flag" { fileinto "spam"; # Stop here so that we do not reply on spams stop; } include :personal "sieve-vacation"; ------------------------------------- And the "" file looks like: ------------------------------------- require ["vacation"]; vacation # Reply at most once a day to a same sender :days 1 :subject "Out of office reply" # List of recipient addresses which are included in the auto replying. # If a mail's recipient is not on this list, no vacation reply is sent for it. :addresses "user(a)example.com" "blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah, user(a)example.com"; ------------------------------------- If I comment out the "include :personal "sieve-vacation";" line, then it works, but obviously not the vacation portion. Deliver does at least not die horribly. I get the same error if I move the content of the vacation include file into the main ".dovecot.sieve" file. # ls -la /usr/local/libexec/dovecot/lda/ -rwsr-xr-x 1 root root 802824 Aug 12 18:12 deliver # ls -la /usr/libexec/dovecot/ total 5728 drwxr-xr-x 2 root root 4096 Jul 31 04:04 . drwxr-xr-x 11 root root 4096 Jul 25 04:39 .. -rwxr-xr-x 1 root root 58416 Jul 24 06:32 checkpassword-reply -rwxr-xr-x 1 root root 666128 Jul 24 06:32 convert-tool -rwxr-xr-x 1 root root 802824 Jul 24 06:32 deliver -rwxr-xr-x 1 root root 164176 Jul 24 06:32 dict -rwxr-xr-x 1 root root 350384 Jul 24 06:32 dovecot-auth -rwxr-xr-x 1 root root 674176 Jul 24 06:32 expire-tool -rwxr-xr-x 1 root root 59200 Jul 24 06:32 gdbhelper -rwxr-xr-x 1 root root 245872 Jul 24 06:32 idxview -rwxr-xr-x 1 root root 854488 Jul 24 06:32 imap -rwxr-xr-x 1 root root 157216 Jul 24 06:32 imap-login -rwxr-xr-x 1 root root 61248 Jul 24 06:32 listview -rwxr-xr-x 1 root root 61800 Jul 24 06:32 logview -rwxr-xr-x 1 root root 74200 Jul 24 06:32 maildirlock -rwxr-xr-x 1 root root 931 Jul 24 06:27 mkcert.sh -rwxr-xr-x 1 root root 787464 Jul 24 06:32 pop3 -rwxr-xr-x 1 root root 149152 Jul 24 06:32 pop3-login -rwxr-xr-x 1 root root 83968 Jul 24 06:32 rawlog -rwxr-xr-x 1 root root 165152 Jun 11 03:21 sievec -rwxr-xr-x 1 root root 157216 Jun 11 03:21 sieved -rwxr-xr-x 1 root root 62584 Jul 24 06:32 ssl-build-param (output of "yum list") dovecot.x86_64 :1.1.2-2_77.el5 installed dovecot-sieve.x86_64 1.1.5-8.el5 installed I'm not finding any AVC errors in the SELinux audit.log file. And I'm not sure what other switches I can turn on to get better error information as to what sendmail/deliver are choking on in this virtual (setuid) environment where we use Dovecot as the LDA.

3 3