- dovecot - dovecot.org

Re: [Dovecot] fts-solr plugin issue (Marked invalid)
by Nikolai Derzhak 05 May '09

05 May '09

Send mail body with wrong Content-Type. If put this in maildir indexing raise 500 error. If fix Content-Type to "application/octet-stream" all OK. Because TEXT/*; content dovecot try to index and solr raise 500, on each search. On Tue, 05/05/2009 at 3:01pm, "Nikolai Derzhak" <nikolai(a)6zap.com> wrote: > On Mon, 05/04/2009 at 6:23pm, "Timo Sirainen" <tss(a)iki.fi> wrote: >> On May 4, 2009, at 10:16 AM, Nikolai Derzhak wrote: >> >>> OK. Concentrating problem in one question. >>> How to ignore "bad" message and index next one in indexing procedure >>> (fts plugin) ?. >>> Now, one "error 500" from solr and dovecot (# 1.1.11: /etc/dovecot/ >>> dovecot.conf >>> # OS: Linux 2.6.21.7-2.fc8xen i686 Ubuntu 8.04.2 ext3 >>> ) stop and each next search query repeat the story. >> >> Maybe simply using 1.1.14 would help? I already fixed one Solr issue: http:/ >> / >> hg.dovecot.org/dovecot-1.1/rev/678c3252a454 >> > I've merge back this commit to 1.1.11 code, cause this is my request about > special chars ;). > >> If that's not the problem, it would help to have one of those mails >> that breaks it. > Yes. Now i have two new issue: > 1. "Content-Type:" header in mail wrong - so it's TEXT/mspowerpoint for > example, but it's binary *.ppt file. > 2. many html from repository change-set(many commit's to html) in text/plain > mail > > In both cases we have "Error 500" from solr (HTMLStripReader class) (in solr > log: "Marked inlalid"). > And in both cases we can not detect this in dovecot code (or it's hard to do) > . > So for now i need some method to ignore "bad" messages. > Because we can not catch all variants, but indexing die on first error like > this. > > > > Powered by the 6zap. Sign up at http://www.6zap.com for an account that > provides advanced e-mail, calendar and contacts capabilities. Powered by the 6zap. Sign up at http://www.6zap.com for an account that provides advanced e-mail, calendar and contacts capabilities.

1 0

Re: [Dovecot] fts-solr plugin issue (Marked invalid)
by Nikolai Derzhak 05 May '09

05 May '09

On Mon, 05/04/2009 at 6:23pm, "Timo Sirainen" <tss(a)iki.fi> wrote: > On May 4, 2009, at 10:16 AM, Nikolai Derzhak wrote: > >> OK. Concentrating problem in one question. >> How to ignore "bad" message and index next one in indexing procedure >> (fts plugin) ?. >> Now, one "error 500" from solr and dovecot (# 1.1.11: /etc/dovecot/ >> dovecot.conf >> # OS: Linux 2.6.21.7-2.fc8xen i686 Ubuntu 8.04.2 ext3 >> ) stop and each next search query repeat the story. > > Maybe simply using 1.1.14 would help? I already fixed one Solr issue: http:// > hg.dovecot.org/dovecot-1.1/rev/678c3252a454 > I've merge back this commit to 1.1.11 code, cause this is my request about special chars ;). > If that's not the problem, it would help to have one of those mails > that breaks it. Yes. Now i have two new issue: 1. "Content-Type:" header in mail wrong - so it's TEXT/mspowerpoint for example, but it's binary *.ppt file. 2. many html from repository change-set(many commit's to html) in text/plain mail In both cases we have "Error 500" from solr (HTMLStripReader class) (in solr log: "Marked inlalid"). And in both cases we can not detect this in dovecot code (or it's hard to do). So for now i need some method to ignore "bad" messages. Because we can not catch all variants, but indexing die on first error like this. Powered by the 6zap. Sign up at http://www.6zap.com for an account that provides advanced e-mail, calendar and contacts capabilities.

1 0

[Dovecot] Maildir bugs
by Timo Sirainen 05 May '09

05 May '09

I just found out that when using deliver or saving mails with IMAP APPEND command there was all kinds of race condition bugs that could cause e.g.: dovecot-uidlist: Duplicate file entry at line 18: 1241457213.M690227P32531.host,W=3359:2,T (uid 170019 -> 170020) And who knows what else. If you've run into those problems, these patches should help: http://hg.dovecot.org/dovecot-1.1/rev/6bbe27c32cb8 http://hg.dovecot.org/dovecot-1.1/rev/003bebac2c76 http://hg.dovecot.org/dovecot-1.1/rev/96c8632775df They are pretty large though, hopefully won't break anything. But at least my stress tests can no longer break it. I'll probably release v1.1.15 RC in a few days so at least some people can try to break it before the final v1.1.15 :)

1 0

[Dovecot] questions about compiling Sieve plugin
by MercuryShadow 04 May '09

04 May '09

I'm using the info on the wiki to try to compile the sieve plugin for dovecot on a server running Debian Etch, but I'm getting nowhere because I'm stuck trying to find the directory to use in the "--with-dovecot=" argument when running the configure script. The wiki says: Compilation is identical among the CMUSieve and the new Sieve plugin. Use --with-dovecot=<path> to point to dovecot-config file's directory. There are two possibilities where this could exist: 1. If you configured Dovecot with --enable-header-install, you'll have dovecot-config installed in $prefix/lib/dovecot/ directory. 2. Compiled Dovecot sources' root directory. The problem is that there is no such file called "dovecot-config" on the server, and dovecot was installed from the Debian repositories, not compiled. There is no /usr/lib/dovecot or /usr/local/lib/dovecot. I figured that, as is the case with many Debian packages, the libraries needed to compile would be included in a "dovecot-dev" package, but I checked and there is no such package in the Debian repos. Does anyone know what I need to do in order to get this plugin compiled? -- View this message in context: http://www.nabble.com/questions-about-compiling-Sieve-plugin-tp23285989p232… Sent from the Dovecot mailing list archive at Nabble.com.

2 8

[Dovecot] Problem with BODYSTRUCTURE (wrong charset)
by Craig Craig 04 May '09

04 May '09

Hi, I've changed a program that fetches mail via IMAP from an Exchange Server to use our Dovecot machine. But now I'm experiencing strange problems with seem to be caused by a wrong BODYSTRUCTURE answer that causes wrong charset handling. When fetchting the BODYSTRUCTURE I see this: Dovecot: 88 UID fetch 32 (BODYSTRUCTURE) * 1 FETCH (UID 32 BODYSTRUCTURE ("text" "plain" ("charset" "us-ascii") NIL NIL "7bit" 725664 9385 NIL NIL NIL NIL)) 88 OK Fetch completed. Exchange: 90 UID fetch 10 (BODYSTRUCTURE) * 2 FETCH (BODYSTRUCTURE (("TEXT" "PLAIN" ("charset" "iso-8859-1") NIL NIL "QUOTED-PRINTABLE" 2784 47 NIL NIL NIL)("IMAGE" "JPEG" ("name" "197254.jpg") NIL NIL "BASE64" 498806 NIL ("attachment" ("filename" "197254.jpg")) NIL)("IMAGE" "JPEG" ("name" "197254_1.jpg") NIL NIL "BASE64" 188424 NIL ("attachment" ("filename" "197254_1.jpg")) NIL)("IMAGE" "JPEG" ("name" "197254_2.jpg") NIL NIL "BASE64" 31878 NIL ("attachment" ("filename" "197254_2.jpg")) NIL)("TEXT" "PLAIN" ("name" "pix.xml") NIL NIL "QUOTED-PRINTABLE" 2934 49 NIL ("attachment" ("filename" "pix.xml")) NIL)("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "7BIT" 0 1 NIL ("attachment" NIL) NIL) "mixed" ("Message-ID:" "<d2788bc14788ff16e67e63941937c1e1(a)foo.de>" "boundary" "000XMAIL000") NIL NIL) UID 10) 90 OK FETCH completed. Mail contents were: [...] To: blah(a)foo.de Subject: Pictures From: blah2(a)foo.de CC: Errors-To: blah2(a)foo.de Sender: foobar(a)foo.de MIME-Version: 1.0 Content-Type: multipart/mixed; Message-ID:=<d2788bc14788ff16e67e63941937c1e1(a)foo.de>; boundary="000XMAIL000" Message-Id: <20090504120233.58AFC346FC(a)foo.de> Date: Mon, 4 May 2009 14:02:33 +0200 (CEST) --000XMAIL000 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: Quoted-Printable <?xml version=3D"1.0" encoding=3D"ISO-8859-1" ?> <foo> [...] --000XMAIL000 Content-Type: image/jpeg; name="197254.jpg" Content-Transfer-Encoding: Base64 Content-Disposition: attachment; filename="197254.jpg" [...] --000XMAIL000 Content-Type: text/plain; name="pix.xml" Content-Transfer-Encoding: Quoted-Printable Content-Disposition: attachment; filename="pix.xml" <?xml version=3D"1.0" encoding=3D"ISO-8859-1" ?>=0D <blah>=0D [...] Here is my dovecot -n output: # 1.1.14: /etc/dovecot/dovecot.conf # OS: Linux 2.6.21-gentoo-r1 i686 Gentoo Base System release 1.12.11.1 ext3 log_path: /var/log/dovecot protocols: imap imaps pop3 pop3s listen: [::] ssl_cert_file: /etc/ssl/dovecot/server.pem ssl_key_file: /etc/ssl/dovecot/server.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login mail_privileged_group: vmail mail_location: maildir:/var/mail/%d/%n mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/lib/dovecot/imap mail_plugin_dir(imap): /usr/lib/dovecot/imap mail_plugin_dir(pop3): /usr/lib/dovecot/pop3 auth default: mechanisms: plain login digest-md5 cram-md5 rpa apop username_translation: AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz verbose: yes debug: yes debug_passwords: yes passdb: driver: passwd-file args: /etc/passwd.dovecot userdb: driver: passwd-file args: /etc/passwd.dovecot socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix master: path: /var/run/dovecot/auth-master mode: 432 user: root group: vmail So is this a parsing error in dovecot or was the Exchange Server just parsing incorrect MIME-Headers?! Any hint would be appreciated... Thanks in advance, Craig

2 1

Re: [Dovecot] fts-solr plugin issue (Marked invalid)
by Nikolai Derzhak 04 May '09

04 May '09

OK. Concentrating problem in one question. How to ignore "bad" message and index next one in indexing procedure (fts plugin) ?. Now, one "error 500" from solr and dovecot (# 1.1.11: /etc/dovecot/dovecot.conf # OS: Linux 2.6.21.7-2.fc8xen i686 Ubuntu 8.04.2 ext3 ) stop and each next search query repeat the story. I've explored fts, and ftp-solr directories in src, without success for now. Timo, you understand code much bettter, can you help me and point to place in code, or probably create some patch, if possible ?. On Fri, 05/01/2009 at 5:56pm, "Nikolai Derzhak" <nikolai(a)6zap.com> wrote: > Hi. > > We user ftp-solr plugin and have problem with solr-1.3+ > with HTMLStripWhitespaceTokenizerFactory (Solr schema in attachments). > In some maildir's present messages with wrong "Content-Type: " fields in > attachments. > For example: > " > Content-Type: TEXT/mspowerpoint; name="Zapatec_6zap_netvibes_1.ppt" > " > Indexing for this messages is stop with "fts_solr: Indexing failed: 500 > Internal Server Error". > In solr log is: > " > SEVERE: java.io.IOException: Mark invalid > at java.io.BufferedReader.reset(BufferedReader.java:485) > " > (mail list with discussion: http://markmail.org/message/2fnfiwygvehjngyr# > query:SEVERE%3A%20java.io.IOException%3A%20Mark%20invalid%20lucene+page:1+ > mid:2fnfiwygvehjngyr+state:results) > > Look's like dovecot try to index attachments like this. > Also for some messages we have same error. > Dovecot stop indexing of box and each search we have lag and CPU load on > server. > > So we need to make dovecot more "stable" to this error. > For first time , will be good, just ignore problematic messages with error > from solr. > > Let's discuss this issue, because this is general problem. > We ready to explore code where needed , etc. > > Regards, > Nikolai > > Powered by the 6zap. Sign up at http://www.6zap.com for an account that > provides advanced e-mail, calendar and contacts capabilities. Powered by the 6zap. Sign up at http://www.6zap.com for an account that provides advanced e-mail, calendar and contacts capabilities.

3 2

Re: [Dovecot] fts-solr plugin issue (Marked invalid)
by Nikolai Derzhak 04 May '09

04 May '09

On Mon, 05/04/2009 at 5:36pm, "Rui Carneiro" <rui.carneiro(a)portugalmail.net> wrote: > I do not have sure if I understood your problem correctly. > > Are you trying to index attachments from messages? Or Dovecot is indexing > some "bad" parts and you just do not know why? > Read first post, please, with description. But in sum: when dovecot try to index some mail's, that solr tokenizer not eat (error 500, Marked invalid), dovecot stop indexing of box and retry attempts in each next search with same result. > Regards, > Rui Carneiro > -- > Portugalmail, Comunicações S.A. > www.portugalmail.net > > Citando Nikolai Derzhak <nikolai(a)6zap.com>: > >> OK. Concentrating problem in one question. >> How to ignore "bad" message and index next one in indexing procedure (fts >> plugin) ?. >> Now, one "error 500" from solr and dovecot (# 1.1.11: >> /etc/dovecot/dovecot.conf >> # OS: Linux 2.6.21.7-2.fc8xen i686 Ubuntu 8.04.2 ext3 >> ) stop and each next search query repeat the story. >> I've explored fts, and ftp-solr directories in src, without success for now. >> >> Timo, you understand code much bettter, can you help me and point to place >> in >> code, >> or probably create some patch, if possible ?. Powered by the 6zap. Sign up at http://www.6zap.com for an account that provides advanced e-mail, calendar and contacts capabilities.

2 1

[Dovecot] login_max_processes_count and logs
by Dominique Fournier 04 May '09

04 May '09

Hello, Our server reach today the login_max_processes_count. Dovecot works well and don't accept any new connection. But it doesn't log anything about this problem : we are blind why the connection is refused for the clients. Could you add a log when the connection limit is reached or (maybe better) a log with the occupation for the beginning of each connection ? Thanks Dom Dovecot version 1.0.15 on Debian AMD64

3 3

[Dovecot] Strange virtual users issues
by Ovi C 02 May '09

02 May '09

Hi i'm using dovecot with virtual users from a postgresql database backend on a Debian Lenny system. I'm using qmail as MTA. Mostly everything seems to work ok , except the fact that the mail delivered never reaches virtual user's INBOX. Here's the dovecot config: dovecot -n # 1.0.15: /etc/dovecot/dovecot.conf base_dir: /var/run/dovecot/ log_path: /var/log/dovecot.log log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap ssl_disable: yes login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login login_greeting: Server ready to process requests. login_process_per_connection: no max_mail_processes: 512 verbose_proctitle: yes first_valid_gid: 61 last_valid_gid: 2003 mail_privileged_group: mail mail_location: maildir:~/Maildir mail_debug: yes mail_plugins: quota autocreate imap_client_workarounds: outlook-idle delay-newmail auth default: mechanisms: plain login digest-md5 passdb: driver: pam passdb: driver: passwd passdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: passwd userdb: driver: sql args: /etc/dovecot/dovecot-sql.conf userdb: driver: prefetch socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 384 user: root group: mail plugin: quota: maildir acl: vfile:/etc/dovecot-acls trash: /etc/dovecot-trash.conf lazy_expunge: .EXPUNGED/ .DELETED/ .DELETED/.EXPUNGED/ autocreate: Trash autocreate2: Spam The setup is as follows: I've created a system user named vmail in /home with a group also called vmail. The virtual users are created in /home/vmail/username when i configure an account from thunderbird. If I send an message from a virtual user to an system user everythings is ok. The system user get's the mail in his Maildir. But if I send a message from the system user to the virtual user the MTA's log says that the message has been delivered but it never reaches virtual user's INBOX. Here's a snip from logs both qmail's and dovecot's: cat /var/log/qmail.log status: local 0/10 remote 0/20 new msg 155041 info msg 155041: bytes 263 from <caretaker@localhost.> qp 4560 uid 1000 starting delivery 1: msg 155041 to remote akkersson@localhost. status: local 0/10 remote 1/20 new msg 155147 info msg 155147: bytes 438 from <caretaker@localhost.> qp 4563 uid 64011 starting delivery 2: msg 155147 to local akkersson@localhost status: local 1/10 remote 1/20 delivery 1: success: 127.0.0.1_accepted_message./Remote_host_said:_250_ok_1241272738_qp_4563/ status: local 1/10 remote 0/20 end msg 155041 delivery 2: deferral: status: local 0/10 remote 0/20 cat /var/log/dovecot.log dovecot: 2009-05-02 16:30:46 Info: Dovecot v1.0.15 starting up dovecot: 2009-05-02 16:30:46 Info: auth(default): pgsql: Connected to maildb dovecot: 2009-05-02 16:32:00 Info: IMAP(akkersson): Loading modules from directory: /usr/lib/dovecot/modules/imap dovecot: 2009-05-02 16:32:00 Info: IMAP(akkersson): Module loaded: /usr/lib/dovecot/modules/imap/lib10_quota_plugin.so dovecot: 2009-05-02 16:32:00 Info: IMAP(akkersson): Module loaded: /usr/lib/dovecot/modules/imap/lib11_autocreate_plugin.so dovecot: 2009-05-02 16:32:00 Info: IMAP(akkersson): Effective uid=2003, gid=61, home=/home/vmail/akkersson dovecot: 2009-05-02 16:32:00 Info: IMAP(akkersson): maildir: data=/home/vmail/akkersson/Maildir dovecot: 2009-05-02 16:32:00 Info: IMAP(akkersson): maildir: root=/home/vmail/akkersson/Maildir, index=/home/vmail/akkersson/Maildir, control=, inbox= dovecot: 2009-05-02 16:32:00 Info: imap-login: Login: user=<akkersson>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured dovecot: 2009-05-02 16:58:11 Info: IMAP(akkersson): Disconnected: Logged out but akkersson's INBOX is empty.. Any hints ??

1 0

[Dovecot] deliver_log_format not working with 1.2.rc3?
by Anton Dollmaier 01 May '09

01 May '09

Hi list, just a small issue with deliver_log_format. Although, the parameter is altered in the config and is correctly shown with "dovecot -n", deliver uses only its default log_format. > root@terra:~# dovecot -n | grep -e deliver -e "^#" > # 1.2.rc3: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.26-1-686 i686 Debian 5.0.1 > deliver_log_format: %m:%$:%f:%s this is from a recent mail: > deliver(antondollmaier(a)aditsystems.de): 2009-04-30 16:13:45 Info: sieve: msgid=<20090430141309.GN16305(a)katherina.student.utwente.nl>: stored mail into mailbox 'Dovecot-List' Subject and From are obviously not logged. Any hints? thanks in advance. best regards, Anton

2 1