Hello,

Am 01.08.2012 19:24, schrieb Timo Sirainen:

On 1.8.2012, at 9.09, Matthias Rieber wrote:

...

I've upgraded from dovecot 2.0 to 2.1. When I perform a header search in a virtual folder dovecot crashes.

Here's the stacktrace:

http://pastebin.com/urSDDzku

Looks like fts-squat isn't currently compatible with virtual folders. I'd suggest moving to fts-lucene. But I guess this also fixes it: http://hg.dovecot.org/dovecot-2.1/rev/8d59874e02ad

thanks, but in the meanwhile I moved to Solr, but I've still some, maybe different, crashes. I'll send a report later after upgrading to v2.1.9 (with config and log).

Is lucence the better choice? According to the wiki Solr seems to be the most preferred solution.

Regards, Matthias

Hi,

Am 01.08.2012 19:24, schrieb Timo Sirainen:

On 1.8.2012, at 9.09, Matthias Rieber wrote:

...

I've upgraded from dovecot 2.0 to 2.1. When I perform a header search in a virtual folder dovecot crashes.

Here's the stacktrace:

http://pastebin.com/urSDDzku

Looks like fts-squat isn't currently compatible with virtual folders. I'd suggest moving to fts-lucene. But I guess this also fixes it: http://hg.dovecot.org/dovecot-2.1/rev/8d59874e02ad

here are the crashes with fts_solr:

Configuration

# 2.1.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.26-2-vserver-amd64 x86_64 Debian 5.0.10 auth_master_user_separator = * disable_plaintext_auth = no listen = * mail_location = maildir:~/Maildir mail_plugins = virtual fts fts_solr zlib acl managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { hidden = yes inbox = no list = no location = prefix = mail separator = type = private } namespace { hidden = no inbox = yes location = prefix = separator = . type = private } namespace { hidden = yes inbox = no list = no location = prefix = INBOX. separator = . type = private } namespace { location = maildir:/home/sharedbox/Maildir prefix = shared. separator = . type = public } namespace { list = yes location = maildir:/home/%%n/Maildir prefix = common.%%u. separator = . subscriptions = no type = shared } namespace { hidden = no list = yes location = virtual:~/Maildir/virtual prefix = virtual. separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } passdb { args = /etc/dovecot/passwd.masterusers driver = passwd-file master = yes } plugin { acl = vfile:/etc/dovecot/acls:cache_secs=7200 acl_shared_dict = file:/var/lib/dovecot/shared-mailboxes.db fts = solr fts_solr = url=http://localhost:8983/solr/ sieve_dir = ~/Maildir/sieve } protocols = imap pop3 sieve service imap { executable = imap vsz_limit = 1 G } service managesieve-login { inet_listener sieve { port = 4190 } } ssl_cert = </etc/dovecot/imap.scheff32.de.pem ssl_key = </etc/dovecot/imap.scheff32.de.key userdb { driver = passwd override_fields = system_groups_user=%u blocking=yes } protocol lda { mail_plugins = fts fts_solr zlib acl virtual sieve } protocol imap { mail_max_userip_connections = 32 mail_plugins = virtual fts fts_solr zlib acl imap_acl imap_zlib } protocol sieve { mail_plugins = virtual fts fts_solr zlib acl sieve }

Crash 1

What I did:

a LOGIN matthias XXXX a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE SEARCH=FUZZY COMPRESS=DEFLATE ACL RIGHTS=texk] Logged in a SELECT virtual.all

• FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk

LearnedAsNonJunk $has_cal $MDNSent 2TDB 3TBD $Forwarded $label1 $DONE $WEB $Label4 $MAILING $PBS $WAA $CHRISTOPH receipt-handled $PART foo redirected Junk LearnedAsJunk 1done test)

• OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk

LearnedAsNonJunk $has_cal $MDNSent 2TDB 3TBD $Forwarded $label1 $DONE $WEB $Label4 $MAILING $PBS $WAA $CHRISTOPH receipt-handled $PART foo redirected Junk LearnedAsJunk 1done test *)] Flags permitted.

• 365142 EXISTS
• 1 RECENT
• OK [UNSEEN 1270] First unseen.
• OK [UIDVALIDITY 1312063141] UIDs valid
• OK [UIDNEXT 799982] Predicted next UID
• OK [HIGHESTMODSEQ 249937] Highest

a OK [READ-WRITE] Select completed. a SEARCH HEADER FROM foobar (server disconects)

Log:

Aug 1 21:04:01 smtp dovecot: imap-login: Login: user=<matthias>, method=PLAIN, rip=10.0.1.3, lip=10.0.1.3, mpid=18561, TLS, session=<O8w+9TjGCAAKAAED> Aug 1 21:05:37 smtp dovecot: imap(matthias): Disconnected: Logged out in=689 out=239091 Aug 1 21:05:40 smtp dovecot: imap(matthias): Error: fts_solr: Lookup failed: 413 FULL head Aug 1 21:05:45 smtp dovecot: imap(matthias): Panic: Leaked t_pop() call Aug 1 21:05:45 smtp dovecot: imap(matthias): Error: Raw backtrace: /usr/local/lib/dovecot/libdovecot.so.0 [0xf7636d71] -> /usr/local/lib/dovecot/libdovecot.so.0 [0xf7636def] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0xf76370c4] -> /usr/local/lib/dov] Aug 1 21:05:45 smtp dovecot: imap(matthias): Fatal: master: service(imap): child 18507 killed with signal 6 (core dumped)

Backtrace:

Core was generated by `dovecot/imap'. Program terminated with signal 6, Aborted. [New process 18507] #0 0xf74d6556 in raise () from /lib/libc.so.6 #0 0xf74d6556 in raise () from /lib/libc.so.6 No symbol table info available. #1 0xf74d7d78 in abort () from /lib/libc.so.6 No symbol table info available. #2 0xf7636d85 in default_fatal_finish (type=<value optimized out>, status=0) at failures.c:191 backtrace = 0x961f2b0 "/usr/local/lib/dovecot/libdovecot.so.0 [0xf7636d71] -> /usr/local/lib/dovecot/libdovecot.so.0 [0xf7636def] -> /usr/local/lib/dovecot/libdovecot.so.0(i_error+0) [0xf76370c4] -> /usr/local/lib/dovecot/l"... #3 0xf7636def in i_internal_fatal_handler (ctx=0xffb97268, format=0xf76620b1 "Leaked t_pop() call", args=0xffb97284 "\024") at failures.c:649 status = 0 #4 0xf76370c4 in i_panic (format=0xf76620b1 "Leaked t_pop() call") at failures.c:263 ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0} #5 0xf76348e1 in t_pop_check (id=0xffb972c8) at data-stack.c:271 No locals. #6 0x08056c6a in client_handle_input (client=0x9670768) at imap-client.c:787 _data_stack_cur_id = 3 ret = true remove_io = <value optimized out> handled_commands = false __FUNCTION__ = "client_handle_input" #7 0x0805765f in client_input (client=0x9670768) at imap-client.c:825 cmd = <value optimized out> output = (struct ostream *) 0x9671094 bytes = <value optimized out> __FUNCTION__ = "client_input" #8 0xf7644f12 in io_loop_call_io (io=0x96710f8) at ioloop.c:379 ioloop = (struct ioloop *) 0x96273f8 t_id = 2 #9 0xf76460d0 in io_loop_handler_run (ioloop=0x96273f8) at ioloop-epoll.c:213 ctx = (struct ioloop_handler_context *) 0x96275e0 event = (const struct epoll_event *) 0x9627620 list = (struct io_list *) 0x9671128 io = (struct io_file *) 0x6 tv = {tv_sec = 1799, tv_usec = 998952} msecs = <value optimized out> ret = 1 i = 0 j = 0 call = false #10 0xf7644ea0 in io_loop_run (ioloop=0x96273f8) at ioloop.c:398 No locals. #11 0xf762d87d in master_service_run (service=0x9627328, callback=0x80607c0 <client_connected>) at master-service.c:543 No locals. #12 0x08060684 in main (argc=1329791025, argv=0x4749464e) at main.c:389 login_set = {auth_socket_path = 0x961f068 "/usr/local/var/run/dovecot/auth-master", postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = 0x8060960 <login_client_connected>, failure_callback = 0x8060900 <login_client_failed>} service_flags = <value optimized out> storage_service_flags = MAIL_STORAGE_SERVICE_FLAG_DISALLOW_ROOT username = 0x0 c = <value optimized out> set_roots = {0x8063340, 0x0}

Crash 2

What I did:

• OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE

IDLE STARTTLS AUTH=PLAIN] Dovecot ready. a LOGIN matthias XXXX a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE SEARCH=FUZZY COMPRESS=DEFLATE ACL RIGHTS=texk] Logged in a SELECT virtual.spam.disagree.spamassassin

• FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
• OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft *)]

Flags permitted.

• 198 EXISTS
• 6 RECENT
• OK [UNSEEN 1] First unseen.
• OK [UIDVALIDITY 1343840294] UIDs valid
• OK [UIDNEXT 417] Predicted next UID
• OK [NOMODSEQ] No permanent modsequences

a OK [READ-WRITE] Select completed. a SEARCH OR HEADER FROM foobar HEADER SUBJECT barfoos (server disconnects)

Log:

Aug 1 21:10:01 smtp dovecot: imap-login: Login: user=<matthias>, method=PLAIN, rip=10.0.1.3, lip=10.0.1.3, mpid=20709, TLS, session=<JP+1CjnGdgAKAAED> Aug 1 21:10:15 smtp dovecot: imap(matthias): Fatal: master: service(imap): child 20642 killed with signal 11 (core dumped)

Backtrace:

Core was generated by `dovecot/imap'. Program terminated with signal 11, Segmentation fault. [New process 20642] #0 fts_search_lookup_level (fctx=0x897cfc8, args=0x8995590, and_args=true) at fts-search.c:125 125 for (i = 0; result->box_results[i].box != NULL; i++) { #0 fts_search_lookup_level (fctx=0x897cfc8, args=0x8995590, and_args=true) at fts-search.c:125 _data_stack_cur_id = 4 #1 0xf734447b in fts_search_lookup (fctx=0x897cfc8) at fts-search.c:350 last_uid = 409 seq1 = 192 seq2 = 198 __FUNCTION__ = "fts_search_lookup" #2 0xf7345dc7 in fts_mailbox_search_init (t=0x897d910, args=0x8995518, sort_program=0x0, wanted_fields=0, wanted_headers=0x0) at fts-storage.c:199 flist = (struct fts_mailbox_list *) 0x8949898 ctx = (struct mail_search_context *) 0x8953068 fctx = (struct fts_search_context *) 0x897cfc8 #3 0xf776cf47 in mailbox_search_init (t=0x897d910, args=0x8995518, sort_program=0x0, wanted_fields=0, wanted_headers=0x0) at mail-storage.c:1402 No locals. #4 0x0805d741 in imap_search_start (ctx=0x894c238, sargs=0x8995518, sort_program=0x0) at imap-search.c:550 cmd = (struct client_command_context *) 0x894c180 #5 0x08053ac6 in cmd_search (cmd=0x894c180) at cmd-search.c:45 ctx = <value optimized out> sargs = (struct mail_search_args *) 0x8995518 args = (const struct imap_arg *) 0x894ea88 charset = 0x8061738 "UTF-8" ret = <value optimized out> #6 0x08057b03 in command_exec (cmd=0x894c180) at imap-commands.c:148 hook = (const struct command_hook *) 0x8903260 ret = false #7 0x08056a3f in client_command_input (cmd=0x894c180) at imap-client.c:682 client = (struct client *) 0x894b768 command = <value optimized out> __FUNCTION__ = "client_command_input" #8 0x08056ae1 in client_command_input (cmd=0x894c180) at imap-client.c:733 client = (struct client *) 0x894b768 command = (struct command *) 0x0 __FUNCTION__ = "client_command_input" #9 0x08056c5d in client_handle_input (client=0x894b768) at imap-client.c:774 _data_stack_cur_id = 3 ret = false remove_io = <value optimized out> handled_commands = false __FUNCTION__ = "client_handle_input" #10 0x0805765f in client_input (client=0x894b768) at imap-client.c:825 cmd = <value optimized out> output = (struct ostream *) 0x894c094 bytes = <value optimized out> __FUNCTION__ = "client_input" #11 0xf76c7f12 in io_loop_call_io (io=0x894c0f8) at ioloop.c:379 ioloop = (struct ioloop *) 0x89023f8 t_id = 2 #12 0xf76c90d0 in io_loop_handler_run (ioloop=0x89023f8) at ioloop-epoll.c:213 ctx = (struct ioloop_handler_context *) 0x89025e0 event = (const struct epoll_event *) 0x8902620 list = (struct io_list *) 0x894c128 io = (struct io_file *) 0x0 tv = {tv_sec = 1789, tv_usec = 618880} msecs = <value optimized out> ret = 1 i = 0 j = 0 call = 16 #13 0xf76c7ea0 in io_loop_run (ioloop=0x89023f8) at ioloop.c:398 No locals. #14 0xf76b087d in master_service_run (service=0x8902328, callback=0x80607c0 <client_connected>) at master-service.c:543 No locals. #15 0x08060684 in main (argc=1329791025, argv=0x4749464e) at main.c:389 login_set = {auth_socket_path = 0x88fa068 "/usr/local/var/run/dovecot/auth-master", postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = 0x8060960 <login_client_connected>, failure_callback = 0x8060900 <login_client_failed>} service_flags = <value optimized out> storage_service_flags = MAIL_STORAGE_SERVICE_FLAG_DISALLOW_ROOT username = 0x0 c = <value optimized out> set_roots = {0x8063340, 0x0}

Regards, Matthias

Hi,

Am 01.08.2012 21:45, schrieb Timo Sirainen:

On 1.8.2012, at 22.21, Matthias Rieber wrote:

...

here are the crashes with fts_solr:

..

...

Crash 1 Aug 1 21:05:40 smtp dovecot: imap(matthias): Error: fts_solr: Lookup failed: 413 FULL head

The crash happens because the Solr lookup failed. I don't know what this Solr error means. Anyway, the crash itself is fixed by: http://hg.dovecot.org/dovecot-2.1/rev/d499f6d0ca68

I thought non-fulltext searches were done by dovecot itself?

Matthias

Am 01.08.2012 22:16, schrieb Timo Sirainen:

On 1.8.2012, at 23.06, Matthias Rieber wrote:

...

...

...

here are the crashes with fts_solr:

..

...

Crash 1 Aug 1 21:05:40 smtp dovecot: imap(matthias): Error: fts_solr: Lookup failed: 413 FULL head

The crash happens because the Solr lookup failed. I don't know what this Solr error means. Anyway, the crash itself is fixed by: http://hg.dovecot.org/dovecot-2.1/rev/d499f6d0ca68

I thought non-fulltext searches were done by dovecot itself?

They are, but header searches are part of full text search.

alright. Some resources say that "413 FULL head" means request too large. Maybe that's caused by the pretty big folder virtual.all which contains more than 360000 mails.

Matthias

Hello everyone,

I had the same errors on my server, and I fixed it by increasing the header size buffer of my server to 65535, for instance.

For jetty, the option is named headerBufferSize. See:

<Call name="addConnector"> <Arg> <New class="org.mortbay.jetty.nio.SelectChannelConnector"> <Set name="host"><SystemProperty name="jetty.host" /></Set> <Set name="port"><SystemProperty name="jetty.port" default="8080"/></Set> <Set name="maxIdleTime">30000</Set> <Set name="Acceptors">2</Set> <Set name="statsOn">false</Set> <Set name="confidentialPort">8443</Set> <Set name="headerBufferSize">65536</Set> <Set name="lowResourcesConnections">5000</Set> <Set name="lowResourcesMaxIdleTime">5000</Set> </New> </Arg> </Call>

On 3 August 2012 15:14, Timo Sirainen tss@iki.fi wrote:

On 1.8.2012, at 23.23, Matthias Rieber wrote:

...

alright. Some resources say that "413 FULL head" means request too large. Maybe that's caused by the pretty big folder virtual.all which contains more than 360000 mails.

Probably it means that the virtual folder consists of too many real folders. Dovecot's Solr query adds each real folder's GUID to the query. I guess there should be some limit and it would have to send more than one query and merge the results.

No need to do this.