I can confirm the report posted in http://dovecot.org/list/dovecot/2011-November/062263.html that fts_squat no longer seems to be used after moving from 2.0.16->2.1 rc 1. I don't see crash reports in the logs, just "0 messages indexed". My search test tool just does a normal IMAP SEARCH for a long non-existent string. If there's another way to trigger re-indexing in 2.1, I don't see anything in the documentation for it.

I've enabled mail_debug, but no log entries that shed light on the problem are available. Has the configuration for fts_squat changed?

If anyone has a working fts_squat setup with Dovecot 2.1rc1, I'd appreciate hearing how you have it setup and working.

During a large mail import with 2.0.16 today, I ran across a worrying message in the logs during an fts_squat reindex: out of memory. The plugin doesn't obey the mmap_disable configuration directive, which I've confirmed in the plugin source.

The mailbox in question has only 17GB (mdbox style), with about 90,000 emails in it. Its "index" (for the purposes of normal IMAP retrieval as opposed to IMAP TEXT/BODY searching) is fine and uncorrupted. I freshly import these mailboxes between test iterations and any version changes anyway, so if there's corruption, it's happening within dovecot only. I'm using use Mail::IMAPClient to create + append mail over localhost, not any direct mdbox conversion trickery.

In looking through the code, I see that mmap() is called for the *ENTIRE FILE*, which is guaranteed to fail on large indexes. I assume this was done out of expedience, but it's a "risky" sort of thing to do in a server process, even if 8GB RAM systems do seem to grow on trees. I intend to put this to work in a large installation (>10K users), so this IS of some concern for me in the long-term.

Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188084224) failed: Cannot allocate memory Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed: Cannot allocate memory Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188280832) failed: Cannot allocate memory Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed: Cannot allocate memory Dec 12 22:50:47 linuxcode dovecot: imap(user1001): Error: Corrupted squat uidlist file .../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids: uidlist not found

================

dovecot -n output:

# 2.0.16: /etc/dovecot/dovecot.conf # OS: Linux 2.6.35.7-smp i686 Slackware 13.1.0 auth_mechanisms = plain cram-md5 digest-md5 apop default_vsz_limit = 192 M disable_plaintext_auth = no first_valid_gid = 100 hostname = linuxcode info_log_path = /tmp/dovecot.log last_valid_gid = 60000 last_valid_uid = 60000 listen = * mail_location = mdbox:~/mdbox mail_plugins = " zlib acl" mdbox_preallocate_space = yes mdbox_rotate_interval = 1 days mmap_disable = yes passdb { args = scheme=plain /etc/cram-md5.pwd driver = passwd-file } plugin { acl = vfile } postmaster_address = postmaster@linuxcode quota_full_tempfail = yes service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 0 vsz_limit = 64 M } service imap { vsz_limit = 512 M } service lmtp { unix_listener lmtp { mode = 0666 } } ssl = no userdb { args = blocking=no driver = passwd } protocol lmtp { mail_plugins = " zlib acl" } protocol imap { mail_plugins = " zlib acl fts fts_squat imap_acl imap_zlib" plugin { fts = squat fts_squat = partial=4 full=10 } }

CONFIGURE

LIBS=-lnsl CFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector -fomit-frame-pointer'
CXXFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector -fomit-frame-pointer'
LDFLAGS=-s ./configure --prefix=/usr --sysconfdir=/etc
--with-mysql --with-sqlite --with-pgsql --without-pam --with-sql
--with-libwrap --with-libcap -with-ssl=openssl --with-solr
--with-mem-align=16 --with-bzlib --with-zlib --localstatedir=/var

OS: Slackware 13.1 (32-bit, 2GB physical RAM, kernel setup for 2G/2G split) fully patched up

=R=