[Dovecot] Dovecot 2.1rc1 + 2.0.16 woes regarding fts_squat

[Robin]

I can confirm the report posted in http://dovecot.org/list/dovecot/2011-November/062263.html that fts_squat no longer seems to be used after moving from 2.0.16->2.1 rc 1.  I don't see crash reports in the logs, just "0 messages indexed". My search test tool just does a normal IMAP SEARCH for a long non-existent string. If there's another way to trigger re-indexing in 2.1, I don't see anything in the documentation for it.

I've enabled mail_debug, but no log entries that shed light on the problem are available.  Has the configuration for fts_squat changed?

If anyone has a working fts_squat setup with Dovecot 2.1rc1, I'd appreciate hearing how you have it setup and working.

During a large mail import with 2.0.16 today, I ran across a worrying message in the logs during an fts_squat reindex: out of memory. The plugin doesn't obey the mmap_disable configuration directive, which I've confirmed in the plugin source.

The mailbox in question has only 17GB (mdbox style), with about 90,000 emails in it.  Its "index" (for the purposes of normal IMAP retrieval as opposed to IMAP TEXT/BODY searching) is fine and uncorrupted. I freshly import these mailboxes between test iterations and any version changes anyway, so if there's corruption, it's happening within dovecot only.  I'm using use Mail::IMAPClient to create + append mail over localhost, not any direct mdbox conversion trickery.

In looking through the code, I see that mmap() is called for the *ENTIRE FILE*, which is guaranteed to fail on large indexes.  I assume this was done out of expedience, but it's a "risky" sort of thing to do in a server process, even if 8GB RAM systems do seem to grow on trees.  I intend to put this to work in a large installation (>10K users), so this IS of some concern for me in the long-term.

Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188084224) failed: Cannot allocate memory
Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed: Cannot allocate memory
Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188280832) failed: Cannot allocate memory
Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed: Cannot allocate memory
Dec 12 22:50:47 linuxcode dovecot: imap(user1001): Error: Corrupted squat uidlist file .../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids: uidlist not found

================

dovecot -n output:

# 2.0.16: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.35.7-smp i686 Slackware 13.1.0 
auth_mechanisms = plain cram-md5 digest-md5 apop
default_vsz_limit = 192 M
disable_plaintext_auth = no
first_valid_gid = 100
hostname = linuxcode
info_log_path = /tmp/dovecot.log
last_valid_gid = 60000
last_valid_uid = 60000
listen = *
mail_location = mdbox:~/mdbox
mail_plugins = " zlib acl"
mdbox_preallocate_space = yes
mdbox_rotate_interval = 1 days
mmap_disable = yes
passdb {
  args = scheme=plain /etc/cram-md5.pwd
  driver = passwd-file
}
plugin {
  acl = vfile
}
postmaster_address = postmaster at linuxcode
quota_full_tempfail = yes
service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 0
  vsz_limit = 64 M
}
service imap {
  vsz_limit = 512 M
}
service lmtp {
  unix_listener lmtp {
    mode = 0666
  }
}
ssl = no
userdb {
  args = blocking=no
  driver = passwd
}
protocol lmtp {
  mail_plugins = " zlib acl"
}
protocol imap {
  mail_plugins = " zlib acl fts fts_squat imap_acl imap_zlib"
  plugin {
    fts = squat
    fts_squat = partial=4 full=10
  }
}
================

CONFIGURE
================
LIBS=-lnsl CFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector -fomit-frame-pointer' \
CXXFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector -fomit-frame-pointer' \
LDFLAGS=-s ./configure --prefix=/usr --sysconfdir=/etc \
--with-mysql --with-sqlite --with-pgsql --without-pam --with-sql \
--with-libwrap --with-libcap -with-ssl=openssl --with-solr \
--with-mem-align=16 --with-bzlib --with-zlib --localstatedir=/var
================

OS: Slackware 13.1 (32-bit, 2GB physical RAM, kernel setup for 2G/2G split) fully patched up

=R=