December 2011 - dovecot

[Dovecot] dovecot penalties and anvil
by Leonardo Rodrigues 20 Dec '11

20 Dec '11

Hi, I'm strugling to find some documentation on dovecot's anvil service and/or its penalties (that can be checked with doveadm penalty) ... but i'm not finding anything on that. is there any documentation on anvil/penalties that i should check other than a few messages on the mailing list archive ? thanks ! -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertrudes(a)solutti.com.br My SPAMTRAP, do not email it

2 1

[Dovecot] Dovecot 2.1rc1 + 2.0.16 woes regarding fts_squat
by Robin 20 Dec '11

20 Dec '11

I can confirm the report posted in http://dovecot.org/list/dovecot/2011-November/062263.html that fts_squat no longer seems to be used after moving from 2.0.16->2.1 rc 1. I don't see crash reports in the logs, just "0 messages indexed". My search test tool just does a normal IMAP SEARCH for a long non-existent string. If there's another way to trigger re-indexing in 2.1, I don't see anything in the documentation for it. I've enabled mail_debug, but no log entries that shed light on the problem are available. Has the configuration for fts_squat changed? If anyone has a working fts_squat setup with Dovecot 2.1rc1, I'd appreciate hearing how you have it setup and working. During a large mail import with 2.0.16 today, I ran across a worrying message in the logs during an fts_squat reindex: out of memory. The plugin doesn't obey the mmap_disable configuration directive, which I've confirmed in the plugin source. The mailbox in question has only 17GB (mdbox style), with about 90,000 emails in it. Its "index" (for the purposes of normal IMAP retrieval as opposed to IMAP TEXT/BODY searching) is fine and uncorrupted. I freshly import these mailboxes between test iterations and any version changes anyway, so if there's corruption, it's happening within dovecot only. I'm using use Mail::IMAPClient to create + append mail over localhost, not any direct mdbox conversion trickery. In looking through the code, I see that mmap() is called for the *ENTIRE FILE*, which is guaranteed to fail on large indexes. I assume this was done out of expedience, but it's a "risky" sort of thing to do in a server process, even if 8GB RAM systems do seem to grow on trees. I intend to put this to work in a large installation (>10K users), so this IS of some concern for me in the long-term. Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188084224) failed: Cannot allocate memory Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed: Cannot allocate memory Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: mremap_anon(188280832) failed: Cannot allocate memory Dec 12 22:48:52 linuxcode dovecot: imap(user1001): Error: read(.../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids) failed: Cannot allocate memory Dec 12 22:50:47 linuxcode dovecot: imap(user1001): Error: Corrupted squat uidlist file .../mdbox/mailboxes/INBOX2010/dbox-Mails/dovecot.index.search.uids: uidlist not found ================ dovecot -n output: # 2.0.16: /etc/dovecot/dovecot.conf # OS: Linux 2.6.35.7-smp i686 Slackware 13.1.0 auth_mechanisms = plain cram-md5 digest-md5 apop default_vsz_limit = 192 M disable_plaintext_auth = no first_valid_gid = 100 hostname = linuxcode info_log_path = /tmp/dovecot.log last_valid_gid = 60000 last_valid_uid = 60000 listen = * mail_location = mdbox:~/mdbox mail_plugins = " zlib acl" mdbox_preallocate_space = yes mdbox_rotate_interval = 1 days mmap_disable = yes passdb { args = scheme=plain /etc/cram-md5.pwd driver = passwd-file } plugin { acl = vfile } postmaster_address = postmaster@linuxcode quota_full_tempfail = yes service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 0 vsz_limit = 64 M } service imap { vsz_limit = 512 M } service lmtp { unix_listener lmtp { mode = 0666 } } ssl = no userdb { args = blocking=no driver = passwd } protocol lmtp { mail_plugins = " zlib acl" } protocol imap { mail_plugins = " zlib acl fts fts_squat imap_acl imap_zlib" plugin { fts = squat fts_squat = partial=4 full=10 } } ================ CONFIGURE ================ LIBS=-lnsl CFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector -fomit-frame-pointer' \ CXXFLAGS='-O2 -march=core2 -mtune=core2 -fstack-protector -fomit-frame-pointer' \ LDFLAGS=-s ./configure --prefix=/usr --sysconfdir=/etc \ --with-mysql --with-sqlite --with-pgsql --without-pam --with-sql \ --with-libwrap --with-libcap -with-ssl=openssl --with-solr \ --with-mem-align=16 --with-bzlib --with-zlib --localstatedir=/var ================ OS: Slackware 13.1 (32-bit, 2GB physical RAM, kernel setup for 2G/2G split) fully patched up =R=

2 1

[Dovecot] Question about index reset in dovecot 2.0.15
by Maria Arrea 20 Dec '11

20 Dec '11

I have seen some strange error log messages in our dovecot server (2.0.15, RHEL 5.7x64, mdbox+zlib, local storage, high load) Dec 14 22:01:27 buzon dovecot: imap(fmcarrasco): Error: /indices_dovecot/indices/07/47/fmcarrasco/mailboxes/INBOX/dovecot.index reset, view is now inconsistent Dec 14 22:00:27 buzon dovecot: imap(cfp): Error: /indices_dovecot/indices/16/58/cfp/mailboxes/Sent/dovecot.index reset, view is now inconsistent Dec 14 20:56:46 buzon dovecot: imap(rcam): Error: Corrupted index cache file /indices_dovecot/indices/03/6b/rcam/mailboxes/INBOX/dovecot.index.cache: Broken virtual size for mail UID 132197 Dec 14 21:50:38 buzon dovecot: imap(cfp): Error: Recent flags state corrupted for mailbox INBOX I have been googling without luck, why does this happen? I have tried with doveadm force-resync, maybe this could solve this issues. doveconf -n output # 2.0.15: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-274.7.1.el5 x86_64 Red Hat Enterprise Linux Server release 5.7 (Tikanga) ext4 auth_cache_negative_ttl = 10 secs auth_cache_size = 10 M auth_cache_ttl = 2 mins auth_master_user_separator = * auth_mechanisms = plain login auth_worker_max_count = 3500 base_dir = /var/run/dovecot/ default_client_limit = 5000 default_process_limit = 6500 disable_plaintext_auth = no imap_client_workarounds = tb-extra-mailbox-sep delay-newmail tb-lsub-flags lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_fsync = never mail_gid = entrega mail_home = /buzones/domain/%2.26Hn/%2.200Hn/%n/home_usuario/ mail_location = mdbox:/buzones/domain/%2.26Hn/%2.200Hn/%n:INDEX=/indices_dovecot/indices/%2.26Hn/%2.200Hn/%n mail_max_userip_connections = 15000 mail_plugins = " zlib acl quota autocreate" mail_uid = entrega managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date mdbox_rotate_interval = 1 days mdbox_rotate_size = 60 M passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } passdb { args = /etc/usuario_maestro.txt driver = passwd-file master = yes } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { acl = vfile autocreate = SPAM autocreate2 = Sent autocreate3 = Drafts autocreate4 = Trash autosubscribe = SPAM autosubscribe2 = Sent autosubscribe3 = Drafts autosubscribe4 = Trash lda_mailbox_autosubscribe = yes quota = dict:Cuota de usuario::file:/buzones/cuotas/%n quota_rule2 = Trash:storage=+10%% quota_warning = storage=90%% aviso_cuota 90 %u sieve = /buzones/domain/%2.26Hn/%2.200Hn/%n/home_usuario/dovecot.sieve sieve_dir = /buzones/domain/%2.26Hn/%2.200Hn/%n/home_usuario/sieve/ zlib_save = gz zlib_save_level = 9 } pop3_no_flag_updates = yes protocols = pop3 imap sieve service anvil { client_limit = 25000 } service auth { client_limit = 28000 unix_listener auth-master { user = entrega } unix_listener auth-userdb { user = entrega } user = root } service aviso_cuota { executable = script /usr/local/bin/quota-warning.sh unix_listener aviso_cuota { mode = 0666 } user = entrega } service imap-login { executable = /usr/libexec/dovecot/imap-login group = dovenull service_count = 0 vsz_limit = 1 G } service imap { executable = /usr/libexec/dovecot/imap process_limit = 6000 } service managesieve-login { executable = /usr/libexec/dovecot/managesieve-login inet_listener sieve { port = 2000 } process_limit = 2000 vsz_limit = 1 G } service managesieve { executable = /usr/libexec/dovecot/managesieve process_limit = 5000 } service pop3-login { executable = /usr/libexec/dovecot/pop3-login process_limit = 4000 service_count = 0 vsz_limit = 1 G } service pop3 { executable = /usr/libexec/dovecot/pop3 process_limit = 4000 } ssl_ca = </etc/pki/generico/cacert.crt.pem ssl_cert = </etc/pki/generico/wildcard-us_es.crt ssl_key = </etc/pki/generico/wildcard-key.pem userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } userdb { args = /etc/dovecot/dovecot-ldap-userdb.conf driver = ldap } verbose_proctitle = yes protocol sieve { managesieve_implementation_string = dovecot managesieve_logout_format = bytes=%i/%o managesieve_max_line_length = 65536 } protocol lda { hostname = domain info_log_path = log_path = mail_fsync = optimized mail_plugins = sieve zlib quota postmaster_address = evcorreo@domain syslog_facility = mail } protocol imap { mail_plugins = zlib autocreate quota imap_quota } protocol pop3 { mail_plugins = zlib pop3_enable_last = yes pop3_uidl_format = %g } Regards Maria

2 1

[Dovecot] doveadm quota does not list all accounts
by Nikolaos Milas 20 Dec '11

20 Dec '11

Hello, I am running dovecot-2.0.13-1_128.el5 x86_64 RPM on CentOS 5.7. All accounts are virtual, hosted on LDAP Server. My problem is that the command: doveadm quota get -A stopped listing all accounts. I think this problem started after I changed in LDAP lookup configuration from "scope = subtree" to "scope = onelevel", because it did not occur before (I did no other changes). Now, the above command only lists 12 accounts. If I query for a particular user: doveadm quota get -u userx this works fine, but userx (and all users except those 12) is NOT listed when trying to display all users. How can I run the above command without problems again? Thanks in advance, Nick

3 12

[Dovecot] Quota PostgreSQL INSERT trigger
by Adis Nezirovic 20 Dec '11

20 Dec '11

Wiki instructions for creating insert trigger for PostgreSQL are wrong http://wiki.dovecot.org/Quota/Dict Calling INSERT on the table from within BEFORE INSERT trigger creates cascading trigger. http://www.postgresql.org/docs/current/interactive/trigger-definition.html Instead, one should return NEW record from the trigger, and it will be inserted after trigger execution. --- CREATE OR REPLACE FUNCTION merge_quota2() RETURNS TRIGGER AS $$ BEGIN IF NEW.messages < 0 OR NEW.messages IS NULL THEN -- ugly kludge: we came here from this function, really do try to insert IF NEW.messages IS NULL THEN NEW.messages = 0; ELSE NEW.messages = -NEW.messages; END IF; return NEW; END IF; LOOP UPDATE quota2 SET bytes = bytes + NEW.bytes, messages = messages + NEW.messages WHERE username = NEW.username; IF found THEN RETURN NULL; END IF; BEGIN IF NEW.messages = 0 THEN RETURN NEW; ELSE NEW.messages = - NEW.messages; RETURN NEW; END IF; EXCEPTION WHEN unique_violation THEN -- someone just inserted the record, update it END; END LOOP; END; $$ LANGUAGE plpgsql; DROP TRIGGER IF EXISTS mergequota2 ON quota2; CREATE TRIGGER mergequota2 BEFORE INSERT ON quota2 FOR EACH ROW EXECUTE PROCEDURE merge_quota2(); --- Best regards, Adis

2 1

[Dovecot] Tuning Login processes for High-performance Mode
by Alessio Cecchi 20 Dec '11

20 Dec '11

Hi, I'm working for tuning a busy cluster of dovecot servers. Heach server as 4 core, 8GB of RAM and provide POP3 and IMAP access to mailbox. OS is Debian 6.0 and dovecot version is 1.2.15. I am unsure how to set "login_max_processes_count" and "login_max_connections" for reach the best situation when login_process_per_connection is set to no. I don't know what value to set login_max_processes_count. For example is better the first or the second configuration? login_process_per_connection=no login_processes_count=4 login_max_processes_count=8 login_max_connections=512 login_process_size=128 login_process_per_connection=no login_processes_count=4 login_max_processes_count=16 login_max_connections=256 login_process_size=128 Which is a suggested ratio from login_max_processes_count and login_max_connections ? Other values are correct? Thanks -- Alessio Cecchi is: @ ILS -> http://www.linux.it/~alessice/ on LinkedIn -> http://www.linkedin.com/in/alessice Assistenza Sistemi GNU/Linux -> http://www.cecchi.biz/ @ PLUG -> ex-Presidente, adesso senatore a vita, http://www.prato.linux.it @ LOLUG -> Socio http://www.lolug.net

2 4

[Dovecot] Calculation of suggested vsz_limit values
by Nikolaos Milas 20 Dec '11

20 Dec '11

Hello, I would like to ask on suggested vsz_limit values. Is there any way of calculating such values? How should we determine them? I don't want to risk wrong values (and possible errors) on a production system, if I upgrade to 2.0.16 where vsz_limit values are enforced (I read release notes)! If there is no standard way of calculation, could someone please check the following info and suggest some values? I am running dovecot-2.0.13-1_128.el5 x86_64 RPM on CentOS 5.7 (as a VM on a KVM Host). All accounts are virtual, hosted on LDAP Server. We have about 250 mailboxes of 4GB each (in practice, only a couple of mailboxes ever reach 1GB so we are "over-booking"). It is a relatively low-load server. Memory status and config follows below. Thanks, Nick The server memory condition is as follows: # vmstat procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 92 62400 212324 1299604 0 0 397 235 70 39 4 2 89 4 0 # free -m total used free shared buffers cached Mem: 2010 1951 59 0 207 1269 -/+ buffers/cache: 474 1536 Swap: 2015 0 2015 ---------- Config Start ------------- # 2.0.13: dovecot.conf protocols = imap pop3 mail_location = maildir:~/Maildir/ mail_gid = 500 mail_uid = 500 auth_mechanisms = plain login auth_username_format = %Lu auth_verbose = yes disable_plaintext_auth = no mail_plugins = quota protocol imap { imap_client_workarounds = "delay-newmail " mail_plugins = quota imap_quota } protocol pop3 { mail_max_userip_connections = 3 mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } protocol lda { auth_socket_path = /var/run/dovecot/auth-master info_log_path = log_path = mail_plugins = quota postmaster_address = sysadmin(a)example.com sendmail_path = /usr/lib/sendmail } userdb { args = /etc/dovecot/dovecot-usrdb-ldap.conf driver = ldap } passdb { args = /etc/dovecot/dovecot-passdb-ldap.conf driver = ldap } plugin { quota = maildir:User quota quota_rule = *:storage=4G quota_rule2 = Trash:storage=+3%% quota_warning = storage=75%% quota-warning 75 %u quota_warning2 = storage=90%% quota-warning 90 %u } service quota-warning { executable = script /opt/mail1.sh user = vmail unix_listener quota-warning { user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { group = vmail mode = 0660 user = vmail } user = root } service imap-login { service_count = 1 vsz_limit = 64 M } service pop3-login { service_count = 1 vsz_limit = 64 M } ssl_ca = </etc/pki/CA/certs/chain.pem ssl_cert = </etc/pki/CA/certs/cert.pem ssl_key = </etc/pki/CA/private/key.pem syslog_facility = local1 ---------- Config End -------------

3 4

[Dovecot] Strange Login / Timeout Sessions
by Carlos Mennens 20 Dec '11

20 Dec '11

I have my webmail application tied to Postfix / Dovecot server & now for some reason the IMAP sessions oddly time out throughout the day and then logging back into the mail system over the web is painfully slow. The only thing I've seen that fixes it is to reload the Dovecot daemon in Linux: /etc/init.d/dovecot reload Once that command is ran, my webmail login is super fast and moving from folder to folder is extremely fast again. Then in about a few hours, the session will time out and this problem repeats. The problem is the same for everyone and not just a few user accounts. Anyone have any idea what I should be looking for or looking at? My Dovecot logs or mixed in /var/log/maillog which is extremely hard to watch live & gets mixed with Postfix, Amavisd, and Postgrey. Any ideas or suggestions to narrow this problem out? I'm running Dovecot-1.0.7-7.el5_7.1. I saw this load on the console today: Error: Dovecot is already running with PID 2334 (read from /var/run/dovecot/master.pid) Fatal: Invalid configuration in /etc/dovecot.conf Below is my configuration file: [root@mail ~]# dovecot -n # 1.0.7: /etc/dovecot.conf protocols: imap ssl_ca_file: /etc/ssl/certs/digicert.crt ssl_cert_file: ***OMIT*** ssl_key_file: ***OMIT*** login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login mail_location: maildir:~/Maildir auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd socket: type: listen client: path: /var/spool/postfix/private/auth mode: 432 user: postfix group: postfix

2 1

[Dovecot] One important question!
by I M 19 Dec '11

19 Dec '11

Hello!Please, give me a feedback as soon as you can! I need some advice regarding "dovecot self signed certificate" configuration!If you have any experience in working with php IMAP-module that would be useful for me to know what would you say about below: Problem: I can connect to my INBOX through telnet, but I cannot do it through imap-functions (Certificate failure for ...)? My best,Edward.

3 3

[Dovecot] Two important new Pigeonhole v0.3 features
by Stephan Bosch 19 Dec '11

19 Dec '11

Hello Dovecot users, Pigeonhole v0.3 for Dovecot v2.1 has two new features I would like you to play with before I make a first release: 1) The new Sieve editheader extension (http://tools.ietf.org/html/rfc5293) This allows modifying the headers of the message before delivery. This is actually the first implemented extension that can modify the message itself in any way. This feature adds quite a bit of new code that I'd like to see tested more thoroughly. 2) The new sieve-filter tool (documented in the sieve-filter(1) man page upon installation). This allows (re-)filtering messages that are already stored in a particular mailbox. It preserves the assigned IMAP flags and keywords and using the imap4flags extension these can also be modified. It also plays nice with the editheader extension. Judging by postings in this mailing list from the last few years, this is a highly-demanded feature and I'm glad that I finally got around to finish this thing. If the new sieve-filter tool is lacking certain features in your opinion, just let me know. Needless to say: read the man page carefully and watch out with important mail. This is currently only available from the mercurial repository, but if you prefer to have an .rc1 release first, just tell me and I'll make one. The automatic Debian repository already includes these features. Regards, Stephan.

3 4