January 2010 - dovecot

[Dovecot] quick question
by David Halik 18 Feb '10

18 Feb '10

Timo (and anyone else who feels like chiming in), I was just wondering if you'd be able to tell me if the amount of corruption I see on a daily basis is what you consider "average" for our current setup and traffic. Now that we are no longer experiencing any core dumps with the latest patches since our migration from courier two months ago, I'd like to know what is expected as operational norms. Prior to this we had never used Dovecot, so I have nothing to go on. Our physical setup is 10 Centos 5.4 x86_64 IMAP/POP servers, all with the same NFS backend where the index, control, and Maildir's for the users reside. Accessing this are direct connections from clients, plus multiple squirrelmail webservers, and pine users, all at the same time with layer4 switch connection load balancing. Each server has an average of about 400 connections, for a total of around concurrent 4000 during a normal business day. This is out of a possible user population of about 15,000. All our dovecot servers syslog to one machine, and on average I see about 50-75 instances of file corruption per day. I'm not counting each line, since some instances of corruption generate a log message for each uid that's wrong. This is just me counting "user A was corrupted once at 10:00, user B was corrupted at 10:25" for example. Examples of the corruption are as follows: ########### Corrupted transaction log file ..../dovecot/.INBOX/dovecot.index.log seq 28: Invalid transaction log size (32692 vs 32800): ...../dovecot/.INBOX/dovecot.index.log (sync_offset=32692) Corrupted index cache file ...../dovecot/.Sent Messages/dovecot.index.cache: Corrupted physical size for uid=624: 0 != 53490263 Corrupted transaction log file ..../dovecot/.INBOX/dovecot.index.log seq 66: Unexpected garbage at EOF (sync_offset=21608) Corrupted transaction log file ...../dovecot/.Trash.RFA/dovecot.index.log seq 2: indexid changed 1264098644 -> 1264098664 (sync_offset=0) Corrupted index cache file ...../dovecot/.INBOX/dovecot.index.cache: invalid record size Corrupted index cache file ...../dovecot/.INBOX/dovecot.index.cache: field index too large (33 >= 19) Corrupted transaction log file ..../dovecot/.INBOX/dovecot.index.log seq 40: record size too small (type=0x0, offset=5788, size=0) (sync_offset=5812) ########## These are most of the unique messages I could find, although the majority are the same as the first two I posted. So, my question, is this normal for a setup such as ours? I've been arguing with my boss over this since the switch. My opinion is that with a setup such as ours where a user can be logged in using Thunderbird, Squirrelmail, and their Blackberry all concurrently at the same time, there will always be the occasional index/log corruption. Unfortunately, he is of the opinion that there should rarely be any and there is a design flaw in how Dovecot is designed to work with multiple services with an NFS backend. What has been your experience so far? Thanks, -Dave -- ================================ David Halik System Administrator OIT-CSS Rutgers University dhalik(a)jla.rutgers.edu ================================

7 44

[Dovecot] Anyone successfully setup Continous Backup of mailboxes using rsync ?
by CoolAtt NNA 18 Feb '10

18 Feb '10

Hi All.. Am working on continous backup of mailboxes using rsync(for e.g by running rsync every 2 min) Things gets more complicated when users create Subfolders in INBOX , SENT , etc.. If anyone among you did that plz guide & advise. Thanks CoolAtt _________________________________________________________________ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. https://signup.live.com/signup.aspx?id=60969

17 41

[Dovecot] Capability COMPRESS implemented?
by Ed W 15 Feb '10

15 Feb '10

Hi, I just read that http://tools.ietf.org/html/rfc4978 ..is being implemented in Thunderbird 3. This is great news! It's been a long time since we last discussed this and I don't think it's currently implemented in Dovecot, but since Cyrus just added it, any chance of getting support in Dovecot please? (Fastmail seems to be supporting this effort) On a similar note we had a previous thread about supporting TLS deflate - any chance you could investigate this again and add support on the Dovecot side so at least it would be feasible to submit a bug fix on the Thunderbird side? (It looked like a one liner change back when we last discussed it?) Thanks Ed W (Obviously for those looking for an external high performance compressor you can use our compressing email proxy from http://www.mailasail.com ...)

3 14

[Dovecot] Next message unexpectedly corrupted in mbox file
by Justin Piszcz 08 Feb '10

08 Feb '10

Hi, I am seeing this on one of my servers: Jan 8 10:08:58 l1 dovecot: IMuser(user): Next message unexpectedly corrupted in mbox file /home/user/Mailbox at 10558 I have never seen this before, and google yields no results except when this warning was added to the source code here: http://www.dovecot.org/list/dovecot-cvs/2009-December/015177.html The user has deleted all of the mail in the Mailbox but the problem persists. Any ideas? Justin.

4 9

[Dovecot] managesieve does call a wrong chown
by Wolfgang.Friebel＠desy.de 07 Feb '10

07 Feb '10

After upgrading from dovecot 1.2.3, managesieve 0.11.8, sieve 0.1.11 to dovecot 1.2.9, managesieve 0.11.10 and sieve 0.1.14 users who do not have sieve rules yet and use managesieve for the first time do trigger error messages in the log file (for the authenticated user ggg in group syspr): Jan 4 17:35:26 apollo dovecot: MANAGESIEVE(ggg): chown(/var/spool/sieve/ggg, -1, 0(root)) failed: Operation not permitted (egid=203(syspr)) Jan 4 17:35:26 apollo dovecot: MANAGESIEVE(ggg): sieve-storage: sieve_storage_verify_dir(/var/spool/sieve/ggg/tmp) failed: Operation not permitted Jan 4 17:35:26 apollo dovecot: MANAGESIEVE(ggg): Fatal: Failed to create sieve storage with data: /var/spool/sieve/ggg Anyway the filter rules get created (used avelsieve) and the filter does get compiled when the first mail hits the rule. Subsequent invocations of managesieve do not try to change the group ownership of the directory. Looks like an annoyance only, but I would feel more comfortable w.o. the error messages. the dovecot -n output: # 1.2.9: /etc/dovecot.conf # OS: Linux 2.6.18-164.6.1.el5 x86_64 Scientific Linux SL release 5.4 (Boron) protocols: imap imaps managesieve listen(default): * listen(imap): * listen(managesieve): *:2009 shutdown_clients: no login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login login_processes_count: 5 login_max_processes_count: 512 verbose_proctitle: yes mail_location: maildir:~/Maildir mmap_disable: yes fsync_disable: yes lock_method: dotlock maildir_copy_with_hardlinks: no mbox_read_locks: dotlock flock mbox_write_locks: dotlock flock mail_drop_priv_before_exec: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_plugins(default): quota imap_quota zlib acl imap_acl listescape mail_plugins(imap): quota imap_quota zlib acl imap_acl listescape mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(managesieve): /usr/lib64/dovecot/managesieve imap_client_workarounds(default): delay-newmail netscape-eoh tb-extra-mailbox-sep outlook-idle imap_client_workarounds(imap): delay-newmail netscape-eoh tb-extra-mailbox-sep outlook-idle imap_client_workarounds(managesieve): namespace: type: private separator: / location: maildir:~/Maildir inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: #mbox/ location: mbox:~/mail:CONTROL=~/mboxctrl hidden: yes list: no subscriptions: yes lda: postmaster_address: postmaster(a)ifh.de mail_plugins: sieve quota acl sieve_global_path: /var/spool/sieve/default.sieve quota_full_tempfail: yes auth_socket_path: /var/run/dovecot/auth-master fsync_disable: no auth default: mechanisms: gssapi gss-spnego login plain realms: IFH.DE DESY.DE cache_size: 32 passdb: driver: pam args: max_requests=1 cache_key=%u userdb: driver: passwd-file args: /etc/userdb socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 438 plugin: quota: maildir:User Quota quota_rule: *:storage=1000MB quota_rule2: INBOX:storage=1000MB quota_rule3: Trash:ignore acl: vfile:/etc/dovecot-acls:cache_secs=300 acl_shared_dict: file:/var/spool/shared/shared-mailboxes.db sieve_storage: /var/spool/sieve/%u sieve: /var/spool/sieve/%u.sieve quota_warning: storage=95%% /usr/local/bin/mailquota-warning.sh 95 quota_warning2: storage=80%% /usr/local/bin/mailquota-warning.sh 80 quota_warning3: storage=60%% /usr/local/bin/mailquota-warning.sh 60 -- Wolfgang Friebel Deutsches Elektronen-Synchrotron DESY Phone/Fax: +49 33762 77372/216 Platanenallee 6 Mail: Wolfgang.Friebel AT desy.de D-15738 Zeuthen Germany

3 5

[Dovecot] blocked for xx secs
by Andrew Hearn 06 Feb '10

06 Feb '10

Hello, Occasionally (a few times a day) we get batches of Warnings in the log such as: dovecot: Warning: chdir(/maildir/x.co.uk/users/y) blocked for 18 secs This tends to last a minute or so for different users. /maildir is a NFS mount, and I'm suspecting it's the NFS server being busy and dovecot having to wait. Other than replacing the NFS server with faster hardware is there anything else we can do? Indexes are local, and Maildir is on the NFS. Dovecot 1.2.5 using: mail_nfs_storage = yes mmap_disable = no NFS mounted as; /maildir on /.automount/a.diskless/root/mail type nfs (nosuid,nodev,noatime,vers=3,proto=tcp) TIA -- Andrew

2 1

[Dovecot] dovecot-auth stops responding
by Jonathan Siegle 05 Feb '10

05 Feb '10

Running dovecot 1.2.4(imap only) on AIX out of inetd. I accept passwords(PAM for auth) and kerberos tickets. Occasionally we see dovecot-auth stop responding. I just kill it off and another process is spawned which works fine. It doesn't look like it is hitting any ulimits(11MB for memory). I issued a kill -11 and ran it through the debugger. Here is what it says: dbx dovecot-auth core Type 'help' for help. [using memory image in core] reading symbolic information ... Segmentation fault in __fd_poll at 0x900000000117634 ($t1) 0x900000000117634 (__fd_poll+0x98) e8410028 ld r2,0x28(r1) (dbx) where __fd_poll(??, ??, ??) at 0x900000000117634 ioloop-poll.poll(__listptr = 0x00000001104dd330, __nfds = 145, __timeout = 1051), line 121 in "poll.h" io_loop_handler_run(ioloop = 0x000000011001bfb0), line 157 in "ioloop- poll.c" io_loop_run(ioloop = 0x000000011001bfb0), line 335 in "ioloop.c" main(argc = 1, argv = 0x0ffffffffffffad8), line 347 in "main.c" Here are my ulimits: ulimit -a time(seconds) unlimited file(blocks) 2097151 data(kbytes) 1048576 stack(kbytes) 65536 memory(kbytes) 65536 coredump(blocks) 2097151 nofiles(descriptors) 32768 Any help would be appreciated. Thanks, Jonathan

3 12

[Dovecot] Fatal: chdir( ) failed: Permission denied
by Vallo Kallaste 05 Feb '10

05 Feb '10

Hi Dovecot fails to chdir() into users home directory giving the following error: Oct 9 16:43:49 jura dovecot: dovecot: Fatal: chdir(/a/home/mina) failed: Permission denied (euid=1118 0(mina) egid=11332(devel) missing +x perm: /a/home) Oct 9 16:43:49 jura dovecot: dovecot: child 14441 (imap) returned error 89 (Fatal failure) The reason seems to be that user primary group is "devel" and directory /a/home has ACL which gives execute permission only to the users in "domain users" group. Now the catch is that the user _is_ member of "domain users" group and is happily accessing his home directory over CIFS(Samba) for long time. Also, the user has no problem entering his home directory when using shell access. It just seems like the user lost membership of a bunch of supplementary groups. The group membership info is coming from M$ PDC via winbind and NSS. We have few users and can change the users primary group and/or relax the /a/home directory ACL, but overall this is not right. Shall I shut up or can we hope for a fix or at least explanation? Thanks jura:/a # ls -la drwxrws--T+ 3 root domain admins 18 Oct 8 15:10 home jura:/a # getfacl home/ # file: home # owner: root # group: domain\040admins user::rwx group::rwx group:domain\040users:--x mask::rwx other::--- -- Vallo

4 8

[Dovecot] v2.0.beta2 released
by Timo Sirainen 02 Feb '10

02 Feb '10

http://dovecot.org/releases/2.0/beta/dovecot-2.0.beta2.tar.gz http://dovecot.org/releases/2.0/beta/dovecot-2.0.beta2.tar.gz.sig I tried to keep the configuration and APIs compatible with v2.0.beta1, but unfortunately there were a few things that really had to be changed: 1. passdbs and userdbs have now a separate "driver" setting inside them. So for example if you had: userdb passwd { } You'll now need to use instead: userdb { driver = passwd } 2. Services now have default inet_listeners. You need to either remove your existing ones, or override the defaults by giving the listeners's name like shown in the new example master.conf. For example if you had: service imap-login { inet_listener { port = 1234 .. change it to: service imap-login { inet_listener imap { port = 1234 If you want to disable a default listener, you can set its port to 0. Some new features: - Added auth_verbose_passwords = no|plain|sha1. If authentication fails, this setting logs the used password. If you don't really need to know what the password itself was, but are more interested in knowing if the user is simply trying to use the wrong password every single time or if it's a brute force attack, you can set this to "sha1" and only the SHA1 of the password is logged. That's enough to know if the password is same or different between login attempts. - lazy-expunge: Support a single-namespace configuration. In this setup if mailbox has some expunged messages and then the whole mailbox gets deleted, they get merged into the same deleted mailbox. - protocol !imap { .. } adds configuration to all protocols except imap. - imap/pop3 proxying supports now async DNS lookups for hosts. There's a separate dns-client service doing the lookups using OS's own blocking resolver. - dovecot-config file changed in a way that it'll be easier to create external software linking against dovecot libraries. This made Pigeonhole's configure script a lot simpler. And of course there have been lots of fixes.

3 4

[Dovecot] Want to have some users with Maildir, some with mbox
by Stewart Dean 02 Feb '10

02 Feb '10

Want to have some users with Maildir, some with mbox, as I migrate from mbox format to maildir over the space of a month. After everyone is converted to maildir, I'll change the mail_location in dovecot.conf. During the interim, can I use the Custom mailbox location script (at the bottom of the Mail Location DC Wiki page)? > if [ -d $HOME/.maildir ]; then > export MAIL=maildir:$HOME/.maildir > else > export MAIL=mbox:$HOME/mail:INBOX=/var/mail/$USER > fi > Questions: 1) What does explicitly specifying MAIL this way override? The dovecot.conf mail_location variable? 2) My mail_location var is defined as: mail_location: mbox:~/mail:INBOX=/var/spool/mail/%u:INDEX=/var/dcindx/%u and my prospective maildir INBOX and mail folder root is to be ~/Maildir. So would I change the above to: > if [ -d $HOME/Maildir ]; then > export MAIL=maildir:$HOME/Maildir:INDEX=/var/dcindx/$USER > else > export MAIL=mbox:$HOME/mail:INBOX=/var/spool/mail/$USER: > INDEX=/var/dcindx/$USER > fi Question 3: When converting existing users from mbox to maildir, should I delete the index contents? Thanks in advance! dovecot -n output # 1.2.8: /usr/local/etc/dovecot.conf # OS: AIX 3 0001378F4C00 listen: *:143 ssl_listen: *:993 disable_plaintext_auth: no verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_processes_count: 12 login_max_processes_count: 774 max_mail_processes: 1280 mail_max_userip_connections: 12 verbose_proctitle: yes first_valid_uid: 200 mail_location: mbox:~/mail:INBOX=/var/spool/mail/%u:INDEX=/var/dcindx/%u mbox_write_locks: fcntl mbox_dirty_syncs: no lda: postmaster_address: postmaster(a)example.com auth default: passdb: driver: pam userdb: driver: passwd -- ==== Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows.... ==== Stewart Dean, Unix System Admin, Bard College, New York 12504 sdean(a)bard.edu voice: 845-758-7475, fax: 845-758-7035

3 9