January 2009 - dovecot

[Dovecot] Dovecot performance on GFS clustered filesystem
by Allen Belletti 06 Feb '09

06 Feb '09

Hello All, We are using Dovecot 1.1.3 to serve IMAP on a pair of clustered Postfix servers which share a fiber array via the GFS clustered filesystem. This all works very well for the most part, with the exception that certain operations are so inefficient on GFS that they generate significant I/O load and hurt performance. We are using the Maildir format on disk. We're also using Dovecot's deliver from Postfix to handle local delivery. As best I can determine, the worst problems occur when certain users with very large Inboxes (~10k messages) receive new mail and their client looks up information about that message. GFS doesn't seem to efficiently handle the large directories that contain folders like this. As a result, lots of I/O ops are generated and performance suffers for everyone. I am beginning to wonder if it might be more efficient to revert to the old mbox format, with one file per folder (plus whatever indices are creates.) It seems that this ought to work better with GFS which is geared toward smaller numbers of larger files. Is anyone on the list currently doing that? Alternately, any thoughts regarding tuning or other options would be appreciated. Thanks, Allen -- Allen Belletti allen(a)isye.gatech.edu 404-894-6221 Phone Industrial and Systems Engineering 404-385-2988 Fax Georgia Institute of Technology

3 4

[Dovecot] 1.1.6: PAM passdb/userdb (mis)configuration
by Oved Ben-Aroya 05 Feb '09

05 Feb '09

I'm sure I'm missing something obvious :-( Dovecot version 1.1.6, pam authentication via ldap (openldap). basicly, we use: passdb pam userdb passwd which work fine, except for Outlook/OL Express users that are asked for their password whenever they "send/receive"... We've had also "passdb shadow" that somehow "fixed" this but allowed also users with expired passwords to login :-( re-added for now, untill the correct configuration is achived). Here is the output of dovecot -n: # 1.1.6: /usr/local/etc/dovecot.conf # OS: Linux 2.6.9-55.ELsmp x86_64 Red Hat Enterprise Linux AS release 4 (Nahant Update 7) info_log_path: /var/log/dovecot protocols: imap imaps pop3 pop3s listen(default): * listen(imap): * listen(pop3): *:110 ssl_listen(default): ssl_listen(imap): ssl_listen(pop3): *:995 ssl_ca_file: /usr/local/etc/dovecot/certs/IPS-IPSCABUNDLE.CRT ssl_cert_file: /usr/local/etc/dovecot/certs/dovecot.pem ssl_key_file: /usr/local/etc/dovecot/private/dovecot.pem disable_plaintext_auth: no verbose_ssl: yes login_dir: /usr/local/var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login first_valid_uid: 50 mail_debug: yes mail_full_filesystem_access: yes mmap_disable: yes lock_method: dotlock mbox_read_locks: dotlock mbox_write_locks: dotlock mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): %08Xv%08Xu pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh namespace: type: private separator: / prefix: mail/ location: mbox:%h/mail list: yes subscriptions: yes namespace: type: private separator: / location: maildir:%h/Maildir:INDEX=/var/dovecot/index/%u:CONTROL=/var/dovecot/control/%u inbox: yes list: yes subscriptions: yes auth default: verbose: yes debug: yes worker_max_request_count: 10 passdb: driver: pam args: dovecot passdb: driver: shadow userdb: driver: passwd args: blocking=yes Thank you for your help. -- \Oved Dr. Oved Ben-Aroya, Head Unix group, Taub Computer Center, Technion Phone: +972 (4) 829 3688 FAX: +972 (4) 823 6212 oved(a)technion.ac.il PGP key at http://tx.technion.ac.il/~oved/pgp/pubkey PGP Key fingerprint: A9 52 46 04 E8 70 41 99 60 E3 DA 8F BA 39 C2 C8

2 4

[Dovecot] Script: Moving Maildirs between servers
by Peter Lindgren 05 Feb '09

05 Feb '09

Hi, I just migrated from dovecot 1.0.rc29 to 1.1.7, from Sparc platform to Intel, it went very well. In the process I merged folders and excluded folders from being copied. User rights on folders were kept/set. I thought I'd share my migration script with you folks. If there's an interest to put it on the wiki I can do that, but I'm not sure in what page. Use it as a starter if you need to do this yourselves. Prerequisites: * User names are local unix accounts * Old server's /home must be mounted with nfs * Structure and user names must match between the two systems * Source dovecot must be at least version 1.0.x * Destination dovecot must be 1.1.x * Only dovecot data folders and files are copied, neither dovecot indices nor Sieve scripts etc. Here it goes, have fun! /Peter #!/bin/sh # # Migrates dovecot Maildir folders and settings from older dovecot on # old server to new dovecot on new server. # Prerequisites: # * User names are local unix accounts # * Old server's /home must be mounted with nfs # * Structure and user names must match between the two systems # * Source dovecot must be at least version 1.0.x # * Destination dovecot must be 1.1.x # * Only dovecot data folders and files are copied, neither dovecot indices nor Sieve scripts etc. # # Originally created by Peter Lindgren <peter(a)norrskenkonsult.com> # Use this script at your own risk. No warranties whatsoever, of any kind! # Basic settings: The first one you must definitely change, it's where you have mounted the old server's home: SOURCEBASE=/mnt/casiopea-home DESTBASE=/home # Just an abbreviation: MD=Maildir # Change field separator, for find's sake as there might be folders with spaces: # (doesn't play well together with the for loop otherwise) OLDIFS=${IFS} IFS=' ' # User loop: # Depending on your source system's setup, # you probably have non-interactive users with home directories (one of them # is probably named _dovecot) that shall not be copied. Edit the username # exclusion list according to your needs. # for username in `ls $SOURCEBASE`; do { # This is your username exclude list: case $username in (maildir|_dovecot) continue;; esac # Here starts the real work: SOURCEDIR=$SOURCEBASE/$username/$MD DESTDIR=$DESTBASE/$username/$MD echo "================================" echo "Copying user "$username"..." echo "From "$SOURCEDIR" to "$DESTDIR":" # IMAP Folder loop: # Will create the Maildir folder too, the first echo $SOURCEDIR"/." does that. This way all the copying is in one place only. for sourceimap in `{ echo $SOURCEDIR"/." ; find $SOURCEDIR -type d -name \.\* -maxdepth 1 -print ; } | sort -f`; do { # Folder exlusion and special actions: # .Draft is replaced by .Drafts, so content of .Draft must be copied to .Drafts. # (Probably there were a client in use for a while that used .Draft instead of .Drafts.) # Junk mail folders with various names from clients using different languages shall not be copied. # .Trash shall not be copied either. # Folder .INBOX is a leftover from earlier versions of dovecot, and shall not be copied either. # Add your own exceptions to the list. Be sure to quote spaces and dots. basename=`basename $sourceimap` action=copy case $basename in (\.Junk|\.Junk\ E-mail|\.Correo\ electr\&APM-nico\ no\ deseado|\.Skr\&AOQ-ppost|\.Trash|\.INBOX) action=skip;; esac echo -n "$basename" "$action" if [ $action"X" == copy"X" ]; then { echo -n ":" destname=$basename # Merge contents of .Draft and .Drafts to .Drafts: if [ $basename"X" == .Draft"X" ]; then destname=${destname}"s"; fi destimap=${DESTDIR}/${destname} # If this is the Maildir folder (.), a clean desination name is needed for mkdir: if [ $basename"X" == ."X" ]; then destimap=${DESTDIR}; fi mkdir -m 700 "${destimap}" echo -n "." chown ${username} "${destimap}" echo -n "." chmod go-rx "${destimap}" echo -n "." cp -p "${sourceimap}/dovecot-uidlist" "${destimap}" echo -n "." cp -p "${sourceimap}/dovecot-keywords" "${destimap}" echo -n "." cp -pR "${sourceimap}/cur" "${destimap}" echo -n "." cp -pR "${sourceimap}/new" "${destimap}" echo -n "." cp -pR "${sourceimap}/tmp" "${destimap}" echo " done." } else { echo "" } fi }; done # Copy root folder files last in user loop, since the Maildir is created in the imap loop above: cp -p $SOURCEDIR/subscriptions $DESTDIR echo "User "${username}" done." }; done IFS=${OLDIFS} -- Peter Lindgren http://www.norrskenkonsult.com

7 11

[Dovecot] Dovecot on Mac - Entourage cannot fetch emails
by Bijo 03 Feb '09

03 Feb '09

Hi, Dovecot 1.1.3 is running on Mac OS X 10.5 latest version, with Maildir (LAYOUT=fs). I have following issues when Entourage 2008 connects it, over imap/imaps. Entourage does not cache the mails. After every restart the folder contents are fetched fresh. I have enabled the Entourage option to get entire messages. Entourage caches the gmail IMAP folders fine. Also Outlook 2007 does not have this problem while connecting to the same IMAP server. I have some big folders with a few thousands of emails. Entourage fails with Error: 4362 (Mail could not be retrieved this time) while fetching mails from these folders. Restarting and rebuilding the Entourage database did not fix this issue. Any tips to resolve these issues or to debug what is going on would be greatly appreciated. Thanks, Bijoy -- View this message in context: http://www.nabble.com/Dovecot-on-Mac---Entourage-cannot-fetch-emails-tp2175… Sent from the Dovecot mailing list archive at Nabble.com.

2 3

[Dovecot] Dovecot 1.1.10 and chroot
by Michal Soltys 03 Feb '09

03 Feb '09

As it's been a while since I upgraded (1.1rc10, jump to 1.1.10), this might be present in earlier versions as well. Anyway, if one enables mail_chroot in global config file, dovecot will try to access /tmp there. Per-user overrides behave correctly. For example, with: mail_chroot=/home/vmail dovecot will abort with: Fatal: chdir(/home/vmail/tmp) failed with uid 65534: No such file or directory Error: imap dump-capability process returned 89 Fatal: Invalid configuration in /etc/dovecot.conf

2 1

[Dovecot] Vacation: Missing file .dovecot.lda-dupes ?
by Tobias Balle-Petersen 03 Feb '09

03 Feb '09

Hello.... I'm using the vacation feature, but the vacation reply is sent when ever a mail is received. What I want is, that the reply should only be sent one time to each user. My understanding is, that a list of people who have received the vacation message should be stored in the file .dovecot.lda-dupes in the users folder. This file is not present in my users fould. I have tried creating the file, but this makes no difference. Any ideas? Output from dovecot -n: # 1.0.10: /etc/dovecot/dovecot.conf log_timestamp: %Y-%m-%d %H:%M:%S protocols: imaps imap listen: localhost:143 ssl_listen: *:993 ssl_cert_file: /etc/ssl/certs/XX.crt ssl_key_file: /etc/ssl/private/XX.key ssl_key_password: XXXXXXXXXXXXXXXXXXXX disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/lib/dovecot/imap-login login_greeting: System ready. login_process_per_connection: no valid_chroot_dirs: /home/vmail mail_extra_groups: mail mail_location: maildir:/home/vmail/%n/mail:INDEX=/var/indexes/%n maildir_copy_with_hardlinks: yes mail_plugins: quota imap_quota trash imap_client_workarounds: delay-newmail auth default: username_format: %n verbose: yes debug: yes passdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot/dovecot-ldap.conf socket: type: listen client: path: /var/spool/postfix/private/auth-client mode: 432 user: postfix group: vmail master: path: /var/run/dovecot/auth-master mode: 432 user: vmail group: vmail plugin: quota: maildir:storage=10485760 quota2: maildir:user quota trash: /etc/dovecot/dovecot-trash.conf sieve: /home/vmail/%n/.dovecot.sieve Thanks, Tobias

2 2

[Dovecot] Using dovecot sasl for authenticating in openldap
by Nathan Huesken 03 Feb '09

03 Feb '09

Hi, I am using dovecot imaps and exim, both authenticating using dovecot-sasl. To keep passwords in one place I would like to authenticate openldap users over dovecot-sasl. So looking at this page: http://www.openldap.org/doc/admin24/security.html I figure it could be done using Pass-Through authentication. For this, I need to create /usr/lib/sasl2/slapd.conf And adding a few config paremeters. This is a example mech_list: plain pwcheck_method: saslauthd saslauthd_path: /var/run/sasl2/mux I understand the mech_list parameter, but I have no Idea what to put in for the rest. Any help? Thanks! Nathan

2 2

[Dovecot] Differences when expire plugin and cronjob?
by Nicolas Letellier 02 Feb '09

02 Feb '09

Hello. I would like to see the differnces between the expire plugin and a cronjob which make a find/delete. Is there performances differences? What's the best to use? I use dovecot 1.1.8. Is expire plugin update the maildirsize and other indexes files? Thanks. Regards, -- -Nicolas.

4 5

[Dovecot] Dovecot-1.1.9 fails to work with Thunderbird
by David Favor 02 Feb '09

02 Feb '09

When Thunderbird starts up 1.1.8 log entries reflect: Jan 24 06:44:56 net1 dovecot: imap-login: Login: user=<yemiah(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:44:56 net1 dovecot: imap-login: Login: user=<david-xfer(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:44:56 net1 dovecot: imap-login: Login: user=<david-lf(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:44:56 net1 dovecot: imap-login: Login: user=<thea(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS When Thunderbird starts up with 1.1.9 server begins logging errors forever for every user of the form: Jan 24 06:34:29 net1 dovecot: uid specified multiple times for david-xfer(a)coolsurf.com Jan 24 06:34:29 net1 dovecot: imap-login: Internal login failure (auth failed, 1 attempts): user=<david-xfer(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:34:29 net1 dovecot: uid specified multiple times for david(a)coolsurf.com Jan 24 06:34:29 net1 dovecot: imap-login: Internal login failure (auth failed, 1 attempts): user=<david(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:34:29 net1 dovecot: uid specified multiple times for aaron(a)coolsurf.com Jan 24 06:34:29 net1 dovecot: imap-login: Internal login failure (auth failed, 1 attempts): user=<aaron(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:34:29 net1 dovecot: uid specified multiple times for yemiah(a)coolsurf.com Jan 24 06:34:29 net1 dovecot: imap-login: Internal login failure (auth failed, 1 attempts): user=<yemiah(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS Jan 24 06:34:29 net1 dovecot: uid specified multiple times for david-lf(a)coolsurf.com Jan 24 06:34:29 net1 dovecot: imap-login: Internal login failure (auth failed, 1 attempts): user=<david-lf(a)coolsurf.com>, method=PLAIN, rip=71.155.185.14, lip=69.94.137.124, TLS -- Love feeling your best ever, all day, every day? Click http://RadicalHealth.com/join for the easy way.

6 10

[Dovecot] Config rewrite for v1.3
by Timo Sirainen 02 Feb '09

02 Feb '09

Still broken, but I committed the current changes to http://hg.dovecot.org/dovecot-config-rewrite/ The idea is something like: - src/config/ contains a doveconf binary which can be used to dump the current configuration. It's the only binary that reads dovecot.conf directly. Currently the code is kind of ugly because I wanted it to be able to parse the existing v1.2 dovecot.conf which required some kludging around. Perhaps in v2.0 the dovecot.conf could have larger changes and these kludges could be removed. - master currently executes doveconf 3 times: for default settings, for imap settings, for pop3 settings. The result is saved into an array of KEY=value pairs in an array. Eventually they're still sent to child processes via environment. This sending via environment is again mainly for backwards compatibility with post-login scripts. Although they are useful there, so I'm not sure if it should be kept that way. Although post-login scripting doesn't really work anymore with Apple's multiple-connections-per-mail-process code, so perhaps that needs some rethinking in any case.. - deliver also currently forks and executes doveconf. I hate this. But I don't like the alternative of having a libconfig that deliver could use to read the configuration inside the binary as is done now, because then it's not as easy anymore to update the doveconf binary (or replace it entirely). But I'm thinking about making doveconf a long running process and deliver could ask configuration from it via UNIX socket if it's running and fallback to forking doveconf. The last possibility would be not to fork, but instead just have doveconf exec() deliver, so either something like "doveconf -e deliver -deliver-params" or if calling deliver directly it could exec() doveconf which would then exec() deliver back. Hmm. Perhaps this is actually better than forking? - %variable expansion is somewhat broken with imap/pop3, but works with deliver. The annoying thing about this is that %vars from doveconf should be expanded, but if userdb returns anything containing %vars they shouldn't be (because e.g. home directory could contain %). So either master should expand everything internally or it should somehow tell child processes about what to expand and what not. It can't really expand everything because it doesn't know all the settings that should be expanded. So that only leaves the possibility of telling child process what to expand. One possibility is to prefix each environment with e.g. 0 or 1 specifying if it should be expanded. This would break all existing postlogin scripts and in general would make it kind of ugly. Another possibility would be to have an environment that lists the settings that shouldn't be expanded. Perhaps that would work better, although if postlogin script changes/adds environments they'd also potentially have to modify that environment too. - Plugins will probably have to keep using something similar to getenv() to get their settings. It would be possible to have them use the nice new pre-filled settings structures, but that would require configuration changes. For example instead of quota=.. quota2=.. you'd have two quota {} blocks. Probably better to leave that change to v2.0. - I'm not entirely sure where all the setting verifications should go. Having them done by master when starting up is nice because it can give error messages immediately and write them to screen, instead of having to look them up from the log. Or when reloading configuration it can cancel it if the new configuration isn't ok. Maybe doveconf could do the checks. Maybe the verifications could go to all the different *settings.c files where they logically belong to, and then the build scripts would copy those functions to doveconf code where it could check all of them when the config is being read. The only annoying thing is that this then required doveconf to be run as root so that it can do all the access() etc. checks. But maybe that's not too bad.

2 3