January 2012 - dovecot

[Dovecot] doveadm + dsync merging
by Timo Sirainen 06 Jan '12

06 Jan '12

doveadm already supports some nice things, such as being able to remotely launch a doveadm command via TCP socket. It also supports executing a command for all users or to some specific users using a wildcard. dsync could use these features, so I merged dsync and doveadm into same binary for v2.1. I'll still install "dsync" symlink pointing to "doveadm", and running that way it should be fully backwards compatible with the old dsync binary and its parameters. I'm mainly now wondering about the command naming for running dsync via doveadm. Any suggestions? a) Use "doveadm dsync" prefix, and otherwise keep the names same: dsync mirror -> doveadm dsync mirror dsync backup -> doveadm dsync backup dsync server -> doveadm dsync server (for running dsync remotely via ssh/etc.) b) Don't have the dsync prefix: dsync mirror -> doveadm mirror dsync backup -> doveadm backup dsync server -> doveadm dsync-server (could be hidden from the doveadm commands list) c) Either a) or b), but rename "mirror" to "sync" or "dsync" or "replicate"? d) Something else?

5 5

[Dovecot] Deduplication active - but how good does it perform?
by Ralf Hildebrandt 06 Jan '12

06 Jan '12

I have deduplication active in my first mdbox: type mailbox, but how do I find out how well the deduplication works? Is there a way of finding out how much disk space I saved (if I saved some :) )? -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

4 4

[Dovecot] dovecot-lda error
by Wesley M. 05 Jan '12

05 Jan '12

Hi, I use Dovecot 2.0.13 on OpenBSD 5.0 When i try to send emails i have the following error in /var/log/maillog Jan 5 11:23:49 mail50 postfix/pipe[29423]: D951842244C: to=, relay=dovecot, delay=0.02, delays=0.01/0/0/0.01, dsn=5.3.0, status=bounced (command line usage error. Command output: deliver: unknown option -- n Usage: dovecot-lda [-c ] [-a ] [-d ] [-p ] [-f ] [-m ] [-e] [-k] ) Jan 5 11:23:49 mail50 postfix/qmgr[13787]: D951842244C: removed In my /etc/postfix/master.cf : # Dovecot LDA dovecot unix - n n - - pipe flags=ODR user=_dovecot:_dovecot argv=/usr/local/libexec/dovecot/deliver -f ${ sender} -d ${user}@${nexthop} -n -m ${extension} How can i resolve that ? Thank you very much for your replies. Cheers, Wesley.

2 1

[Dovecot] POP3 problems
by sottilette＠rfx.it 04 Jan '12

04 Jan '12

Migrated a 1.0.2 server to 2.0.16 (same old box). IMAP seems working Ok. POP3 give problems with some clients (Outlook 2010 and Thunderbird reported). Seems authentication problem Below my doveconf -n (debug enbled, but no answer found to the problems) Any hints? Thanks, P. # doveconf -n # 2.0.16: /etc/dovecot/dovecot.conf doveconf: Warning: NOTE: You can get a new clean config file with: doveconf -n > dovecot-new.conf doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:791: add auth_ prefix to all settings inside auth {} and remove the auth {} section completely doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:864: passdb {} has been replaced by passdb { driver= } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:935: userdb passwd {} has been replaced by userdb { driver=passwd } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:998: auth_user has been replaced by service auth { user } doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:1131: ssl_disable has been renamed to ssl # OS: Linux 2.6.9-42.0.10.ELsmp i686 CentOS release 4.9 (Final) auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login auth_verbose = yes disable_plaintext_auth = no info_log_path = /var/log/mail/dovecot.info.log listen = * log_path = /var/log/mail/dovecot.log mail_full_filesystem_access = yes mail_location = mbox:~/:INBOX=/var/mail/%u mbox_read_locks = dotlock fcntl passdb { driver = pam } protocols = pop3 imap service auth { user = root } ssl = no ssl_cert = /etc/pki/dovecot/certs/dovecot.pem ssl_key = /etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } userdb { driver = passwd } protocol lda { postmaster_address = postmaster(a)example.com }

4 4

[Dovecot] assertion failed in mail-index.c
by Attila Nagy 04 Jan '12

04 Jan '12

Hi, I have this: Jan 04 15:55:21 pop3(jfm47): Panic: file mail-index.c: line 257 (mail_index_keyword_lookup_or_create): assertion failed: (*keyword != '\0') Jan 04 15:55:21 master: Error: service(pop3): child 3391 killed with signal 6 (core not dumped - set service pop3 { drop_priv_before_exec=yes }) I don't know why this happened, but wouldn't be the "self healing" (seen in the wiki I think :) kick in here? I mean it's even better to completely remove the index than dying and make the mailbox inaccessible. Thanks,

2 1

[Dovecot] Small LOGIN_MAX_INBUF_SIZE for GSSAPI with samba4 (AD)
by Ludek Finstrle 04 Jan '12

04 Jan '12

Hello, I faced the problem with samba (AD) + mutt (gssapi) + dovecot (imap). From dovecot log: Jan 2 17:58:42 server dovecot: imap-login: Disconnected: Input buffer full (no auth attempts): rip=192.167.14.16, lip=192.167.14.16, secured My situation: CentOS 6.2 IMAP: dovecot --version: 2.0.9 (CentOS 6.2) MUA: mutt 1.5.20 (CentOS 6.2) Kerberos: samba4 4.0.0alpha17 as AD PDC $ klist -e Ticket cache: FILE:/tmp/krb5cc_1002_Mmg2Rc Default principal: luf@TEST Valid starting Expires Service principal 01/02/12 15:56:16 01/03/12 01:56:16 krbtgt/TEST@TEST renew until 01/03/12 01:56:16, Etype (skey, tkt): arcfour-hmac, arcfour-hmac 01/02/12 16:33:19 01/03/12 01:56:16 imap/server.test@TEST Etype (skey, tkt): arcfour-hmac, arcfour-hmac I fixed this problem with enlarging LOGIN_MAX_INBUF_SIZE. I also red about wrong lower/uppercase but it's not definitely my problem (I tried all possibilities of lower/uppercas in login). I sniffed the plain communication and the "a0000 AUTHENTICATE GSSAPI" line has around 1873 chars. When I enlarged the LOGIN_MAX_INBUF_SIZE to 2048 the problem disappeared and I'm now able to login to dovecot using gssapi in mutt client. I use also thunderbird (on windows with sspi) and it works ok with LOGIN_MAX_INBUF_SIZE = 1024. Does anybody have any idea why it's so large or how to fix it another way? It's terrible to patch each version of dovecot rpm package. Or is there any possibility to change constant? I have no idea how much this should affect memory usage. The simple patch I have to use is attached. Please cc: to me (luf at pzkagis dot cz) as I'm not member of the this list. Best regards, Ludek Finstrle

2 2

[Dovecot] doveadm move from one user's mailbox to another user's mailbox?
by Ralf Hildebrandt 04 Jan '12

04 Jan '12

Is something along the lines: doveadm move -u sourceuser destinationuser:/inbox search_query possible with 2.0.16? I want to move mails from a backup mailbox (which has no valid password assigned) to a "restore" mailbox (which *HAS* a password assigned to it). -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt(a)charite.de | http://www.charite.de

2 2

[Dovecot] migrate dovecot files 1.2.16 to 2.0.13 (OpenBSD 5.0)
by Wesley M. 04 Jan '12

04 Jan '12

Hi, I have a mailserver(Postfix+MySql) on OpenBSD 4.9 with Dovecot 1.2.16, all works fine. Now i want to do the same but on OpenBSD 5.0. I meet problems using dovecot 2.0.13 on OpenBSD 5.0. Some tests (on the box): telnet 127.0.0.1 110 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. Connection closed by foreign host. telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. Connection closed by foreign host. Seems that pop3/imap doesn't work 'netstat -anf inet' tcp 0 0 *.993 *.* LISTEN tcp 0 0 *.143 *.* LISTEN tcp 0 0 *.995 *.* LISTEN tcp 0 0 *.110 *.* LISTEN Therefore, ports are open. When i use Roundcube webmail, i have errors : error imap connection If someone can help me on. Thank you very much. Files to migrate (already tried to modify them) : dovecot.conf / dovecot-sql.conf / and 'dovecot -n ' ###############::::::::dovecot.conf:::::::::::################################# base_dir = /var/dovecot/ protocols = imap pop3 ssl_cert = /etc/ssl/dovecotcert.pem ssl_key = /etc/ssl/private/dovecot.pem ssl_cipher_list = HIGH:MEDIUM:+TLSv1:!SSLv2:+SSLv3 disable_plaintext_auth = yes default_login_user = _dovecot default_internal_user = _dovecot login_process_per_connection = no login_process_size = 64 mail_location = maildir:/var/mailserv/mail/%d/%n first_valid_uid = 1000 mmap_disable = yes protocol imap { mail_plugins = quota imap_quota autocreate imap_client_workarounds = delay-newmail } protocol pop3 { pop3_uidl_format = %08Xv%08Xu mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh } protocol lda { mail_plugins = sieve quota postmaster_address = postmaster(a)mailr130.localdomain sendmail_path = /usr/sbin/sendmail auth_socket_path = /var/run/dovecot-auth-master } auth default { mechanisms = plain login digest-md5 cram-md5 apop passdb { driver=sql args = /etc/dovecot/dovecot-sql.conf } userdb { driver=sql args = /etc/dovecot/dovecot-sql.conf } user = root socket listen { client { path = /var/spool/postfix/private/auth mode = 0660 user = _postfix group = _postfix } master { path = /var/run/dovecot-auth-master mode = 0600 user = _dovecot # User running Dovecot LDA group = _dovecot # Or alternatively mode 0660 + LDA user in this group } } } plugin { sieve=~/.dovecot.sieve sieve_storage=~/sieve } plugin { quota = maildir quota_rule = *:storage=5G quota_rule2 = Trash:storage=100M quota_warning = storage=95%% /usr/local/bin/quota-warning.sh 95 quota_warning2 = storage=80%% /usr/local/bin/quota-warning.sh 80 } plugin { autocreate = Trash autocreate2 = Spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = Spam autosubscribe3 = Sent autosubscribe4 = Drafts } plugin { antispam_signature = X-Spam-Flag antispam_signature_missing = move # move silently without training antispam_trash = trash;Trash;Deleted Items; Deleted Messages antispam_spam = SPAM;Spam;spam;Junk;junk antispam_mail_sendmail = /usr/local/bin/sa-learn antispam_mail_sendmail_args = --username=%u antispam_mail_spam = --spam antispam_mail_notspam = --ham antispam_mail_tmpdir = /tmp } ###############::::::::dovecot-sql.conf:::::::################################## driver = mysql connect = host=localhost dbname=mail user=postfix password=postfix default_pass_scheme = PLAIN password_query = SELECT email as user, password FROM users WHERE email = '%u' user_query = SELECT id as uid, id as gid, home, concat('*:storage=', quota, 'M') AS quota_rule FROM users WHERE email = '%u' ################### dovecot -n######################################## # 2.0.13: /etc/dovecot/dovecot.conf # OS: OpenBSD 5.0 i386 ffs auth_mechanisms = plain login digest-md5 cram-md5 apop base_dir = /var/dovecot/ default_internal_user = _dovecot default_login_user = _dovecot first_valid_uid = 1000 mail_location = maildir:/var/mailserv/mail/%d/%n mmap_disable = yes passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { antispam_mail_notspam = --ham antispam_mail_sendmail = /usr/local/bin/sa-learn antispam_mail_sendmail_args = --username=%u antispam_mail_spam = --spam antispam_mail_tmpdir = /tmp antispam_signature = X-Spam-Flag antispam_signature_missing = move antispam_spam = SPAM;Spam;spam;Junk;junk antispam_trash = trash;Trash;Deleted Items; Deleted Messages autocreate = Trash autocreate2 = Spam autocreate3 = Sent autocreate4 = Drafts autosubscribe = Trash autosubscribe2 = Spam autosubscribe3 = Sent autosubscribe4 = Drafts quota = maildir quota_rule = *:storage=5G quota_rule2 = Trash:storage=100M quota_warning = storage=95%% /usr/local/bin/quota-warning.sh 95 quota_warning2 = storage=80%% /usr/local/bin/quota-warning.sh 80 sieve = ~/.dovecot.sieve sieve_storage = ~/sieve } protocols = imap pop3 service auth { unix_listener /var/run/dovecot-auth-master { group = _dovecot mode = 0600 user = _dovecot } unix_listener /var/spool/postfix/private/auth { group = _postfix mode = 0660 user = _postfix } user = root } service imap-login { service_count = 0 vsz_limit = 64 M } service pop3-login { service_count = 0 vsz_limit = 64 M } ssl_cert = /etc/ssl/dovecotcert.pem ssl_cipher_list = HIGH:MEDIUM:+TLSv1:!SSLv2:+SSLv3 ssl_key = /etc/ssl/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { imap_client_workarounds = delay-newmail mail_plugins = quota imap_quota autocreate } protocol pop3 { mail_plugins = quota pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xv%08Xu } protocol lda { auth_socket_path = /var/run/dovecot-auth-master mail_plugins = sieve quota postmaster_address = postmaster(a)mailr130.localdomain sendmail_path = /usr/sbin/sendmail } Cheers, Wesley. M www.mouedine.net

1 0

[Dovecot] error bad file number with compressed mbox files
by Jürgen Obermann 04 Jan '12

04 Jan '12

Hello, can dsync convert from compressed mbox to compressed mdbox format? When I use compressed mbox files, either with gzip or with bzip2, I can read the mails as usual, but I find the following errors in dovecots log file: imap(userxy): Error: nfs_flush_fcntl: fcntl(/home/hrz/userxy/Mail/mymbox.gz, F_RDLCK) failed: Bad file number imap(userxy): Error: nfs_flush_fcntl: fcntl(/home/hrz/userxy/Mail/mymbox.bz2, F_RDLCK) failed: Bad file number These errors also appear when I use dsync to convert the compressed mbox to mdbox format on a second dovecot server: /opt/local/bin/dsync -v -u userxy backup mdbox:/sanpool/mail/home/hrz/userxy/mdbox dsync(userxy): Error: nfs_flush_fcntl: fcntl(/home/hrz/userxy/Mail/mymbox.gz, F_RDLCK) failed: Bad file number But now dovecot does not find the mails in the folder mymbox.gz on the second dovecot server in mdbox format! The relevant part of the dovcot configuration is: # 2.0.16: /opt/local/etc/dovecot/dovecot.conf # OS: SunOS 5.10 sun4v mail_fsync = always mail_location = mbox:~/Mail:INBOX=/var/mail/%u mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = mail_log notify zlib mmap_disable = yes Thank you, -- Jürgen Obermann Hochschulrechenzentrum der Justus-Liebig-Universität Gießen Heinrich-Buff-Ring 44 Tel. 0641-9913054

2 2

Re: [Dovecot] GPFS for mail-storage (Was: Re: Compressing existing maildirs)
by list＠airstreamcomm.net 04 Jan '12

04 Jan '12

Great information, thank you. Could you remark on GPFS services hosting mail storage over a WAN between two geographically separated data centers? ----- Reply message ----- From: "Jan-Frode Myklebust" <janfrode(a)tanso.net> To: "Stan Hoeppner" <stan(a)hardwarefreak.com> Cc: "Timo Sirainen" <tss(a)iki.fi>, <dovecot(a)dovecot.org> Subject: [Dovecot] GPFS for mail-storage (Was: Re: Compressing existing maildirs) Date: Tue, Jan 3, 2012 2:14 am On Sat, Dec 31, 2011 at 01:54:32AM -0600, Stan Hoeppner wrote: > Nice setup. I've mentioned GPFS for cluster use on this list before, > but I think you're the only operator to confirm using it. I'm sure > others would be interested in hearing of your first hand experience: > pros, cons, performance, etc. And a ball park figure on the licensing > costs, whether one can only use GPFS on IBM storage or if storage from > others vendors is allowed in the GPFS pool. I used to work for IBM, so I've been a bit uneasy about pushing GPFS too hard publicly, for risk of being accused of being biased. But I changed job in November, so now I'm only a satisfied customer :-) Pros: Extremely simple to configure and manage. Assuming root on all nodes can ssh freely, and port 1191/tcp is open between the nodes, these are the commands to create the cluster, create a NSD (network shared disks), and create a filesystem: # echo hostname1:manager-quorum > NodeFile # "manager" means this node can be selected as filesystem manager # echo hostname2:manager-quorum >> NodeFile # "quorum" means this node has a vote in the quorum selection # echo hostname3:manager-quorum >> NodeFile # all my nodes are usually the same, so they all have same roles. # mmcrcluster -n NodeFile -p $(hostname) -A ### sdb1 is either a local disk on hostname1 (in which case the other nodes will access it over tcp to ### hostname1), or a SAN-disk that they can access directly over FC/iSCSI. # echo sdb1:hostname1::dataAndMetadata:: > DescFile # This disk can be used for both data and metadata # mmcrnsd -F DescFile # mmstartup -A # starts GPFS services on all nodes # mmcrfs /gpfs1 gpfs1 -F DescFile # mount /gpfs1 You can add and remove disks from the filesystem, and change most settings without downtime. You can scale out your workload by adding more nodes (SAN attached or not), and scale out your disk performance by adding more disks on the fly. (IBM uses GPFS to create scale-out NAS solutions http://www-03.ibm.com/systems/storage/network/sonas/ , which highlights a few of the features available with GPFS) There's no problem running GPFS on other vendors disk systems. I've used Nexsan SATAboy earlier, for a HPC cluster. One can easily move from one disksystem to another without downtime. Cons: It has it's own page cache, staticly configured. So you don't get the "all available memory used for page caching" behaviour as you normally do on linux. There is a kernel module that needs to be rebuilt on every upgrade. It's a simple process, but it needs to be done and means we can't just run "yum update ; reboot" to upgrade. % export SHARKCLONEROOT=/usr/lpp/mmfs/src % cp /usr/lpp/mmfs/src/config/site.mcr.proto /usr/lpp/mmfs/src/config/site.mcr % vi /usr/lpp/mmfs/src/config/site.mcr # correct GPFS_ARCH, LINUX_DISTRIBUTION and LINUX_KERNEL_VERSION % cd /usr/lpp/mmfs/src/ ; make clean ; make World % su - root # export SHARKCLONEROOT=/usr/lpp/mmfs/src # cd /usr/lpp/mmfs/src/ ; make InstallImages > > To this point IIRC everyone here doing clusters is using NFS, GFS, or > OCFS. Each has its downsides, mostly because everyone is using maildir. > NFS has locking issues with shared dovecot index files. GFS and OCFS > have filesystem metadata performance issues. How does GPFS perform with > your maildir workload? Maildir is likely a worst case type workload for filesystems. Millions of tiny-tiny files, making all IO random, and getting minimal controller read cache utilized (unless you can cache all active files). So I've concluded that our performance issues are mostly design errors (and the fact that there were no better mail storage formats than maildir at the time these servers were implemented). I expect moving to mdbox will fix all our performance issues. I *think* GPFS is as good as it gets for maildir storage on clusterfs, but have no number to back that up ... Would be very interesting if we could somehow compare numbers for a few clusterfs'. I believe our main limitation in this setup is the iops we can get from the backend storage system. It's hard to balance the IO over enough RAID arrays (the fs is spread over 11 RAID5 arrays of 5 disks each), and we're always having hotspots. Right now two arrays are doing <100 iops, while others are doing 4-500 iops. Would very much like to replace it by something smarter where we can utilize SSDs for active data and something slower for stale data. GPFS can manage this by itself trough it's ILM interface, but we don't have the very fast storage to put in as tier-1. -jf

2 1