July 2014 - dovecot - dovecot.org

Dovecot mailstore performance tuning
by Murray Trainer 29 Jul '14

29 Jul '14

HI All, We have a couple of dovecot director proxies and six backed mailstores each accessing mailboxes stored on five NFSv4 filsystems with about 1TB of mail on each in maildir format. We have about 800 max users on each mailstore at peak times and performance appears to starting to degrade at these times. The mailstores are pretty recent hardware with 64GB of RAM and 24 cores. The NFS storage is EMC VNX and we are doing about 250 I/O per sec upto max of 500 on each filesystem. I need to squeeze more performance out of these servers whether that is in the NFS client, Dovecot or Linux OS/kernel areas. We use LDAP for auth and I am doing some tuning in that area. The NFS filesystems are mounted with the options below: 10.11.0.238:/mailbox_store_01 on /home1 type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,nordirplus,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.11.0.96,local_lock=none,addr=10.11.0.238) My dovecot config is below. I am not seeing any obvious issues in the server logs. Any suggestions on how to improve performance would be appreciated. Thanks Murray # doveconf -n # 2.2.9: /etc/dovecot/dovecot.conf doveconf: Warning: service auth { client_limit=40960 } is lower than required under max. load (41280) doveconf: Warning: service anvil { client_limit=40970 } is lower than required under max. load (41183) # OS: Linux 3.13-0.bpo.1-amd64 x86_64 Debian 7.5 auth_cache_size = 64 M auth_cache_ttl = 2 hours auth_failure_delay = 0 auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!#$-=?^_{}~./@+%" auth_username_translation = +@ auth_worker_max_count = 256 base_dir = /var/run/dovecot/ disable_plaintext_auth = no first_valid_gid = 1001 first_valid_uid = 1001 mail_fsync = always mail_location = maildir:~/ managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mmap_disable = yes namespace { inbox = yes location = prefix = INBOX. type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile quota = maildir quota_rule = INBOX.Trash:ignore } protocols = " imap lmtp sieve pop3" service anvil { client_limit = 40970 } service auth-worker { user = dovecot } service auth { client_limit = 40960 unix_listener auth-userdb { group = mail mode = 0666 user = dovecot } } service imap-login { chroot = login inet_listener imap { address = *, [::] port = 143 } inet_listener imaps { address = * port = 993 ssl = yes } process_limit = 20480 process_min_avail = 32 service_count = 1 user = dovecot vsz_limit = 256 M } service imap { process_limit = 40960 vsz_limit = 256 M } service lmtp { inet_listener lmtp { address = 27.54.95.41 port = 24 } process_min_avail = 32 } service managesieve-login { client_limit = 40960 process_limit = 120 process_min_avail = 5 service_count = 0 vsz_limit = 64 M } service managesieve { process_limit = 4096 vsz_limit = 256 M } service pop3-login { chroot = login inet_listener pop3 { address = *, [::] port = 110 } inet_listener pop3s { address = * port = 995 ssl = yes } process_limit = 20480 process_min_avail = 32 service_count = 1 user = dovecot vsz_limit = 256 M } service pop3 { process_limit = 40960 vsz_limit = 256 M } ssl_cert =

2 1

Re: identify MUA connecting?
by Joseph Tam 29 Jul '14

29 Jul '14

Juan Pablo <juanpablo83(a)openmailbox.org> writes: > That is what I was not sure. I did not know what information was > possible available. I was thinking or hoping something same as http logs My dovecot 2 installation logs this information. E.g. Jul 26 07:13:02 server dovecot: imap(user): ID sent: name=iPad Mail, version=11D201, os=iOS, os-version=7.1.1 (11D201) Jul 26 07:13:38 server dovecot: imap-login: ID sent: name=com.google.android.email, os=android, os-version=4.4.4; KTU84P, vendor=LGE, x-android-device-model=Nexus 4, x-android-mobile-net-operator=Koodo, AGUID=...: user=<>, rip={client-ip}, lip={server-ip}, TLS, session=<BH7SSxn/CQBFrJG2> If you want this info, upgrade, as a later poster suggests. Joseph Tam <jtam.home(a)gmail.com>

1 0

Dovecot pigeonhole help sought
by Timothy Murphy 29 Jul '14

29 Jul '14

I'm running postfix + dovecot + dovecot-pigeonhole on my CentOS-7 home server. I would like spam to finish up in ~/Maildir/.Spam/ . Spam is being marked, but is not separated - it ends up with all the other email in ~/Maildir/cur/ . Evidently there is some step I have omitted to take. I give the output of "sudo doveconf -n" below. The file ~/.dovecot.sieve reads: ------------------------------ # Sieve Filter require ["fileinto","regex","envelope","vacation"]; if header :contains "X-Spam-Flag" "YES" { fileinto "/home/tim/Maildir/.Spam/"; } ------------------------------ When I run "sudo systemctl restart dovecot" I see the line Jul 27 18:54:23 alfred dovecot: master: Dovecot v2.2.10 starting up for imap, pop3, lmtp, sieve (core dumps disabled) in /var/log/maillog Output of "sudo doveconf -n > dovecot.conf" ------------------------------ # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-123.4.4.el7.x86_64 x86_64 CentOS Linux release 7.0.1406 (Core) listen = * mail_debug = yes mail_location = maildir:~/Maildir managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { driver = pam } plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } protocols = imap pop3 lmtp sieve ssl = required ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { driver = passwd } protocol lmtp { mail_plugins = " sieve" } protocol lda { mail_plugins = " sieve" } ------------------------------ Any advice or suggestions gratefully received.

5 4

Error after Upgrade
by Jim Knuth 29 Jul '14

29 Jul '14

Hello, after upgrade (Debian wheezy) from dovecot 2:2.2.13-1~auto+103 => 2:2.2.13-1~auto+113 I discovered that in my log and all mails stuck in the Queue. I’ve had downgrade v 2:2.2.13-1~auto+103 an everything is fine. What’s wrong? Any ideas or solution greatly appreciated. Thanks. Jul 25 11:03:01 server2 dovecot: lmtp(25638): Fatal: master: service(lmtp): child 25638 killed with signal 11 (core dumps disabled) Jul 25 11:03:01 server2 dovecot: lmtp(25639): Connect from local Jul 25 11:03:01 server2 dovecot: lmtp(25639): Fatal: master: service(lmtp): child 25639 killed with signal 11 (core dumps disabled) Jul 25 11:03:01 server2 postfix/lmtp[25617]: 20773D10003: to=<web7p1(a)server1.art-domains.de>, orig_to=<jk(a)jkart.de>, relay=server1.art-domains.de[private/dovecot-lmtp], delay=0.05, delays=0.03/ 0/0.01/0.02, dsn=4.4.2, status=deferred (lost connection with server1.art-domains.de[private/dovecot-lmtp] while sending end of data -- message may be sent more than once) Jul 25 11:03:01 server2 amavis[25541]: (25541-01) Passed CLEAN {RelayedInbound}, [213.203.238.6]:53401 [193.175.143.182] <postfixbuch-users-bounces(a)listen.jpberlin.de> -> <jk(a)jkart.de>, Queue-I D: 62B8BD10001, Message-ID: <53D21B91.2070109(a)fh-landshut.de>, mail_id: AE71HPGl9ZR5, Hits: -5, size: 13518, queued_as: 20773D10003, 2771 ms Jul 25 11:03:01 server2 amavis[25541]: (25541-01) Passed CLEAN, <postfixbuch-users-bounces(a)listen.jpberlin.de> -> <jk(a)jkart.de>, Hits: -5, tag=-9999.9, tag2=2.5, kill=3, queued_as: 20773D10003, L/Y/0/0 Jul 25 11:03:01 server2 postfix/lmtp[25607]: 62B8BD10001: to=<jk(a)jkart.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=3.9, delays=1.2/0/0.01/2.8, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[ 127.0.0.1]:10025): 250 2.0.0 Ok: queued as 20773D10003) Jul 25 11:03:01 server2 postfix/qmgr[25508]: 62B8BD10001: removed Jul 25 11:03:02 server2 postfix/lmtp[25615]: 20773D10003: to=<web1p9(a)server1.art-domains.de>, orig_to=<admin(a)art-domains.de>, relay=server1.art-domains.de[private/dovecot-lmtp], delay=0.06, del ays=0.03/0/0.03/0.01, dsn=4.4.2, status=deferred (lost connection with server1.art-domains.de[private/dovecot-lmtp] while sending end of data -- message may be sent more than once) Jul 25 11:03:02 server2 dovecot: lmtp(25653): Connect from local Jul 25 11:03:02 server2 dovecot: lmtp(25653): Fatal: master: service(lmtp): child 25653 killed with signal 11 (core dumps disabled) -- Mit freundlichen Grüßen, with kind regards, Jim Knuth --------- Die Freiheit der Rede hat den Nachteil, daß immer wieder Dummes, Häßliches und Bösartiges gesagt wird. Wenn wir aber alles in allem nehmen, sind wir doch eher bereit, uns damit abzufinden, als sie abzuschaffen. [Churchill]

3 5

Re: "Corrupted dbox file [...] purging found mismatched offsets"
by Jesus Cea 29 Jul '14

29 Jul '14

On 29/07/14 01:21, Daniel Parthey wrote: > Are you using dovecot director and a shared filesystem like NFS? > > You not only need to run imap and pop3 service via director, > but also the doveadm service, otherwise different hosts might > access the mdbox simultaneously, which will corrupt your index > files and you will lose flags. No, this is a dovecot server running in my own laptop. I guess some computer crash left the mail in an inconsistent state, and I need to assert the extend of the problem. A well designed system should not lose email, even when crashes are involved. -- Jesús Cea Avión _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz

1 0

Segfault dovecot 2.2.9
by Simon Gareste 29 Jul '14

29 Jul '14

Hi all, When trying to set up dovecot, I somehow managed to get a segfault. Trying to launch dovecot from command line (simply execute 'dovecot') results in the error message: Segmentation fault (core dumped) Trying dovecot -n results in pretty much the same: # 2.2.9: /etc/dovecot/dovecot.conf Segmentation fault (core dumped) Looking at syslog, I see doveconf[3676]: segfault at 200 ip 00007fbb93c4fcb3 sp 00007fff005b4c10 error 4 in libc-2.19.so[7fbb93c04000+1bc000] I tried but couldn't find the core dump anywhere on the server. Any help on this would be appreciated too. I (trying to ) run it with postfix, on an Ubuntu 14.04 (Linux mail2.arturia.com 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux), brand new server. What troubles me even more is that I just set up successfully another mail server a week ago, with the same configuration (diffing the conf folders) except for log or mailbox paths (driver and stuff are the same). In the google search I've done, I saw some stuff about openssl: both servers are running 1.0.1f, since one is running fine, I guess this is not the culprit. One difference I noticed between the two servers is that on the failing one, there is no /var/run/dovecot. I presume this folder and its files are created once dovecot is launched, so that's why it's not present, and is only a symptom of the failure, not a cause? Thanks for your help, and please tell me what other information I could provide to help. Simon

3 5

Multiple passwords with sql authentication
by BlackVoid 29 Jul '14

29 Jul '14

I'm currently working on a control panel which is using postfix, dovecot and other applications and I want to add application specific passwords to increase security. I found one solution [1], however it requires the password to be included in the query which is something I do not want to do, because the query may be written in clear-text to log-files. So I'm wondering if there is a way to have multiple passwords with dovecot without risking passwords being leakied in clear-text to log-files. [1] http://wiki2.dovecot.org/AuthDatabase/SQL#Password_verification_by_SQL_serv…

4 11

doveadm json formatter
by James Devine 29 Jul '14

29 Jul '14

Any thoughts to adding a json formatter to the doveadm output?

1 0

Migration from mbox/maildir++ to mdbox
by Nima Saed-Samii 28 Jul '14

28 Jul '14

Dear list, dovecot version: 2.1.7 I have an old mail server that uses mboxes for the INBOX and maildirs for subfolders. I want to migrate our mail handling to a new setup using mdboxes for everything. The current config looks like this: # 2.1.7: /etc/dovecot/dovecot.conf doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (1127) # OS: Linux 3.2.0-0.bpo.4-686-pae i686 Debian 6.0.10 auth_mechanisms = plain login disable_plaintext_auth = no first_valid_uid = 114 log_path = /var/log/dovecot.log log_timestamp = "%Y-%m-%d %H:%M:%S " mail_location = mbox:~/Mail/MBox:INBOX=/var/mail/%u mail_max_userip_connections = 30 mail_privileged_group = mail namespace { hidden = yes inbox = yes list = no location = prefix = "#mbox/" separator = / } namespace { location = maildir:~/Mail/Maildir prefix = separator = / } passdb { args = dovecot driver = pam } protocols = imap service auth { client_limit = 1024 user = root } service imap-login { inet_listener imap { port = 0 } process_limit = 1024 } ssl_cert = </etc/ssl/certs/cert-imap.ikp.pem ssl_key = </etc/ssl/private/key-imap.ikp.pem ssl_key_password = ********* userdb { driver = passwd } I would like to use dsync to migrate all to mdboxes and transfer them to the new system. Unfortunately I can't get dsync to do that. Using a config that specifies only one namespace allows me to convert either mboxes or Maildirs to mdboxes but not both. Using the above config dsync somehow thinks the source and destination are the same: Syncing mailbox for user nima (136 MB) Setup done. doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so doveadm(nima): Debug: Effective uid=5224, gid=1162, home=/home/nima doveadm(nima): Debug: Namespace mboxes: type=private, prefix=#mbox/, sep=/, inbox=yes, hidden=no, \ list=yes, subscriptions=yes location=mbox:~/Mail/MBox:INBOX=/var/mail/nima doveadm(nima): Debug: fs: root=/home/nima/Mail/MBox, index=, control=, inbox=/var/mail/nima, alt= doveadm(nima): Debug: Namespace maildirs: type=private, prefix=, sep=/, inbox=no, hidden=no, list=yes, \ subscriptions=yes location=maildir:~/Mail/Maildir doveadm(nima): Debug: maildir++: root=/home/nima/Mail/Maildir, index=, control=, inbox=, alt= dsync(nima): Debug: Effective uid=5224, gid=1162, home=/home/nima dsync(nima): Debug: Namespace mboxes: type=private, prefix=#mbox/, sep=/, inbox=yes, hidden=no, list=yes, \ subscriptions=yes location=mdbox:/scratch/mail-convert/nima dsync(nima): Debug: fs: root=/scratch/mail-convert/nima, index=, control=, inbox=, alt= dsync(nima): Debug: Namespace #mbox/: Using permissions from /scratch/mail-convert/nima: mode=0700 gid=-1 dsync(nima): Debug: Namespace maildirs: type=private, prefix=, sep=/, inbox=no, hidden=no, list=yes, \ subscriptions=yes location=maildir:~/Mail/Maildir dsync(nima): Debug: maildir++: root=/home/nima/Mail/Maildir, index=, control=, inbox=, alt= dsync(nima): Fatal: Both source and destination mail_location points to same directory: /home/nima/Mail/Maildir What am I doing wrong? Best regards, Nima Saed-Samii -- Institute for Nuclear Physics, University of Cologne Zuelpicher Str. 77 50937 Cologne Germany Tel.: +49-221-470-3617 Fax.: +49-221-470-5168 -- Institute for Nuclear Physics, University of Cologne Zuelpicher Str. 77 50937 Cologne Germany Tel.: +49-221-470-3617 Fax.: +49-221-470-5168

1 0

ACLs on Top level mailboxes.
by Dan Carroll 28 Jul '14

28 Jul '14

Hello, I’m in the process of migrating a mail server to new hardware. At the same time, I’ve made the decision to switch from courier to dovecot (v2.2.13) and from what I have seen so far I’m pretty happy. I have two questions for the list. I’d like my users mailboxes (Maildir format) to look like this: Inbox UserCreatedSubFolder1 UserCreatedSubFolder2 … Sent UserCreatedSubFolder3 Drafts Trash Junk Archives The Top level folders (Sent, Drafts etc) are all “Special” and I do not wish the users to be able to delete them. I also don’t want them to create new Top level folders. This will force all custom folders to be under a Top Level folder. The Maildir format currently looks like this: ~/Maildir/cur ~/Maildir/new ~/Maildir/tmp ~/Maildir/.INBOX/.UserCreatedSubFolder1/cur ~/Maildir/.INBOX/.UserCreatedSubFolder1/new ~/Maildir/.INBOX/.UserCreatedSubFolder1/tmp ~/Maildir/.Sent/cur ~/Maildir/.Sent/new ~/Maildir/.Sent/tmp etc…. I can get ACLs working with dovecot-acl files, and the permission I need (I think is) “owner lrwstipe”. But if I put it into the ~/Maildir directory, then it applies to Inbox. I’ve tried creating an .INBOX dir and putting it in there but that did not work. I also tried some global-acl file settings but I could not make that work either. Basically, I’m wondering how to apply ACLs to the Top level and have different ACLs for the actual inbox when using this scheme. I’m not opposed to changing the Maildir format, but that would mean a lot of work re-doing proemial scripts. My second question relates to migration. Is it enough that I copy the Maildir structure and simply delete the courier files or is it a lot better to use a courier->dovecot migration tool? I realise folder subscriptions would be lost but that’s OK, I’ll recreate them manually anyway. Thanks for your time. -Dan

2 1