- dovecot - dovecot.org

High memory usage
by Hanna Ruotsalainen 17 Oct '14

17 Oct '14

Hi I recently (about 2 weeks ago) moved over my users over to a new version of dovecot (2.0.9) and now the memory usage on the new server seems much higher than it was before. The new server is a CentOS 6 VM with 4 GB of RAM and 4 CPUs and according to top, it's used up all of the RAM and all of the Swap. Here's a sample of the output from top (sorted by allocated memory). top - 10:42:36 up 22 days, 21:57, 1 user, load average: 1.04, 1.40, 1.52 Tasks: 1643 total, 1 running, 1642 sleeping, 0 stopped, 0 zombie Cpu(s): 0.7%us, 0.7%sy, 0.0%ni, 93.3%id, 5.1%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3924432k total, 3779236k used, 145196k free, 23740k buffers Swap: 4063228k total, 3181164k used, 882064k free, 156420k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16738 root 20 0 399m 182m 436 S 0.0 4.8 5:50.58 config 18790 dovenull 20 0 54528 11m 2036 S 0.0 0.3 0:00.05 imap-login 22277 dovenull 20 0 54528 11m 2156 S 0.0 0.3 0:00.04 imap-login 22796 dovenull 20 0 54528 11m 2140 S 0.0 0.3 0:00.03 imap-login 22749 dovenull 20 0 54528 11m 2148 S 0.0 0.3 0:00.03 imap-login 22825 dovenull 20 0 54528 11m 2156 S 0.0 0.3 0:00.03 imap-login This is with 1015 imap processes and 484 imap-login processes. The old version was 1.0.rc15 and running on a CentOS 4 box with 2GB of RAM. It was overloaded but the RAM usage wasn't maxed out like this one is (not much swap usage and a large amount of cached memory). Is this amount of memory usage normal? What could be causing it, if it isn't? Thanks for your help, Hanna Here is the output of dovecot -n: # 2.0.9: /etc/dovecot/dovecot.conf # OS: Linux 2.6.32-431.29.2.el6.x86_64 x86_64 CentOS release 6.5 (Final) nfs auth_verbose = yes disable_plaintext_auth = no listen = * login_greeting = Ready. mail_debug = yes mail_location = maildir:/mail/%1Lu/%u:INDEX=/var/cache/dovecot/indexes/%1Lu/%u mail_plugins = " mail_log notify" maildir_very_dirty_syncs = yes mbox_lazy_writes = no namespace { inbox = yes location = prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-ldap-pass.conf driver = ldap } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size } protocols = imap pop3 service anvil { client_limit = 1250 } service auth { client_limit = 4300 user = root } service imap-login { process_limit = 600 service_count = 1 } service imap { process_limit = 1536 } service pop3-login { process_limit = 600 service_count = 1 } service pop3 { process_limit = 1536 } ssl_ca = </etc/pki/tls/certs/ca-bundle.crt ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = </etc/pki/dovecot/private/dovecot.pem userdb { args = /etc/dovecot/dovecot-ldap-user.conf driver = ldap } version_ignore = yes protocol imap { mail_max_userip_connections = 10 ssl_cert = </etc/pki/dovecot/certs/imap.ecuad.ca.pem ssl_key = </etc/pki/dovecot/private/imap.ecuad.ca.pem } protocol pop3 { pop3_uidl_format = %08Xv%08Xu ssl_cert = </etc/pki/dovecot/certs/pop.ecuad.ca.pem ssl_key = </etc/pki/dovecot/private/pop.ecuad.ca.pem } protocol lda { postmaster_address = _postmaster(a)example.com_ } -- Hanna Ruotsalainen Coordinator of Information Applications, IT Services | tel 604 844 3875 local 3875 emily carr university of art + design <http://www.ecuad.ca> | 1399 Johnston Street, Vancouver BC V6H 3R9

1 0

replicating (manage)sieve - no symlink on replica?
by Philipp 17 Oct '14

17 Oct '14

2.2.13/0.4.2, both from source-tarball, no fancy options. tcp-replication between two nodes and following plugin parameters (same on both sides, except the ip-addr) plugin { sieve_before = /home/ssfn/etc/dovecot/sieve.default sieve_global_dir = /home/ssfn/etc/dovecot sieve = ~/.dovecot.sieve sieve_dir = ~/.sieve sieve_extensions = +vnd.dovecot.duplicate sieve_duplicate_period = 6h mail_replica = tcp:10.66.50.15:9993 } mail_location = sdbox:/home/ssfn/var/mail/boxes/%n mail_home = /home/ssfn/var/mail/boxes/%n Testbed: user getting first email delivered, the subdir %n/.sieve/tmp is automatically created as expected. User logs into managesieve on node1 and uploads a script. The %n/.sieve/managesieve.sieve and the symlink .dovecot.sieve -> .sieve/managesieve.sieve is created as expected. One node2 (via replica) happens: automatically create %n/.sieve/tmp automatically create/sync %n/.sieve/managesieve.sieve just NOT the symlink .dovecot.sieve -> .sieve/managesieve.sieve Logfiles for lmtp,doveadm,dsync-local dont indicate an error why it could not create the link or such, to me it looks like it wouldnt even try. mail-syncing works flawlessly (for delivery, just read about the expunge .. hum.) I tried to force a full-resync via doveadm replicator afterwards, to no change on the matter. Any input? Log from node2 after the moment managesieve saves everything correctly on node1: Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: auth input: user=ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: auth input: ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: Added userdb setting: plugin/=yes Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: Effective uid=993, gid=993, home=/home/ssfn/var/mail/boxes/ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=sdbox:/home/ssfn/var/mail/boxes/ssfntest:INDEX=/home/ssfn/var/mail/index/ssfntest:ALT=/home/ssfn/var/mail/alternate/ssfntest Sep 11 15:20:12 doveadm(10.66.30.15,ssfntest): Debug: fs: root=/home/ssfn/var/mail/boxes/ssfntest, index=/home/ssfn/var/mail/index/ssfntest, indexpvt=, control=, inbox=, alt=/home/ssfn/var/mail/alternate/ssfntest Sep 11 15:20:12 dsync-server(ssfntest): Debug: Namespace : Using permissions from /home/ssfn/var/mail/boxes/ssfntest: mode=0700 gid=default Sep 11 15:20:12 dsync-server(ssfntest): Debug: doveadm-sieve: Iterating Sieve mailbox attributes Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve: Pigeonhole version 0.4.2 initializing Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: using active sieve script path: /home/ssfn/var/mail/boxes/ssfntest/.dovecot.sieve Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: using sieve script storage directory: /home/ssfn/var/mail/boxes/ssfntest/.sieve Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: permission lookup failed from /home/ssfn/var/mail/boxes/ssfntest/.sieve Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: created storage directory /home/ssfn/var/mail/boxes/ssfntest/.sieve/tmp Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve-storage: relative path to sieve storage in active link: .sieve/ Sep 11 15:20:12 dsync-server(ssfntest): Debug: sieve: script file /home/ssfn/var/mail/boxes/ssfntest/.sieve/managesieve.sieve not found Sep 11 15:20:12 dsync-server(ssfntest): Debug: doveadm-sieve: Value missing for key `vendor/vendor.dovecot/pvt/server/sieve/files/managesieve' (last change: 1970-01-01 01:00:00) Sep 11 15:20:13 dsync-server(ssfntest): Debug: doveadm-sieve: Assigned value for key `vendor/vendor.dovecot/pvt/server/sieve/files/managesieve' FS tree node2 after that: [..boxes]# find ssfntest -ls|grep -v mailb 24442 4 drwx------ 4 ssfnimp dlagimp 4096 Sep 11 15:20 ssfntest 21048 4 drwx------ 3 ssfnimp dlagimp 4096 Sep 11 15:20 ssfntest/.sieve 21052 4 drwx------ 2 ssfnimp dlagimp 4096 Sep 11 15:20 ssfntest/.sieve/tmp 21056 4 -rw------- 1 ssfnimp dlagimp 69 Sep 11 15:20 ssfntest/.sieve/managesieve.sieve 21016 0 lrwxrwxrwx 1 ssfnimp dlagimp 38 Sep 11 15:05 ssfntest/dbox-alt-root -> /home/dlag/var/mail/alternate/ssfntest FS tree node1 "before": [..boxes]# find ssfntest -ls|grep -v mailb 1835841 4 drwx------ 4 ssfnimp ssfnimp 4096 Sep 11 15:20 ssfntest 1835861 0 lrwxrwxrwx 1 ssfnimp ssfnimp 24 Sep 11 15:05 ssfntest/.dovecot.sieve -> .sieve/managesieve.sieve 1835854 4 -rw------- 1 ssfnimp ssfnimp 8 Sep 11 15:05 ssfntest/dovecot-uidvalidity 1835856 4 drwx------ 3 ssfnimp ssfnimp 4096 Sep 11 15:20 ssfntest/.sieve 1835857 4 drwx------ 2 ssfnimp ssfnimp 4096 Sep 11 15:20 ssfntest/.sieve/tmp 1835860 4 -rw------- 1 ssfnimp ssfnimp 69 Sep 11 15:20 ssfntest/.sieve/managesieve.sieve 1835846 0 lrwxrwxrwx 1 ssfnimp ssfnimp 38 Sep 11 15:05 ssfntest/dbox-alt-root -> /home/ssfn/var/mail/alternate/ssfntest

3 2

Re: Unable to get virtual users set up with database auth
by Gedalya 17 Oct '14

17 Oct '14

When you reply, try to make sure you reply to the list (dovecot(a)dovecot.org) in Thunderbird you should have a Reply List button, or just use reply to all. btw I meant: the output of the 'dovecot -n' command, sorry :D OK so, first of all, lda doesn't do any passdb lookup at all, only userdb. It doesn't have any password to check. Just as a comment, your password query could just as well be: password_query = SELECT uPass as password FROM users WHERE uName = '%n' AND uDomain = '%d' auth_username_format = %Lu -- which I believe is set by default, would lowercase the username for you on the way in from the client, so if all your usernames are meant to be purely lowercase, there is no need to return these fields from the database. Set it explicitly just to be sure. You're using a somewhat old version, I'm not sure how it was back then. Either way, this shouldn't be affecting lda. Now, this seems to be wrong. mail_location = mbox:/var/spool/mail/%u:INBOX=/var/spool/mail/%u You probably want it more like: mbox:~/mail:INBOX=/var/spool/mail/%u The first value, ~/mail, will be a /mail directory under the 'home' directory set in your static userdb, and that's where mailboxes (folders) other than INBOX will be stored. It must be a directory, not a file! I'm a little confused about some of the other details in your log. Let's try to correct your mail_location first and see where that takes us. Also, under namespace, you might as well comment out the 'location = ' and 'prefix = ' lines, though I'm not sure it matters. On 10/16/2014 12:51 PM, Cliff Hayes wrote: > Here it is: > > [root@avalon dovecot]# dovecot -n output > # 2.0.9: /etc/dovecot/dovecot.conf > # OS: Linux 2.6.32-431.29.2.el6.x86_64 x86_64 Scientific Linux release > 6.5 (Carbon) ext4 > auth_debug = yes > auth_debug_passwords = yes > auth_verbose = yes > auth_verbose_passwords = plain > first_valid_uid = 496 > mail_access_groups = mail > mail_debug = yes > mail_location = mbox:/var/spool/mail/%u:INBOX=/var/spool/mail/%u > mbox_write_locks = fcntl > namespace { > hidden = no > inbox = yes > list = yes > location = > prefix = > separator = . > subscriptions = yes > type = private > } > passdb { > args = /etc/dovecot/dovecot-sql.conf.ext > driver = sql > } > protocols = imap > ssl_ca = </etc/pki/dovecot/certs/intermediate.crt > ssl_cert = </etc/pki/dovecot/certs/avalon20140929.crt > ssl_key = </etc/pki/dovecot/private/avalon20140929.key > userdb { > args = uid=497 gid=12 home=/var/email/%u > driver = static > } > verbose_ssl = yes > protocol imap { > imap_id_log = * > } > > [root@avalon dovecot]# cat dovecot-sql.conf.ext > > driver = mysql > > default_pass_scheme = PLAIN > > connect = host=localhost dbname=email user=blah password=blah > > password_query = SELECT uName as user, uNameDomain as username, > uDomain as domain, uPass as password FROM users WHERE uName = '%n' AND > uDomain = '%d' > > > > > On 10/16/2014 11:24 AM, Gedalya wrote: >> On 10/16/2014 12:19 PM, Cliff Hayes wrote: >>> well i don't want to use system users. but how do I assign a UID and >>> GID to virtual users to eliminate the permissions errors? >>> >> For dovecot, it is like this: >> http://wiki2.dovecot.org/AuthDatabase/SQL#User_database_lookups >> >> I don't know if this would be OK with your sendmail side. >> >> You should really post your complete dovecot config (dovecot -n output + >> any ".ext" files) so that we can get a better picture. >>

2 1

ssd for /home
by Luciano Gabriel Andino 16 Oct '14

16 Oct '14

Hi, I am thinking to change sata disk for /home and I want to know if change to a SSD hd, is a good option. I have 30-40 accounts with 30-50K email in boxes. -- Saludos!! Luciano Andino GNU/Linux user #185103 Santa Fe - Argentina -----------------------------------------------

4 3

dsync duplicating identical messages
by Bruno Viaris 16 Oct '14

16 Oct '14

Dear list, I’m trying to setup dovecot/postfix/getmail for a single user application: I have 3 email accounts and 2 main computers (home and work). I want to have an imap server on each machine with sync/replication so that read/reply/delete status and sorting into folders are identical. This part is achieved via dsync and seems to work. Messages are fetched from my ISPs with getmail on both machines. The problem I have is that dsync does not see that a message already fetched on both sides is identical, and duplicates it. I use sdbox (because I like the compression option and the separate storage of attachments). The content of the messages (headers and body) are identical, but the sdbox files have differences in the header and/or trailer. In fact, for each incoming message, I get 4 slightly different files (2 on each machine). Do you know where I could find documentation on how dsync compares messages, and also on sdbox file format ? If you explain me that there is no way to prevent this duplication and that anyway dsync does the right thing, I will probably use getmail only on one machine and let dsync propagate the messages to the other site, but I will miss the redundancy (and some speed) of my wished setup... Thanks -- Bruno P.S. : Background: Home has DSL with 10 MBit/s down and 1 MBit/s up, and full control over NAT / firewall Work internet access is limited by the 100 MBit LAN and has strict firewall and complicated VPN. So it is not easy for my home email client to access the imap server of my work machine, and it is slow for my work email client to access my home imap server. I don’t trust any company for archiving my emails.

2 1

Re: Unable to get virtual users set up with database auth
by Gedalya 16 Oct '14

16 Oct '14

On 10/16/2014 12:19 PM, Cliff Hayes wrote: > well i don't want to use system users. but how do I assign a UID and > GID to virtual users to eliminate the permissions errors? > For dovecot, it is like this: http://wiki2.dovecot.org/AuthDatabase/SQL#User_database_lookups I don't know if this would be OK with your sendmail side. You should really post your complete dovecot config (dovecot -n output + any ".ext" files) so that we can get a better picture.

1 0

Unable to get virtual users set up with database auth
by Cliff Hayes 16 Oct '14

16 Oct '14

I can't seem to get imap virtual users to work with database authentication on new Scientific Linux 6.5 server with Sendmail 8.14.4 & Dovecot 2.0.9 Apparently I have to also create system users because sendmail/procmail/dovecot want to check directory ownership. So I created a database, made entry for the user, created the user chayes on the system. But %u always contains just chayes instead of chayes(a)domain.tld even though I tried different select statements to put the entire username in the username field. This causes the following errors: Oct 16 00:10:26 avalon sendmail[2536]: s9G5APo9002535: forward /var/spool/mail/chayes(a)domain.tld/chayes/.forward.avalon: Group writable directory Oct 16 00:10:26 avalon sendmail[2536]: s9G5APo9002535: forward /var/spool/mail/chayes(a)domain.tld/chayes/.forward: Group writable directory Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: Effective uid=497, gid=497, home=/var/spool/mail/chayes(a)domain.tld/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: Home dir not found: /var/spool/mail/chayes(a)domain.tld/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: Namespace : type=private, prefix=, sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:/var/spool/mail/chayes:INBOX=/var/spool/mail/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: fs: root=/var/spool/mail/chayes, index=, control=, inbox=/var/spool/mail/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Error: user chayes: Initialization failed: Namespace '': mbox: mbox root directory can't be a file: /var/spool/mail/chayes (http://wiki.dovecot.org/MailLocation/Mbox) Oct 16 00:10:26 avalon dovecot: lda(chayes): Fatal: Invalid user settings. Refer to server log for more information. protocols = imap mail_location = mbox:/var/spool/mail/%u:INBOX=/var/spool/mail/%u password_query = SELECT uNameDomain as username, uDomain as domain, uPass as password FROM users WHERE uName = '%n' AND uDomain = '%d' also tried password_query = SELECT uName as username, uDomain as domain, uPass as password FROM users WHERE uName = '%n' AND uDomain = '%d' # file: /etc/procmailrc # system-wide settings for procmail SHELL="/bin/bash" SENDMAIL="/usr/sbin/sendmail -oi -t" LOGFILE="/var/log/procmail.log" DROPPRIVS=yes DELIVER="/usr/libexec/dovecot/deliver" #MAILDIR=$HOME/ #DEFAULT=$HOME/ :0 w * ^X-Spam-Status: Yes | $DELIVER -m spam :0 w | $DELIVER Thanks in advance

3 4

Unable to get virtual users set up with database auth
by Cliff Hayes 16 Oct '14

16 Oct '14

I can't seem to get imap virtual users to work with database authentication on new Scientific Linux 6.5 server with Sendmail 8.14.4 & Dovecot 2.0.9 Apparently I have to also create system users because sendmail/procmail/dovecot want to check directory ownership. So I created a database, made entry for the user, created the user chayes on the system. But %u always contains just chayes instead of chayes(a)domain.tld even though I tried different select statements to put the entire username in the username field. This causes the following errors: Oct 16 00:10:26 avalon sendmail[2536]: s9G5APo9002535: forward /var/spool/mail/chayes(a)domain.tld/chayes/.forward.avalon: Group writable directory Oct 16 00:10:26 avalon sendmail[2536]: s9G5APo9002535: forward /var/spool/mail/chayes(a)domain.tld/chayes/.forward: Group writable directory Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: Effective uid=497, gid=497, home=/var/spool/mail/chayes(a)domain.tld/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: Home dir not found: /var/spool/mail/chayes(a)domain.tld/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: Namespace : type=private, prefix=, sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=mbox:/var/spool/mail/chayes:INBOX=/var/spool/mail/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Debug: fs: root=/var/spool/mail/chayes, index=, control=, inbox=/var/spool/mail/chayes Oct 16 00:10:26 avalon dovecot: lda(chayes): Error: user chayes: Initialization failed: Namespace '': mbox: mbox root directory can't be a file: /var/spool/mail/chayes (http://wiki.dovecot.org/MailLocation/Mbox) Oct 16 00:10:26 avalon dovecot: lda(chayes): Fatal: Invalid user settings. Refer to server log for more information. protocols = imap mail_location = mbox:/var/spool/mail/%u:INBOX=/var/spool/mail/%u password_query = SELECT uNameDomain as username, uDomain as domain, uPass as password FROM users WHERE uName = '%n' AND uDomain = '%d' also tried password_query = SELECT uName as username, uDomain as domain, uPass as password FROM users WHERE uName = '%n' AND uDomain = '%d' # file: /etc/procmailrc # system-wide settings for procmail SHELL="/bin/bash" SENDMAIL="/usr/sbin/sendmail -oi -t" LOGFILE="/var/log/procmail.log" DROPPRIVS=yes DELIVER="/usr/libexec/dovecot/deliver" #MAILDIR=$HOME/ #DEFAULT=$HOME/ :0 w * ^X-Spam-Status: Yes | $DELIVER -m spam :0 w | $DELIVER Thanks in advance

1 0

[Dovecot/VirtualUsers/VirtualMailBoxes] No mails into mua
by aleskandro 16 Oct '14

16 Oct '14

Hi all, I've configured postfix with multiple host and virtual mailboxes (main.cf is here: http://pastebin.com/798VVxbv) So I can receive email to vuser(a)mydomain.tld into /var/mail/vhosts/mydomain.tld/vuser file. Now I'm configuring dovecot 2.2.13-r1; this is my dovecot -n and /etc/dovecot/passwddir/mydomain.tld/passwd: http://pastebin.com/b4xcjj7t - The auth and server subscribe from a mua is ok and in the vuser file there are some mails but I can't see them in my mua. mail.log doesn't report any error. Where I'm wrong? Aleskandro

2 1

Strange quota warning behavior
by Gordon Grubert 16 Oct '14

16 Oct '14

Hi, we are using dovecot 2.2.14. So far, we used the following quota warning configuration quota_rule = *:storage=10G quota_warning = storage=90%% quota-warning 90 %n quota_warning2 = storage=80%% quota-warning 80 %n quota_warning3 = storage=70%% quota-warning 70 %n This works fine. Now, we wanted to add the all clear feature for the quota warning by adding this line: quota_warning4 = -storage=70%% quota-warning 70 %n below The result is really strange. Test 1: 70% exceeded: warning3 executed => OK fall below 70%: quota_warning2 executed => NOT OK quota_warning4 executed => OK Test 2: 70% exceeded: warning3 executed => OK 80% exceeded: warning2 executed => OK fall below 80% (but still more than 70%): quota_warning executed => NOT OK fall below 70%: quota_warning2 executed => NOT OK quota_warning4 executed => OK At the moment, we are running the minimal configuration quota_warning = storage=70%% quota-warning 70 %n quota_warning2 = -storage=70%% quota-warning 70 %n below working like expected (warning when exceeding 70% and all clear when falling below 70%). Does anybody know if there is an error in our configuration or maybe, could this be a bug in dovecot? Best regards, Gordon -- Technischer Leiter & stellv. Direktor Universitätsrechenzentrum (URZ) E.-M.-Arndt-Universität Greifswald Felix-Hausdorff-Str. 12 17489 Greifswald Germany Tel. +49 3834 86 1456 Fax. +49 3834 86 1401

1 0