- dovecot - dovecot.org

auth-worker goes wild
by Hans Morten Kind 14 Oct '14

14 Oct '14

dovecot-2.2.13 running on Solaris-10, app 5000 concurrent connections, and a pretty standard config where passdb is set up with driver = pam. We have had some situations where an auth-worker gets into a loop, the load on the server rises and the auth-master seems blocked untill finally no new imap nor pop session are accepted. A 'doveadm reload' restores normal operations. With debug logging enabled we are observing this: auth: Debug: pam(,10.10.30.33,<7WMu3UkEGQCBsR4h>): cache miss auth-worker(24556): Debug: pam(,10.10.30.33): lookup service=dovecot auth-worker(24556): Debug: pam(,10.10.30.33): #1/1 style=2 msg=Please enter user name: ... auth-worker(24556): Debug: pam(,10.10.30.33): #1/1 style=2 msg=Please enter user name: etc etc After some time dovecot only logs Warning: Auth process not responding, delayed sending initial response (greeting) Disconnected: Auth process broken (disconnected before auth was ready, waited 26 secs) 'truss -wall -aep PID' on the looping auth-worker shows the loop: 0.0757 write(2, 0x080AE0C0, 71) = 71 0101 2 4 5 5 6 p a m ( , 1 0 . 1 0 . 3 0 . 3 3 ) : # 1 / 1 s t y l e = 2 m s g = P l e a s e e n t e r u s e r n a m e : \n 0.0826 write(2, 0x080AE0C0, 71) = 71 0101 2 4 5 5 6 p a m ( , 1 0 . 1 0 . 3 0 . 3 3 ) : # 1 / 1 s t y l e = 2 m s g = P l e a s e e n t e r u s e r n a m e : \n The host at 10.10.30.33 is running RoundCube for app 1000 concurrent users. This problem started after we provided our users RoundCube, and every time we end up in this stuck situation, it is the RoundCube-server making the connections where the username has zero length. I am unable to reproduce how RoundCube puts dovecot into this mess. With the enclosed patch installed, the loop terminates and normal operation are restored immediately: auth: Debug: pam(,10.10.30.33,<Q8kV6AEFYwCBsR4h>): cache miss auth-worker(18658): Debug: pam(,10.10.30.33): lookup service=dovecot auth-worker(18658): Debug: pam(,10.10.30.33): #1/1 style=2 msg=Please enter user name: auth-worker(18658): Fatal: NO USER? auth-worker(18658): Error: Raw backtrace: 0xfef0798c -> 0xfef06d3b -> 0x807ca6e -> 0xfecc460e -> 0xfecc1d1e -> 0xfe3c0af4 -> 0xfecc2608 -> 0xfecc27f1 -> 0x807cc5e -> 0x807d162 -> 0x807d310 -> 0x8069ca1 -> 0x806ab72 -> 0x806aeec -> 0xfef1e866 -> 0xfef2001c -> 0xfef1e9ff -> 0xfef1e956 -> 0xfeeb0172 -> 0x8071bf2 -> 0x805a650 auth: Error: auth worker: Aborted PASSV request for : Worker process died unexpectedly I think however, that the pam driver should not have been initiated when the username has zero length, definitely there is nobody around to answer the prompt for a username ... hmk

2 2

Doveadm quota recalc sets quota for wrong quota root
by Rick van den Hof 14 Oct '14

14 Oct '14

Hi, In our setup, we use two quota roots. One for user quota and one for domain quota. If a user has no quota, then the domain quota is applied. For a user with user quota this is how it looks: # doveadm quota get -u test(a)shellz.nl Quota name Type Value Limit % Domain quota STORAGE 1439155 2560000 56 Domain quota MESSAGE 21257 - 0 User quota STORAGE 0 102400 0 User quota MESSAGE 0 - 0 In this case, the whole domain contains 21257 messages but the account itself contains 0. For a user with only domain quota, this is how it looks: # doveadm quota get -u rick(a)shellz.nl Quota name Type Value Limit % Domain quota STORAGE 1439155 2560000 56 Domain quota MESSAGE 21257 - 0 User quota STORAGE 693299 - 0 User quota MESSAGE 12876 - 0 So far so good. My account contains 12876 messages. Now I've sent a message containing a 1mb.bin as attachment to test(a)shellz.nl: # doveadm quota get -u test(a)shellz.nl Quota name Type Value Limit % Domain quota STORAGE 1440540 2560000 56 Domain quota MESSAGE 21258 - 0 User quota STORAGE 1384 102400 1 User quota MESSAGE 1 - 0 Seems to be fine, the message on disk is indeed 1.4MB and this has been added to both domain and user quota. Now see what happens when I run the following command: # doveadm quota recalc -u test(a)shellz.nl # doveadm quota get -u test(a)shellz.nl Quota name Type Value Limit % Domain quota STORAGE 1384 2560000 0 Domain quota MESSAGE 1 - 0 User quota STORAGE 1384 102400 1 User quota MESSAGE 1 - 0 The recalc action has updated the domain usage to reflect the specific user's usage (my 12876 messages are no longer counted). We log domain usage (usage_domain) in a seperate table from mailbox usage (usage_mailbox). These are the queries that set the usage: 256 Query BEGIN 256 Query DELETE FROM usage_domain WHERE domain = 'shellz.nl' 256 Query DELETE FROM usage_domain WHERE domain = 'shellz.nl' 256 Query INSERT INTO usage_domain (bytes,domain) VALUES ('1418118','shellz.nl') ON DUPLICATE KEY UPDATE bytes='1418118' 256 Query INSERT INTO usage_domain (messages,domain) VALUES ('1','shellz.nl') ON DUPLICATE KEY UPDATE messages='1' 256 Query COMMIT 257 Query BEGIN 257 Query DELETE FROM usage_mailbox WHERE userdomain = 'test(a)shellz.nl' 257 Query DELETE FROM usage_mailbox WHERE userdomain = 'test(a)shellz.nl' 257 Query INSERT INTO usage_mailbox (bytes,userdomain) VALUES ('1418118','test(a)shellz.nl') ON DUPLICATE KEY UPDATE bytes='1418118' 257 Query INSERT INTO usage_mailbox (messages,userdomain) VALUES ('1','test(a)shellz.nl') ON DUPLICATE KEY UPDATE messages='1' 257 Query COMMIT How do I prevent this from happening? It should only update the usage_mailbox table when I run the recalc command. Could this be because we use seperate tables for domain and user usage? dovecot -n: # 2.2.9: /etc/dovecot/dovecot.conf # OS: Linux 3.13.0-37-generic x86_64 Ubuntu 14.04.1 LTS auth_mechanisms = plain login auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890+=_.-@& dict { expire = mysql:/etc/dovecot/dovecot-sql-expire.conf quotadomaindict = mysql:/etc/dovecot/dovecot-sql-quota-domain.conf quotauserdict = mysql:/etc/dovecot/dovecot-sql-quota-user.conf } disable_plaintext_auth = no listen = *,[::] mail_fsync = never mail_location = maildir:~/Maildir mail_plugins = quota expire managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave passdb { args = /etc/dovecot/dovecot-sql-user.conf driver = sql } plugin { expire = Trash expire2 = Spam expire_dict = proxy::expire mail_log_events = delete copy save expunge mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:Domain quota:%d:proxy::quotadomaindict quota_rule2 = Trash:ignore quota_warning = storage=99%% doquotawarning 99 %u quota_warning2 = storage=95%% doquotawarning 95 %u quota_warning3 = storage=75%% doquotawarning 75 %u quota2 = dict:User quota::proxy::quotauserdict quota2_rule2 = Trash:ignore quota2_warning = storage=99%% doquotawarning 99 %u quota2_warning2 = storage=95%% doquotawarning 95 %u quota2_warning3 = storage=75%% doquotawarning 75 %u } protocols = imap pop3 service auth-worker { unix_listener auth-worker { mode = 0600 user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-master { mode = 0600 user = vmail } unix_listener auth-userdb { mode = 0600 user = vmail } user = vmail } service dict { unix_listener dict { mode = 0600 user = vmail } } service doquotawarning { executable = script /usr/bin/doquotawarning.py unix_listener doquotawarning { user = vmail } user = vmail } service imap-login { chroot = process_limit = 200 process_min_avail = 2 } service imap { executable = imap postlogin } service managesieve-login { chroot = } service pop3-login { chroot = process_limit = 50 process_min_avail = 2 } service pop3 { executable = pop3 postlogin } service postlogin { executable = script-login /usr/local/postfixint/dolastlogin.py user = $default_internal_user } ssl_cert = </etc/ssl/certs/wild.totaal.net/wild.totaal.net.pem ssl_cipher_list = ALL:!LOW:!SSLv2:ALL:!aNULL:!ADH:!eNULL:!EXP:RC4+RSA:+HIGH:+MEDIUM ssl_key = </etc/ssl/certs/wild.totaal.net/wild.totaal.net.key userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql-user.conf driver = sql } protocol imap { imap_client_workarounds = delay-newmail tb-extra-mailbox-sep imap_logout_format = bytes=%i/%o mail_max_userip_connections = 50 mail_plugins = quota imap_quota mail_log notify expire } protocol pop3 { mail_max_userip_connections = 50 mail_plugins = quota expire pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_logout_format = top=%t/%p, retr=%r/%b, del=%d/%m, size=%s, bytes=%i/%o pop3_uidl_format = %f } protocol lda { deliver_log_format = msgid=%m: %$ mail_fsync = optimized mail_plugins = quota sieve expire postmaster_address = hostmaster(a)vps04.totaal.net rejection_reason = Your message to <%t>' was automatically rejected:%n%r } dovecot-sql-quota-domain.conf: connect = xxx map { pattern = priv/quota/storage table = usage_domain username_field = domain value_field = bytes } map { pattern = priv/quota/messages table = usage_domain username_field = domain value_field = messages } dovecot-sql-quota-user.conf: connect = xxx map { pattern = priv/quota/storage table = usage_mailbox username_field = userdomain value_field = bytes } map { pattern = priv/quota/messages table = usage_mailbox username_field = userdomain value_field = messages } Thanks in advance to anyone who might be able to shed some light in this situation :). Kind regards, Rick van den Hof -- Manager Engineering Totaalnet Internet Works B.V. IJsselburcht 4e 6825 BP Arnhem +31(0)26-3844944 | r.vandenhof(a)tiw.eu (PGP Key: 0x5A66E935)

2 1

Re: delete/archive old mail
by F. 14 Oct '14

14 Oct '14

>>>> Hi everybody, >>>> I store user's mail in the old mbox format. >>>> I have many scripts to manage users that works fine with mbox. >>>> I use the very old Expire_mail.pl script to delete mail older than NN >>>> days for selected users (nightly cron job). Still works fine with my >>>> CentOS dovecot-2.0.16. >>>> Now I want move the mail to a sort of archive folder instead of simply >>>> deleting it from the inbox for some users. >>>> Is there a similar script or some simple triks? >>>> >>>> Thanks, F. >>> >>> perhaps you will find some ideas here >>> >>> https://sys4.de/de/blog/2013/02/07/mailarchiv-mit-dovecot-und-postfix-sorti… >>> >>> >>> ... >>> #/bin/bash >>> YEAR=`/bin/date +%Y` >>> /usr/bin/doveadm expunge -d -u *(a)backup.server.org mailbox >>> user-backup\*$YEAR*\* savedbefore 10d >>> ... >> >> Thanks for replay. >> My German is really poor, but from a quick look of the example above and >> man doveadm[-expunge], seems that it can be used to replace the old >> Expire_mail.pl (I need to study the right syntax ...) >> >> But still there is the move/archive old mail activity not solved. > > i use the bc feature of postfix to move a copy of all mail per > maildomain into a backup maildomain archive folder, as this is done via lmtp > i am able to sort mails via sieve per user and date folder, depending > on how you like to setup you may use imap acl to give seen rights to the > orig user, so the may restore their own mail whenever they want and/or > their domain postmaster can do it for them. > > But thats only one chance you can do it, you have to decide what fits > best to your needs. > > >> >> doveadm-altmove seems interesting, but in the man page it is reported >> only work with dbox (dbox-only), and my inboxes are in mbox format. >> >> Any other advice? Sorry for the replay to sender instead of the list. As I originally wrote, my need is script or a command like the old Expire_mail.pl that parse a local mbox folder and move/copy mail older than NN days to another local mbox folder. Probably for a Perl programmer is only matter of modify this script or something similar. I searched in docs, wiki, ... but not noticied something that can simply be inserted in a cron job to do the work. Only migration tools or complex (for me), library that can be used to manipulate folders :-) Thanks, F.

1 0

Re: Mailbox format
by Robert Schetterer 14 Oct '14

14 Oct '14

Am 13.10.2014 um 23:29 schrieb Jorge Bastos: > Guys, > > Whats the best mailbox format: > - dbox > - mbox > - maildir > - ?!? there is no universal answer to this a minimum answer might be, maildir is ok in a small non cluster setup ever, please study wiki and archive for more info Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

2 1

[Pigeonhole Sieve interpreter] :create argument and lda_mailbox_autosubscribe setting
by Vivien Lacourba 14 Oct '14

14 Oct '14

Hi, I am using dovecot and dovecot-sieve packages from Debian Wheezy (2.1.7-7+deb7u1). I noticed that while using ":create" (from the mailbox extension) in my Sieve filter and "lda_mailbox_autosubscribe = yes" in Dovecot's LDA config it automatically subscribes to the destination mailbox even if that mailbox already existed. Reading Dovecot's documentation and the mailbox extension RFC it seems subscription should happen only if it is a newly created mailbox and should not happen if the mailbox already existed. http://tools.ietf.org/html/rfc5490#section-3.2 [[ 3.2. ":create" Argument to "fileinto" Command (...) If the mailbox already exists, this argument is ignored. (...) ]] /etc/dovecot/conf.d/15-lda.conf [[ # Should automatically created mailboxes be also automatically subscribed? lda_mailbox_autosubscribe = yes ]] Also I noticed that if I don't use ":create" in my filters and instead set "lda_mailbox_autocreate = yes" in Dovecot's LDA config, then this time the behavior looks correct as the automatic subscription will only happen for a newly created mailbox. Thanks, Vivien -- Vivien Lacourba World Wide Web Consortium Systems & Network Engineer http://www.w3.org vivien(a)w3.org Tel: +33.4.92.38.78.89 http://www.w3.org/People/Vivien Fax: +33.4.92.38.78.22 -- Vivien Lacourba World Wide Web Consortium Systems & Network Engineer http://www.w3.org vivien(a)w3.org Tel: +33.4.92.38.78.89 http://www.w3.org/People/Vivien Fax: +33.4.92.38.78.22

2 1

Re: Mailbox format
by Gedalya 14 Oct '14

14 Oct '14

On 10/13/2014 05:29 PM, Jorge Bastos wrote: > Guys, > > Whats the best mailbox format: > - dbox Dovecot-specific. Higher performance, saves I/O. > - mbox Don't bother. > - maildir Simple, compatible, resilient. > - ?!? > > Pros and cons? > No short answer is the right answer. Try to thoroughly read http://wiki2.dovecot.org/MailboxFormat and child pages.

1 0

Re: Compile dovecot with sieve/managesieve support
by Daniel Parthey 14 Oct '14

14 Oct '14

Hi, please have a look at http://wiki2.dovecot.org/Pigeonhole/Installation Kind regards Daniel

1 0

Mailbox format (was: RE: Compile dovecot with sieve/managesieve support)
by Jorge Bastos 14 Oct '14

14 Oct '14

Guys, Whats the best mailbox format: - dbox - mbox - maildir - ?!? Pros and cons? > -----Original Message----- > From: dovecot [mailto:dovecot-bounces@dovecot.org] On Behalf Of Jorge > Bastos > Sent: segunda-feira, 13 de Outubro de 2014 21:35 > To: dovecot(a)dovecot.org > Subject: RE: Compile dovecot with sieve/managesieve support > > I see, > > If i need anything else i'll post, > > Thanks, > > > -----Original Message----- > > From: Larry Rosenman [mailto:larryrtx@gmail.com] > > Sent: segunda-feira, 13 de Outubro de 2014 21:05 > > To: Jorge Bastos > > Subject: Re: Compile dovecot with sieve/managesieve support > > > > Pigeonhole does ManageSieve as well...... > > > > On 10/13/14, Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: > > > Larry & Pascal, > > > > > > Thanks for the reply. > > > > > > Ok this is the lib for sieve, and the sieved, I see in debian > > packages > > > that there is a dovecot-managesieved, the daemon, does this > > pigeonhole > > > is the daemon also? > > > > > >> -----Original Message----- > > >> From: Larry Rosenman [mailto:larryrtx@gmail.com] > > >> Sent: segunda-feira, 13 de Outubro de 2014 20:12 > > >> To: Jorge Bastos > > >> Subject: Re: Compile dovecot with sieve/managesieve support > > >> > > >> look at pigeonhole. That's the Sieve implementation for Dovecot > > >> 2.x > > >> > > >> On 10/13/14, Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: > > >> > Hi guys, > > >> > > > >> > > > >> > > > >> > I'd like to have dovecot with sieve support, but can't find any > > >> > information on how to enable the sieve/managesieve support. > > >> > > > >> > Does anyone can light me up on this? > > >> > > > >> > > > >> > > > >> > Thanks, > > >> > > > >> > > > >> > > > >> > > >> > > >> -- > > >> Larry Rosenman http://www.lerctr.org/~ler > > >> Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com > > >> US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 > > > > > > > > > -- > > Larry Rosenman http://www.lerctr.org/~ler > > Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com > > US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

1 0

RE: Compile dovecot with sieve/managesieve support
by Jorge Bastos 13 Oct '14

13 Oct '14

I see, If i need anything else i'll post, Thanks, > -----Original Message----- > From: Larry Rosenman [mailto:larryrtx@gmail.com] > Sent: segunda-feira, 13 de Outubro de 2014 21:05 > To: Jorge Bastos > Subject: Re: Compile dovecot with sieve/managesieve support > > Pigeonhole does ManageSieve as well...... > > On 10/13/14, Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: > > Larry & Pascal, > > > > Thanks for the reply. > > > > Ok this is the lib for sieve, and the sieved, I see in debian > packages > > that there is a dovecot-managesieved, the daemon, does this > pigeonhole > > is the daemon also? > > > >> -----Original Message----- > >> From: Larry Rosenman [mailto:larryrtx@gmail.com] > >> Sent: segunda-feira, 13 de Outubro de 2014 20:12 > >> To: Jorge Bastos > >> Subject: Re: Compile dovecot with sieve/managesieve support > >> > >> look at pigeonhole. That's the Sieve implementation for Dovecot 2.x > >> > >> On 10/13/14, Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: > >> > Hi guys, > >> > > >> > > >> > > >> > I'd like to have dovecot with sieve support, but can't find any > >> > information on how to enable the sieve/managesieve support. > >> > > >> > Does anyone can light me up on this? > >> > > >> > > >> > > >> > Thanks, > >> > > >> > > >> > > >> > >> > >> -- > >> Larry Rosenman http://www.lerctr.org/~ler > >> Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com > >> US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 > > > > > -- > Larry Rosenman http://www.lerctr.org/~ler > Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com > US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

1 0

RE: Compile dovecot with sieve/managesieve support
by Jorge Bastos 13 Oct '14

13 Oct '14

Larry & Pascal, Thanks for the reply. Ok this is the lib for sieve, and the sieved, I see in debian packages that there is a dovecot-managesieved, the daemon, does this pigeonhole is the daemon also? > -----Original Message----- > From: Larry Rosenman [mailto:larryrtx@gmail.com] > Sent: segunda-feira, 13 de Outubro de 2014 20:12 > To: Jorge Bastos > Subject: Re: Compile dovecot with sieve/managesieve support > > look at pigeonhole. That's the Sieve implementation for Dovecot 2.x > > On 10/13/14, Jorge Bastos <mysql.jorge(a)decimal.pt> wrote: > > Hi guys, > > > > > > > > I'd like to have dovecot with sieve support, but can't find any > > information on how to enable the sieve/managesieve support. > > > > Does anyone can light me up on this? > > > > > > > > Thanks, > > > > > > > > > -- > Larry Rosenman http://www.lerctr.org/~ler > Phone: +1 214-642-9640 (c) E-Mail: larryrtx(a)gmail.com > US Mail: 108 Turvey Cove, Hutto, TX 78634-5688

1 0