- dovecot - dovecot.org

Hitting wall at 2048 IMAP connections
by Ron Cleven 04 Feb '15

04 Feb '15

We are gradually rolling out Dovecot (IMAP only, no POP3) to our customer base. We are replicating between a pair of CentOS 7 boxes. All has been working wonderfully. However, to be sure our rollout continues to go smoothly, we put together a simple benchmark client program to fire up X persistent IMAP connections (using hundreds of mailboxes) that login, list the folders, select the INBOX, periodically FETCH an email from the INBOX, and otherwise sit in an IDLE loop. As long as we keep the number of concurrent IMAP connections under 2k, everything works fine. 2,000 processes generate a minimal load on the server. However, 2048 is a "hard limit" for number of IMAP processes on a box the way things are configured right now. The number (2048) is a bit too magical to be anything but a misconfiguration of either some kernel limit or a dovecot configuration limit. I have been unable to figure out where else to look. I have pasted in my dovecot config below along with the kernel setting for max_user_instances. Hoping someone can tell me what stupid mistake I have made or what else to check. Just an aside, we front-end this with a separate set of proxy servers that provide an SSL front-end, hence, no SSL in the dovecot config. /proc/sys/fs/inotify/max_user_instances = 4096 # 2.2.10: /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-123.13.1.el7.x86_64 x86_64 CentOS Linux release 7.0.1406 (Cor e) auth_master_user_separator = * auth_mechanisms = plain login default_internal_user = sysadm default_login_user = sysadm default_vsz_limit = 512 M disable_plaintext_auth = no doveadm_password = ************** doveadm_port = 12345 imap_max_line_length = 128 k listen = * mail_gid = 2001 mail_location = maildir:~/Maildir mail_plugins = notify replication mail_uid = 2001 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mbox_write_locks = fcntl namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = scheme=PLAIN /z/imap/usr/master-users driver = passwd-file master = yes pass = yes } passdb { args = scheme=PLAIN username_format=%u /z/imap/usr/users driver = passwd-file } plugin { mail_replica = tcp:imap0s sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_max_actions = 32 sieve_max_redirects = 4 sieve_max_script_size = 1M } protocols = imap pop3 sieve service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = sysadm } unix_listener replication-notify { mode = 0666 user = sysadm } } service auth { unix_listener auth-userdb { mode = 0777 } } service doveadm { inet_listener { port = 12345 } user = sysadm } service imap { process_limit = 4096 } service managesieve-login { inet_listener sieve { port = 4190 } } service managesieve { process_limit = 4096 } service replicator { process_min_avail = 1 } ssl = no userdb { args = username_format=%u /z/imap/usr/users driver = passwd-file } protocol lda { mail_plugins = sieve notify replication } protocol imap { mail_max_userip_connections = 40 } protocol sieve { managesieve_max_line_length = 65536 }

5 4

Corruption of index files
by Oliver Welter 04 Feb '15

04 Feb '15

Hi All, after upgrading my mail server (dovecot 1.1.7 -> 2.2.13) I get tons of messages about corrupted index files in the syslog ("Error: Corrupted transaction log" and "Warning: fscking index file .. dovecot.index". I tried flock and even dotlock, but the problems persist. The system is a VPS with an overlay filesystem which did work flawlessly with the old dovecot install and is not reporting any errors in the host machine. From time to time I get warnings about an exceeded inotify limit for the dovecot user - might this be a problem? Anybody has an idea what might be the cause those problems and how to track them down. Oliver -- Protect your environment - close windows and adopt a penguin!

2 4

Upstart script for Poolmon
by Listas＠Adminlinux 03 Feb '15

03 Feb '15

Hi My servers are Ubuntu 14.04 and I'm needing to make a Upstart script for Poolmon. Does someone already made this and could to share with us ? Thanks! -- Thiago henrique

1 1

Master user without pass=yes error
by Thomas HUMMEL 03 Feb '15

03 Feb '15

Hello Timo, I just tried the master user feature with a very simple setup (Dovecot v2.2.15) : !include auth-master.conf.ext -> passwd-file passdb !include auth-ldap.conf.ext -> ldap passdb (userdb prefetched) without auth_bind=yes without pass=yes I get this userdb lookup error : dovecot: auth: passwd-file(masteruser,157.99.64.42,master,<4Pgesh0OygCdY0Aq>): Master user logging in as normaluser dovecot: auth: Error: prefetch(normaluser,157.99.64.42,<4Pgesh0OygCdY0Aq>): userdb lookup not possible with only userdb prefetch dovecot: imap: Error: Internal auth failure (client-pid=10449 client-id=1) dovecot: imap-login: Internal login failure (pid=10449 id=1) (internal failure, 1 successful auths): user=<normaluser>, method=PLAIN, rip=157.99.64.42, lip=157.99.64.81, mpid=10570, TLS, session=<4Pgesh0OygCdY0Aq> with pass=yes, it works. Feb 2 17:51:24 langres dovecot: auth: passwd-file(masteruser,157.99.64.42,master,<YmjAwx0O0gCdY0Aq>): Master user logging in as normaluser Feb 2 17:51:24 langres dovecot: imap-login: Login: user=<normaluser>, method=PLAIN, rip=157.99.64.42, lip=157.99.64.81, mpid=11647, TLS, session=<YmjAwx0O0gCdY0Aq> I dont quite understand why because the documentation states that 'pass=yes' "means that Dovecot verifies that the login user really exists before allowing the master user to log in. Without the setting if a nonexistent login username is given,[...]" Here, 'normaluser' exists in the ldap passdb so, even with pass=no, I'm not supposed to be in the 'nonesxistent login username' case. Can you help ? thanks. -- Thomas Hummel | Institut Pasteur <hummel(a)pasteur.fr> | Groupe Exploitation et Infrastructure

1 2

quote strings passed to sql
by Juan Bernhard 03 Feb '15

03 Feb '15

Hello list. I'm thinking to migrate the hole user db from system users to mysql. I already did it in a test environment, but something is annoying my OCD... I don't quote the variables username and password sent to the mysql server. I know, the mysql user that dovecot uses only has select rights, but it stills bother me, because its possible to do an useless sql code injection. Is there a way to quote that? Something like exim's quote_mysql? Saludos, Juan.

3 4

Re: Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 02 Feb '15

02 Feb '15

Can no one give some adivse on this issue? Thanks Christian

1 0

LDAP: "unused" & "uid missing"
by Leander Schäfer 02 Feb '15

02 Feb '15

Does someone know what this means? It looks to me like it got the UID, yet it complains ... ? Also, why does it complain about attributes being unused, yet I use prefetch, so they will be used?! Log: [...] dovecot: auth: Debug: ldap(test(a)mydomain.tld,192.168.10.233,<Rl2CF+UNhgDAqArp>): result: mailStorageDirectory=/var/vmail mailUidNumber=2110 mailGidNumber=2110 mailQuotaStorage=10240 mailQuotaMessages=20; mailQuotaMessages,mailStorageDirectory,mailUidNumber,mailQuotaStorage,mailGidNumber unused dovecot: auth: Debug: ldap(test(a)mydomain.tld,192.168.10.233,<Rl2CF+UNhgDAqArp>): result: mailStorageDirectory=/var/vmail mailUidNumber=2110 mailGidNumber=2110 mailQuotaStorage=10240 mailQuotaMessages=20; uid missing [...] As mentioned, I use prefetch, so this is how my dovecot/dovecot-ldap.conf.ext is looking: [...] #user_attrs = mailStorageDirectory=home=%$/%u, mailStorageDirectory=mail=maildir:%$/%u/maildir, mailUidNumber=uid, mailGidNumber=gid, mailQuotaStorage=quota_rule=*:storage=%$, mailQuotaMessages=quota_rule2=*:messages=%$ pass_attrs = uid=user, userPassword=password, mailStorageDirectory=userdb_home=%$/%u, mailStorageDirectory=userdb_mail=maildir:%$/%u/maildir, mailUidNumber=userdb_uid, mailGidNumber=userdb_gid, mailQuotaStorage=userdb_quota_rule=*:storage=%$, mailQuotaMessages=userdb_quota_rule2=*:messages=%$ [...] Thanks

2 1

Sieve permissions issue following update
by David Gessel 02 Feb '15

02 Feb '15

I recently updated dovecot and my sieve filters stopped working. Checking the logs I see: Dec 9 00:09:59 mailhost dovecot: lda(gessel(a)domain.com): Error: sieve: binary save: failed to create temporary file: open(/usr/local/etc/dovecot/sieve/10-move-spam.svbin.mailhost.domain.com.114.) failed: Permission denied (euid=5000(vmail) egid=5000(vmail) missing +w perm: /usr/local/etc/dovecot/sieve, we're not in group 6(mail), dir owned by 143:6 mode=0775) Dec 9 00:09:59 mailhost dovecot: lda(gessel(a)domain.com): Error: sieve: The LDA Sieve plugin does not have permission to save global Sieve script binaries; global Sieve scripts like `/usr/local/etc/dovecot/sieve/10-move-spam.sieve' need to be pre-compiled using the sievec tool However this fairly clear advice on the failure seems to be contradicted by: # id vmail uid=5000(vmail) gid=5000(vmail) groups=5000(vmail),6(mail) ? dovecot-pigeonhole-0.4.6 = up-to-date with index dovecot2-2.2.15_1 = up-to-date with index uname -a FreeBSD host.domain.com 9.3-RELEASE FreeBSD 9.3-RELEASE #0 r268932: Mon Jul 21 15:51:38 PDT 2014 gessel@host1.domain.com:/usr/obj/usr/src/sys/BARCELONA-13-08 amd64

8 22

managesieve & checkpassword authdb interface
by Peter Hodur 02 Feb '15

02 Feb '15

Hello, does anybody knows what managesieve addon sets in SERVICE env when autentificating to checkpassword? thanks Pete

2 1

Domain quota
by Evgeny Basov 02 Feb '15

02 Feb '15

Hello. I'm tried to use domain quota: plugin { quota = dict:Domain quota:%d:redis:host=127.0.0.1:prefix=domain/ quota_rule = *:storage=0 quota2 = dict:User quota::redis:host=127.0.0.1:prefix=user/ quota2_rule = *:storage=0 quota3_rule = Trash:storage=+5%% } but got next problem with it. Init state: # doveadm quota get -u "testacc(a)m.tld" Quota name Type Value Limit % Domain quota STORAGE 27070 204800 13 Domain quota MESSAGE 21 - 0 User quota STORAGE 27068 - 0 User quota MESSAGE 18 - 0 # doveadm quota get -u "testacc2(a)m.tld" Quota name Type Value Limit % Domain quota STORAGE 27070 204800 13 Domain quota MESSAGE 21 - 0 User quota STORAGE 2 - 0 User quota MESSAGE 3 - 0 Make recalc: # doveadm quota recalc -u "testacc2(a)m.tld" and got: # doveadm quota get -u "testacc2(a)m.tld" Quota name Type Value Limit % Domain quota STORAGE 2 204800 0 Domain quota MESSAGE 3 - 0 User quota STORAGE 2 - 0 User quota MESSAGE 3 - 0 # doveadm quota get -u "testacc(a)m.tld" Quota name Type Value Limit % Domain quota STORAGE 2 204800 0 Domain quota MESSAGE 3 - 0 User quota STORAGE 27068 - 0 User quota MESSAGE 18 - 0 All values of all domain quota are equals. Execute recalc one more time: # doveadm quota recalc -u "testacc(a)m.tld" # doveadm quota get -u "testacc(a)m.tld" Quota name Type Value Limit % Domain quota STORAGE 27068 204800 13 Domain quota MESSAGE 18 - 0 User quota STORAGE 27068 - 0 User quota MESSAGE 18 - 0 # doveadm quota get -u "testacc2(a)m.tld" Quota name Type Value Limit % Domain quota STORAGE 27068 204800 13 Domain quota MESSAGE 18 - 0 User quota STORAGE 2 - 0 User quota MESSAGE 3 - 0 How can I get fast realtime changed domain quota? Used dovecot 2.2.13.

2 1

auth: Error: auth worker: Aborted request: Lookup timed out
by ML mail 01 Feb '15

01 Feb '15

Hello, I was checking the errors on my mailbox server using "doveadm log errors" and found that around 2-3 times a day I get the following error: Jan 29 22:14:43 auth: Error: auth worker: Aborted request: Lookup timed out Any ideas if this is a bad sign? For example some timeout on my side which is too low or is it just an aborted connection from the side of the IMAP client? The configuration on my Dovecot mailbox server can be found here: http://article.gmane.org/gmane.mail.imap.dovecot/80823 It is a Debian 7 box with standard debian packages. Best regards ML

4 6

[Bug ?] IMAP keywords are not preserved during mailbox conversion
by Alexander Moisseev 01 Feb '15

01 Feb '15

IMAP keywords (Thunderbird Tags) are not preserved during Maildir to mdbox conversion. For mailbox conversion I am using the command: # doveadm -o mail_location=maildir:~/Maildir sync -u user(a)example.com mdbox:~/mdbox After conversion all Thunderbird Tags are lost. Is it a bug or expected behavior? -- Alexander

1 0

LDAP Quota Limits & Warnings
by Leander Schäfer 30 Jan '15

30 Jan '15

I found this on the dovecot wiki: The following limit names are supported: * *storage*: Quota limit in kilobytes, 0 means unlimited. * *bytes*: Quota limit in bytes, 0 means unlimited. * *messages*: Quota limit in number of messages, 0 means unlimited. This probably isn't very useful. * *backend*: Quota backend-specific limit configuration. * *ignore*: Don't include the specified mailbox in quota at all. As I understood it, all 5 limit categories are valid to use with maildir format while only "storage" and "messages" are valid to use with dmbox format, right? Also the "ignore" limit may appear more than one time for a user - like Trash:ignore and Spam:ignore, right? So I assumed following should work - and it does - except for "ignore" - even though LDAP does not hold multiple values for it - only a single one for each limit: user_attrs = [...] mailQuotaStorage=quota_rule=*:storage=%$, mailQuotaMessages=quota_rule2=*:messages=%$, mailQuotaIgnore=quota_rule3=*:ignore=%$ pass_attrs = [...] mailQuotaStorage=userdb_quota_rule=*:storage=%$, mailQuotaMessages=userdb_quota_rule2=*:messages=%$, mailQuotaIgnore=userdb_quota_rule3=*:ignore=%$ And unfortunately /var/log/maillog tells me: dovecot: imap(abuse(a)netocean.local): Error: user abuse(a)netocean.local: Initialization failed: Failed to initialize quota: Invalid quota root quota: Invalid rule *:ignore=Trash: Unknown rule limit name: ignore Yet /var/log/debug.log sounds a bit more promising ... but then again not really ... dovecot: imap: Debug: Added userdb setting: mail=maildir:/usr/local/www/default/mail/user@mydomain.tld dovecot: imap: Debug: Added userdb setting: plugin/quota_rule=*:storage=1048576 dovecot: imap: Debug: Added userdb setting: plugin/quota_rule2=*:messages=10 dovecot: imap: Debug: Added userdb setting: plugin/quota_rule3=*:ignore=Trash dovecot: imap(abuse(a)netocean.local): Debug: Effective uid=2110, gid=2110, home=/var/vmail/user(a)mydomain.tld/mail dovecot: imap(abuse(a)netocean.local): Debug: Quota root: name=User quota backend=maildir args= dovecot: imap(abuse(a)netocean.local): Debug: Quota rule: root=User quota mailbox=* bytes=1073741824 messages=0 dovecot: imap(abuse(a)netocean.local): Debug: Quota rule: root=User quota mailbox=* bytes=1073741824 messages=10 dovecot: imap(abuse(a)netocean.local): Debug: Quota rule: root=User quota mailbox=* bytes=1073741824 messages=10 Additionally, if one of the two working quotas (storage & messages) is exceeded than Dovecot sends out a warning from this setting: plugin { quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u # What message to send to IMAP clients (and SMTP senders) when quota is exceeded? quota_exceeded_message = Quota exceeded, please go to http://MyDomain.TLD for instructions on how to fix this. } No my question is how to provide two different warnings. One for storage like: Sorry, you're mail storage is full. Please delete some. And a second for messages like: Sorry, you've too many messages. Please delete some. Thanks & Best Regards Leander

1 0

Troubleshooting doveadm for Solr Configuration
by Kevin Laurie 30 Jan '15

30 Jan '15

Hey guys, I am trying to pinpoint where the problem with my dovecot-FTS installation . I am trying to rule out between the Dovecot and the Solr Module. I think if the output below looks OK then the dovecot is fine, and will focus more on the Solr. Appreciate if someone could check and confirm. Thanks root@mail:/etc/dovecot/conf.d# doveadm -D -v fts rescan -u admin(a)email.net doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so doveadm(admin(a)email.net): Debug: auth input: admin(a)email.net master_user= admin(a)email.net home=/var/vmail/vmail1/ email.net/t/h/a/admin-2014.10.25.07.15.15/ quota_rule=*:bytes=26214400000 doveadm(admin(a)email.net): Debug: Added userdb setting: plugin/master_user= admin(a)email.net doveadm(admin(a)email.net): Debug: Added userdb setting: plugin/quota_rule=*:bytes=26214400000 doveadm(admin(a)email.net): Debug: Effective uid=2000, gid=2000, home=/var/vmail/vmail1/email.net/t/h/a/thai-2014.10.25.07.15.15/ doveadm(admin(a)email.net): Debug: Quota root: name=user backend=dict args=:proxy::quotadict doveadm(admin(a)email.net): Debug: Quota rule: root=user mailbox=* bytes=26214400000 messages=0 doveadm(admin(a)email.net): Debug: Quota warning: bytes=22282240000 (85%) messages=0 reverse=no command=quota-warning 85 admin(a)email.net doveadm(admin(a)email.net): Debug: Quota warning: bytes=23592960000 (90%) messages=0 reverse=no command=quota-warning 90 admin(a)email.net doveadm(admin(a)email.net): Debug: Quota warning: bytes=24903680000 (95%) messages=0 reverse=no command=quota-warning 95 admin(a)email.net doveadm(admin(a)email.net): Debug: Quota grace: root=user bytes=2621440000 (10%) doveadm(admin(a)email.net): Debug: dict quota: user=admin(a)email.net, uri=proxy::quotadict, noenforcing=0 doveadm(admin(a)email.net): Debug: Namespace : type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir://var/vmail/vmail1/ email.net/t/h/a/thai-admin.10.25.07.15.15//Maildir/:INDEX=//var/vmail/vmail… doveadm(admin(a)email.net): Debug: maildir++: root=//var/vmail/vmail1/ email.net/t/h/a/admin-2014.10.25.07.15.15//Maildir, index=, indexpvt=, control=, inbox=//var/vmail/vmail1/ sicl.net/t/h/a/thai-2014.10.25.07.15.15//Maildir, alt= doveadm(admin(a)email.net): Debug: Namespace : type=shared, prefix=Shared/%u/, sep=/, inbox=no, hidden=no, list=children, subscriptions=yes location=maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/Shared/%u doveadm(admin(a)email.net): Debug: shared: root=/var/run/dovecot, index=, indexpvt=, control=, inbox=, alt= doveadm(admin(a)email.net): Debug: fts: Indexes disabled for namespace 'Shared/%u/'

2 1

Re: Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 30 Jan '15

30 Jan '15

If mailbox guids should be unique systemwide, would it be safe to delete all 'dovecot-uidlist' files of the user's mailboxes and then do a doveadm force-resync after mailbox migration? Thanks! Christian ----------------ursprüngliche Nachricht----------------- Von: Christian Binder Stadt Freilassing [christian.binder(a)freilassing.de ] An: "dovecot dovecot.org" [dovecot(a)dovecot.org ] Datum: Thu, 29 Jan 2015 09:24:12 +0100 ------------------------------------------------- > Sorry if this is a duplicate message. I'm new in this mailing list and not sure if > the first message really got posted ... > > Dovecot version: 2.2.9 > > We're using doveadm backup to migrate user mailboxes from cyrus to dovecot. I > defined a master user for the old cyrus mailserver in imapc backend to get access > to all the mailboxes. Now I realized that after migration every user's "Inbox" or > "Trash" on the new dovecot server has the same mailbox-guid, e.g: > > User a: > inbox dd9456... > trash 5549cc... > > User b: > inbox dd9456... > trash 5549cc... > > Is this a wanted behaviour? If i do not migrate with doveadm backup and just > manually create mailboxes, the mailbox-guids are globally unique, e.g: > > User a: > inbox dd9456... > trash 5549cc... > > User b: > inbox c543fe... > trash t49ccc... > > Thanks! > Christian >

1 0

Re: Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 30 Jan '15

30 Jan '15

Update: seems to me that the old cyrus mailserver already uses these non-unique mailbox guids. Is it necessary to have them unique over all private namespaces? Thanks Christian ----------------ursprüngliche Nachricht----------------- Von:Christian Binder Stadt Freilassing [christian.binder(a)freilassing.de ] An:"dovecot dovecot.org" [dovecot(a)dovecot.org ] Datum:Thu, 29 Jan 2015 09:24:12 +0100 ------------------------------------------------- > Sorry if this is a duplicate message. I'm new in this mailing list and not sure if > the first message really got posted ... > > Dovecot version:2.2.9 > > We're using doveadm backup to migrate user mailboxes from cyrus to dovecot. I > defined a master user for the old cyrus mailserver in imapc backend to get access > to all the mailboxes. Now I realized that after migration every user's "Inbox" or > "Trash" on the new dovecot server has the same mailbox-guid, e.g: > > User a: > inbox dd9456... > trash 5549cc... > > User b: > inbox dd9456... > trash 5549cc... > > Is this a wanted behaviour? If i do not migrate with doveadm backup and just > manually create mailboxes, the mailbox-guids are globally unique, e.g: > > User a: > inbox dd9456... > trash 5549cc... > > User b: > inbox c543fe... > trash t49ccc... > > Thanks! > Christian >

1 0

Info on shared folders
by Bobber 30 Jan '15

30 Jan '15

I'm interested in setting up shared folders in a small office environment.Â Can a folder be created and shared with other users from an IMAP client such as Thunderbird? -- *Bob Wooldridge* Blog: http://kc0dxf.net/blog/

2 1

Re: mdbox: Broken virtual size for mail UID ...
by Dennis Schridde 29 Jan '15

29 Jan '15

Hello everyone! I originally asked Timo for help in this case, but I understand he is very busy these days, so I am now posting my questions to this mailinglist. On Sunday 28 Dec 2014 10:02:08 Timo Sirainen wrote: > On 28 Dec 2014, at 07:06, Dennis Schridde <devurandom(a)gmx.net> wrote: > > My harddrives crashed, and now I get "Broken virtual size for mail UID …" > > messages in the dovecot logs from some mdbox folders. I assume the logged > > emails are destroyed forever, but I would like to restore as much as > > possible of them and keep them in their corrupted / incomplete state. How > > do I do that? > > > > I guess what I need is some kind of "doveadm fsck" command, that tries its > > best to limit the damage to a minimum: Inform me about the broken emails > > and then ask me whether they should be stored in truncated form, be > > deleted, etc. Is there such a thing? > > http://dovecot.org/tools/mdbox-recover.pl Thank you! >From what I read in the Wiki and how I understand the script, I should proceed like this: #!/bin/bash broken_storage=$PWD/storage recovered_storage=$PWD/storage-recovered cd ${recovered_storage} n=1 for f in ${broken_storage}/m.* ; do mdbox-recover.pl ${f} for f in msg.* ; do mv ${f} m.${n} $((n++)) done done mv ${broken_storage} storage-backup mv ${recovered_storage} storage exit 0 Afterwards dovecot will now automatically regenerate the dovecot.map.index files from the now bare m.* files, and as the dovecot.index files in the actual mailboxes reference the GUID, and the messages in the m.* files have the GUID embedded, nothing will be lost. Is that correct? To create proper assessment of the damage, I need to know a bit more: 1. In the script on line 43, the comment says: # end of metadata block missing, finish the previous mail That seems to be wrong, since idx2!=-1 and state==STATE_BODY on that line, while I understand it should be -1 and STATE_META if we were still reading the metadata block. I guess it should be something like: # still in data block, finish the previous mail 2. Line 47 reads: # truncated / broken data? just keep writing to previous file That is the only error which could be detected in that script, right? 3. A message could also be corrupted by loosing its complete data block from \001\003 to \001\002, which cannot be detected in the script, right? 4. The only case to detect this and find other unexpected oddities would be to compare each msg.* in size and offset with the messages referenced in the dovecot.map.index file, correct? How do I do that, if the dovecot.map.index does not contain the GUID of the message? Do I need to fuzzy match the file_id, offset and size with my current values, or is there a more reliable way to do this? 5. Metadata refers to the RFC822 mail headers? Or are they also included in the body, and the mentioned metadata is dovecot-only metadata? If the latter, do they also contain the size of the body and similar information? Best regards, Dennis

1 0

Dovecot antispam
by David Mehler 29 Jan '15

29 Jan '15

Hello, This is not specifically a dovecot issue, but to it's antispam plugin, that being the case if private responses are deemed so please do so. I've got a Postfix, amavisd-new, dovecot (delivery and sieve), setup with spamassassin. It's for the most part working fine, here's the problem, false positives and false negatives. The sieve moves spamassassin-tagged messages in to the Spam folder, sometimes it misses either a false positive or negative. What I'm wanting to do and this is where the dovecot-antispam plugin comes in is automatically retrain messages, a false positive or negative needs to be moved and reclassified accordingly. I've found a lot of google results indicating dspam, when I used this the results were terrible after three months of usage I gave up on it. I've thought about using spamc with dovecot-antispam to handle retraining or crm114, but not much on getting this going. If anyone has any information on this or any other non-dspam setup that would work I'd appreciate knowing about it. Thanks. Dave.

2 3

Re: Indexing Mail faster
by Joseph Tam 29 Jan '15

29 Jan '15

Kevin writes: > Appreciate if you could help with this. I have been trying to address this > "slow search" issue for a while with very limited success(I was trying to > implement FTS also), so I will appreciate if you could support. When I'm stumped, one of the diagnostic tools I use is process tracing. Connect via IMAP, in another window/session process trace the IMAP worker process (Linux: strace -r -otrace.dump {pid})), initiate the search and quit. Then you can look through the trace and see whether it gets hung on a particular operation. Joseph Tam <jtam.home(a)gmail.com>

3 6

Re: Indexing Mail faster
by Kevin Laurie 29 Jan '15

29 Jan '15

Hi Micheal, Here is how it should look:- ~# telnet localhost imap Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. . login user(a)domain.net password . OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY QUOTA] Logged in . select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded \*)] Flags permitted. * 49319 EXISTS * 12 RECENT * OK [UNSEEN 46791] First unseen. * OK [UIDVALIDITY 1414214135] UIDs valid * OK [UIDNEXT 106554] Predicted next UID * OK [NOMODSEQ] No permanent modsequences . OK [READ-WRITE] Select completed (0.128 secs). . search text hello * OK Searched 6% of the mailbox, ETA 2:34 Finally at the end you logout by . logout On Thu, Jan 29, 2015 at 4:39 PM, Kevin Laurie <superinterstellar(a)gmail.com> wrote: > Dear Michael, > You will need to telnet locally to imap. So run the following commands in > your server > > 1. telnet localhost imap > 2. . login email(a)domain.com password > 3. . select inbox > 4. . search text "passport" (This command here will search your inbox and > give you the time needed) > 5. . logout > > Notice the period(.) You need to use it . > > Hope this is clear. > Regards > Kevin > > > On Thu, Jan 29, 2015 at 4:32 PM, M. Koehler <michael.koehler(a)makomi.de> > wrote: > >> Hi Kevin, >> >> I want to test it with my dovecot but I don´t know how to measure it. >> Could you please provide my the configuration/terminal commands to get an >> reliable output? I just have mailboxes up to 4GB in VM on a proxmox server >> (Debian Wheezy at the moment). >> >> regards, >> Michael >> >> >> >> > Am 29.01.2015 um 06:29 schrieb Kevin Laurie < >> superinterstellar(a)gmail.com>: >> > >> > Dear Peter, >> > Noted. Thanks for your input. Appreciate it. >> > At this point my urgent most priority is to get FTS. Waiting 3 mins for >> a >> > body search email is bad(but of course my email box is large) >> > I need to have this sorted out by today as I have been putting this off >> for >> > too long mostly because of lack of troubleshooting documentation >> online(if >> > i do figure this out will create some for guidance) >> > Regards >> > Kevin >> > >> > On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur <petehodur(a)gmail.com> >> wrote: >> > >> >> >> >> >> >> On Thursday, January 29, 2015, Kevin Laurie < >> superinterstellar(a)gmail.com> >> >> wrote: >> >> >> >>> Dear Peter, >> >>> Oh. Sorry(didnt know you were addressing someone else) my apologies. >> But >> >>> as you can see , I am desperately trying to address this issue. >> >>> >> >> >> >> No problem ;))) i wrote because my search result is good but not so >> >> good as someone wrote here couple of days ago. >> >> >> >> >> >> >> >>> I have 1 disk with 200GB running on a VPS. >> >>> The file system is ext4. >> >>> >> >> >> >> Im not dovecot expert, but i think this is your problem. You need more >> >> iops. If you need performance, VPS may be (if connected to fast SAN) >> but >> >> basicly is not a good choice. >> >> >> >> >> >>> I think I will need to implement FTS to fix this as I will need body >> >>> searches. >> >>> >> >> >> >> FTS could be a solution. You are right. >> >> >> >> >> >>> Have you tried FTS before? >> >>> >> >> >> >> Im sorry, never :( >> >> >> >> But may be someone more skilled than me could answer the main question, >> >> how much dovecot index helps in case imap search agains body. >> >> >> >> Im not sure, but my opinion is that it helps little (may be not at all) >> >> >> >> Ps: may be you use it, but do not forget to setup delivering from mail >> >> server via LDA or LMTP instead of direct to maildirs. This is better >> >> because dovecot updates your indexes at the time of message arrival. >> Later >> >> accesses should be faster. >> >> >> >> Peter >> >> >> >> >> >> >> >>> >> >>> >> >>> Regards >> >>> Kevin >> >>> >> >> >> >>> >> >>> On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur <petehodur(a)gmail.com> >> >>> wrote: >> >>> >> >>>> Kevin, >> >>>> >> >>>> My message was not addressed to you ;) because my results are NOT so >> >>>> good as someone wrote it here. I can fulltext cca 8k messages in 7-8 >> >>>> seconds. >> >>>> >> >>>> Someone wrote that he has results cca 22k messages in 4 seconds :( >> >>>> >> >>>> Im not sure, but the answer is may be in the index and disk >> subsystem. >> >>>> >> >>>> If and only if dovecot index does not have saved in the index >> keywords >> >>>> from BODY of messages the problem is your/my disk subsystem. >> >>>> >> >>>> Generally, messages from mailing list like this are pretty small. The >> >>>> problem is that dovecot must read all messages. If you have maildir >> its >> >>>> mean open and read many files. >> >>>> >> >>>> But my test search was agains archive of my personal inbox - so not >> onoy >> >>>> small messages like this but also messages with big attachments etc. >> >>>> >> >>>> If dovecot index only headers, than it in case of full text search >> read >> >>>> whole messages - mime parts allows that plain text body can be after >> >>>> attachment etc. >> >>>> >> >>>> And here is it all about iops and throughput. >> >>>> >> >>>> How many disks you have? And what setup? >> >>>> >> >>>> Generally, the only good setup is many smaller disks in RAID 10 >> >>>> (stripped mirrors) and if your fs allows it, with added read cache - >> in >> >>>> case of zfs: l2arc on ssd. >> >>>> >> >>>> >> >>>> >> >>>> Pete >> >>>> >> >>>> >> >>>> >> >>>> On Thursday, January 29, 2015, Kevin Laurie < >> superinterstellar(a)gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Hi Peter, >> >>>>> Sorry, >> >>>>> >> >>>>> I think its maildir >> >>>>> Output of my dovecot -n is listed below:- >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> # 2.2.9: /etc/dovecot/dovecot.conf >> >>>>> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS >> >>>>> auth_master_user_separator = * >> >>>>> auth_mechanisms = PLAIN LOGIN >> >>>>> dict { >> >>>>> acl = mysql:/etc/dovecot/dovecot- >> >>>>> share-folder.conf >> >>>>> quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf >> >>>>> } >> >>>>> first_valid_uid = 2000 >> >>>>> last_valid_uid = 2000 >> >>>>> listen = * >> >>>>> log_path = /var/log/dovecot.log >> >>>>> mail_debug = yes >> >>>>> mail_gid = 2000 >> >>>>> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ >> >>>>> mail_plugins = quota fts >> >>>>> mail_uid = 2000 >> >>>>> managesieve_notify_capability = mailto >> >>>>> managesieve_sieve_capability = fileinto reject envelope >> >>>>> encoded-character vacation subaddress comparator-i;ascii-numeric >> relational >> >>>>> regex imap4flags copy include variables body enotify environment >> mailbox >> >>>>> date ihave >> >>>>> namespace { >> >>>>> inbox = yes >> >>>>> location = >> >>>>> mailbox Drafts { >> >>>>> auto = subscribe >> >>>>> special_use = \Drafts >> >>>>> } >> >>>>> mailbox Junk { >> >>>>> auto = subscribe >> >>>>> special_use = \Junk >> >>>>> } >> >>>>> mailbox Sent { >> >>>>> auto = subscribe >> >>>>> special_use = \Sent >> >>>>> } >> >>>>> mailbox "Sent Messages" { >> >>>>> auto = no >> >>>>> special_use = \Sent >> >>>>> } >> >>>>> mailbox Spam { >> >>>>> auto = no >> >>>>> special_use = \Junk >> >>>>> } >> >>>>> mailbox Trash { >> >>>>> auto = subscribe >> >>>>> special_use = \Trash >> >>>>> } >> >>>>> prefix = >> >>>>> separator = / >> >>>>> type = private >> >>>>> } >> >>>>> namespace { >> >>>>> list = children >> >>>>> location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u >> >>>>> prefix = Shared/%%u/ >> >>>>> separator = / >> >>>>> subscriptions = yes >> >>>>> type = shared >> >>>>> } >> >>>>> passdb { >> >>>>> args = /etc/dovecot/dovecot-mysql.conf >> >>>>> driver = sql >> >>>>> } >> >>>>> passdb { >> >>>>> args = /etc/dovecot/dovecot-master-users-password >> >>>>> driver = passwd-file >> >>>>> master = yes >> >>>>> } >> >>>>> plugin { >> >>>>> acl = vfile >> >>>>> acl_shared_dict = proxy::acl >> >>>>> auth_socket_path = /var/run/dovecot/auth-master >> >>>>> quota = dict:user::proxy::quotadict >> >>>>> quota_rule = *:storage=1G >> >>>>> quota_warning = storage=85%% quota-warning 85 %u >> >>>>> quota_warning2 = storage=90%% quota-warning 90 %u >> >>>>> quota_warning3 = storage=95%% quota-warning 95 %u >> >>>>> sieve = /%Lh/sieve/dovecot.sieve >> >>>>> sieve_default = /var/vmail/sieve/dovecot.sieve >> >>>>> sieve_dir = /%Lh/sieve >> >>>>> sieve_global_dir = /var/vmail/sieve >> >>>>> } >> >>>>> protocols = pop3 imap sieve lmtp >> >>>>> service auth { >> >>>>> unix_listener /var/spool/postfix/private/dovecot-auth { >> >>>>> group = postfix >> >>>>> mode = 0666 >> >>>>> user = postfix >> >>>>> } >> >>>>> unix_listener auth-master { >> >>>>> group = vmail >> >>>>> mode = 0666 >> >>>>> user = vmail >> >>>>> } >> >>>>> unix_listener auth-userdb { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> service dict { >> >>>>> unix_listener dict { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> service imap-login { >> >>>>> process_limit = 500 >> >>>>> service_count = 1 >> >>>>> } >> >>>>> service lmtp { >> >>>>> executable = lmtp -L >> >>>>> inet_listener lmtp { >> >>>>> port = 24 >> >>>>> } >> >>>>> process_min_avail = 5 >> >>>>> unix_listener /var/spool/postfix/private/dovecot-lmtp { >> >>>>> group = postfix >> >>>>> mode = 0600 >> >>>>> user = postfix >> >>>>> } >> >>>>> user = vmail >> >>>>> } >> >>>>> service pop3-login { >> >>>>> service_count = 1 >> >>>>> } >> >>>>> service quota-warning { >> >>>>> executable = script /usr/local/bin/dovecot-quota-warning.sh >> >>>>> unix_listener quota-warning { >> >>>>> group = vmail >> >>>>> mode = 0660 >> >>>>> user = vmail >> >>>>> } >> >>>>> } >> >>>>> ssl = required >> >>>>> ssl_cert = </etc/ssl/certs/sicl.net.crt >> >>>>> ssl_key = </etc/ssl/private/sicl.net.key >> >>>>> userdb { >> >>>>> args = /etc/dovecot/dovecot-mysql.conf >> >>>>> driver = sql >> >>>>> } >> >>>>> protocol lda { >> >>>>> auth_socket_path = /var/run/dovecot/auth-master >> >>>>> lda_mailbox_autocreate = yes >> >>>>> log_path = /var/log/dovecot-sieve.log >> >>>>> mail_plugins = quota fts sieve >> >>>>> postmaster_address = root >> >>>>> } >> >>>>> protocol lmtp { >> >>>>> info_log_path = /var/log/dovecot-lmtp.log >> >>>>> lmtp_save_to_detail_mailbox = yes >> >>>>> mail_plugins = quota sieve >> >>>>> postmaster_address = postmaster >> >>>>> recipient_delimiter = + >> >>>>> } >> >>>>> protocol imap { >> >>>>> imap_client_workarounds = tb-extra-mailbox-sep >> >>>>> mail_plugins = quota fts imap_quota >> >>>>> } >> >>>>> protocol pop3 { >> >>>>> mail_plugins = quota fts >> >>>>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh >> >>>>> pop3_uidl_format = %08Xu%08Xv >> >>>>> } >> >>>>> >> >>>>> >> >>>>> On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie < >> >>>>> superinterstellar(a)gmail.com> wrote: >> >>>>> >> >>>>>> Dear Peter, >> >>>>>> >> >>>>>> My inbox is MDA_external >> >>>>>> Storage: 17GB of 24GB >> >>>>>> >> >>>>>> Subject / From / To is fast but FTS(Full Text Search) for body is >> >>>>>> horrible. I suppose this is where we need Apache Solr. >> >>>>>> >> >>>>>> Do you think my mail storage format is bad? Do I need to change for >> >>>>>> better performance? >> >>>>>> Please advise >> >>>>>> Kevin >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur(a)gmail.com >> > >> >>>>>> wrote: >> >>>>>> >> >>>>>>>> * Kevin Laurie <superinterstellar(a)gmail.com> 2015.01.24 19:41: >> >>>>>>>> >> >>>>>>>>> Currently the time it takes to search 25,000mails is 4mins. If >> >>>>>>> indexed >> >>>>>>>> how >> >>>>>>>>> much faster are we looking at? >> >>>>>>>> >> >>>>>>>> With a current version of Dovecot a search is pretty fast >> _without_ >> >>>>>>> using >> >>>>>>>> external indexes. I have a view defined (virtual plugin) with >> >>>>>>> around 22.000 >> >>>>>>>> messages in it, and searching the full view only takes 2.5 >> seconds: >> >>>>>>>> >> >>>>>>>> >> >>>>>>> hmmm, could you please tell me more about your setup? What storage >> >>>>>>> format >> >>>>>>> do you use? Maildir or? What is the cumulative size of your >> messages? >> >>>>>>> >> >>>>>>> My results without FTS on ZFS FS with SSD L2ARC are not so good: >> >>>>>>> >> >>>>>>> a4 select INBOX._OLD-OUTLOOK >> >>>>>>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk >> >>>>>>> JunkRecorded $Forwarded) >> >>>>>>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft >> $Junk >> >>>>>>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. >> >>>>>>> ** 7748 EXISTS* >> >>>>>>> * 0 RECENT >> >>>>>>> * OK [UIDVALIDITY 1421198037] UIDs valid >> >>>>>>> * OK [UIDNEXT 11509] Predicted next UID >> >>>>>>> * OK [HIGHESTMODSEQ 12204] Highest >> >>>>>>> a4 OK [READ-WRITE] Select completed (0.001 secs). >> >>>>>>> >> >>>>>>> a5 search charset utf-8 body "mall" >> >>>>>>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 >> 240 >> >>>>>>> 249 >> >>>>>>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 >> 385 >> >>>>>>> 389 390 >> >>>>>>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 >> 531 >> >>>>>>> 532 533 >> >>>>>>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 >> 672 >> >>>>>>> 677 692 >> >>>>>>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 >> 821 >> >>>>>>> 833 855 >> >>>>>>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 >> 1010 >> >>>>>>> 1011 >> >>>>>>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 >> 1137 >> >>>>>>> 1140 >> >>>>>>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 >> 1282 >> >>>>>>> 1286 >> >>>>>>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 >> 1471 >> >>>>>>> 1494 >> >>>>>>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 >> 1606 >> >>>>>>> 1611 >> >>>>>>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 >> 1735 >> >>>>>>> 1781 >> >>>>>>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 >> 2002 >> >>>>>>> 2006 >> >>>>>>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 >> 2208 >> >>>>>>> 2210 >> >>>>>>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 >> 2377 >> >>>>>>> 2378 >> >>>>>>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 >> 2539 >> >>>>>>> 2551 >> >>>>>>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 >> 2754 >> >>>>>>> 2780 >> >>>>>>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 >> 2936 >> >>>>>>> 2938 >> >>>>>>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 >> 3161 >> >>>>>>> 3178 >> >>>>>>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 >> 3386 >> >>>>>>> 3428 >> >>>>>>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 >> 3801 >> >>>>>>> 3808 >> >>>>>>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 >> 3953 >> >>>>>>> 3954 >> >>>>>>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 >> 4059 >> >>>>>>> 4063 >> >>>>>>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 >> 4145 >> >>>>>>> 4146 >> >>>>>>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 >> 4258 >> >>>>>>> 4259 >> >>>>>>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 >> 4502 >> >>>>>>> 4538 >> >>>>>>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 >> 4594 >> >>>>>>> 4604 >> >>>>>>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 >> 4787 >> >>>>>>> 4788 >> >>>>>>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 >> 4901 >> >>>>>>> 4905 >> >>>>>>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 >> 5129 >> >>>>>>> 5131 >> >>>>>>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 >> 5276 >> >>>>>>> 5277 >> >>>>>>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 >> 5391 >> >>>>>>> 5392 >> >>>>>>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 >> 5642 >> >>>>>>> 5647 >> >>>>>>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 >> 5866 >> >>>>>>> 5867 >> >>>>>>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 >> 5958 >> >>>>>>> 5990 >> >>>>>>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 >> 6233 >> >>>>>>> 6271 >> >>>>>>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 >> 6482 >> >>>>>>> 6492 >> >>>>>>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 >> 6852 >> >>>>>>> 6859 >> >>>>>>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 >> 7042 >> >>>>>>> 7108 >> >>>>>>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 >> 7317 >> >>>>>>> 7322 >> >>>>>>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 >> 7417 >> >>>>>>> 7418 >> >>>>>>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 >> 7502 >> >>>>>>> 7503 >> >>>>>>> 7558 7588 7589 7628 7685 7695 7699 7703 7723 >> >>>>>>> *a5 OK Search completed (7.846 secs).* >> >>>>>>> >> >>>>>>> >> >>>>>>> Searching against "subject" is pretty fast, few miliseconds ... >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> >> >

1 0

Indexing Mail faster
by Kevin Laurie 29 Jan '15

29 Jan '15

Hi, I am trying to get faster search results on our webmail client(Roundcube). Besides using Lucene for FTS are there other options? Would having all mails indexed give fast results? Currently the time it takes to search 25,000mails is 4mins. If indexed how much faster are we looking at? Really appreciate if someone could advise about this. Thanks Kevin

8 28

Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 29 Jan '15

29 Jan '15

Sorry if this is a duplicate message. I'm new in this mailing list and not sure if the first message really got posted ... Dovecot version: 2.2.9 We're using doveadm backup to migrate user mailboxes from cyrus to dovecot. I defined a master user for the old cyrus mailserver in imapc backend to get access to all the mailboxes. Now I realized that after migration every user's "Inbox" or "Trash" on the new dovecot server has the same mailbox-guid, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox dd9456... trash 5549cc... Is this a wanted behaviour? If i do not migrate with doveadm backup and just manually create mailboxes, the mailbox-guids are globally unique, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox c543fe... trash t49ccc... Thanks! Christian

1 0

Re: Indexing Mail faster
by Kevin Laurie 29 Jan '15

29 Jan '15

Dear Peter, Noted. Thanks for your input. Appreciate it. At this point my urgent most priority is to get FTS. Waiting 3 mins for a body search email is bad(but of course my email box is large) I need to have this sorted out by today as I have been putting this off for too long mostly because of lack of troubleshooting documentation online(if i do figure this out will create some for guidance) Regards Kevin On Thu, Jan 29, 2015 at 1:18 PM, Peter Hodur <petehodur(a)gmail.com> wrote: > > > On Thursday, January 29, 2015, Kevin Laurie <superinterstellar(a)gmail.com> > wrote: > >> Dear Peter, >> Oh. Sorry(didnt know you were addressing someone else) my apologies. But >> as you can see , I am desperately trying to address this issue. >> > > No problem ;))) i wrote because my search result is good but not so > good as someone wrote here couple of days ago. > > > >> I have 1 disk with 200GB running on a VPS. >> The file system is ext4. >> > > Im not dovecot expert, but i think this is your problem. You need more > iops. If you need performance, VPS may be (if connected to fast SAN) but > basicly is not a good choice. > > >> I think I will need to implement FTS to fix this as I will need body >> searches. >> > > FTS could be a solution. You are right. > > >> Have you tried FTS before? >> > > Im sorry, never :( > > But may be someone more skilled than me could answer the main question, > how much dovecot index helps in case imap search agains body. > > Im not sure, but my opinion is that it helps little (may be not at all) > > Ps: may be you use it, but do not forget to setup delivering from mail > server via LDA or LMTP instead of direct to maildirs. This is better > because dovecot updates your indexes at the time of message arrival. Later > accesses should be faster. > > Peter > > > >> >> >> Regards >> Kevin >> > >> >> On Thu, Jan 29, 2015 at 12:55 PM, Peter Hodur <petehodur(a)gmail.com> >> wrote: >> >>> Kevin, >>> >>> My message was not addressed to you ;) because my results are NOT so >>> good as someone wrote it here. I can fulltext cca 8k messages in 7-8 >>> seconds. >>> >>> Someone wrote that he has results cca 22k messages in 4 seconds :( >>> >>> Im not sure, but the answer is may be in the index and disk subsystem. >>> >>> If and only if dovecot index does not have saved in the index keywords >>> from BODY of messages the problem is your/my disk subsystem. >>> >>> Generally, messages from mailing list like this are pretty small. The >>> problem is that dovecot must read all messages. If you have maildir its >>> mean open and read many files. >>> >>> But my test search was agains archive of my personal inbox - so not onoy >>> small messages like this but also messages with big attachments etc. >>> >>> If dovecot index only headers, than it in case of full text search read >>> whole messages - mime parts allows that plain text body can be after >>> attachment etc. >>> >>> And here is it all about iops and throughput. >>> >>> How many disks you have? And what setup? >>> >>> Generally, the only good setup is many smaller disks in RAID 10 >>> (stripped mirrors) and if your fs allows it, with added read cache - in >>> case of zfs: l2arc on ssd. >>> >>> >>> >>> Pete >>> >>> >>> >>> On Thursday, January 29, 2015, Kevin Laurie <superinterstellar(a)gmail.com> >>> wrote: >>> >>>> Hi Peter, >>>> Sorry, >>>> >>>> I think its maildir >>>> Output of my dovecot -n is listed below:- >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> # 2.2.9: /etc/dovecot/dovecot.conf >>>> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS >>>> auth_master_user_separator = * >>>> auth_mechanisms = PLAIN LOGIN >>>> dict { >>>> acl = mysql:/etc/dovecot/dovecot- >>>> share-folder.conf >>>> quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf >>>> } >>>> first_valid_uid = 2000 >>>> last_valid_uid = 2000 >>>> listen = * >>>> log_path = /var/log/dovecot.log >>>> mail_debug = yes >>>> mail_gid = 2000 >>>> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/ >>>> mail_plugins = quota fts >>>> mail_uid = 2000 >>>> managesieve_notify_capability = mailto >>>> managesieve_sieve_capability = fileinto reject envelope >>>> encoded-character vacation subaddress comparator-i;ascii-numeric relational >>>> regex imap4flags copy include variables body enotify environment mailbox >>>> date ihave >>>> namespace { >>>> inbox = yes >>>> location = >>>> mailbox Drafts { >>>> auto = subscribe >>>> special_use = \Drafts >>>> } >>>> mailbox Junk { >>>> auto = subscribe >>>> special_use = \Junk >>>> } >>>> mailbox Sent { >>>> auto = subscribe >>>> special_use = \Sent >>>> } >>>> mailbox "Sent Messages" { >>>> auto = no >>>> special_use = \Sent >>>> } >>>> mailbox Spam { >>>> auto = no >>>> special_use = \Junk >>>> } >>>> mailbox Trash { >>>> auto = subscribe >>>> special_use = \Trash >>>> } >>>> prefix = >>>> separator = / >>>> type = private >>>> } >>>> namespace { >>>> list = children >>>> location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u >>>> prefix = Shared/%%u/ >>>> separator = / >>>> subscriptions = yes >>>> type = shared >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-mysql.conf >>>> driver = sql >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-master-users-password >>>> driver = passwd-file >>>> master = yes >>>> } >>>> plugin { >>>> acl = vfile >>>> acl_shared_dict = proxy::acl >>>> auth_socket_path = /var/run/dovecot/auth-master >>>> quota = dict:user::proxy::quotadict >>>> quota_rule = *:storage=1G >>>> quota_warning = storage=85%% quota-warning 85 %u >>>> quota_warning2 = storage=90%% quota-warning 90 %u >>>> quota_warning3 = storage=95%% quota-warning 95 %u >>>> sieve = /%Lh/sieve/dovecot.sieve >>>> sieve_default = /var/vmail/sieve/dovecot.sieve >>>> sieve_dir = /%Lh/sieve >>>> sieve_global_dir = /var/vmail/sieve >>>> } >>>> protocols = pop3 imap sieve lmtp >>>> service auth { >>>> unix_listener /var/spool/postfix/private/dovecot-auth { >>>> group = postfix >>>> mode = 0666 >>>> user = postfix >>>> } >>>> unix_listener auth-master { >>>> group = vmail >>>> mode = 0666 >>>> user = vmail >>>> } >>>> unix_listener auth-userdb { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> service dict { >>>> unix_listener dict { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> service imap-login { >>>> process_limit = 500 >>>> service_count = 1 >>>> } >>>> service lmtp { >>>> executable = lmtp -L >>>> inet_listener lmtp { >>>> port = 24 >>>> } >>>> process_min_avail = 5 >>>> unix_listener /var/spool/postfix/private/dovecot-lmtp { >>>> group = postfix >>>> mode = 0600 >>>> user = postfix >>>> } >>>> user = vmail >>>> } >>>> service pop3-login { >>>> service_count = 1 >>>> } >>>> service quota-warning { >>>> executable = script /usr/local/bin/dovecot-quota-warning.sh >>>> unix_listener quota-warning { >>>> group = vmail >>>> mode = 0660 >>>> user = vmail >>>> } >>>> } >>>> ssl = required >>>> ssl_cert = </etc/ssl/certs/sicl.net.crt >>>> ssl_key = </etc/ssl/private/sicl.net.key >>>> userdb { >>>> args = /etc/dovecot/dovecot-mysql.conf >>>> driver = sql >>>> } >>>> protocol lda { >>>> auth_socket_path = /var/run/dovecot/auth-master >>>> lda_mailbox_autocreate = yes >>>> log_path = /var/log/dovecot-sieve.log >>>> mail_plugins = quota fts sieve >>>> postmaster_address = root >>>> } >>>> protocol lmtp { >>>> info_log_path = /var/log/dovecot-lmtp.log >>>> lmtp_save_to_detail_mailbox = yes >>>> mail_plugins = quota sieve >>>> postmaster_address = postmaster >>>> recipient_delimiter = + >>>> } >>>> protocol imap { >>>> imap_client_workarounds = tb-extra-mailbox-sep >>>> mail_plugins = quota fts imap_quota >>>> } >>>> protocol pop3 { >>>> mail_plugins = quota fts >>>> pop3_client_workarounds = outlook-no-nuls oe-ns-eoh >>>> pop3_uidl_format = %08Xu%08Xv >>>> } >>>> >>>> >>>> On Thu, Jan 29, 2015 at 12:37 PM, Kevin Laurie < >>>> superinterstellar(a)gmail.com> wrote: >>>> >>>>> Dear Peter, >>>>> >>>>> My inbox is MDA_external >>>>> Storage: 17GB of 24GB >>>>> >>>>> Subject / From / To is fast but FTS(Full Text Search) for body is >>>>> horrible. I suppose this is where we need Apache Solr. >>>>> >>>>> Do you think my mail storage format is bad? Do I need to change for >>>>> better performance? >>>>> Please advise >>>>> Kevin >>>>> >>>>> >>>>> >>>>> On Thu, Jan 29, 2015 at 12:25 PM, Peter Hodur <petehodur(a)gmail.com> >>>>> wrote: >>>>> >>>>>> > * Kevin Laurie <superinterstellar(a)gmail.com> 2015.01.24 19:41: >>>>>> > >>>>>> > > Currently the time it takes to search 25,000mails is 4mins. If >>>>>> indexed >>>>>> > how >>>>>> > > much faster are we looking at? >>>>>> > >>>>>> > With a current version of Dovecot a search is pretty fast _without_ >>>>>> using >>>>>> > external indexes. I have a view defined (virtual plugin) with >>>>>> around 22.000 >>>>>> > messages in it, and searching the full view only takes 2.5 seconds: >>>>>> > >>>>>> > >>>>>> hmmm, could you please tell me more about your setup? What storage >>>>>> format >>>>>> do you use? Maildir or? What is the cumulative size of your messages? >>>>>> >>>>>> My results without FTS on ZFS FS with SSD L2ARC are not so good: >>>>>> >>>>>> a4 select INBOX._OLD-OUTLOOK >>>>>> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk $NotJunk >>>>>> JunkRecorded $Forwarded) >>>>>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $Junk >>>>>> $NotJunk JunkRecorded $Forwarded \*)] Flags permitted. >>>>>> ** 7748 EXISTS* >>>>>> * 0 RECENT >>>>>> * OK [UIDVALIDITY 1421198037] UIDs valid >>>>>> * OK [UIDNEXT 11509] Predicted next UID >>>>>> * OK [HIGHESTMODSEQ 12204] Highest >>>>>> a4 OK [READ-WRITE] Select completed (0.001 secs). >>>>>> >>>>>> a5 search charset utf-8 body "mall" >>>>>> * SEARCH 2 49 101 117 158 171 185 192 197 202 207 223 228 234 236 240 >>>>>> 249 >>>>>> 279 280 281 287 288 289 290 297 321 327 337 344 351 360 370 373 385 >>>>>> 389 390 >>>>>> 391 398 405 413 424 444 458 463 470 474 480 482 505 513 520 530 531 >>>>>> 532 533 >>>>>> 543 559 560 561 562 563 566 588 593 597 625 630 639 644 656 671 672 >>>>>> 677 692 >>>>>> 720 723 734 738 741 745 752 755 757 765 775 777 784 791 818 820 821 >>>>>> 833 855 >>>>>> 863 864 868 881 896 910 917 922 926 928 931 991 996 997 998 1000 1010 >>>>>> 1011 >>>>>> 1012 1014 1018 1019 1026 1047 1068 1077 1095 1101 1105 1122 1136 1137 >>>>>> 1140 >>>>>> 1155 1160 1166 1171 1179 1180 1197 1208 1229 1239 1258 1263 1271 1282 >>>>>> 1286 >>>>>> 1290 1298 1319 1364 1365 1370 1386 1408 1410 1429 1463 1465 1470 1471 >>>>>> 1494 >>>>>> 1518 1522 1529 1530 1536 1541 1548 1571 1581 1585 1588 1594 1605 1606 >>>>>> 1611 >>>>>> 1612 1619 1620 1625 1652 1666 1667 1729 1730 1731 1732 1733 1734 1735 >>>>>> 1781 >>>>>> 1782 1817 1818 1897 1900 1921 1940 1946 1960 1972 1981 1995 1998 2002 >>>>>> 2006 >>>>>> 2028 2049 2057 2095 2100 2157 2168 2181 2185 2192 2203 2204 2207 2208 >>>>>> 2210 >>>>>> 2220 2225 2255 2273 2282 2283 2288 2289 2317 2320 2340 2367 2374 2377 >>>>>> 2378 >>>>>> 2379 2384 2389 2402 2409 2436 2459 2475 2476 2488 2504 2519 2538 2539 >>>>>> 2551 >>>>>> 2566 2572 2597 2599 2603 2617 2629 2664 2698 2716 2731 2733 2753 2754 >>>>>> 2780 >>>>>> 2805 2808 2815 2818 2850 2861 2862 2867 2886 2896 2900 2914 2931 2936 >>>>>> 2938 >>>>>> 2939 2950 2969 2990 3017 3019 3062 3075 3094 3101 3115 3138 3159 3161 >>>>>> 3178 >>>>>> 3185 3190 3204 3217 3218 3248 3263 3265 3266 3273 3282 3288 3295 3386 >>>>>> 3428 >>>>>> 3453 3476 3478 3479 3511 3548 3606 3629 3693 3694 3737 3793 3799 3801 >>>>>> 3808 >>>>>> 3812 3814 3815 3834 3849 3860 3862 3880 3910 3917 3930 3932 3952 3953 >>>>>> 3954 >>>>>> 3957 3959 3968 3971 3973 3978 3979 3980 4008 4022 4040 4057 4058 4059 >>>>>> 4063 >>>>>> 4064 4066 4069 4070 4075 4096 4112 4131 4132 4133 4141 4143 4144 4145 >>>>>> 4146 >>>>>> 4147 4167 4174 4199 4201 4202 4203 4206 4211 4217 4218 4226 4229 4258 >>>>>> 4259 >>>>>> 4267 4287 4357 4359 4363 4364 4365 4367 4390 4391 4462 4475 4497 4502 >>>>>> 4538 >>>>>> 4540 4552 4557 4558 4561 4563 4567 4571 4572 4573 4575 4577 4593 4594 >>>>>> 4604 >>>>>> 4611 4619 4628 4638 4639 4662 4672 4678 4679 4692 4696 4785 4786 4787 >>>>>> 4788 >>>>>> 4789 4792 4793 4794 4802 4817 4818 4819 4820 4836 4857 4874 4887 4901 >>>>>> 4905 >>>>>> 4906 4907 4908 4911 4925 4928 4940 4941 4953 5060 5103 5116 5118 5129 >>>>>> 5131 >>>>>> 5136 5158 5163 5182 5184 5203 5212 5216 5269 5270 5271 5272 5273 5276 >>>>>> 5277 >>>>>> 5278 5286 5301 5302 5303 5304 5320 5341 5358 5371 5385 5389 5390 5391 >>>>>> 5392 >>>>>> 5395 5409 5412 5424 5425 5437 5544 5587 5600 5602 5613 5615 5620 5642 >>>>>> 5647 >>>>>> 5666 5668 5687 5696 5700 5736 5740 5749 5764 5783 5809 5814 5853 5866 >>>>>> 5867 >>>>>> 5877 5888 5895 5896 5897 5898 5899 5908 5910 5911 5912 5939 5950 5958 >>>>>> 5990 >>>>>> 6000 6059 6074 6095 6097 6112 6137 6141 6189 6193 6212 6228 6229 6233 >>>>>> 6271 >>>>>> 6273 6275 6285 6310 6317 6335 6383 6384 6397 6427 6430 6459 6463 6482 >>>>>> 6492 >>>>>> 6506 6565 6585 6620 6670 6673 6675 6705 6715 6716 6741 6812 6826 6852 >>>>>> 6859 >>>>>> 6895 6896 6907 6913 6919 6935 6943 6948 6979 7023 7025 7035 7039 7042 >>>>>> 7108 >>>>>> 7131 7145 7163 7171 7172 7194 7198 7199 7203 7256 7257 7294 7303 7317 >>>>>> 7322 >>>>>> 7343 7344 7347 7348 7352 7386 7390 7391 7392 7393 7407 7408 7409 7417 >>>>>> 7418 >>>>>> 7419 7420 7421 7426 7432 7437 7462 7467 7468 7473 7474 7475 7488 7502 >>>>>> 7503 >>>>>> 7558 7588 7589 7628 7685 7695 7699 7703 7723 >>>>>> *a5 OK Search completed (7.846 secs).* >>>>>> >>>>>> >>>>>> Searching against "subject" is pretty fast, few miliseconds ... >>>>>> >>>>> >>>>> >>>> >>

1 0

Dovecot & gethostbyname() vulnerability
by Timo Sirainen 28 Jan '15

28 Jan '15

So just a quick note: Dovecot in general doesn't do DNS lookups, except the ones in configuration files and such. So I don't think there are any Dovecot setups which do DNS lookups for untrusted hostnames. Also, gethostbyname() is used only if IPv6 support isn't compiled into Dovecot. And IPv6 support is enabled by default if the libc is detected to support it, so pretty much all systems use it.

1 0

Uniqueness of dovecot mailbox-guids
by Christian Binder Stadt Freilassing 28 Jan '15

28 Jan '15

Dovecot version: 2.2.9 I'm using doveadm backup to migrate about hundred user mailboxes from cyrus to dovecot. I defined a master user for the old cyrus mailserver in imapc backend to get access to all the mailboxes. Now I realized that after migration every user's "Inbox" or "Trash" on the new dovecot server has the same mailbox-guid, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox dd9456... trash 5549cc... Is this a wanted behaviour? If i do not migrate with doveadm backup and just manually create mailboxes, the mailbox-guids are globally unique, e.g: User a: inbox dd9456... trash 5549cc... User b: inbox c543fe... trash t49ccc... Thanks! Christian

1 0

dovecot migration
by mkawada＠redhat.com 28 Jan '15

28 Jan '15

Dear List, To migrate a dovecot server from dovecot-1.x to dovecot-2.x, then I wonder if the UIDL format "pop3_uidl_format" setting is required in the configuration file for clients not to re-download the same mails from the migrated mail server. Currently, no UIDL format is set in /etc/dovecot file of the one using dovecot-1.x server which means it is commented out as default like below. ]# grep pop3_uidl_format /etc/dovecot.conf #pop3_uidl_format = %08Xu%08Xv <---- Does this mean the "pop3_uidl_format" is in use as "%08Xu%08Xv" by default? I see the same in /etc/dovecot/conf.d/20-imap.conf of the one using dovecot-2.x, and have not been able to find any information saying it's the defualt setting. Here are the questions: 1.Can we say the "pop3_uidl_format = %08Xu%08Xv" is the enabled default setting? Or any ways to check the default? 2.Do they need to specify anything for the "pop3_uidl_format =" field for their migration in this case? Env infomation: dovecot-1.x dovecot-2.x mail_location = mbox Any information/clues would be greatly appreciated. Thanks,

3 4

sendmail not invoking dovecot-lda
by alvin 27 Jan '15

27 Jan '15

hi dovecot mailinglist - Configuration FreeBSD-9.3 sendmail -d0.1 == sendmail-8.14.9 <<-- dovecot --version == dovecot-2.2.15 # =================================================================== # I'm trying to get sendmail to invoke dovecot.m4 ( dovecot-lda ) to # deliver emails to dovecot's virtual users ( /etc/dovecot/passwd ) # or mysql/postgresql virtual users # # # sendmail -bv dovecot.VirtualUser@localhost --> returns "User Unknown" # # cat test.msg.txt | dovecot-lda -d dovecot.VirtualUser ===> works # email is delivered to dovecot.VirtualUser inbox # # =================================================================== - there are tons of howto and comments for dovecot + postfix and very little for dovecot + sendmail ... therefore, it'd be great if we/i can get this dovecot.m4 problem resolved ------------------------------------------------------------------------------ - status of what works and what does NOT work while testing dovecot's LDA ... ------------------------------------------------------------------------------ # # fyi.. recepient user is in the form of user, user@localhost and user(a)domain.com # sendmail -bv user.in/etc/passwd = deliverable sendmail -bv user.in/etc/mail/virtusertable = deliverable to sendmail virtual user -->> sendmail -bv dovecot.user.in/dovecot/virtual.passwd.file == user unknown dovecot user user.in/etc/passwd == returns UID/GID/home/mailbox ( deliverable ) dovecot user dovecot.user == returns UID/GID/home/mailbox ( deliverable ) # -------------------------------------------------------------------- # dovecot's LDA delivers the test email to the dovecot/mysql recepient # -------------------------------------------------------------------- cat test-email-with-headers.txt | dovecot-lda -d dovecot.user ( email delivered ) # ---------------------------------------------------------------- -->> # mail/mutt/sendmail canNOT deliver to dovecot recepient == FAILS # ---------------------------------------------------------------- -->> sendmail -bv dovecot.user.in/dovecot/virtual.passwd.file == user unknown 3 echo "testing mail to dovecot" | mail -v -s "testing mail to dovecot" doveccot.user == user unknown == ------------------------------------------------------- - my (relevant to dovecot ) sendmail.mc file looks like: ------------------------------------------------------- dnl # dnl # local mail delivery dnl # define(`PROCMAIL_MAILER_PATH', `/usr/local/bin/procmail')dnl FEATURE(local_procmail,`',`procmail -t -Y -a $h -d $u')dnl dnl # dnl # I'm trying to get sendmail to use dovecot.m4 to deliver local mail dnl # to dovecot's LDA ( dovecot-lda ) for mysql virtual users dnl # dnl uncomment and use either feature(...dovecot-lda) or mailer(dovecot.m4) dnl dnl FEATURE(`local_procmail', `/usr/local/libexec/dovecot/dovecot-lda',`/usr/local/libexec/dovecot/dovecot-lda -d $u') dnl MAILER(local) MAILER(smtp) dnl dnl MAILER(dovecot)dnl # did NOT work either dnl dnl # trying to use procmail to deliver local mail to system users ( /etc/passwd ) MAILER(procmail) dnl dnl # is the ordering of dovecot.m4 important relative to (local) and (procmail) dnl dnl # trying to use dovecot-lda to deliver email to dovecot's virtual users dnl # defined in /etc/dovecot/password dnl MAILER(dovecot)dnl # see below dnl dnl End of File ------------------------------------------------------- - my (relevant to dovecot ) sendmail.cf file looks like: ------------------------------------------------------- # # lots of deleted ... ??not?? important until Mlocal ---> MAILER(local) # Mlocal, P=/usr/libexec/mail.local, F=lsDFMAw5:/|@qPSXmnz9, S=EnvFromSMTP/HdrFromL, R=EnvToL/ HdrToL, T=DNS/RFC822/SMTP, A=mail.local -l Mprog, P=/bin/sh, F=lsDFMoqeu9, S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, D=$z:/, T=X-Unix/X-Unix/X-Unix, A=sh -c $u ##################################### ### SMTP Mailer specification ### ##################################### # # lots of deleted ... ??not?? important until procmail stuff ---> MAILER(procmail) # ######################*****############## ### PROCMAIL Mailer specification ### ##################*****################## ##### $Id: procmail.m4,v 8.23 2013-11-22 20:51:14 ca Exp $ ##### Mprocmail, P=/usr/local/bin/procmail, F=DFMSPhnu9, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP/HdrFr omSMTP, T=DNS/RFC822/X-Unix, A=procmail -Y -m $h $f $u ######################*****######################### ### ### ### 10-Jan-15 amo Dovecot.m4 ### ### ### ### dovecot.m4 stuff ### ### ### ##############################*****################# Mdovecot, P=/usr/local/libexec/dovecot/dovecot-lda, F=DFMPhnu9, S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, T=DNS/RFC822/X-Unix, A=/usr/local/libexec/dovecot/dovecot-lda -d $u # # End of sendmail.cf ===================================================================== - dovecot has an example dovecot.m4 ... but it doesn't allow sendmail to call dovecot to deliver local mail with dovecot-lda ===================================================================== http://wiki2.dovecot.org/LDA/Sendmail ######################################### ### DOVECOT Mailer specification ### ######################################### # # # dovecot virtual users defined in: /etc/dovecot/password # # # # cp /etc/mail/dovecot.m4 /usr/share/sendmail/cf/mailer # # # # cd /etc/mail && make # # /etc/rc.d/sendmail restart # # # # the dovecot.m4 stanza are in the resulting sendmail.cf # # # ######################################### Mdovecot, P=/usr/local/libexec/dovecot/dovecot-lda, F=DFMPhnu9, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP/HdrFromSMTP, T=DNS/RFC822/X-Unix, A=/usr/local/libexec/dovecot/dovecot-lda -d $u # # i've tried using these other options in lieu of the above : # # F=DFMPShnu9 # added "S" option # F=lsDFMAw5:/|@qSPfhn9, # # S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, # # # # tried adding U= between S= and T= lines # U=root:mail U=dovecot:mail U=dovenull:mail # # End of dovecot.m4 http://wiki2.dovecot.org/LDA/Sendmail also states that one could use these 3 statements in lieu of dovecot.m4 # # this replaces sendmail.cf Mlocal P=mail.local with P=dovecot-lda # FEATURE(`local_procmail', `/usr/local/libexec/dovecot/dovecot-lda', \ `/usr/local/libexec/dovecot/dovecot-lda -d $u') dnl MODIFY_MAILER_FLAGS(`LOCAL', `-f') MAILER(procmail) vi /etc/mail/mailertable # # cd /etc/mail && make && /etc/rc.d/sendmail restart # # make and restart sendmail after editing # virtualdomain.example.com vmail:vmail # other options where sendmail still wasn't able to execute dovecot-lda # virtualdomain.example.com dovecot:local # virtualdomain.example.com local:mail # virtualdomain.example.com root:mail ------------------------------------ - my relevant dovecot config options ------------------------------------ # # are there any other major options i'm missing in the dovecot config files # doveconf dovecot -a # # for /etc/passwd passdb { args = driver = pam .. } # # ------------------------------------------------------------------------ # need to get these dovecot's virtual users to be able send/receive emails # ------------------------------------------------------------------------ # sendmail -bv dovecot.VirtualUser@localhost # dovecot.VirtuaUser@localhost... User unknown # passdb { args = /etc/dovecot/passwd # # dovecot.VirtualUser:{PLAIN}password::::: # # end of file driver = passwd-file .. } # passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql .. } ... ... ... userdb { args = driver = passwd .. } # userdb { args = /etc/dovecot/password driver = passwd-file .. } # userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql .. } # # --------------------------------------------------------------------- # -- How do i get sendmail to execute dovecot-lda for incoming emails # -- directed to dovecot virtual users and/or mysql users ?? # --------------------------------------------------------------------- # # end of txt

2 2

How to quote the special wildcard characters '*' and '?'
by David Davidov 27 Jan '15

27 Jan '15

Hello, I am searching for a way to execute doveadm command against mailbox containing a special wildcard character in its name. For example: # doveadm expunge -u 'me?(a)example.com' mailbox 'INBOX.Junk Mail' SAVEDBEFORE 7 The result is: doveadm(root): Error: User listing returned failure doveadm: Error: Failed to iterate through some users with exit code 75 In the mail log: dovecot: auth: Error: Trying to iterate users, but userdbs don't support it To be clear: the question mark is a part of the mailbox's name. Not an attempt to use it as wildcard. Is there a way 1. To quote the special characters or 2. To disable wildcard matcing in the -u parameter regards, -- David Davidov | System Administrator | ICDSoft Ltd.

2 1

"error: number of redirect actions exceeds policy limit"
by Kai Sommer 27 Jan '15

27 Jan '15

Greetings, after testing a lot with my system I have to write to this list to ask for help from the cummunity. My system: * Ubuntu 12.04.5 LTS * Dovecot 2.0.19 * Dovecot config (extract) ––––– 8< ––––– plugin { sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_max_redirects = 10 } ––––– >8–––––– * Sieve filter rules ––––– 8< ––––– require ["copy"]; # rule:[Alles weiterleiten] if true { redirect "mail_1@mydomain"; redirect "mail_2@mydomain"; redirect "mail_3@mydomain"; redirect "mail_4@mydomain"; redirect "mail_5@mydomain"; redirect "mail_6@mydomain"; keep; stop; } ––––– >8–––––– Problem scenario: With the redirect-filter (see above) Dovecot redirects a incoming email to the first address and stops withe the message "error: number of redirect actions exceeds policy limit" (.dovecot.sieve.log). In my mail.log I see the message "Error: sieve: execution of script $PATH/.dovecot.sieve failed, but implicit keep was successful (user logfile $PATH/.dovecot.sieve.log may reveal additional details)". If I reduce the count of recipients to four – which is the Dovecot default number – everything will redireced as expected. I only could find an old problem report here [0] where as the posted solution doesn't work for me (as you can see above). (Also I tried the rule "redirect :copy" that doesn't changed something …) [0] <http://dovecot.org/pipermail/dovecot/2012-August/085038.html> → Is this problem known? – Is there a solution for that problem? Thanks a lot and best regards! – Kai -- Kai Sommer sig: 5E2E FA58 43BD 5599 4068 | pubkey: http://ma.ximi.se/mekey

2 1

Thunderbird: improper command pipelining after EHLO
by Leander Schäfer 27 Jan '15

27 Jan '15

I couldn't find working solutions for this anomalie on the net. What does this mean and does someone know how to fix this? postfix/smtpd[18757]: improper command pipelining after EHLO from unknown[192.168.10.233]: QUIT\r\n Thanks Best Regards, Leander

4 4

imap-login: Fatal: pipe() failed: Too many open files
by Leander Schäfer 26 Jan '15

26 Jan '15

Hi I keep on getting errors and can't connect/login to Dovecot. I did my research but unfortunately without success. It is for sure not ulimit because ulimit is set to unlimited per default already. Still , it complains about "Too many open files" but this is a test system and the service dovecot and postfix have just been started. No one except me is testing on this system. This is very weired ;/ root@WM-01 [~]$ service dovecot stop; service postfix stop ; echo "" > /var/log/maillog ; echo "" > /var/log/slapd.log ; service postfix start ; service dovecot start Stopping dovecot. Waiting for PIDS: 3245. postfix/postfix-script: stopping the Postfix mail system postfix/postfix-script: starting the Postfix mail system Starting dovecot. root@WM-01 [~]$ telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. root@WM-01 [~]$ cat /var/log/mail Jan 26 01:39:12 WM-01 dovecot: imap-login: Fatal: pipe() failed: Too many open files Jan 26 01:39:12 WM-01 dovecot: master: Error: service(imap-login): command startup failed, throttling for 4 secs If I uncoment port = 0 in POP and IMAP then it looks even worse then above. Telnet is not even closed by remote host and three of the "Too many open files" errors are showing up in maillog: root@WM-01 [~]$ telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ^] telnet> Connection closed. root@WM-01 [~]$ cat /var/log/maillog Jan 26 01:45:19 WM-01 postfix/postfix-script[60560]: starting the Postfix mail system Jan 26 01:45:19 WM-01 postfix/master[60562]: daemon started -- version 1.0, configuration /usr/local/etc/postfix Jan 26 01:45:19 WM-01 dovecot: master: Dovecot v2.2.15 starting up for imap, pop3, lmtp Jan 26 01:45:24 WM-01 dovecot: imap-login: Error: socket(login) failed: Too many open files Jan 26 01:45:24 WM-01 dovecot: imap-login: Error: auth: connect(login) failed: Too many open files Jan 26 01:45:24 WM-01 dovecot: imap-login: Error: net_accept() failed: Too many open files root@WM-01 [~]$ Does anyone have an idea what could be wrong here? I attached the relevant configuration below. I use OpenLDAP as backend. Please let me know if you need more information. Thank you very much. Best Regards, Leander # ========================== dovecot -n ============================== # # 2.2.15: /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.6 (3e924b1b6c5c+) # OS: FreeBSD 10.1-RELEASE amd64 ufs auth_debug = yes auth_debug_passwords = yes auth_failure_delay = 3 secs auth_mechanisms = plain login auth_verbose = yes auth_verbose_passwords = yes base_dir = /var/run/dovecot/ disable_plaintext_auth = no imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes login_greeting = Welcome to Mail Server. login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c %k mail_debug = yes mail_gid = vmail mail_location = maildir:/var/mail/%d/%n mail_plugins = quota expire mail_log notify mail_privileged_group = mail mail_temp_dir = /tmp/Dovecot mail_uid = vmail managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave duplicate namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { expire = Trash mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = maildir:User quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_rule = *:storage=1G quota_rule2 = Trash:storage=+30%% quota_rule3 = Sent:storage=+30%% quota_warning = storage=90%% quota-warning 90 %u quota_warning2 = storage=75%% quota-warning 75 %u } pop3_client_workarounds = outlook-no-nuls oe-ns-eoh postmaster_address = info(a)netocean.de sendmail_path = /usr/local/sbin/sendmail service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service imap-login { inet_listener imaps { port = 0 } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } user = vmail } service pop3-login { inet_listener pop3s { port = 0 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl = required ssl_cert = </etc/ssl/RootCA/certs/192.168.50.101.pem ssl_cipher_list = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:+CAMELLIA256:+AES256:+CAMELLIA128:+AES128:SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!ECDSA:CAMELLIA256-SHA:AES256-SHA:CAMELLIA128-SHA:AES128-SHA ssl_key = </etc/ssl/RootCA/certs/192.168.50.101.key ssl_prefer_server_ciphers = yes ssl_protocols = !SSLv2 !SSLv3 userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } verbose_ssl = yes protocol lda { mail_plugins = sieve } protocol imap { mail_max_userip_connections = 3 mail_plugins = imap_quota imap_zlib } protocol pop3 { mail_max_userip_connections = 3 } # ======================== dovecot/conf.d/auth-ldap.conf.ext =================== # passdb { driver = ldap # Path for LDAP configuration file, see example-config/dovecot-ldap.conf.ext args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext } # "prefetch" user database means that the passdb already provided the # needed information and there's no need to do a separate userdb lookup. # <doc/wiki/UserDatabase.Prefetch.txt> userdb { driver = prefetch } userdb { driver = ldap args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext # Default fields can be used to specify defaults that LDAP may override #default_fields = home=/home/virtual/%u } # ======================== dovecot/dovecot-ldap.conf.ext =================== # hosts = 127.0.0.1:389 dn = uid=dovecot,ou=systemuser,ou=mail,dc=MyDomain,dc=TLD dnpass = TopSecret tls = yes tls_cert_file = /etc/ssl/RootCA/certs/192.168.50.101.pem tls_key_file = /etc/ssl/RootCA/certs/192.168.50.101.key debug_level = -1 auth_bind = yes ldap_version = 3 base = ou=accounts,ou=mail,dc=MyDomain,dc=TLD scope = subtree user_attrs = mailStorageDirectory=home, mailUidNumber=uid, mailGidNumber=gid, mailQuotaSize=quota_rule=*:bytes=%$, mailQuotaCount=quota_rule2 user_filter = (&(objectClass=mailAccount)(mailAddress=%u)) pass_attrs = mailAddress=user,mailPassword=password pass_filter = (&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=%u)) default_pass_scheme = SSHA

3 8

Re: LDA input validation
by Joseph Tam 26 Jan '15

26 Jan '15

St?phane Cottin writes: >> You're stilling going to lose contents. If dspam fails, the mail >> is dumped, the LDA returns exit code 75, and the MTA will probably >> issue a bounce Email to the sender. > > from dovecot-lda man page : > > 75 A temporary failure. This is returned for almost all failures. See the log file for > details. (EX_TEMPFAIL) > > The mta keep the mail and retry delivery later, which is what I need. Ah, my mistake. But ... > I don't want to deliver email on dspam error, I prefer TEMPFAIL and > future retries, so I have a chance to fix the issue later without > loosing or deliver unprocessed mails ... you only have a limited time to fix it. Depending on the sending MTA's retry schedule, you might have only a few days or less to to deal with it before the sending MTA stops trying. If it's important not to lose mail, then the deliver on failure strategy would be more appropriate. You can even copy the input message that caused dpam to crash to an inspection queue for later perusal. Joseph Tam <jtam.home(a)gmail.com>

2 1

Re: LDA input validation
by Joseph Tam 26 Jan '15

26 Jan '15

St?phane Cottin writes: > dspam already send errors to syslog, the point here is to never loose > email contents. This was a wrong design, i'm now use a wrapper instead > ( see my previous post for details ). You're stilling going to lose contents. If dspam fails, the mail is dumped, the LDA returns exit code 75, and the MTA will probably issue a bounce Email to the sender. If you really don't want the recipient to lose Email, you should buffer the input into a file, run dspam on it, and if the output is not-null, pipe it to dovecot-lda, otherwise pipe the original input. Joseph Tam <jtam.home(a)gmail.com>

4 6

Re: sieve filter not working
by Christian Kivalo 24 Jan '15

24 Jan '15

Am 23. Jänner 2015 22:02:06 MEZ, schrieb Michael Williamson <michael.h.williamson(a)gmail.com> Hello, >> Another question: >> Local mail delivery of received mail from external sources does >indeed >> work? > >Yes. > That is good. >> If so, was that log excerpt in one of the former mails an example of >> such delivery? > >There are dovecot messages in "/var/log/maillog" showing only >imap-login & disconnected. >When I monitor "/var/log/maillog" while sending spam, there are no >messages containing 'dovecot'. I attached that output to a previous >post. It has only messages from postfix and amavis. > Ther should be messages from dovecot when receiving mail at least when dovecot lda has something to do with delivery. >home_mailbox = Maildir/ This tells postfix where to deliver the mails. dovecot is not configured to delivet mails. Take a look at dovecot wiki there are examples of how to incorporate dovecot lda into thendelivery process. >> I believe dovecot has actually nothing to do with delivering your >mails >> so no sieve filters are applied... > >Is it postfix? > yes, seems so >>> Thanks, >>> -Mike - christian

3 3

from Cyrus to Dovecot, seen status on cyrus.index files
by Norberto Bensa 24 Jan '15

24 Jan '15

Hello, Using cyrus2dovecot (https://github.com/a-schild/cyrus2dovecot) I have migrated some test mailboxes from Cyrus (2.4.17) to Dovecot (2.2.9). The problem I'm having is that the seen status is in cyrus.index files, which cyrus2dovecot doesn't understand. Does anyone use cyrus2dovecot for migrations? If so, how do you deal with seen status when it is stored in the .index files? Many thanks in advance, Norberto

4 6

Re: LDA input validation
by Joseph Tam 24 Jan '15

24 Jan '15

St?phane Cottin writes: > A typical .qmail file : > > | /usr/bin/dspam --client --deliver=stdout --user "$EXT@$USER" | /usr/bin/preline -f /usr/lib/dovecot/dovecot-lda -d "$EXT@$USER" -a "$EXT@$USER" -m "$EXT2" > > When dspam segfault or is killed, preline receive an empty content and > only pass to dovecot-lda two headers ( Return-Path and Delivered-To ). > Then dovecot-lda delivers successfully and the original message > contents are lost. Steffen Kaiser rightly points out: > That's the problem of the used Unix pipe and broken by design. The right > member of the pipe (preline) runs independendly of dspam and recieves no > content via pipe. Why should preline or dovecot-lda believe there is an > error? [... and suggests a wrapper script that aborts delivery on empty > dspam output.] I'm not sure if it's appropriate under your circumstances, but you can also pass stderr to stdout as well, and the recipient will get some diagnostic message, which may or may not be helpful. | /usr/bin/dspam --client --deliver=stdout --user "$EXT@$USER" 2>&1 | ... Joseph Tam <jtam.home(a)gmail.com>

2 1

LDA input validation
by Stéphane Cottin 24 Jan '15

24 Jan '15

Hi, I'm using qmail + dspam + dovecot-lda. A typical .qmail file : | /usr/bin/dspam --client --deliver=stdout --user "$EXT@$USER" | /usr/bin/preline -f /usr/lib/dovecot/dovecot-lda -d "$EXT@$USER" -a "$EXT@$USER" -m "$EXT2" When dspam segfault or is killed, preline receive an empty content and only pass to dovecot-lda two headers ( Return-Path and Delivered-To ). Then dovecot-lda delivers successfully and the original message contents are lost. Should dovecot-lda returns an error in such conditions ( invalid contents ) ? Qmail will keep it in the queue and retry delivery later. Stéphane

2 2

Mailbox corruption (dovecot 2.2.13, Debian 8)
by Santiago Vila 24 Jan '15

24 Jan '15

Hello. The attached mbox folder becomes corrupted in the server when being retrieved by fetchmail using the --folder option. This happens on a system running Debian jessie (i.e. dovecot 2.2.13) using the default configuration. I've reported this to Debian here: https://bugs.debian.org/776094 but I thought it would be good to post it here as well in case it is still a bug in the latest version (well, if it's not anymore a bug in the latest version, a patch against 2.2.13 will surely help the Debian maintainer in either case). Thanks.

1 0

sieve filter not working
by Michael Williamson 23 Jan '15

23 Jan '15

Hi, I have a question. I have dovecot 2.0.9 running on a CentOS 6.6 email server for a small department, ~15 users. amavis and postfix are also enabled. It appears that amavis invokes spamassassin, which tags incoming spam email. All email is then put into users local inbox directory, regardless of spam tag X-Spam_Flag value of YES or NO. I want instead to redirect spam to a special directory. I put a sieve filter in users home directory, "/home/user/.dovecot.sieve": require ["fileinto"]; # Move spam to spam folder if header :contains "X-Spam-Flag" ["YES"] { fileinto "Maildir/.SPAM"; stop; } I try different target directory for fileinto, ".SPAM", "/home/user/Maildir/.SPAM/cur", etc.. I know that those directories exist. The configure file "/etc/dovecot/dovecot.conf" is in original state. I changed one line in configure file "/etc/dovecot/conf.d/15-lda.conf": protocol lda { # Space separated list of plugins to load (default is global mail_plugins). mail_plugins = sieve $mail_plugins } I changed one line in configure file "/etc/dovecot/conf.d/20-lmtp.conf": protocol lmtp { # Space separated list of plugins to load (default is global mail_plugins). mail_plugins = sieve $mail_plugins } The configure file "/etc/dovecot/conf.d/90-sieve.conf" is unchanged, the relevant part is: plugin { # The path to the user's main active script. sieve = ~/.dovecot.sieve ... I restarted dovecot. I checked that it is running. The configuration command # doveconf -a output is attached. It is not working. How do I diagnose the problem? Thanks, -Mike

5 8

Client shows null Sender & date
by John Hendrich 23 Jan '15

23 Jan '15

I'm using Postfix and Dovecot 2.0.19 and Virtual domains & users (mysql). Incoming mail is handled by Postfix and then handed off to Dovecot LMTP for delivery. However, the Sender and Date are essentially null when viewing the email with either the POP3 or IMAP client. The logs (below) show this. I changed the sender and recipient addresses for privacy. Jan 23 08:41:48 klsrv postfix/smtpd[10836]: 1F907F00276: client=unknown[10.0.0.19] Jan 23 08:42:07 klsrv postfix/cleanup[10842]: 1F907F00276: message-id=<> Jan 23 08:42:07 klsrv postfix/qmgr[4889]: 1F907F00276: from=<sender(a)example.com>, size=217, nrcpt=1 (queue active) Jan 23 08:42:07 klsrv dovecot: lmtp(10845): Connect from local Jan 23 08:42:07 klsrv dovecot: lmtp(10845, recipient(a)example.net) wvU2Gq9PwlRdKgAAjuDoqw: subject=test to client: from=: msgid=unspecified: status=saved mail to INBOX: Jan 23 08:42:07 klsrv postfix/lmtp[10844]: 1F907F00276: to=<recipient(a)example.net>, relay=klsrv.airhop.net[private/dovecot-lmtp], delay=28, delays=28/0.01/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 <recipient(a)example.net> wvU2Gq9PwlRdKgAAjuDoqw Saved) Jan 23 08:42:07 klsrv dovecot: lmtp(10845): Disconnect from local: Client quit (in reset) Jan 23 08:42:07 klsrv postfix/qmgr[4889]: 1F907F00276: removed You can see that Postfix accepts the incoming mail (from sender(a)example.com) and then the message is handed to Dovecot LMTP. But the from=: should be "from=sender(a)example.com:". Postfix / qmgr has the sender listed (from=<sender(a)example.com>) but no sender (from=) is seen in LMTP log output. My LMTP is setup to use UNIX sockets, not the INET socket. Any help is appreciated.

3 2

Easiest way to compile dovecot on Ubuntu 14.04
by Kevin Laurie 23 Jan '15

23 Jan '15

Hello, I need to recompile dovecot on ubuntu 14.04 What would be the easiest way to do this? I am trying to follow this tutorial:- https://blog.vbonhomme.fr/add-full-text-search-fts-to-your-dovecot-using-so… When I run :- apt-get source dovecot Reading package lists... Done Building dependency tree Reading state information... Done NOTICE: 'dovecot' packaging is maintained in the 'Git' version control system at: git://git.debian.org/git/collab-maint/dovecot.git Need to get 5759 kB of source archives. Get:1 http://mirrors.gandi.net/ubuntu/ trusty-updates/main dovecot 1:2.2.9-1ubuntu2.1 (dsc) [3136 B] Get:2 http://mirrors.gandi.net/ubuntu/ trusty-updates/main dovecot 1:2.2.9-1ubuntu2.1 (tar) [4541 kB] Get:3 http://mirrors.gandi.net/ubuntu/ trusty-updates/main dovecot 1:2.2.9-1ubuntu2.1 (diff) [1215 kB] Fetched 5759 kB in 0s (8141 kB/s) sh: 1: dpkg-source: not found Unpack command 'dpkg-source -x dovecot_2.2.9-1ubuntu2.1.dsc' failed. Check if the 'dpkg-dev' package is installed. E: Child process failed Any faster means? Also cant get dpkg-dev installed on my system. Thanks Kevin

3 6

[serious help request!] Desperately trying to get Apache Solr working with Dovecot.
by Kevin Laurie 23 Jan '15

23 Jan '15

Hello, I am desperately trying to get Apache Solr to work with Dovecot FTS. I would really appreciate if someone could please help me! I have already done the following:- 1. I can ssh into my server and see that Apache Solr is up and running. ssh -t -L 8983:localhost:8983 user(a)mydomain.com 2. In the collection1 core selector I have the following files:- solrconfig.xml and schema.xml The schema.xml output is as follows(see link):- http://pastebin.com/thGw2pQj 3. I have installed the dovecot-solr already. 4. Configured dovecot to run solr-fts as follows:- In 10-mail.conf: # Space separated list of plugins to load for all services. Plugins specific to # IMAP, LDA, etc. are added to this list in their own .conf files. #mail_plugins = mail_plugins = fts fts_solr In 90-plugin.conf:- plugin { fts = solr fts_solr = break-imap-search url=http://localhost:8983/solr/ } *Desperately looking for some help! Please help!!!* Thanks Kevin

1 0

Postfix Dovecot LDAP setup
by Leander Schäfer 23 Jan '15

23 Jan '15

[EDIT]: Sorry about the second mail. ignore the first one - it had try & and error mistakes in. Problem is still the very same as explaned below. Thanks a lot. Hi, since days I'm trying to substitude my SQL based Postfix/Dovecot mail server by a LDAP based setup. Everything seems to go smooth so far - thanks to the many how to and Dovecot wiki. Unfortunately I'm struggeling, since I have to create individual a LDAP schema which suits the currently used LDAP structure. The current tructure looks like this: => dc=MyDomain,dc=TLD ==> ou=People ===> uid=User-1 ====> ou=mail ===> uid=User-2 ====> ou=mail ... and so on ... Within ou=mail should be the individual mail account(s) information of a user. So in the end I want to add a(nother) mail account by something like this: cat << EOF > ./newUser.ldif dn: mailAddress=Test(a)Domain.TLD,ou=mail,uid=User-1,ou=people,dc=MyDomain,dc=TLD objectclass: top objectclass: mailAccount mailAddress: Test(a)Domain.TLD MailPassword: {SSHA}SomePassword MailAccountStatus: active [...] EOF Therefore I setup a LDAP schema like the following, but it seems to ignore the attributes "MailPassword" and "noMailAccountStatus". Why? I don't understand what I'm missing here on my objectclass? ;/ # ====================== LDAP schema ======================= # # # OID Macros (10001 should be IANA-registered) # objectidentifier nameSpace 1.3.6.1.4.1.10001 objectidentifier mail nameSpace:1 objectidentifier objectClassAccount mail:1 objectidentifier objectClassAccountInfo mail:2 # # Attributes: objectClass[NAME]:1.[SERIAL] # attributetype ( objectClassAccount:1.1 NAME 'mailAddress' DESC 'The hosted mail addresses' EQUALITY caseIgnoreIA5Match SUBSTR caseIgnoreIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26{256} SINGLE-VALUE ) attributetype ( objectClassAccount:1.2 NAME 'MailPassword' DESC 'The hosted mail password' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40{128} ) attributetype ( objectClassAccount:1.3 NAME 'MailAccountStatus' DESC 'The status of a user account: active, noaccess, disabled, deleted' EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE ) [...] # # Objects: objectClass[NAME]:2.[SERIAL] # objectclass ( objectClassAccount:2.1 NAME 'mailAccount' SUP ( top ) STRUCTURAL DESC 'Mail account' MUST ( mailAddress ) MAY ( MailPassword $ MailAccountStatus ) ) # ====================================================== # Thanks Best Regards, Leander

1 1

doveadm sync different users
by Jonas Plitt 23 Jan '15

23 Jan '15

Hi all, The command below works to Sync a specific user between Server A and Server B. > doveadm sync -u user(a)mydomain.me tcp:new-server.domain.de How do i need to modify the command, to sync user1(a)domain.me at ServerA and user2domain.me at ServerB? In my case the usernames needed to sync, are not the same. thanks in advance, Jonas

1 0

Postfix Dovecot LDAP setup
by Leander Schäfer 22 Jan '15

22 Jan '15

Hi, since days I'm trying to substitude my SQL based Postfix/Dovecot mail server by a LDAP based setup. Everything seems to go smooth so far - thanks to the many how to and Dovecot wiki. Unfortunately I'm struggeling, since I have to create individual a LDAP schema which suits the currently used LDAP structure. The current tructure looks like this: => dc=MyDomain,dc=TLD ==> ou=People ===> uid=User-1 ====> ou=mail ===> uid=User-2 ====> ou=mail ... and so on ... Within ou=mail should be the individual mail account(s) information of a user. So in the end I want to add a(nother) mail account by something like this: cat << EOF > ./newUser.ldif dn: mailAddress=Test(a)Domain.TLD,ou=mail,uid=User-1,ou=people,dc=MyDomain,dc=TLD objectclass: top objectclass: mailAccount mailAddress: Test(a)Domain.TLD noMailPassword: {SSHA}SomePassword noMailAccountStatus: active [...] EOF Therefore I setup a LDAP schema like the following, but it seems to ignore the attributes "MailPassword" and "noMailAccountStatus". Why? I don't understand what I'm missing here on my objectclass? ;/ # ====================== LDAP schema ======================= # # # OID Macros (10001 should be IANA-registered) # objectidentifier nameSpace 1.3.6.1.4.1.10001 objectidentifier mail nameSpace:1 objectidentifier objectClassAccount mail:1 objectidentifier objectClassAccountInfo mail:2 # # Attributes: objectClass[NAME]:1.[SERIAL] # attributetype ( objectClassAccount:1.1 NAME 'mailAddress' DESC 'The hosted mail addresses' EQUALITY caseIgnoreIA5Match SUBSTR caseIgnoreIA5SubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.26{256} SINGLE-VALUE ) attributetype ( objectClassAccount:1.2 NAME 'MailPassword' DESC 'The hosted mail password' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40{128} ) attributetype ( objectClassAccount:1.3 NAME 'MailAccountStatus' DESC 'The status of a user account: active, noaccess, disabled, deleted' EQUALITY caseIgnoreIA5Match SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 SINGLE-VALUE ) [...] # # Objects: objectClass[NAME]:2.[SERIAL] # objectclass ( objectClassAccount:2.1 NAME 'mailAccount' SUP ( top ) STRUCTURAL DESC 'Mail account' MUST ( mailAddress ) MAY ( MailPassword $ MailAccountStatus ) ) # ====================================================== # Thanks Best Regards, Leander

1 0

Fatal: master: service(imap): child 19549 killed with signal 11 (core dumped)
by Gabriel Almaguer 22 Jan '15

22 Jan '15

hello. Today I had a problem (core dumped) in a dovecot installation update yesterday. After many hours I found that when one settings "protocol imap" the option "replication" causes (core dumped). In this server settings "replication" was removed leaving only the "protocol imap" (my error) I send my documentation error so they can add a validation and not generate a Fatal Error end users. Any other info you need please email me Thank You. dovecot-2.2.15 ---------------------/src/plugins/replication/replication-plugin.c replication_mail_transaction_commit(void *txn, struct mail_transaction_commit_changes *changes) { struct replication_mail_txn_context *ctx = (struct replication_mail_txn_context *)txn; struct replication_user *ruser = REPLICATION_USER_CONTEXT(ctx->ns->user); enum replication_priority priority; if (ctx->new_messages || changes->changed) { priority = !ctx->new_messages ? REPLICATION_PRIORITY_LOW : <== linea 258 look dgb ruser->sync_secs == 0 ? REPLICATION_PRIORITY_HIGH : REPLICATION_PRIORITY_SYNC; replication_notify(ctx->ns, priority); } i_free(ctx); } --------------------------------------------------------------------------------------- dovecot -n protocol imap { mail_plugins = quota notify fts fts_squat zlib acl imap_zlib imap_acl quota imap_quota replication } ------------------------------------------------------------- /var/log/maillog Jan 21 20:38:17 xxxxxx dovecot: imap(etrevino(a)xxxxxxx.com.mx): Fatal: master: service(imap): child 19549 killed with signal 11 (core dumped) --------------------------------------------- /var/log/messages Jan 21 20:38:17 xxxxxx kernel: imap[19549]: segfault at 0000000000000018 rip 00002aacb45ea645 rsp 00007fffc2db7350 error 4 -------------------------------------------------------------------- strace [pid 18265] fstat(4, {st_mode=S_IFREG|0600, st_size=32712, ...}) = 0 [pid 18265] write(4, "\200\200\200\205\4\0\0\0R\2\0\0R\2\0\0\10\0\0\0", 20) = 20 [pid 18265] fcntl(4, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 [pid 18265] stat("/var/spool/vdovecot/xxxxxx.com.mx/cduran/dovecot.index.log", {st_mode=S_IFREG|0600, st_size=32732, ...}) = 0 [pid 18265] fstat(4, {st_mode=S_IFREG|0600, st_size=32732, ...}) = 0 [pid 18265] munmap(0x2ab13f335000, 32712) = 0 [pid 18265] pread(4, "\200\200\200\205\4\0\0\0R\2\0\0R\2\0\0\10\0\0\0", 8192, 32712) = 20 [pid 18265] pread(4, "", 8192, 32732) = 0 [pid 18265] fstat(4, {st_mode=S_IFREG|0600, st_size=32732, ...}) = 0 [pid 18265] mmap(NULL, 32732, PROT_READ, MAP_SHARED, 4, 0) = 0x2ab13f335000 [pid 18265] madvise(0x2ab13f335000, 32732, MADV_SEQUENTIAL) = 0 [pid 18265] fstat(4, {st_mode=S_IFREG|0600, st_size=32732, ...}) = 0 [pid 18265] --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 18265 detached --------------------------------------------------------------------------------- gdb Core was generated by `dovecot/imap'. Program terminated with signal 11, Segmentation fault. #0 replication_mail_transaction_commit (txn=0x18610d70, changes=0x7fff80ac1440) at replication-plugin.c:258 258 replication-plugin.c: No such file or directory. in replication-plugin.c (gdb) bt full #0 replication_mail_transaction_commit (txn=0x18610d70, changes=0x7fff80ac1440) at replication-plugin.c:258 ctx = <value optimized out> priority = <value optimized out> #1 0x00002ab29a975749 in notify_contexts_mail_transaction_commit (t=0x18610000, changes=0x7fff80ac1440) at notify-plugin.c:132 ctx = 0x1859af70 mail_txn = 0x18610800 #2 0x00002ab29a975d60 in notify_transaction_commit (t=0x18610000, changes_r=0x7fff80ac1440) at notify-storage.c:182 lt = 0x18610d50 #3 0x00002ab29a569beb in quota_mailbox_transaction_commit (ctx=0x18610000, changes_r=0x7fff80ac1440) at quota-storage.c:134 qt = 0x18610d90 #4 0x00002ab299bce134 in mailbox_transaction_commit_get_changes (_t=<value optimized out>, changes_r=0x7fff80ac1440) at mail-storage.c:1916 _data_stack_cur_id = 3 t = 0x18610000 box = 0x18609610 save_count = 1 ret = <value optimized out> __FUNCTION__ = "mailbox_transaction_commit_get_changes" #5 0x000000000040b85c in cmd_append_parse_new_msg () No symbol table info available. #6 0x000000000040b65b in cmd_append_continue_message () No symbol table info available. #7 0x0000000000416aa7 in command_exec () No symbol table info available. #8 0x000000000040c191 in client_input_append () No symbol table info available. #9 0x00002ab299ee3689 in io_loop_call_io (io=0x1859c010) at ioloop.c:498 ioloop = 0x18585740 t_id = 2 __FUNCTION__ = "io_loop_call_io" #10 0x00002ab299ee4a45 in io_loop_handler_run_internal (ioloop=<value optimized out>) at ioloop-epoll.c:220 ctx = <value optimized out> event = 0x18587240 list = 0x18587e30 io = 0x20 tv = {tv_sec = 1799, tv_usec = 999036} msecs = <value optimized out> ret = 1 i = 0 call = false __FUNCTION__ = "io_loop_handler_run_internal" #11 0x00002ab299ee3729 in io_loop_handler_run (ioloop=0x20) at ioloop.c:545 No locals. #12 0x00002ab299ee39a8 in io_loop_run (ioloop=0x18585740) at ioloop.c:522 __FUNCTION__ = "io_loop_run" #13 0x00002ab299e8a233 in master_service_run (service=0x185855d0, callback=0x28) at master-service.c:566 No locals. #14 0x000000000041f3ce in main ()

1 0

Trying to get lda to deliver into public sub-sub-folder
by Claudius 21 Jan '15

21 Jan '15

Hi, I have this setup: recipient_delimiter = + mail_location = sdbox:~/sdbox namespace { type = public prefix = public/ location = sdbox:/home/vmail/shared/sdbox subscriptions = no } Postfix delivers via dovecot-lda: dovecot_shared unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/bin/env HOME=/home/vmail/shared /usr/lib/dovecot/dovecot-lda -f ${sender} -m ${user}/${extension} This works fine, ACL are manually defined and also work great. If I want to deliver into a subfolder I tell postfix to rewrite it to: folder+subfolder@dovecot_shared This is how it looks like: /home/vmail/shared/sdbox/mailboxes/{user}/{extension}/dbox-Mails But with this setup I couldn't get it to deliver into a subfolder of "subfolder/extension". Is it at all possible? Changing the HOME directory does not work because of the sdbox folder structure. I tried to generate a service for each subfolder like that: -m ${user}/subfolder/${extension} But when I did not have an extension I got: save failed to open mailbox {user}/subfolder/: Invalid mailbox name even though it exists. I think it doesn't like the trailing slash, but I don't know how to get rid of it. Hope I explained it well enough :) Regards, Claudius

1 0

Shared folders - Namespace definition
by Peter Hodur 21 Jan '15

21 Jan '15

Hello, I'm trying to configure shared mailboxes with ACL. My problem is FS layout. Our maildirs is completely outside of home dirs (home dirs is on pure SSD zpool, maildirs on separate HDD zpool). We are using checkpassword auth, which sets mailbox_location for each user. Layout is following: maildirs: /dpool/mail/maldirs/user-uuid/ home is: /dpool/mail/home/user-uuid/ index & control is under home: /dpool/mail/home/user-uuid/[index|control] the problem is how to specify path in NAMESPACE definition. I can use %% variables, but maildirs is not under home and also does not have "fixed" structure (so I can not use %%u for example). Is it possible to for example set some special env variable from checkpassword and than use it in namespace definition like home variable? Or can someone help to find the code in dovecot, which evaluates namespace location patter so I can patch it? thanks Pete

2 2

UID changes every few seconds
by Adam Przybylski 20 Jan '15

20 Jan '15

Hi, I'm trying to use Gmail-style conversation view, as described in http://wiki2.dovecot.org/Plugins/Virtual#Examples >From a client based on Roundcube Framework I'm fetching message list from "virtual/INBOX" folder and that works fine to some degree, but the problem is that every few seconds or so (probably when new message is delivered or bounced on the mail server) UID of those messages changes. When I'm trying to display the details of given message (fetch body by UID), I'm getting an error because message with given UID do not exist on the server. When I only refresh the list (fetch messages one more time) and compare UIDs of the same messages between fetches, there are completely different numbers - growing every fetch. When I switch from virtual folder to real INBOX everything works fine, UID stays unchanged over time. Is this the intended behavior for virtual folders or some kind of bug? Or am I doing something wrong? I'm using Dovecot 2.1.7 on Debian GNU/Linux 7 (wheezy). The output of dovecot -n is attached. Thanks in advance, Adam

2 1