[Dovecot] When should dictionary entries for the expire plugin be added/updated?
I've set up Dovecot 2.0 with the expire plugin using an SQLite DB, and when mail is delivered via dovecto-lda, the DB gets updated as expected. However, when I use Thunderbird 3.1.2 for Mac OS X to move a message to any of the folders known to the expire plugin (e.g. the 'tmp' folder), the SQLite DB is not updated.
Here is an excerpt of my current configuration:
# /etc/dovecot/dovecot.conf dict { expire = sqlite:/etc/dovecot/dict-expire.conf } plugin { expire = Junk expire2 = Trash expire3 = tmp expire4 = *.News expire5 = *.Reports expire_dict = proxy::expire sieve = /etc/dovecot/sieve/%u.sieve } service dict { unix_listener dict { # Ensure that there is no permission-related problem mode = 0666 } } protocol lda { mail_plugins = expire sieve postmaster_address = postmaster@domain.tld sendmail_path = /usr/sbin/sendmail }
# /etc/dovecot/dict-expire.conf connect = /var/lib/dovecot/expires.sqlite map { pattern = shared/expire/$user/$mailbox table = expires value_field = expire fields { username = $user mailbox = $mailbox } }
# Schema dump of /var/lib/dovecot/expires.sqlite PRAGMA foreign_keys=OFF; BEGIN TRANSACTION; CREATE TABLE expires ( username VARCHAR(75) NOT NULL, mailbox VARCHAR(255) NOT NULL, expire integer NOT NULL, PRIMARY KEY (username, mailbox) ); CREATE TRIGGER mergeexpire BEFORE INSERT ON expires FOR EACH ROW BEGIN UPDATE expires SET expire=NEW.expire WHERE username=NEW.username AND mailbox=NEW.mailbox; SELECT RAISE(IGNORE) WHERE (SELECT 1 FROM expires WHERE username=NEW.username AND mailbox=NEW.mailbox) IS NOT NULL; END; COMMIT;
With these settings, rows will be inserted to (or updated within) expires.sqlite when dovecot-lda stores messages to any of my "Reports" folders using sieve rules. When I manually move messages to "tmp" or "Junk", the database remains unchanged. Also, when I delete messages in Thunderbird, I'd expect database additions for the folder "Trash", but this does not happen either. What am I doing wrong here?
-Ralph
On Tue, 2010-08-24 at 20:18 +0200, Ralph Seichter wrote:
Here is an excerpt of my current configuration: .. protocol lda { mail_plugins = expire sieve
Your excerpt shows that expire plugin is enabled for lda, but doesn't show if it's enabled for imap.
On 24.08.10 20:27, Timo Sirainen wrote:
Your excerpt shows that expire plugin is enabled for lda, but doesn't show if it's enabled for imap.
Sure enough, I forgot to add
protocol imap { mail_plugins = expire }
to my configuration. Thanks, now the DB gets updated as expected when I move messages around (or delete 'em) with Thunderbird.
-Ralph
On Tue, 2010-08-24 at 21:20 +0200, Ralph Seichter wrote:
Sure enough, I forgot to add
protocol imap { mail_plugins = expire }
to my configuration.
Even better, make it global so all programs (like doveadm) will use it:
mail_plugins = expire protocol lda { mail_plugins = $mail_plugins sieve }
On 24.08.10 22:06, Timo Sirainen wrote:
Even better, make it global so all programs (like doveadm) will use it
Would that be a prerequisite for "doveadm expunge" to use the expire plugin's data? That is certainly worth mentioning in the Wiki pages.
-Ralph
On 24.08.10 20:18, Ralph Seichter wrote:
service dict { unix_listener dict { # Ensure that there is no permission-related problem mode = 0666 } }
I forgot to ask: what mode setting would you suggest in the scenario I described? Would it be sufficient if only the user "dovecot" had write permissions to the dict socket, or does every mail user need to be able to write to the socket when using his/her IMAP client to move or delete messages?
-Ralph
On Tue, 24 Aug 2010 23:04:06 +0200 Ralph Seichter dovecot-ml@seichter.de articulated:
On 24.08.10 20:18, Ralph Seichter wrote:
service dict { unix_listener dict { # Ensure that there is no permission-related problem mode = 0666 } }
I forgot to ask: what mode setting would you suggest in the scenario I described? Would it be sufficient if only the user "dovecot" had write permissions to the dict socket, or does every mail user need to be able to write to the socket when using his/her IMAP client to move or delete messages?
Timo, are the mode settings "decimal" or "octal" in the config file? I thought once upon a time that someone stated that they were octal.
-- Jerry ✌ Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
What this country needs is a dime that will buy a good five-cent bagel.
On 25.8.2010, at 0.51, Jerry wrote:
service dict { unix_listener dict { # Ensure that there is no permission-related problem mode = 0666 } }
Timo, are the mode settings "decimal" or "octal" in the config file? I thought once upon a time that someone stated that they were octal.
For settings that regularly output the value in octal (so that's basically just the mode settings), set=0123 is octal and set=123 is decimal. Just like .. well, at least in C code :)
On 24.8.2010, at 22.04, Ralph Seichter wrote:
On 24.08.10 20:18, Ralph Seichter wrote:
service dict { unix_listener dict { # Ensure that there is no permission-related problem mode = 0666 } }
I forgot to ask: what mode setting would you suggest in the scenario I described? Would it be sufficient if only the user "dovecot" had write permissions to the dict socket, or does every mail user need to be able to write to the socket when using his/her IMAP client to move or delete messages?
Mail processes connect to dict socket, so all mail users executing mail processes need to have access to it. Or you could use mail_access_groups as described by http://wiki2.dovecot.org/Dict
On 25.08.10 01:52, Timo Sirainen wrote:
Mail processes connect to dict socket, so all mail users executing mail processes need to have access to it.
Just as I thought when I configured "mode = 0666". I am uneasy about userA being potentially able to modify dict entries of userB. One can already define per-user sieve scripts in Dovecot 2.0, and I wonder if you have considered per-user dictionaries?
-Ralph
On Wed, 2010-08-25 at 13:00 +0200, Ralph Seichter wrote:
On 25.08.10 01:52, Timo Sirainen wrote:
Mail processes connect to dict socket, so all mail users executing mail processes need to have access to it.
Just as I thought when I configured "mode = 0666". I am uneasy about userA being potentially able to modify dict entries of userB.
Do you have system users? The group way I mentioned would avoid problems with them, but of course not security problems related to Dovecot processes themselves.
One can already define per-user sieve scripts in Dovecot 2.0, and I wonder if you have considered per-user dictionaries?
Well, the whole point of expire database is that a single command can quickly see what users have mails to expunge. So this needs to be a shared dictionary across users.
Of course, having some kind of user authentication would be nice across Dovecot processes.. But I'm not sure if there's a way to make that work.
On 25.08.10 15:03, Timo Sirainen wrote:
The group way I mentioned would avoid problems with them, but of course not security problems related to Dovecot processes themselves.
There are users with shell access to the Dovecot server, and as long as all these users require write permissions for the dict socket, I see some potential for trouble.
the whole point of expire database is that a single command can quickly see what users have mails to expunge. So this needs to be a shared dictionary across users.
Does it really? I use something like this for cleanup:
for user in $MAILUSERS; do doveadm expunge -u $user ... done
As I loop over users anyway, I don't think that a per-user dictionary would be a bad idea. "doveadm expunge -A" might be different, but I think that even with "-A" you'll need to iterate over users.
If I understand the current implementation correctly, there is one expire dictionary (one SQLite DB in my case), and the lookups are performed with a primary key consisting of username-mailbox-pairs. I suggest using one dictionary/DB per username and using only the mailbox as primary key within this dictionary.
The difference, performance-wise, would be that multiple databases need to be opened and closed, which could hurt performance on systems with a large number of users unless the DB connections are cached. Personally, I'd be inclined to accept a performance penalty if this was the price to pay for improved security by means of better user dictionary separation.
-Ralph
On Wed, 2010-08-25 at 16:07 +0200, Ralph Seichter wrote:
On 25.08.10 15:03, Timo Sirainen wrote:
The group way I mentioned would avoid problems with them, but of course not security problems related to Dovecot processes themselves.
There are users with shell access to the Dovecot server, and as long as all these users require write permissions for the dict socket, I see some potential for trouble.
But you don't have to give rw access to them. You have to give rw access to Dovecot processes (this works as long as you don't use dovecot-lda). See http://wiki2.dovecot.org/Dict
the whole point of expire database is that a single command can quickly see what users have mails to expunge. So this needs to be a shared dictionary across users.
Does it really? I use something like this for cleanup:
for user in $MAILUSERS; do doveadm expunge -u $user ... done
If you do that, then there's no point in using expire database at all. Even the wiki page says at the top that you don't really need it.. What the expire plugin optimizes is if you have lots of users and you call it to all users with -A parameter.
As I loop over users anyway, I don't think that a per-user dictionary would be a bad idea. "doveadm expunge -A" might be different, but I think that even with "-A" you'll need to iterate over users.
Yeah, the doveadm_expire plugin makes the iteration skip users who don't have anything to expunge.
On 25.08.10 16:29, Timo Sirainen wrote:
But you don't have to give rw access to them. You have to give rw access to Dovecot processes (this works as long as you don't use dovecot-lda).
My message 4C7433C6.50705@seichter.de was all about this question: is it sufficient to allow rw access to the user "dovecot", or does every user who connects via IMAP need to write to the dict socket. I'm sorry if I did not make this clear enough.
Anyway, I call dovecot-lda as Postfix's mailbox_command, so I understand that the socket needs to be world-writeable? I read the Wiki pages for Dict and Expunge, but did not find them conclusive. You have the unfair advantage of having written the software and thus knowing exactly how it behaves. ;-)
If you do that, then there's no point in using expire database at all. Even the wiki page says at the top that you don't really need it.
Yes, the Wiki states that "Even without this plugin it's possible to use doveadm to expunge messages". Howevery, I don't think the Wiki currently mentions that "doveadm expunge -u" (in contrast to "-A") does not use the expire plugin's data at all?
Of course, I believe you if you tell me that there is no point in using the expire plugin in my particular scenario. I shall disable it.
-Ralph
On Wed, 2010-08-25 at 16:44 +0200, Ralph Seichter wrote:
On 25.08.10 16:29, Timo Sirainen wrote:
But you don't have to give rw access to them. You have to give rw access to Dovecot processes (this works as long as you don't use dovecot-lda).
My message 4C7433C6.50705@seichter.de was all about this question: is it sufficient to allow rw access to the user "dovecot", or does every user who connects via IMAP need to write to the dict socket. I'm sorry if I did not make this clear enough.
It was clear to me what you meant, but you gave only two choices and the answers to them were:
is it sufficient to allow rw access to the user "dovecot",
"no"
or does every user who connects via IMAP need to write to the dict socket.
and "no", because the 3rd possibility was described in the Dict wiki.
Anyway, I call dovecot-lda as Postfix's mailbox_command, so I understand that the socket needs to be world-writeable? I read the Wiki pages for Dict and Expunge, but did not find them conclusive. You have the unfair advantage of having written the software and thus knowing exactly how it behaves. ;-)
You could switch to using LMTP.
If you do that, then there's no point in using expire database at all. Even the wiki page says at the top that you don't really need it.
Yes, the Wiki states that "Even without this plugin it's possible to use doveadm to expunge messages". Howevery, I don't think the Wiki currently mentions that "doveadm expunge -u" (in contrast to "-A") does not use the expire plugin's data at all?
Yeah.. I guess I'll have to figure out some clearer way to explain it.
On 25.08.10 16:51, Timo Sirainen wrote:
It was clear to me what you meant, but you gave only two choices [...]
Ah, I see. ;-)
You could switch to using LMTP.
I used http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP as a starting point and switched from
mailbox_command = /usr/local/dovecot-2.0/libexec/dovecot/dovecot-lda
to
mailbox_transport = lmtp:unix:private/dovecot-lmtp
In my Postfix main.cf. As server.domain.tld is listed in mydestination, I use mailbox_transport instead of virtual_transport. This does not seem to work, though:
postfix/lmtp[20701]: 2C59018E0E8: to=user@server.domain.tld, relay=server.domain.tld[private/dovecot-lmtp], delay=0.04, delays=0.02/0/0/0.01, dsn=5.1.1, status=bounced (host server.domain.tld[private/dovecot-lmtp] said: 550 5.1.1 user@server.domain.tld User doesn't exist: user@server.domain.tld (in reply to RCPT TO command))
What would be required to make Dovecot store mail for user@server.domain.tld in /home/user/.maildir?
BTW, when I do use virtual_transport (which I shouldn't, if I understand Postfix correctly), mail for user@server.domain.tld is delivered locally by Postfix into the proper maildir, but bypasses Dovecot LMTP alltogether.
-Ralph
Am 25.08.2010 um 18:42 schrieb Ralph Seichter:
BTW, when I do use virtual_transport (which I shouldn't, if I understand Postfix correctly), mail for user@server.domain.tld is delivered locally by Postfix into the proper maildir, but bypasses Dovecot LMTP alltogether.
Hi Ralph,
this is covered on the Wiki here:
http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP
and here:
http://wiki2.dovecot.org/HowTo/Virtual%20User%20Flat%20File%20Configuration%...
Regards Thomas
On 25.08.10 18:46, Thomas Leuxner wrote:
this is covered on the Wiki here: http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP and here: http://wiki2.dovecot.org/HowTo/Virtual%20User%20Flat%20File%20Configuration%...
Thanks for the pointers, Thomas, but as I wrote before: I used http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP as a starting point, but I don't think virtual_transport is valid in my particular server setup. What I needed was a means of dropping the host name in user@server.domain.tld when delivering mail via LMTP, and Timo's suggestion "auth_username_format = %Ln" works fine for me.
-Ralph
Am 25.08.2010 um 19:33 schrieb Ralph Seichter:
Thanks for the pointers, Thomas, but as I wrote before: I used http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP as a starting point, but I don't think virtual_transport is valid in my particular server setup. What I needed was a means of dropping the host name in user@server.domain.tld when delivering mail via LMTP, and Timo's suggestion "auth_username_format = %Ln" works fine for me.
Ah ok. Missed the first reference, but glad it worked out anyway. Postfix has a million ways to configure things, so the articles can only help you to find a working setup for you as you correctly pointed out :)
On 25.08.10 19:38, Thomas Leuxner wrote:
Postfix has a million ways to configure things, so the articles can only help you to find a working setup for you as you correctly pointed out :)
I wonder if you might have a suggestion for configuring the drop-the- server-and-domain-part of the recipient address on the Postfix side? Timo's idea works just fine, but as it has consequences for all username lookups within Dovecot, I'm open to look at alternatives.
-Ralph
Am 25.08.2010 um 19:51 schrieb Ralph Seichter:
I wonder if you might have a suggestion for configuring the drop-the- server-and-domain-part of the recipient address on the Postfix side? Timo's idea works just fine, but as it has consequences for all username lookups within Dovecot, I'm open to look at alternatives.
Unless I missed again (double-checked) what authentication backend do you use? Can you post an excerpt? This may get more opinions from the list as I use unspectacular flat files.
On 25.08.10 20:01, Thomas Leuxner wrote:
what authentication backend do you use?
I use LDAP for authentication, and /etc/passwd as the user DB:
passdb { args = /etc/dovecot/ldap.conf driver = ldap } userdb { driver = passwd }
I could provide /etc/dovecot/ldap.conf, but I don't think you'll need this config file for the question at hand?
-Ralph
On Wed, 2010-08-25 at 18:42 +0200, Ralph Seichter wrote:
postfix/lmtp[20701]: 2C59018E0E8: to=user@server.domain.tld, relay=server.domain.tld[private/dovecot-lmtp], delay=0.04, delays=0.02/0/0/0.01, dsn=5.1.1, status=bounced (host server.domain.tld[private/dovecot-lmtp] said: 550 5.1.1 user@server.domain.tld User doesn't exist: user@server.domain.tld (in reply to RCPT TO command))
What would be required to make Dovecot store mail for user@server.domain.tld in /home/user/.maildir?
So you want to drop the domain? I'm not sure if you can do this on Postfix's side, but you can at least enable it globally to Dovecot (also means for imap/pop3 logins users can start to use user@whatever):
auth_username_format = %Ln
On 25.08.10 18:52, Timo Sirainen wrote:
auth_username_format = %Ln
That's a way of using auth_username_format I had not thought of at all. Thanks once again, this works fine and Postfix now delivers mail using Dovecot's LMTP socket.
-Ralph
On 25.08.10 18:52, Timo Sirainen wrote:
So you want to drop the domain? I'm not sure if you can do this on Postfix's side [...]
I don't think so, based on the e-mails I received. It has been rightly pointed that LMTP requires fully qualified e-mail addresses.
Now I try to find a LDAP-only setup for passdb and userdb. I have experimented with both
user_filter = (&(objectClass=posixAccount)(uid=%n))
and/or
pass_filter = (&(objectClass=posixAccount)(uid=%n))
but this only works as long as I enable "auth_username_format = %Ln" aswell. I was hoping that using "%n" instead of the default "%u" would help, but alas, no.
It is frustrating that this how-to-drop-the-domain business is only an issue with LMTP. When I use dovecot-lda, auth_username_format is not required.
-Ralph
On 28.08.10 17:55, Charles Marcus wrote:
So... why not just use LDA?
I you read the complete thread, you'll find that Timo suggested using LMTP instead of LDA to avoid the need for a world-writeable socket.
On 8/28/2010 12:05 PM, Ralph Seichter wrote:
On 28.08.10 17:55, Charles Marcus wrote:
So... why not just use LDA?
I you read the complete thread, you'll find that Timo suggested using LMTP instead of LDA to avoid the need for a world-writeable socket.
Oh, right - I recall seeing that and intended to ask if/why this was necessary, and how much of a security risk it was...
One of the stated goals of dovecot is to be 100% secure, so I'm very curious about the answer...
--
Best regards,
Charles
participants (5)
-
Charles Marcus
-
Jerry
-
Ralph Seichter
-
Thomas Leuxner
-
Timo Sirainen