[Dovecot] best choice of user database file to work with postfix?
I'm setting up a Postfix and Dovecot combination. What I want to do is have a user database that (1) is not running from some engine (so not LDAP or SQL or such) ... and (2) is completely disassociated from system users (e.g. most email users are not in /etc/passwd and most /etc/passwd users are not email users). Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login authentication) can use together. An alternative is some way to get Postfix to go through Dovecot to query for users (at the time of mail arriving on SMTP so it doesn't queue anything that would later be rejected). This is a smallish setup on one server, with probably a max of 50 to 100 users and 50 or so role account mailboxes over the next year or two. Any recommendations?
build as passdb ( http://wiki.dovecot.org/AuthDatabase/PasswdFile ) and write a sript which builds an valid postfix lookuptable usable as local_recipient_maps.
Andreas
Am 21.04.2010 10:32 schrieb Phil Howard:
I'm setting up a Postfix and Dovecot combination. What I want to do is have a user database that (1) is not running from some engine (so not LDAP or SQL or such) ... and (2) is completely disassociated from system users (e.g. most email users are not in /etc/passwd and most /etc/passwd users are not email users). Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login authentication) can use together. An alternative is some way to get Postfix to go through Dovecot to query for users (at the time of mail arriving on SMTP so it doesn't queue anything that would later be rejected). This is a smallish setup on one server, with probably a max of 50 to 100 users and 50 or so role account mailboxes over the next year or two. Any recommendations?
-- Andreas Schulze Internetdienste | P532
DATEV eG 90329 Nürnberg | Telefon +49 911 319-0 | Telefax +49 911 319-3196 E-Mail info @datev.de | Internet www.datev.de Sitz: 90429 Nürnberg, Paumgartnerstr. 6-14 | Registergericht Nürnberg, GenReg Nr.70 Vorstand Prof. Dieter Kempf (Vorsitzender) Dipl.-Kfm. Wolfgang Stegmann (stellvertretender Vorsitzender) Dipl.-Kfm. Michael Leistenschneider Jörg Rabe v. Pappenheim Dipl.-Vw. Eckhard Schwarzer Vorsitzender des Aufsichtsrates: Reinhard Verholen
I think /etc/passwd is as close as it gets to your requirements... why not just add the users as system users, and set their shell to /bin/false?
Patrick
"Phil Howard" <ttiphil@gmail.com> wrote:
I'm setting up a Postfix and Dovecot combination. What I want to do is have a user database that (1) is not running from some engine (so not LDAP or SQL or such) ... and (2) is completely disassociated from system users (e.g. most email users are not in /etc/passwd and most /etc/passwd users are not email users). Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login authentication) can use together. An alternative is some way to get Postfix to go through Dovecot to query for users (at the time of mail arriving on SMTP so it doesn't queue anything that would later be rejected). This is a smallish setup on one server, with probably a max of 50 to 100 users and 50 or so role account mailboxes over the next year or two. Any recommendations?
-- Sent from my Android phone with K-9. Please excuse my brevity.
On Wed, Apr 21, 2010 at 10:47 AM, Patrick Nagel <mail@patrick-nagel.net>wrote:
I think /etc/passwd is as close as it gets to your requirements... why not just add the users as system users, and set their shell to /bin/false?
There would be conflicts in this, especially with multiple domain names (sorry, forgot to mention that ... there will be about 10 domain names).
Phil Howard escribió:
On Wed, Apr 21, 2010 at 10:47 AM, Patrick Nagel <mail@patrick-nagel.net>wrote:
I think /etc/passwd is as close as it gets to your requirements... why not just add the users as system users, and set their shell to /bin/false?
There would be conflicts in this, especially with multiple domain names (sorry, forgot to mention that ... there will be about 10 domain names).
Then I think MySQL will do the job. Both postfix and dovecot support MySQL, and you can use SASL to authenticate SMTP with Dovecot, so Dovecot would do all the auth work. Finally, you could use Postfix's VDA patch if you want to use Maildir++.
Hope this helps.
On Wed, Apr 21, 2010 at 1:30 PM, Rodolfo Gonzalez <rgonzalez@gnt.cc> wrote:
Phil Howard escribió:
On Wed, Apr 21, 2010 at 10:47 AM, Patrick Nagel <mail@patrick-nagel.net
wrote:
I think /etc/passwd is as close as it gets to your requirements... why
not just add the users as system users, and set their shell to /bin/false?
There would be conflicts in this, especially with multiple domain names (sorry, forgot to mention that ... there will be about 10 domain names).
Then I think MySQL will do the job. Both postfix and dovecot support MySQL, and you can use SASL to authenticate SMTP with Dovecot, so Dovecot would do all the auth work. Finally, you could use Postfix's VDA patch if you want to use Maildir++.
Hope this helps.
I don't want to use any other server engine of any kind with this. I'm trying to keep it small and lean, and minimize what the people that have to monitor and fix it need to know. So at the present time, I am excluding all databases like any SQL or LDAP or anything else that needs to run as a daemon/engine/service.
Ideal would be a single file both can read, be it a flat file, or a Berkeley DB file. Next to that would be two files where one is authoritative and the other (for Postfix since it only needs a list) is generated from it. That looks like what I ultimately will do (the script that adds users will just generate the files and test them).
I was hoping there might be a way to get Postfix to use the legacy Unix passwd file format, but with a different file name. It doesn't seem to have that ability. It would make things very simple if it did. It would also be simple if the system /etc/passwd file could be used, but it can't.
I'm running a setup that should be good enough for what you are trying to achieve. All user information is stored in flat files per domain and you may override per user settings individually:
passdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file }
userdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file }
$ cat passwd user@domain.tld:{scheme}<password>:5000:5000::/var/vmail/domain.tld/user::userdb_quota_rule=*:storage=5G userdb_acl_groups=PublicMailboxAdmins
I would vote against storing aliases in these files though. Reason being the Postfix alias files are more flexible, because you would need to setup NULL password/No Login users or similar in the Dovecot backend. Another reason to keep them in Postfix is to completely separate alias management from the user management and use the same for login checks.
See how aliases are used for routing and to authenticate valid mail from senders with one file:
$ cat virtual alias@domain.tld login@domain.tld postmaster@domain.tld login@domain.tld
[main.cf] virtual_mailbox_domains = domain.tld, domain1.tld virtual_mailbox_base = /var/vmail virtual_minimum_uid = 100 virtual_uid_maps = static:5000 virtual_gid_maps = static:5000 virtual_alias_maps = hash:/etc/postfix/virtual virtual_transport = lmtp:unix:private/dovecot-lmtp […] smtpd_sender_login_maps=hash:/etc/postfix/virtual
If this seems suitable I can send more details to you.
Regards Thomas
On Wed, Apr 21, 2010 at 3:30 PM, Thomas Leuxner <tlx@leuxner.net> wrote:
I'm running a setup that should be good enough for what you are trying to achieve. All user information is stored in flat files per domain and you may override per user settings individually:
passdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file }
userdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file }
What does it take to get Postfix to read this?
$ cat passwd
user@domain.tld:{scheme}<password>:5000:5000::/var/vmail/domain.tld/user::userdb_quota_rule=*:storage=5G userdb_acl_groups=PublicMailboxAdmins
In which directory was this?
I would vote against storing aliases in these files though. Reason being the Postfix alias files are more flexible, because you would need to setup NULL password/No Login users or similar in the Dovecot backend. Another reason to keep them in Postfix is to completely separate alias management from the user management and use the same for login checks.
See how aliases are used for routing and to authenticate valid mail from senders with one file:
$ cat virtual alias@domain.tld login@domain.tld postmaster@domain.tld login@domain.tld
I suspect I will want to be maping virtuals between different domains, so I might have
abuse@example.com mailadmin@example.net abuse@example.net mailadmin@example.net postmaster@example.com mailadmin@example.net postmaster@example.net mailadmin@example.net
[main.cf]
virtual_mailbox_domains = domain.tld, domain1.tld virtual_mailbox_base = /var/vmail virtual_minimum_uid = 100 virtual_uid_maps = static:5000 virtual_gid_maps = static:5000 virtual_alias_maps = hash:/etc/postfix/virtual virtual_transport = lmtp:unix:private/dovecot-lmtp […] smtpd_sender_login_maps=hash:/etc/postfix/virtual
One thing I need to watch out for, and am concerned with because the last time I used Postfix there were a bunch of "virtual" configurations that really didn't work for me for a reason I cannot recall right now ... is that the same user name in different domains is NOT always the same user. E.g. bob@example.com is NOT the same person as bob@example.net while bob@example.org doesn't even exist. So there needs to be distinct entries for bob@example.com and bob@example.net (and not any for bob@example.org and have Postfix reject that during incoming SMTP sessions).
There can also be cases where mike@example.com and mike@example.net are the same person, and Mike wants to have mail to these two addresses kept in separate mail boxes (and presumably must do separate logins, so he'd have to set up 2 accounts in his MUA) ... as well as steve@example.com and steve@example.net also being the same person, but Steve wants everything in one mailbox (so he'd have to pick between steve@example.com and steve@example.net and I'd have to set up a virtual map for the other to be delivered to the mailbox of his choice ... in a separate lookup table in Postfix).
If this seems suitable I can send more details to you.
It might well be as long the domains are fully distinct. I'll have to go read up on each of the virtual_* configuration parameters to be sure of the effects. I was thinking to use:
mailbox_command = /usr/lib/dovecot/deliver
in Postfix main.cf. Is that workable instead of "virtual_transport = lmtp:unix:private/dovecot-lmtp" Or would running LMTP be a better way?
On Wed, Apr 21, 2010 at 04:34:30PM -0400, Phil Howard wrote:
userdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file } What does it take to get Postfix to read this?
Basically these two parameters in 'main.cf':
[main.cf] smtpd_sasl_type=dovecot smtpd_sasl_path=private/auth
Since this will have implications when Dovecot is not running/unavailable as Authtentication Backend, Postfix will reject legit incoming mail in that case, it is better to put this in the master configuration actually and have Postfix use a dedicated submission port for SASL clients:
[master.cf] submission inet n - - - - smtpd smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_sasl_path=private/auth -o smtpd_sasl_security_options=noanonymous -o smtpd_sasl_local_domain=$myhostname -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_sender_login_maps=hash:/etc/postfix/virtual -o smtpd_sender_restrictions=reject_sender_login_mismatch -o smtpd_recipient_restrictions=reject_unknown_recipient_domain,reject_non_fqdn_recipient,permit_sasl_authenticated,reject
$ cat passwd
user@domain.tld:{scheme}<password>:5000:5000::/var/vmail/domain.tld/user::userdb_quota_rule=*:storage=5G userdb_acl_groups=PublicMailboxAdmins
In which directory was this?
$ l /var/vmail/auth.d/doamin.tld/ total 4 -r-------- 1 doveauth dovecot 1234 2010-04-10 11:38 passwd
I suspect I will want to be maping virtuals between different domains, so I might have
abuse@example.com mailadmin@example.net abuse@example.net mailadmin@example.net postmaster@example.com mailadmin@example.net postmaster@example.net mailadmin@example.net
No problem to do this.
One thing I need to watch out for, and am concerned with because the last time I used Postfix there were a bunch of "virtual" configurations that really didn't work for me for a reason I cannot recall right now ... is that the same user name in different domains is NOT always the same user. E.g. bob@example.com is NOT the same person as bob@example.net while bob@example.org doesn't even exist. So there needs to be distinct entries for bob@example.com and bob@example.net (and not any for bob@example.org and have Postfix reject that during incoming SMTP sessions).
Yes, this is taken care of in the example. You can have Bob spread all over the domains routing into different mailboxes, or point multiple aliases to the same.
There can also be cases where mike@example.com and mike@example.net are the same person, and Mike wants to have mail to these two addresses kept in separate mail boxes (and presumably must do separate logins, so he'd have to set up 2 accounts in his MUA) ... as well as steve@example.com and steve@example.net also being the same person, but Steve wants everything in one mailbox (so he'd have to pick between steve@example.com and steve@example.net and I'd have to set up a virtual map for the other to be delivered to the mailbox of his choice ... in a separate lookup table in Postfix).
See above, possible too.
It might well be as long the domains are fully distinct. I'll have to go read up on each of the virtual_* configuration parameters to be sure of the effects. I was thinking to use:
mailbox_command = /usr/lib/dovecot/deliver
in Postfix main.cf. Is that workable instead of "virtual_transport = lmtp:unix:private/dovecot-lmtp" Or would running LMTP be a better way?
LMTP would be better long-term as it is more flexible and robust, e.g. allowing multiple recipient deliveries in parallel and has a real protocol handshake compared to piping into the LDA, but both is feasible. Hower LMTP is available with Dovecot 2.0 only.
Deliver flavour in pre-2.0 would look like this:
[main.cf] virtual_transport = dovecot dovecot_destination_recipient_limit = 1
[master.cf] dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -f ${sender} -d ${recipient}
I will look into writing this up for the 2.0 Wiki.
Regards Thomas
On Thu, Apr 22, 2010 at 3:33 AM, Thomas Leuxner <tlx@leuxner.net> wrote:
On Wed, Apr 21, 2010 at 04:34:30PM -0400, Phil Howard wrote:
userdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file } What does it take to get Postfix to read this?
Basically these two parameters in 'main.cf':
[main.cf] smtpd_sasl_type=dovecot smtpd_sasl_path=private/auth
Since this will have implications when Dovecot is not running/unavailable as Authtentication Backend, Postfix will reject legit incoming mail in that case, it is better to put this in the master configuration actually and have Postfix use a dedicated submission port for SASL clients:
[master.cf] submission inet n - - - - smtpd smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_sasl_path=private/auth -o smtpd_sasl_security_options=noanonymous -o smtpd_sasl_local_domain=$myhostname -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_sender_login_maps=hash:/etc/postfix/virtual -o smtpd_sender_restrictions=reject_sender_login_mismatch -o smtpd_recipient_restrictions=reject_unknown_recipient_domain,reject_non_fqdn_recipient,permit_sasl_authenticated,reject
So what would local_recipient_maps look like in this case? At this point, I don't understand what is happening for this. I would be expecting Postfix to be asking Dovecot if a user is valid. This is for mail incoming from outside, to make the rejection decision during the SMTP session. This looks more like a configuration to provide a submission interface and authenticate through Dovecot. That's fine, and probably what is needed. But I'm trying to sort out the local_recipient_maps at this time. Can this solve both issues at the same time?
It might well be as long the domains are fully distinct. I'll have to go
read up on each of the virtual_* configuration parameters to be sure of the effects. I was thinking to use:
mailbox_command = /usr/lib/dovecot/deliver
in Postfix main.cf. Is that workable instead of "virtual_transport = lmtp:unix:private/dovecot-lmtp" Or would running LMTP be a better way?
LMTP would be better long-term as it is more flexible and robust, e.g. allowing multiple recipient deliveries in parallel and has a real protocol handshake compared to piping into the LDA, but both is feasible. Hower LMTP is available with Dovecot 2.0 only.
I'm doing this on Ubuntu 9.10 and it has Dovecot 1.1.11 right now (we need to get this mail server up before we will be ready to eval Ubuntu 10.04 or another distro).
On Thursday 22 April 2010 15:34:40 Phil Howard wrote:
So what would local_recipient_maps look like in this case?
As the suggested setup uses virtual_mailbox_domains for the mailboxes hosted by dovecot, it would be virtual_mailbox_maps. Alternatively one could define relay_domains, relay_transport and relay_recipient_maps. local_recipient_maps is for use with mydestination and system (/etc/passwd) users only. But this aspect is indeed missing in Thomas' suggestion.
At this point, I don't understand what is happening for this. I would be expecting Postfix to be asking Dovecot if a user is valid.
This is not directly possible as there is no postfix lookup table that can query any protocol that dovecot speaks. Neither is there one for the passwd- file format.
Instead of a lookup table one could theoretically use reject_unverified_recipient as described in the Postfix ADDRESS_VERIFICATION_README but I don't know if it works with any non-SMTP transport for the destination domain. I can't tell from the first reading, and have no time to explore that.
Otherwise there's no choice but to generate a hash/btree/cdb file from the dovecot passwd-files for use in virtual_mailbox_maps.
mailbox_command = /usr/lib/dovecot/deliver
in Postfix main.cf. Is that workable instead of "virtual_transport = lmtp:unix:private/dovecot-lmtp" Or would running LMTP be a better way?
If you can't wait for Dovecot 2.0, you need to use dovecot deliver, but you should set it up as a pipe transport in master - see http://wiki.dovecot.org/LDA/Postfix for virtual users. mailbox_command again is for real system users only.
Rainer
On Thu, Apr 22, 2010 at 10:43 AM, Rainer Frey <rainer.frey@inxmail.de>wrote:
If you can't wait for Dovecot 2.0, you need to use dovecot deliver, but you should set it up as a pipe transport in master - see http://wiki.dovecot.org/LDA/Postfix for virtual users. mailbox_command again is for real system users only.
Basically what I expect to be doing is:
Postfix listens on SMTP for incoming MX mail to local (as in virtual, not system) users.
Postfix listens on Submission, encrypted only, and authenticates users to submit mail for delivery anywhere.
Dovecot listens on encrypted IMAPS and POP3S for user authenticated access to mailboxes.
Everything but MX to SMTP on port 25 shall be encrypted only. If I can force the use of STARTTLS on the non-encrypted ports, then it would be OK to use them that way. But I do not want to give any user an option to not be encrypted.
Passwords stored encrypted, such as MD5. And it should be a scheme that both Postfix and Dovecot can use so I don't have keep two different encryption schemes.
I'd prefer not to, but it looks like I will have to copy data from one format to another format so Dovecot can read it and Postfix can read it. I will most likely be using the CDB format (the constant database file format from Dan Bernstein ... which I'd think should be easy enough for a future version of Dovecot to support).
I am wondering if I can trick Postfix into reading virtual user info by running it chrooted where I substitute /etc/passwd and /etc/shadow with stuff I generate from Dovecot files.
On Thursday 22 April 2010 18:15:18 Phil Howard wrote: [ ... all standard stuff that is well documented ... ]
- Passwords stored encrypted, such as MD5. And it should be a scheme that both Postfix and Dovecot can use so I don't have keep two different encryption schemes.
Postfix doesn't need any password directly. It only authenticates a user with a password in one case: SMTP SASL authentication of submission users, and it uses the dovecot auth service for that and does not read the password database itself.
- I'd prefer not to, but it looks like I will have to copy data from one format to another format so Dovecot can read it and Postfix can read it. I will most likely be using the CDB format (the constant database file format from Dan Bernstein ... which I'd think should be easy enough for a future version of Dovecot to support).
It is not about supporting a certain database library. There is simply a difference in what Dovecot and Postfix need from a user/recipient database. Dovecot needs information like path to mailbox, uid/gid with which to put files into the mailboxes, extra configuration fields ... Therefore it uses a structured multi-value format like the passwd-file. CDB or similar don't work like this, so dovecot can't easily support using the same CDB files as postfix.
Postfix only supports name:value tables, either to use the value (table-style lookup) or to check whether there is an entry for a name (list-style lookup). Therefore it only uses file databases with such a mapping. In the case of valid recipients which will be handed off to dovecot for delivery, it is such a simple list lookup.
(If it interests you: the postfix virtual delivery agent needs very similar information to dovecot, but it uses several key:value maps for the different information, all with the recipient as key.)
- I am wondering if I can trick Postfix into reading virtual user info by running it chrooted where I substitute /etc/passwd and /etc/shadow with stuff I generate from Dovecot files.
If you need to generate the files that postfix uses, you can generate supported lookup table formats as well.
Rainer
On Fri, Apr 23, 2010 at 2:49 AM, Rainer Frey <rainer.frey@inxmail.de> wrote:
On Thursday 22 April 2010 18:15:18 Phil Howard wrote: [ ... all standard stuff that is well documented ... ]
- Passwords stored encrypted, such as MD5. And it should be a scheme that both Postfix and Dovecot can use so I don't have keep two different encryption schemes.
Postfix doesn't need any password directly. It only authenticates a user with a password in one case: SMTP SASL authentication of submission users, and it uses the dovecot auth service for that and does not read the password database itself.
OK, so all that is in whatever SASL configuration, and so I'll be doing that in Dovecot.
It is not about supporting a certain database library. There is simply a
difference in what Dovecot and Postfix need from a user/recipient database. Dovecot needs information like path to mailbox, uid/gid with which to put files into the mailboxes, extra configuration fields ... Therefore it uses a structured multi-value format like the passwd-file. CDB or similar don't work like this, so dovecot can't easily support using the same CDB files as postfix.
Why not have all those fields combined into the value string you get from CDB? If it can do it from a line out of /etc/passwd then it can do it from CDB (the value could simply be that same line ... or defined to be something else more convenient).
Postfix only supports name:value tables, either to use the value
(table-style lookup) or to check whether there is an entry for a name (list-style lookup). Therefore it only uses file databases with such a mapping. In the case of valid recipients which will be handed off to dovecot for delivery, it is such a simple list lookup.
The value can be a tuple (see above). We're lucky that what Postfix needs is "does this exist". So it can just completely ignore the value and not need any code to parse it.
(If it interests you: the postfix virtual delivery agent needs very similar information to dovecot, but it uses several key:value maps for the different information, all with the recipient as key.)
I've run into that before. I think it was the source of some difficulty. In programming I've done, I've stored structured data in Berkeley DB and never needed but one table/file per class of index.
I have now created a basic How To for the desired configuration on the 2.0 Wiki. Although it has some 2.0 specifics it should work with 1.2 for most parts too. Maybe someone with a higher privilege level on the Wiki than I have can beautify the URL by removing the Whitespaces :)
http://wiki2.dovecot.org/HowTo/Virtual%20User%20Flat%20File%20Configuration%...
Regards Thomas
On Wednesday 21 April 2010 21:30:12 Thomas Leuxner wrote:
I'm running a setup that should be good enough for what you are trying to achieve. All user information is stored in flat files per domain and you may override per user settings individually:
passdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file }
userdb { args = username_format=%u /var/vmail/auth.d/%d/passwd driver = passwd-file }
$ cat passwd user@domain.tld:{scheme}<password>:5000:5000::/var/vmail/domain.tld/user::u serdb_quota_rule=*:storage=5G userdb_acl_groups=PublicMailboxAdmins
[...]
See how aliases are used for routing and to authenticate valid mail from senders with one file:
$ cat virtual alias@domain.tld login@domain.tld postmaster@domain.tld login@domain.tld
[main.cf] virtual_mailbox_domains = domain.tld, domain1.tld virtual_mailbox_base = /var/vmail virtual_minimum_uid = 100 virtual_uid_maps = static:5000 virtual_gid_maps = static:5000 virtual_alias_maps = hash:/etc/postfix/virtual virtual_transport = lmtp:unix:private/dovecot-lmtp
What I don't see here at all (and neither in your Wiki Howto) is how Postfix determines the valid recipients for the domains in virtual_mailbox_domains.
The correct parameter would be virtual_mailbox_maps, but AFAIK there is no lookup table that read the passwd format from an arbitrary file. So a script that generates a hash/whatever postfix lookup file from the passwd-files would still be necessary.
Or do you use recipient validation via LMTP? (I didn't notice a reject_unverified_recipient though) This at least won't work with deliver, I'm not even sure about LMTP.
Regards Thomas
Rainer
On Thu, Apr 22, 2010 at 11:18:09AM +0200, Rainer Frey wrote:
What I don't see here at all (and neither in your Wiki Howto) is how Postfix determines the valid recipients for the domains in virtual_mailbox_domains.
Postfix will expand possible aliases first and determine the final recipient handing over to Dovecot:
virtual_alias_maps = hash:/etc/postfix/virtual
It will query the recipients by connecting to the socket in its chroot provided by Dovecot:
service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = doveauth }
Once it has the homedir it will send it off via LMTP or deliver, whichever you configured via:
virtual_transport = lmtp:unix:private/dovecot-lmtp or
virtual_transport = dovecot
The correct parameter would be virtual_mailbox_maps, but AFAIK there is no lookup table that read the passwd format from an arbitrary file. So a script that generates a hash/whatever postfix lookup file from the passwd-files would still be necessary.
There is no such thing as a correct parameter from my perspective. I did not say that alias creation was to be unified/automated. Instead I said I did not even think this is good practice to me. Anyone with at least a bit of sed/awk knowledge can kludge it from the flat-files anyway.
Or do you use recipient validation via LMTP? (I didn't notice a reject_unverified_recipient though) This at least won't work with deliver, I'm not even sure about LMTP.
This is not required in the example and optional at least:
http://www.postfix.org/ADDRESS_VERIFICATION_README.html
Consider that verification is done differently in the example based on the fact whether the mail is received on Port 25 or injected via the submission port, completely handled in 'master.cf'. So it has a diverse set of restrictions.
Regards Thomas
Please honour the Reply-To header next time. Thanks.
On Thursday 22 April 2010 11:42:01 Thomas Leuxner wrote:
On Thu, Apr 22, 2010 at 11:18:09AM +0200, Rainer Frey wrote:
What I don't see here at all (and neither in your Wiki Howto) is how Postfix determines the valid recipients for the domains in virtual_mailbox_domains.
Postfix will expand possible aliases first and determine the final recipient handing over to Dovecot:
$ cat virtual alias@domain.tld login@domain.tld postmaster@domain.tld login@domain.tld
virtual_alias_maps = hash:/etc/postfix/virtual
Do you define all valid recipients there (e.g. in you example virtual file login@domain.tld)?
It will query the recipients by connecting to the socket in its chroot provided by Dovecot:
service auth {
This is wrong. The auth service is not queried for recipient, only for valid SASL users (that connect to the submission service as *senders*). I'm talking about determining valid *recipients* for the virtual_mailbox_domains.
[...]
Once it has the homedir it will send it off via LMTP or deliver, whichever you configured via:
virtual_transport = lmtp:unix:private/dovecot-lmtp or virtual_transport = dovecot
But this is at the delivery stage, when the mail has already been accepted. This means, if no homedir/mailbox is found, bounce mails are sent, to potentially forged senders. That is backscatter.
The correct parameter would be virtual_mailbox_maps, but AFAIK there is no lookup table that read the passwd format from an arbitrary file. So a script that generates a hash/whatever postfix lookup file from the passwd-files would still be necessary.
There is no such thing as a correct parameter from my perspective. I did not say that alias creation was to be unified/automated.
I'm not talking about aliases, I'm talking about recipient addresses of virtual mailboxes. You need to verify whether a mailbox exists for a recipient address in the SMTP server before accepting the message.
Instead I said I did not even think this is good practice to me. Anyone with at least a bit of sed/awk knowledge can kludge it from the flat-files anyway.
Indeed, but you offered the original poster your solution as one that "should be good enough for what you are trying to achieve", but your solution leaves out the aspect of the valid recipient list for the virtual mailbox domain address class.
Or do you use recipient validation via LMTP? (I didn't notice a reject_unverified_recipient though) This at least won't work with deliver, I'm not even sure about LMTP.
This is not required in the example and optional at least:
Of course, but it would be a viable alternative to a lookup table for the recipients.
Rainer
On Thu, Apr 22, 2010 at 7:12 AM, Rainer Frey <rainer.frey@inxmail.de> wrote:
This is wrong. The auth service is not queried for recipient, only for valid SASL users (that connect to the submission service as *senders*). I'm talking about determining valid *recipients* for the virtual_mailbox_domains.
That's what I was (initially) talking about, too. Of course, having a submission service would be a good idea so out-of-office users don't have to use some remote SMTP submission. And that will have to be authenticated. A combination solution would be good. But I definitely need to have the recipient test done based on a common database of "mail users".
If the wiki can describe how to set up both, together, that is great. If it has to involve extracting usernames out of a file and running makemap in Postfix to build a btree db file, that's OK, though I was checking to see if there was a known way to avoid that.
If Postfix will reject incoming mail when Dovecot is not up, that's could be bad, depending on how it's rejecting. If it can validate from a file, then, in theory, Postfix can queue and deliver safely later (and only users that were deleted in the interim risk a separately mailed reject).
The ideal would be a complete mail server package that handled it all in one ... SMTP, submission, IMAP(S), POP3(S). But what I've seen as attempts to do that so far were less than promising (even though I have no need for what most of the MTAs do).
On Thu, Apr 22, 2010 at 09:48:36AM -0400, Phil Howard wrote:
The ideal would be a complete mail server package that handled it all in one ... SMTP, submission, IMAP(S), POP3(S). But what I've seen as attempts to do that so far were less than promising (even though I have no need for what most of the MTAs do).
The only one like that I know of is Courier, and I found it incredibly frustrating to work with. I use dovecot and postfix, but don't have the virtual user issue -- all of my users (for twelve domains) have real accounts.
Jim
Jim Trigg, Lord High Everything Else O- /"
\ / ASCII RIBBON CAMPAIGN
Hostmaster, Huie Kin family website X HELP CURE HTML MAIL
Verger, All Saints Church - Sharon Chapel / \
On Thu, Apr 22, 2010 at 12:07 PM, Jim Trigg <jtrigg@spamcop.net> wrote:
On Thu, Apr 22, 2010 at 09:48:36AM -0400, Phil Howard wrote:
The ideal would be a complete mail server package that handled it all in one ... SMTP, submission, IMAP(S), POP3(S). But what I've seen as attempts to do that so far were less than promising (even though I have no need for what most of the MTAs do).
The only one like that I know of is Courier, and I found it incredibly frustrating to work with. I use dovecot and postfix, but don't have the virtual user issue -- all of my users (for twelve domains) have real accounts.
I would have done that, except for the fact that there is the issue of same user name users under different domains, and I don't want the mess of mapping those in strange ways to system users (and I don't want "@" characters in system users).
On Thu, Apr 22, 2010 at 01:12:24PM +0200, Rainer Frey wrote:
Do you define all valid recipients there (e.g. in you example virtual file login@domain.tld)?
Yes.
But this is at the delivery stage, when the mail has already been accepted. This means, if no homedir/mailbox is found, bounce mails are sent, to potentially forged senders. That is backscatter.
I'm not talking about aliases, I'm talking about recipient addresses of virtual mailboxes. You need to verify whether a mailbox exists for a recipient address in the SMTP server before accepting the message.
Possibly. But this could then be fixed by adding another recipient restriction, is that what is bothering you?
Indeed, but you offered the original poster your solution as one that "should be good enough for what you are trying to achieve", but your solution leaves out the aspect of the valid recipient list for the virtual mailbox domain address class.
This was not meant to say this is the ultimate one and only solution. See for recipient_restrictions esspecially, everyone may have different needs. But at least someone *may* a starting point. Feel free to refine the setup.
Of course, but it would be a viable alternative to a lookup table for the recipients.
Will look into it, but maybe you can add your thoughts how you would do.
Thomas
On Thursday 22 April 2010 16:36:33 Thomas Leuxner wrote:
On Thu, Apr 22, 2010 at 01:12:24PM +0200, Rainer Frey wrote:
Do you define all valid recipients there (e.g. in you example virtual file login@domain.tld)?
Yes.
So a valid recipient must be in the passwd file and in the postfix virtual alias file? This does not solve the problem of using the same flat-file user database between doevecot and postfix, and of course int that case you can define a virtual_mailbox_map as well, which works well (no kludge like aliasing an address to itself to terminate recursive alias expansion) and is semantically correct.
But this is at the delivery stage, when the mail has already been accepted. This means, if no homedir/mailbox is found, bounce mails are sent, to potentially forged senders. That is backscatter.
I'm not talking about aliases, I'm talking about recipient addresses of virtual mailboxes. You need to verify whether a mailbox exists for a recipient address in the SMTP server before accepting the message.
Possibly.
No, not possibly, but most definitely. Causing backscatter is not acceptable and leads to the server being blacklisted at some sites.
But this could then be fixed by adding another recipient restriction, is that what is bothering you?
But what recipient restriction? There's only:
- reject_unlisted_recipient, which needs a non-empty recipient lookup map for the domain class
- reject_unverified_recipient, the address verification mentioned below
- check_recipient_access, which again needs a postfix lookup table with valid recipients.
Indeed, but you offered the original poster your solution as one that "should be good enough for what you are trying to achieve", but your solution leaves out the aspect of the valid recipient list for the virtual mailbox domain address class.
This was not meant to say this is the ultimate one and only solution. See for recipient_restrictions esspecially, everyone may have different needs. But at least someone *may* a starting point. Feel free to refine the setup.
Well, it leaves out the *one tricky part* of using a flat file database for virtual users with dovecot and postfix: there is no common format that both understand directly.
[ This quotation is missing the doubt whether postfix address verification works with LMTP (or even pipe) ]
Of course, but it would be a viable alternative to a lookup table for the recipients.
Will look into it, but maybe you can add your thoughts how you would do.
If it works, it is a good alternative that is used in similar setups, although mostly with relay_domains and servers like Exchange that speak SMTP. The ADDRESS_VERIFICATION_README details the limitations and drawbacks
Thomas
Rainer
On Thu, 22 Apr 2010 17:03:00 +0200 Rainer <rainer.frey@inxmail.de> articulated:
Well, it leaves out the *one tricky part* of using a flat file database for virtual users with dovecot and postfix: there is no common format that both understand directly.
I have not been following this thread as closely as I probably should have; however, I was wondering what the OP's problem was with using MySQL? It would greatly simplify the job of constructing and maintaining databases. It is even possible to create tables that both Postfix and Dovecot can use jointly if desired. I use MySQL for several projects, and would never go back to using 'flat files" unless there was no other way to achieve my goal.
If the OP does not know how to construct a usable table in MySQL, I would be willing to work with him as I am sure would others on this list.
The thread subject implies that the OP wants the best choice for a database file system and that would be, IMHO, a SQL one.
Just my 2¢.
-- Jerry Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
On Thu, Apr 22, 2010 at 12:12 PM, Jerry <dovecot.user@seibercom.net> wrote:
On Thu, 22 Apr 2010 17:03:00 +0200 Rainer <rainer.frey@inxmail.de> articulated:
Well, it leaves out the *one tricky part* of using a flat file database for virtual users with dovecot and postfix: there is no common format that both understand directly.
I have not been following this thread as closely as I probably should have; however, I was wondering what the OP's problem was with using MySQL? It would greatly simplify the job of constructing and maintaining databases. It is even possible to create tables that both Postfix and Dovecot can use jointly if desired. I use MySQL for several projects, and would never go back to using 'flat files" unless there was no other way to achieve my goal.
The administration is going to be handed off to less technical people, and my goal is to mimize the number of elements in this. It's not about MySQL itself ... it's about not running yet another server/daemon.
On Thu, 22 Apr 2010 12:18:41 -0400 Phil <ttiphil@gmail.com> articulated:
The administration is going to be handed off to less technical people, and my goal is to mimize the number of elements in this. It's not about MySQL itself ... it's about not running yet another server/daemon.
Excuse my ignorance here; however, it appears that what you are trying to accomplish is exceedingly harder to implement than simply using SQL for the job. The are numerous GUIs to administer SQL with. Personally, I use "phpMyAdmin" with MySQL and have found it quite a good fit. There are several others of course, and other flavors of SQL that you might want to investigate.
I have personally set up MySQL with phpMyAdmin for some rather technically challenged individuals and they have not experienced any problems that I am aware of. There is also a PostfixAdmin <http://sourceforge.net/projects/postfixadmin/> that might simplify your life. I don't use it myself; however, I have heard it is quite good.
Again, just my 2¢.
-- Jerry Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
On Thu, Apr 22, 2010 at 12:40 PM, Jerry <dovecot.user@seibercom.net> wrote:
On Thu, 22 Apr 2010 12:18:41 -0400 Phil <ttiphil@gmail.com> articulated:
The administration is going to be handed off to less technical people, and my goal is to mimize the number of elements in this. It's not about MySQL itself ... it's about not running yet another server/daemon.
Excuse my ignorance here; however, it appears that what you are trying to accomplish is exceedingly harder to implement than simply using SQL for the job. The are numerous GUIs to administer SQL with. Personally, I use "phpMyAdmin" with MySQL and have found it quite a good fit. There are several others of course, and other flavors of SQL that you might want to investigate.
I have personally set up MySQL with phpMyAdmin for some rather technically challenged individuals and they have not experienced any problems that I am aware of. There is also a PostfixAdmin <http://sourceforge.net/projects/postfixadmin/> that might simplify your life. I don't use it myself; however, I have heard it is quite good.
The decision to go non-SQL and non-LDAP has already been made. Only if nothing else can work will it be re-opened. The non-SQL/non-LDAP approach is what my current investigation is on.
Phil Howard put forth on 4/22/2010 11:18 AM:
On Thu, Apr 22, 2010 at 12:12 PM, Jerry <dovecot.user@seibercom.net> wrote:
On Thu, 22 Apr 2010 17:03:00 +0200 Rainer <rainer.frey@inxmail.de> articulated:
Well, it leaves out the *one tricky part* of using a flat file database for virtual users with dovecot and postfix: there is no common format that both understand directly.
I have not been following this thread as closely as I probably should have; however, I was wondering what the OP's problem was with using MySQL? It would greatly simplify the job of constructing and maintaining databases. It is even possible to create tables that both Postfix and Dovecot can use jointly if desired. I use MySQL for several projects, and would never go back to using 'flat files" unless there was no other way to achieve my goal.
The administration is going to be handed off to less technical people, and my goal is to mimize the number of elements in this. It's not about MySQL itself ... it's about not running yet another server/daemon.
With this many lookup table types supported by Postfix, is it true that it has no "simple" table type in common with Dovecot?
btree A sorted, balanced tree structure. This is
available on systems with support for Berke-
ley DB databases.
cdb A read-optimized structure with no support
for incremental updates. This is available
on systems with support for CDB databases.
cidr A table that associates values with Class-
less Inter-Domain Routing (CIDR) patterns.
This is described in cidr_table(5).
dbm An indexed file type based on hashing. This
is available on systems with support for DBM
databases.
environ
The UNIX process environment array. The
lookup key is the variable name. Originally
implemented for testing, someone may find
this useful someday.
hash An indexed file type based on hashing. This
is available on systems with support for
Berkeley DB databases.
internal
A non-shared, in-memory hash table. Its con-
tent are lost when a process terminates.
ldap (read-only)
Perform lookups using the LDAP protocol.
This is described in ldap_table(5).
mysql (read-only)
Perform lookups using the MYSQL protocol.
This is described in mysql_table(5).
pcre (read-only)
A lookup table based on Perl Compatible Reg-
ular Expressions. The file format is
described in pcre_table(5).
pgsql (read-only)
Perform lookups using the PostgreSQL proto-
col. This is described in pgsql_table(5).
proxy (read-only)
A lookup table that is implemented via the
Postfix proxymap(8) service. The table name
syntax is type:name.
regexp (read-only)
A lookup table based on regular expressions.
The file format is described in regexp_ta-
ble(5).
sdbm An indexed file type based on hashing. This
is available on systems with support for
SDBM databases.
static (read-only)
A table that always returns its name as
lookup result. For example, static:foobar
always returns the string foobar as lookup
result.
tcp (read-only)
Perform lookups using a simple request-reply
protocol that is described in tcp_table(5).
This feature is not included with the stable
Postfix release.
unix (read-only)
A limited way to query the UNIX authentica-
tion database. The following tables are
implemented:
unix:passwd.byname
The table is the UNIX password data-
base. The key is a login name. The
result is a password file entry in
passwd(5) format.
unix:group.byname
The table is the UNIX group database.
The key is a group name. The result
is a group file entry in group(5)
format.-- Stan
On Thu, Apr 22, 2010 at 3:12 PM, Stan Hoeppner <stan@hardwarefreak.com>wrote:
With this many lookup table types supported by Postfix, is it true that it has no "simple" table type in common with Dovecot?
There are some ... like mysql for example. The ones I call "simple" are ones that have a single file in some readily contructable form. These include btree (Berkeley DB) and cdb (Dan B's CDB). Others could be included but I haven't looked at them, yet (such as cidr, which has specific purposes, only). Things I consider "not simple" involve making a network connection. It can certainly be simple to configure them. But I'm classifying these in terms of the number of things going on to get the answer. Even a DB scheme to use a DNS lookup (not implemented for Postfix) I would consider as "not simple" here, even though it is likely very simple to do. The tcp scheme is probably the simplest of the network schemes (but it requires something to listen somewhere). If I were asked what I think could be added to this list of lookup schemes, it might be a long, long thread. Let's not do that here.
unix (read-only)A limited way to query the UNIX authentica- tion database. The following tables are implemented: unix:passwd.byname The table is the UNIX password data- base. The key is a login name. The result is a password file entry in passwd(5) format. unix:group.byname The table is the UNIX group database. The key is a group name. The result is a group file entry in group(5) format.
This is "simple", too. But the problem is it isn't flexible enough. Dovecot supports this scheme and can use it on any specified file. Postfix is limited to just the system files (and is probably using Unix library calls to access it, hence the source of the limitation).
I would think the general (insert some character here)-separated file should be doable: comma-delimited:/some/file/name colon-delimited:/some/file/name tab-delimited:/some/file/name space-delimited:/some/file/name But it hasn't been done that I can see (some third party patch not withstanding).
Anyway, I need to re-study what is involved with the virtual users, since I may have to be doing this in Postfix just to get submission authentication to work, and then see how that affects using local_recipient_maps. And at this point it's probably more of a SASL issue than a Dovecot issue. So back to reading Postfix docs (including the SASL reference someone gave) and asking on the Postfix M/L if needed (not here, this is for Dovecot). It may well be the description given early to use Dovecot SASL to authenticate Postfix submission users would be just fine, as long as Postfix does a SOFT rejection when Dovecot isn't up, so it requeues files. On the SMTP end, I just want an up front rejection decision to avoid backscatter. The one backscatter hole where an existing user is being deleted while incoming mail is being queued is probably insignificant.
Phil Howard put forth on 4/22/2010 3:07 PM:
I would think the general (insert some character here)-separated file should be doable: comma-delimited:/some/file/name colon-delimited:/some/file/name tab-delimited:/some/file/name space-delimited:/some/file/name But it hasn't been done that I can see (some third party patch not withstanding).
It's not exactly what you're asking for here, but it sounds like SQLite would fit your needs. If only both Postfix and Dovecot had an SQLite driver for user and pwd lookup. At this time I believe neither does.
well be the description given early to use Dovecot SASL to authenticate Postfix submission users would be just fine, as long as Postfix does a SOFT
This is a very populate way to integrate the two, and works well.
rejection when Dovecot isn't up, so it requeues files. On the SMTP end, I just want an up front rejection decision to avoid backscatter. The one backscatter hole where an existing user is being deleted while incoming mail is being queued is probably insignificant.
Postfix is very configurable in this regard and is smart enough to return a 4xx when it can't talk to Dovecot (or any "remote" auth provider) and a 5xx when Dovecot says unknown user. I'm not looking at docs now, but upon a brief scan earlier I recall at least 3 different rejection codes Postfix will return depending on the reason an auth attempt fails. Some of these are configurable. One nice thing about Postfix is that the documentation is _very_ thorough, even if sometimes hard to digest.
Good luck, and please keep us up to date. This is an interesting topic and the solution may be useful to others.
-- Stan
On Thu, Apr 22, 2010 at 5:19 PM, Stan Hoeppner <stan@hardwarefreak.com>wrote:
One nice thing about Postfix is that the documentation is _very_ thorough, even if sometimes hard to digest.
Yes, I would agree. Sometimes a twisty maze of passages, but you can eventually find things.
Good luck, and please keep us up to date. This is an interesting topic and
the solution may be useful to others.
I'll try to write a wiki page on what I do, if it isn't a duplicate of one already there. I'd call it a "minimalist mail server".
Phil Howard skrev:
On Thu, Apr 22, 2010 at 12:12 PM, Jerry <dovecot.user@seibercom.net> wrote:
On Thu, 22 Apr 2010 17:03:00 +0200 Rainer <rainer.frey@inxmail.de> articulated:
Well, it leaves out the *one tricky part* of using a flat file database for virtual users with dovecot and postfix: there is no common format that both understand directly.
I have not been following this thread as closely as I probably should have; however, I was wondering what the OP's problem was with using MySQL? It would greatly simplify the job of constructing and maintaining databases. It is even possible to create tables that both Postfix and Dovecot can use jointly if desired. I use MySQL for several projects, and would never go back to using 'flat files" unless there was no other way to achieve my goal.
The administration is going to be handed off to less technical people, and my goal is to mimize the number of elements in this. It's not about MySQL itself ... it's about not running yet another server/daemon.
Have you looked into Postfix Admin? http://postfixadmin.sourceforge.net/
It might be a good solution for you. I'm using it for a a growing database of users and I'm very happy with it. The setup with postfix, dovecot and mysql was quite straight forward, and this interface requires no particular technical know-how.
Should be perfect if you can do the initial setup, then just give the admisn the password to this interface.
Arne
On Thu, Apr 22, 2010 at 4:31 PM, Arne K. Haaje <arne@drlinux.no> wrote:
Have you looked into Postfix Admin? http://postfixadmin.sourceforge.net/
It might be a good solution for you. I'm using it for a a growing database of users and I'm very happy with it. The setup with postfix, dovecot and mysql was quite straight forward, and this interface requires no particular technical know-how.
Should be perfect if you can do the initial setup, then just give the admisn the password to this interface.
No, I had not looked into this, yet. But it won't fit in here. Adding/deleting users is going to be going though another system where likely the HR person will be doing it. The admin will then be handling the "what's wrong with it" issues (as opposed to the clerical work of adding/deleting users). And it requires database engine.
On 22/04/2010 17:18, Phil Howard wrote:
I have not been following this thread as closely as I probably should have; however, I was wondering what the OP's problem was with using MySQL? It would greatly simplify the job of constructing and maintaining databases. It is even possible to create tables that both Postfix and Dovecot can use jointly if desired. I use MySQL for several projects, and would never go back to using 'flat files" unless there was no other way to achieve my goal.
The administration is going to be handed off to less technical people, and my goal is to mimize the number of elements in this. It's not about MySQL itself ... it's about not running yet another server/daemon.
You need to look at where you are going with this.... One way or another you need a database - call it a banana if you prefer, but it's still a database whether it's a flat file or a BDB file or whatever
Your requirements appear to be badly phrased. What you *appear* to want
is a black box system which is as simple to maintain as possible.
However, you have stated a bunch of hard to meet criteria such as not
allowing any long running code to support your needs (a daemon). I
really don't see that you should exclude additional daemons, simply
evaluate the complete system with and without and choose the one which
is "easiest" to maintain
Yes you CAN setup something which uses a plain text database, I tried it for a while but at 50 users I decided it was too complicated and switched to a DB.
You are optimistic if you don't think some user will accidently add a <CR> halfway through the text file, completely breaking it and have the skills to realise what they did. Or they will add a "," in the middle of a password and change the meaning of all your fields. Or they will miss a field save the file and then never realise their mistake. Better yet are errors which cause some external (grep,sed,awk) function to choke on the input and cause some really wierd effects downstream...
Why should an sql database need any monitoring and maintenance over the next X years? Yes you can corrupt the files, but I would hope for very decent uptimes and after all they aren't going to repair a corrupted boot sector so you need some kind of maintenance plan in place, simply work in a full OS backup into the plan and this saves your DB at the same time? (Offlist we can discuss, but a simple rsync to some separate partition should do it)
If you really, really don't want a daemon based database then you will have to look at bdb (if dovecot supports it?) or sqlite which I think postfix+dovecot both support. You can then add convenience command line functions to examine and alter the data. Those convenience functions can error check the input also.
Good luck
Ed W
P.S. The "idiot" who kept breaking the plain text format file in my original setup was .... da da ... me ... So given I think of myself as reasonably technical, I would claim that text format "databases" are way more fragile than you might expect.... (good luck getting a non technical user not to break the text format file from time to time...)
On Fri, Apr 23, 2010 at 5:23 AM, Ed W <lists@wildgooses.com> wrote:
P.S. The "idiot" who kept breaking the plain text format file in my original setup was .... da da ... me ... So given I think of myself as reasonably technical, I would claim that text format "databases" are way more fragile than you might expect.... (good luck getting a non technical user not to break the text format file from time to time...)
But this is more an issue of exposing the system to human failure than it is a choice of using a flat file. If the flat file is generated by some other system, and maybe merged with other data, and never handled by a human except in extraordinary situations, it should be fine. This is used as a justification for XML being a text format (that a human can read it and even edit it, in non-routine situations).
FYI, I have done hand editing of /etc/{group,passwd,shadow} to add users. So far I've never killed root. I still have to at times. Most recently I did so with Ubuntu because the user management tool couldn't get it right (I needed a specific UID/GID arrangement for compatibility with other systems, and the GUI tool refused ... it would let me do one or the other, but not both at the same time).
On Fri, Apr 23, 2010 at 3:45 AM, Ed W <lists@wildgooses.com> wrote:
You need to look at where you are going with this.... One way or another you
need a database - call it a banana if you prefer, but it's still a database whether it's a flat file or a BDB file or whatever
One must be careful with the term "database". I agree with you that it could be anything. But there are lots of people out there that assume some server daemon or engine when you say "database". And most of them assume "SQL" is in there, too. So I avoid that term.
Your requirements appear to be badly phrased. What you *appear* to want is a black box system which is as simple to maintain as possible. However, you have stated a bunch of hard to meet criteria such as not allowing any long running code to support your needs (a daemon). I really don't see that you should exclude additional daemons, simply evaluate the complete system with and without and choose the one which is "easiest" to maintain
Yes, a black box system would have suited. But it needs to be open source (that's not a contradiction ... I just have't seen one that I can use). It also needs to be able to integrate with what I will have, which is an exported file, which I have reason to suspect will be in a spreadsheet form.
- Yes you CAN setup something which uses a plain text database, I tried it
for a while but at 50 users I decided it was too complicated and switched to a DB.
What complications? Were you hand editing this file?
- You are optimistic if you don't think some user will accidently add a
<CR> halfway through the text file, completely breaking it and have the skills to realise what they did. Or they will add a "," in the middle of a password and change the meaning of all your fields. Or they will miss a field save the file and then never realise their mistake. Better yet are errors which cause some external (grep,sed,awk) function to choke on the input and cause some really wierd effects downstream...
Ah, yes, hand editing. But that won't be happening in my case.
- Why should an sql database need any monitoring and maintenance over the
next X years? Yes you can corrupt the files, but I would hope for very decent uptimes and after all they aren't going to repair a corrupted boot sector so you need some kind of maintenance plan in place, simply work in a full OS backup into the plan and this saves your DB at the same time? (Offlist we can discuss, but a simple rsync to some separate partition should do it)
You rsync the files an SQL database (like MySQL) works from, and don't expect corruption? That's only safe if the database is synced and shut down. I don't want to be doing that. If I did run a database engine, it would have to import everything ... as a single massive transaction (or maybe a live table switch scenario between two tables). To back it up I'd either export the entire table to a file (and send that off to the archive), or just back up the file I used to import with.
- If you really, really don't want a daemon based database then you will
have to look at bdb (if dovecot supports it?) or sqlite which I think postfix+dovecot both support. You can then add convenience command line functions to examine and alter the data. Those convenience functions can error check the input also.
CDB would work, since I would be importing everything all at once every time a change takes place (once every few days perhaps). Berkeley DB would work, too (nearly as fast). A flat file cached in RAM by the OS would be about as fast for these few number of users (that's a "database", too, in its own way).
Good luck
It looks like the way I will be doing this is to let Dovecot handle Postfix's SASL needs and generate a little users table for Postfix to make the accept/reject decision on incoming email and spam. So my task now is to decide what form I will give the data to Dovecot.
On Fri, 23 Apr 2010 09:03:02 -0400 Phil <ttiphil@gmail.com> articulated:
You rsync the files an SQL database (like MySQL) works from, and don't expect corruption? That's only safe if the database is synced and shut down. I don't want to be doing that. If I did run a database engine, it would have to import everything ... as a single massive transaction (or maybe a live table switch scenario between two tables). To back it up I'd either export the entire table to a file (and send that off to the archive), or just back up the file I used to import with.
There are numerous ways to export/backup a live MySQL database. I have employed several of them myself. You might want to check out this URL for starters:
http://www.noupe.com/how-tos/10-ways-to-automatically-manually-backup-mysql-...
Your statement that the only safe way to backup/export an SQL database is to stop it, etc is totally incorrect. No offense, but I question your knowledge of SQL, and MySQL in particular. If you need help with SQL, I know plenty of individuals, including myself, who would be willing to help you design a usable one for your needs.
I am somewhat perplex by you insinuation that the end users of this database system you are designing are total novices. Depending on the table:type you choose, you may very well have to invoke 'postmap' and perhaps 'postaliase' on them to make them usable by Postfix. If your intended audience is so incompetent that they cannot handle a simple SQL database, how do you expect them to handle the intricacies of getting this database you are designing serviceable?
It is my own opinion; however, I think you are basing your decision on a fear of SQL more than on simplifying the final database decision. A simple SQL would greatly simplify maintenance and security. Then again, you are going to be the one who takes the heat if this blows up in your face.
-- Jerry Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
Entropy requires no maintenance.
Markoff ChaneyOn 23/04/2010 15:51, Jerry wrote:
There are numerous ways to export/backup a live MySQL database. I have employed several of them myself. You might want to check out this URL for starters:
http://www.noupe.com/how-tos/10-ways-to-automatically-manually-backup-mysql-...
Your statement that the only safe way to backup/export an SQL database is to stop it, etc is totally incorrect. No offense, but I question your knowledge of SQL, and MySQL in particular.
Err to be fair, I was about to reply and say that it's IS achievable, but on the flip side I would concede that it does require a bit of thought to make this stuff work correctly. I think you are being a bit harsh to suggest it's *that* simple to make an automated push work reliably?
So in summary, it IS possible, but yes it does need a bit of thought, fair enough.
I am somewhat perplex by you insinuation that the end users of this database system you are designing are total novices. Depending on the table:type you choose, you may very well have to invoke 'postmap' and perhaps 'postaliase' on them to make them usable by Postfix. If your intended audience is so incompetent that they cannot handle a simple SQL database, how do you expect them to handle the intricacies of getting this database you are designing serviceable?
It is my own opinion; however, I think you are basing your decision on a fear of SQL more than on simplifying the final database decision. A simple SQL would greatly simplify maintenance and security. Then again, you are going to be the one who takes the heat if this blows up in your face.
The OP has noted (offlist) some extra facts. Basically he has an upstream setup which will generate all the files - these will then be pushed down to the remote sites in an automated fashion, therefore there will be no editing of the local files. The situation is simply a case of finding the most efficient way to push the static data down to the remote sites
From that point of view I still think a DB is a useful option, but the text/BDB options are also much more sensible
Cheers
Ed W
On Fri, 23 Apr 2010 16:07:20 +0100 Ed <lists@wildgooses.com> articulated:
Err to be fair, I was about to reply and say that it's IS achievable, but on the flip side I would concede that it does require a bit of thought to make this stuff work correctly. I think you are being a bit harsh to suggest it's *that* simple to make an automated push work reliably?
Sorry, I did not mean that it was trivial, although it is not exactly rocket science either. I was simply countering the statement that the whole MySQL daemon process would have to be shut down to achieve a reliable backup/export was incorrect.
So in summary, it IS possible, but yes it does need a bit of thought, fair enough.
Again, I did not mean to offend anyone. Sorry!
-- Jerry Dovecot.user@seibercom.net
Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header.
The chief enemy of creativity is "good" sense.
PicassoAm 22.04.2010 um 17:03 schrieb Rainer Frey (Inxmail GmbH):
So a valid recipient must be in the passwd file and in the postfix virtual alias file? This does not solve the problem of using the same flat-file user database between doevecot and postfix, and of course int that case you can define a virtual_mailbox_map as well, which works well (no kludge like aliasing an address to itself to terminate recursive alias expansion) and is semantically correct.
In order to use the same file for 'smtpd_sender_login_maps' yes. If you wouldn't want to use them you may drop the self-aliasing part I guess (untested).
- reject_unverified_recipient, the address verification mentioned below
Added to the Wiki. This at least works for the LMTP-Server and will not fire an NDR.
Thomas
On Thu, Apr 22, 2010 at 5:42 AM, Thomas Leuxner <tlx@leuxner.net> wrote:
On Thu, Apr 22, 2010 at 11:18:09AM +0200, Rainer Frey wrote:
What I don't see here at all (and neither in your Wiki Howto) is how Postfix determines the valid recipients for the domains in virtual_mailbox_domains.
Postfix will expand possible aliases first and determine the final recipient handing over to Dovecot:
virtual_alias_maps = hash:/etc/postfix/virtual
It will query the recipients by connecting to the socket in its chroot provided by Dovecot:
service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } user = doveauth }
Once it has the homedir it will send it off via LMTP or deliver, whichever you configured via:
virtual_transport = lmtp:unix:private/dovecot-lmtp or
virtual_transport = dovecot
The correct parameter would be virtual_mailbox_maps, but AFAIK there is
no
lookup table that read the passwd format from an arbitrary file. So a script that generates a hash/whatever postfix lookup file from the passwd-files would still be necessary.
There is no such thing as a correct parameter from my perspective. I did not say that alias creation was to be unified/automated. Instead I said I did not even think this is good practice to me. Anyone with at least a bit of sed/awk knowledge can kludge it from the flat-files anyway.
So your suggestion for local_recipient_maps is to read from a table extracted from the file used by Dovecot. And then use the previously described setup (now in the Wiki) for a proper submission interface (using SASL and asking Dovecot).'
I have Dovecot 1.1.11, and for now using package versions outside of what Ubuntu offers isn't an option (that's something I'll be doing later with a different distribution, but it isn't an option right now). So I can't have submission ask via LMTP. If I can get Postfix to authenticate via passwords in a btree or cdb, I guess I can just copy BOTH usernames AND passwords from the file Dovecot reads from and build the btree or cdb db file for Postfix from there.
Any chance Dovecot can read either btree or cdb files directly?
On Wed, Apr 21, 2010 at 01:45:35PM -0400, Phil Howard wrote:
Then I think MySQL will do the job. Both postfix and dovecot support MySQL, and you can use SASL to authenticate SMTP with Dovecot, so Dovecot would do all the auth work. Finally, you could use Postfix's VDA patch if you want to use Maildir++.
Hope this helps.
I don't want to use any other server engine of any kind with this. I'm trying to keep it small and lean, and minimize what the people that have to monitor and fix it need to know. So at the present time, I am excluding all databases like any SQL or LDAP or anything else that needs to run as a daemon/engine/service.
Aham, yes, but as soon as you need some management interface, like a web based one, it will be more and more complicated to deal with this decision, you must edit/generate those files with that interface, care about locking because of the possibility of multiple admin access at the same time, you must parse them when you want to show the user list and so on. Sure, if you are very sure that it's not a need and it won't be either, then maybe you're right about "keeping minimal" solution, but based on my experience at an ISP, it's always the situation that sooner or later somebody want to extend the usage of a system which sooner or later needs to use some kind of database to avoid the complexity with dealing local "databases" as flat files or other solution (or keeping them as "system users").
On 23/04/2010 10:30, Gábor Lénárt wrote:
On Wed, Apr 21, 2010 at 01:45:35PM -0400, Phil Howard wrote:
Then I think MySQL will do the job. Both postfix and dovecot support MySQL, and you can use SASL to authenticate SMTP with Dovecot, so Dovecot would do all the auth work. Finally, you could use Postfix's VDA patch if you want to use Maildir++.
Hope this helps.
I don't want to use any other server engine of any kind with this. I'm trying to keep it small and lean, and minimize what the people that have to monitor and fix it need to know. So at the present time, I am excluding all databases like any SQL or LDAP or anything else that needs to run as a daemon/engine/service.
Aham, yes, but as soon as you need some management interface, like a web based one, it will be more and more complicated to deal with this decision, you must edit/generate those files with that interface, care about locking because of the possibility of multiple admin access at the same time, you must parse them when you want to show the user list and so on. Sure, if you are very sure that it's not a need and it won't be either, then maybe you're right about "keeping minimal" solution, but based on my experience at an ISP, it's always the situation that sooner or later somebody want to extend the usage of a system which sooner or later needs to use some kind of database to avoid the complexity with dealing local "databases" as flat files or other solution (or keeping them as "system users").
Yeah, I PROMISE that someone will eventually corrupt the flat file text "database" by accident. So now you need to add a backup system (git? rsync?), plus some editing sanity checks? (a la visudo?)
In the end I personally went for a mysql + small rails frontend (using activescaffold which allows you to generate an admin interface in a couple of lines of code)
We don't know the exact requirements of the OP, but based on what we read here, *I* personally would evaluate either:
- sqlite based database with command line shell scripts to edit/examine
- mysql based system with remote control editing from some centralised location (ie you admin it for them)
- Some black box office distro where you just push the button and out it rolls. I didn't look, but there must be some Centos/Ubuntu derivative thing which comes with a web interface, webmail, etc?
With mysql use innodb databases and all the logging options set to sync writes. It should be near impossible to kill even if you keep ripping the power out.
Good luck
Ed W
Set up the users in /etc/dovecot/passwd for Dovecot authentication and /etc/postfix/vmailbox for postfix users. Not possible to use one file for two of them because the formats are markedly different.
Alexander ----- Original Message ----- From: "Phil Howard" <ttiphil@gmail.com> To: <dovecot@dovecot.org> Sent: Wednesday, April 21, 2010 3:32 PM Subject: [Dovecot] best choice of user database file to work with postfix?
I'm setting up a Postfix and Dovecot combination. What I want to do is have a user database that (1) is not running from some engine (so not LDAP or SQL or such) ... and (2) is completely disassociated from system users (e.g. most email users are not in /etc/passwd and most /etc/passwd users are not email users). Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login authentication) can use together. An alternative is some way to get Postfix to go through Dovecot to query for users (at the time of mail arriving on SMTP so it doesn't queue anything that would later be rejected). This is a smallish setup on one server, with probably a max of 50 to 100 users and 50 or so role account mailboxes over the next year or two. Any recommendations?
postfix:
smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth
dovecot:
auth default { socket listen { client { path = /var/spool/postfix/private/auth user = _postfix group = wheel mode = 0660 } } }
Then configure your favorite auth mechanism for dovecot. Let bake for 20 minutes, add salt for taste.
On 2010 Apr 21 (Wed) at 10:32:36 -0400 (-0400), Phil Howard wrote: :I'm setting up a Postfix and Dovecot combination. What I want to do is have :a user database that (1) is not running from some engine (so not LDAP or SQL :or such) ... and (2) is completely disassociated from system users (e.g. :most email users are not in /etc/passwd and most /etc/passwd users are not :email users). Ideal would be a one-file solution, which can be managed by :text editing or simple command line tools. But what I want is ONE file that :both Postfix (for valid recipients) and Dovecot (for user login :authentication) can use together. An alternative is some way to get Postfix :to go through Dovecot to query for users (at the time of mail arriving on :SMTP so it doesn't queue anything that would later be rejected). This is a :smallish setup on one server, with probably a max of 50 to 100 users and 50 :or so role account mailboxes over the next year or two. Any :recommendations?
-- Banectomy, n.: The removal of bruises on a banana. -- Rich Hall, "Sniglets"
On Wed, Apr 21, 2010 at 10:58 AM, Peter Hessler <phessler@theapt.org> wrote:
postfix:
smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth
dovecot:
auth default { socket listen { client { path = /var/spool/postfix/private/auth user = _postfix group = wheel mode = 0660 } } }
Then configure your favorite auth mechanism for dovecot. Let bake for 20 minutes, add salt for taste.
This looks interesting. That would basically be run through Dovecot with a named socket I take it. How would the lookup table type be specified for local_recipient_maps in Postfix to use this?
On 04/22/10 00:58, Peter Hessler wrote:
postfix:
smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth
dovecot:
auth default { socket listen { client { path = /var/spool/postfix/private/auth user = _postfix group = wheel mode = 0660 } } }
Then configure your favorite auth mechanism for dovecot. Let bake for 20 minutes, add salt for taste.
I'll second this.
Why bother finding a file format that's compatible, when Postfix has built-in support for asking Dovecot to take care of it?
I've been running my own mail server this way for some time now...
-- Curtis
On Wed, Apr 21, 2010 at 10:32:36AM -0400, Phil Howard wrote:
Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login authentication) can use together. An alternative is some way to get Postfix to go through Dovecot to query for users (at the time of mail arriving on SMTP so it doesn't queue anything that would later be rejected). This is a smallish setup on one server, with probably a max of 50 to 100 users and 50 or so role account mailboxes over the next year or two. Any recommendations?
If you can't get postfix to read dovecot's virtual users, try switching to exim. I found exim can be configured to read pretty much any format you like. For example, I got exim to read courier-imap's userdb.dat files:
Lookup in userdb. If the address lookup succeeds, then we set
address_data. If it's forced to fail, we'll drop through to the next
router. If a temporary error occurs (e.g. file not readable), we'll defer.
reset_address_data: driver = redirect address_data = data =
userdb_lookup:
driver = redirect
condition = ${if ! def:address_data}
address_data = source=userdb ${sg
{${lookup {$local_part@$domain} dbmnz {/path/to/userdb.dat} {$value}
{${lookup {@$domain} dbmnz {/path/to/userdb.dat}
{$value} fail }} }}
{([^=]+)=([^|]+)\\|?} {\$1=\${quote:\$2\} } }
note space between \$2\} and the next }
data =
Then use ${extract {home} {$address_data} ...} to get field "home" from the userdb entry.
Basically what it's doing is finding the the .dat entry with key user@domain or @domain (wildcard), finding a value of the form foo=bar|baz=qux and rewriting it to foo="bar" baz="qux", which the ${extract} function expects.
This sort of coding in exim's configure file is not pretty, but it works well.
With only 50-100 users you'd be fine with a plain text file rather (lsearch) than a .dat
HTH,
Brian.
Hello Phil,
Phil Howard <ttiphil@gmail.com> (Mi 21 Apr 2010 16:32:36 CEST):
I'm setting up a Postfix and Dovecot combination. What I want to do is have a user database that (1) is not running from some engine (so not LDAP or SQL or such) ... and (2) is completely disassociated from system users (e.g. most email users are not in /etc/passwd and most /etc/passwd users are not email users). Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login
A recent demonstration of a German postfix expert used a sed-Script to convert (basically cut everything behind the first „:“) the dovecot passdb file to a postfix readable text file (and convert this to a hash(?)).
I'm not sure, if postfix really can't read a passdb (passwd-like) file. Probably it (postfix) isn't flexible enough for doing this, or the expert didn't want to show it.
As an exim user I'd suggest using exim - and enjoing real flexiblity ;-) The solution I'd prefer is (d) - it makes your exim independend on the userdb/passdb used by dovecot, you're just talking to the auth-master. (Something I'd implement additionally is a „softfail“ (4xx error) in case the socket is not usable.)
exim config snipped - the dovecot passdb is /etc/vmail/passwd
# for better readability of the (d) alternative below (using
# exims macro feature
SOCKET = /var/run/dovecot/auth-master
REQUEST = VERSION\t1\t0\nUSER\t$pid\t$local_part\tservice=imap\n
# local user router
# chose (a), (b), (c), (d)
vmail:
driver = accept
#(a) local_parts = lsearch;/etc/vmail/passwd
#(b) condition = ${lookup{$local_part}lsearch{/etc/vmail/passwd}{true}}
#(c) condition = ${lookup{$local_part@$domain}lsearch{/etc/vmail/passwd}{true}}
#(d) condition = ${if match {${readsocket{SOCKET}{REQUEST}}} {(?m)^USER}}
transport = dovecot
# dovecot transport
# dovecot uses uid vmail for accessing all mailboxes (userdb static)
dovecot:
driver = pipe
command = /usr/lib/dovecot/deliver -d $local_part@$domain
user = vmail(…)
smallish setup on one server, with probably a max of 50 to 100 users and 50 or so role account mailboxes over the next year or two. Any recommendations?
Use Exim ;-)
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann-- SCHLITTERMANN.de ---------------------------- internet & unix support - Heiko Schlittermann HS12-RIPE ----------------------------------------- gnupg encrypted messages are welcome - key ID: 48D0359B --------------- gnupg fingerprint: 3061 CFBF 2D88 F034 E8D2 7E92 EE4E AC98 48D0 359B -
On Wed, Apr 21, 2010 at 5:45 PM, Heiko Schlittermann <hs@schlittermann.de>wrote:
Hello Phil,
Phil Howard <ttiphil@gmail.com> (Mi 21 Apr 2010 16:32:36 CEST):
I'm setting up a Postfix and Dovecot combination. What I want to do is have a user database that (1) is not running from some engine (so not LDAP or SQL or such) ... and (2) is completely disassociated from system users (e.g. most email users are not in /etc/passwd and most /etc/passwd users are not email users). Ideal would be a one-file solution, which can be managed by text editing or simple command line tools. But what I want is ONE file that both Postfix (for valid recipients) and Dovecot (for user login
A recent demonstration of a German postfix expert used a sed-Script to convert (basically cut everything behind the first „:“) the dovecot passdb file to a postfix readable text file (and convert this to a hash(?)).
Dozens, maybe millions, of ways to do that. The "cut" command might be enough. I do stuff in C, Bash, Pike, Awk, and even a little Python (just started learning it), as needed. I'd just integrate this copying and conversion into the script used to add mail users.
I'm not sure, if postfix really can't read a passdb (passwd-like) file.
Probably it (postfix) isn't flexible enough for doing this, or the expert didn't want to show it.
It has the code to parse it. It just assumes a specific file location (e.g. the Unix system file /etc/passwd).
As an exim user I'd suggest using exim - and enjoing real flexiblity ;-)
The solution I'd prefer is (d) - it makes your exim independend on the userdb/passdb used by dovecot, you're just talking to the auth-master. (Something I'd implement additionally is a „softfail“ (4xx error) in case the socket is not usable.)
I'm already more familiar with Postfix. It's Dovecot and IMAP that's new to me right now. I can make this work in Postfix. But I'm just checking for shortcuts I don't otherwise see.
participants (18)
-
Alexander
-
Andreas Schulze
-
Arne K. Haaje
-
Brian Candler
-
Curtis Maloney
-
Ed W
-
Gábor Lénárt
-
Heiko Schlittermann
-
Jerry
-
Jim Trigg
-
Patrick Nagel
-
Peter Hessler
-
Phil Howard
-
Rainer Frey
-
Rainer Frey (Inxmail GmbH)
-
Rodolfo Gonzalez
-
Stan Hoeppner
-
Thomas Leuxner