Hi,
Up to now, I used PAM of each user in order to send and receive email. ( BTW, sending email, a use authentication was required and we used the login and passwd of the user on the system
Now, for dovecot, I start to use MD5 passwrd.. and that sounds to be OK
auth_mechanisms = plain login cram-md5
passdb { driver = passwd-file # Path for passwd-file. Also set the default password scheme. args = scheme=cram-md5 /etc/cram-md5.pwd }
But changing the passwrd for the user1.. he can retrieve emails from dovecot, but cannot send anymore, because sending emails kept the old passwrd. ( using the PAM)
How can I says sendmail to use the same passwd file ( with MD5) than dovecot ?
Ideally, I would like to create virtual users for the same mailbox Is that possible ?
like 2 files Users and PAsswrds pointing out the mailbox : maildir :/home/mailbox/user1
ex :
user1@foo.com passwrd1 /home/mailbox/generic_mails
and
user2 passwrd2 home/mailbox/generic_mails
How can I do that ?
Thanks for your help
steph> Up to now, I used PAM of each user in order to send and receive steph> email. ( BTW, sending email, a use authentication was required steph> and we used the login and passwd of the user on the system
So just to be clear, each user has a login on your mail server in /etc/passwd? If so, I would strongly urge you to move to using only virtual users on your mail infrastructure.
steph> Now, for dovecot, I start to use MD5 passwrd.. and that sounds to be OK
steph> auth_mechanisms = plain login cram-md5 steph> passdb { steph> driver = passwd-file steph> # Path for passwd-file. Also set the default password scheme. steph> args = scheme=cram-md5 /etc/cram-md5.pwd steph> }
steph> But changing the passwrd for the user1.. he can retrieve steph> emails from dovecot, but cannot send anymore, because sending steph> emails kept the old passwrd. ( using the PAM)
What is your mail software? I assume you are having your users connect to port 587 to submit emails to be sent out, correct? If so, are you using postfix, exim, sendmail or some other mailer to access email submissions and then send them out? If so, you should be able to configure your mail server to use the same password file as your new md5 password file.
steph> 1) How can I says sendmail to use the same passwd file ( with MD5) than dovecot ?
Ah... just saw this. And I don't know how to configure sendmail for this. I would suggest you look on the sendmail.org site for help.
steph> 2) Ideally, I would like to create virtual users for the same steph> mailbox Is that possible ?
steph> like 2 files Users and PAsswrds pointing out the mailbox : steph> maildir :/home/mailbox/user1 ex : user1@foo.com passwrd1 steph> /home/mailbox/generic_mails and user2 passwrd2 steph> home/mailbox/generic_mails
I do this myself using postfix and dovecot and it works well. I have my users defined in an sqlite3 DB, though for a small number of users I think a flat file is simpler.
The trick is to have the dovecot and postfix/sendmail using the same files for the virtual users and their passwords. There are a number of tutorials out there for doing this.
John
So just to be clear, each user has a login on your mail server in /etc/passwd? If so, I would strongly urge you to move to using only virtual users on your mail infrastructure.
Why? Just disallow login, and that is from the perspective that a mail user should be limited mail resources.
I argue exactly the opposite. Keep as much as possible linux users. As linux has been engineered for allowing multiple user accounts, and most other virtual user providers that are used here, have not.
"Marc" == Marc Marc@f1-outsourcing.eu writes:
So just to be clear, each user has a login on your mail server in /etc/passwd? If so, I would strongly urge you to move to using only virtual users on your mail infrastructure.
Marc> Why? Just disallow login, and that is from the perspective that Marc> a mail user should be limited mail resources.
If the user does NOT need to login to the dovecot/mail servers, then not having these users at all is more secure.
Marc> I argue exactly the opposite. Keep as much as possible linux Marc> users. As linux has been engineered for allowing multiple user Marc> accounts, and most other virtual user providers that are used Marc> here, have not.
I'm having a hard time to parse what you are saying here.
I'm saying that if the mail/dovecot server is only providing mail services, then putting all the users (across multiple domains even) into a virtual user database is more secure and more scalable.
General users don't need accounts on the mail server, and security in depth argues that keeping them off the server entirely is a good thing.
John
Marc> Why? Just disallow login, and that is from the perspective that Marc> a mail user should be limited mail resources.
If the user does NOT need to login to the dovecot/mail servers, then not having these users at all is more secure.
No, because there is a difference between a need to login and the presence of a uid. Lots of daemons run under accounts that cannot login.
Marc> I argue exactly the opposite. Keep as much as possible linux Marc> users. As linux has been engineered for allowing multiple user Marc> accounts, and most other virtual user providers that are used Marc> here, have not.
I'm having a hard time to parse what you are saying here.
I'm saying that if the mail/dovecot server is only providing mail services, then putting all the users (across multiple domains even) into a virtual user database is more secure
No it is not more secure, eg.
if a user does not exist on the os, how can processes be spawned as these uid's. Everything is running under the same uid.
if you do not use separate users, everything is written under the same uid.
most amateurs use a crappy mysql as backend for virtual users. The likelihood of that being compromised compared to the linux os is much and much higher.
Say you are more professional and setup an ldap server (with correct acls (which is not trivial at all)) If you would have dovecot use it as a backend for virtual users. Does dovecot relay that user auth information or does it need some static bind. The static bind is already an increased attack surface. Better is have the os use the ldap backend and have dovecot use the os.
I would even argue that having dovecot 'outsource' the user management to the linux os is more secure. Because dovecot developers are more experienced in programming the email application and have far less experience with authorization, authentication than the linux developers. There is much more scrutiny on the linux os than the dovecot user system.
and more scalable.
Not relevant, that is different discussion.
General users don't need accounts on the mail server, and security in depth argues that keeping them off the server entirely is a good thing.
You constantly apply incorrect logic. You think that "keeping them off the server entirely" equals virtual user. "keeping them off the server entirely" also includes /sbin/nologin. According to your incorrect logic’s, you support my statement because in my case users are kept off.
If your logic’s is incorrect, how can your conclusion be correct? Repeating this does not make it true, the alternative is far worse.
Linux always does a better job on permissions, users, authentication than whatever 3rd party software. And if you outsource this to linux you have even more possibilities by using selinux rules.
"Marc" == Marc Marc@f1-outsourcing.eu writes:
Marc> Why? Just disallow login, and that is from the perspective that Marc> a mail user should be limited mail resources.
If the user does NOT need to login to the dovecot/mail servers, then not having these users at all is more secure.
Marc> No, because there is a difference between a need to login and Marc> the presence of a uid. Lots of daemons run under accounts that Marc> cannot login.
You're missing my point. Yes, the daemons running the services are locked down. But the users using those services have no need to for logins or access to the system. They only need access to the application.
That's why virtual users are good. Also, UIDs used to be limited to under 65,000 seperate logins, but early on large FTP and ISP sites disovered that they wanted to have more user than that, so moving to virtual users was the solution.
Marc> I argue exactly the opposite. Keep as much as possible linux Marc> users. As linux has been engineered for allowing multiple user Marc> accounts, and most other virtual user providers that are used Marc> here, have not.
I'm having a hard time to parse what you are saying here.
I'm saying that if the mail/dovecot server is only providing mail services, then putting all the users (across multiple domains even) into a virtual user database is more secure
Marc> No it is not more secure, eg.
Marc> 1. if a user does not exist on the os, how can processes be Marc> spawned as these uid's. Everything is running under the same Marc> uid.
Yes, the daemons/applications running the service being provided runs under a single UID. Which is more secure becuase now you have just one UID to lock down tight, using apparmour, selinux or other OS level tools.
Marc> 2. if you do not use separate users, everything is written under Marc> the same uid.
So?
Marc> 3. most amateurs use a crappy mysql as backend for virtual Marc> users. The likelihood of that being compromised compared to the Marc> linux os is much and much higher.
How would it be compromised? What makes you think that the backend database is even exposed to the internet at all? In a smart setup, it's configured so that only local access works, or only access from a restricted set of IPs with restricted logins is allowed access.
Marc> 4. Say you are more professional and setup an ldap server (with Marc> correct acls (which is not trivial at all)) If you would have Marc> dovecot use it as a backend for virtual users. Does dovecot Marc> relay that user auth information or does it need some static Marc> bind. The static bind is already an increased attack Marc> surface. Better is have the os use the ldap backend and have Marc> dovecot use the os.
The static bind is fine, because you do not bind to AD as a root user, but only as a user with the minimum needed access to do the queries.
Marc> 5. I would even argue that having dovecot 'outsource' the user Marc> management to the linux os is more secure. Because dovecot Marc> developers are more experienced in programming the email Marc> application and have far less experience with authorization, Marc> authentication than the linux developers. There is much more Marc> scrutiny on the linux os than the dovecot user system.
You really don't know how authentication and access to IMAP mailboxes works, do you? And how postfix submission port works. Regular port 25 SMTP traffic doesn't have access controls, but it's also not where you accept email that gets sent to other domains, you only accept email for your destination domains.
Submission port 587, for accepting outgoing email to be sent outside your your domain, needs and requires authentication. It's part of the specs that mail clients need to implement properly.
and more scalable.
Marc> Not relevant, that is different discussion.
General users don't need accounts on the mail server, and security in depth argues that keeping them off the server entirely is a good thing.
Marc> You constantly apply incorrect logic. You think that "keeping Marc> them off the server entirely" equals virtual user. "keeping them Marc> off the server entirely" also includes /sbin/nologin. According Marc> to your incorrect logic’s, you support my statement because in Marc> my case users are kept off.
Again, you're not being clear here.
Marc> If your logic’s is incorrect, how can your conclusion be Marc> correct? Repeating this does not make it true, the alternative Marc> is far worse.
You're telling me my logic is broken, but I keep giving you reasons why I stand by my assertion that having virtual users is more secure, because it lowers the attack surface.
Marc> Linux always does a better job on permissions, users, Marc> authentication than whatever 3rd party software. And if you Marc> outsource this to linux you have even more possibilities by Marc> using selinux rules.
You need to think of security happening in layers. Keeping users virtual means they have an outer layer to penetrate to get access, but they only get access to the UID of the mail/imap user. Which should also be locked odwn without any system privs beyond that bare minumum needed.
So then the attacker needs to break the application access to get into the OS to successfully attack.
Security by layers is a good thing, and virutal users means you can support lots and lots of users more easily in alot of ways.
John
Marc> Why? Just disallow login, and that is from the perspective that Marc> a mail user should be limited mail resources.
If the user does NOT need to login to the dovecot/mail servers, then not having these users at all is more secure.
Marc> No, because there is a difference between a need to login and Marc> the presence of a uid. Lots of daemons run under accounts that Marc> cannot login.
You're missing my point. Yes, the daemons running the services are locked down. But the users using those services have no need to for logins or access to the system.
I am not missing the point. I am stating that it is very common to have users that are limited, nothing more nothing less.
They only need access to the application.
Incorrect, they also need to have access the file system.
and if I process is spawned under the uid of the logged in user, then only those files having those uid's are available. So if there is some sort of problem in the code of the spawned process it is only limited to the current uid environment.
That's why virtual users are good.
That is why the use of tooth paste is good!
Also, UIDs used to be limited to under 65,000 seperate logins, but early on large FTP and ISP sites disovered that they wanted to have more user than that, so moving to virtual users was the solution.
Again not relevant for the discussion
Marc> 1. if a user does not exist on the os, how can processes be Marc> spawned as these uid's. Everything is running under the same Marc> uid.
Yes, the daemons/applications running the service being provided runs under a single UID. Which is more secure becuase now you have just one UID to lock down tight, using apparmour, selinux or other OS level tools.
Indeed so if you argue the running under separate uid's is more secure. How can you also argue exactly the opposite that running all virtual users as a single uid is more secure. These statements are contradictory.
Marc> 2. if you do not use separate users, everything is written under Marc> the same uid.
So?
So if something is wrong with your application. Say a user can escape, all other users data are available. if the process runs under the uid, even if the user can escape it somehow, the linux os will limit this user to the files it is owning.
Marc> 3. most amateurs use a crappy mysql as backend for virtual Marc> users. The likelihood of that being compromised compared to the Marc> linux os is much and much higher.
How would it be compromised? What makes you think that the backend database is even exposed to the internet at all? In a smart setup, it's configured so that only local access works, or only access from a restricted set of IPs with restricted logins is allowed access.
Is not relevant.
Marc> 4. Say you are more professional and setup an ldap server (with Marc> correct acls (which is not trivial at all)) If you would have Marc> dovecot use it as a backend for virtual users. Does dovecot Marc> relay that user auth information or does it need some static Marc> bind. The static bind is already an increased attack Marc> surface. Better is have the os use the ldap backend and have Marc> dovecot use the os.
The static bind is fine, because you do not bind to AD as a root user, but only as a user with the minimum needed access to do the queries.
Incorrect, a static bind requires more privileges (searching acl etc) then simple auth request, ofter also on a lower ssf. But you are missing the most important point here.
Marc> 5. I would even argue that having dovecot 'outsource' the user Marc> management to the linux os is more secure. Because dovecot Marc> developers are more experienced in programming the email Marc> application and have far less experience with authorization, Marc> authentication than the linux developers. There is much more Marc> scrutiny on the linux os than the dovecot user system.
You really don't know how authentication and access to IMAP mailboxes works, do you?
I think my knowledge exceeds yours, although I do not see myself as an expert.
And how postfix submission port works.
How is this relevant
Regular port 25 SMTP traffic doesn't have access controls, but it's also not where you accept email that gets sent to other domains, you only accept email for your destination domains.
How is this relevant
Submission port 587, for accepting outgoing email to be sent outside your your domain, needs and requires authentication. It's part of the specs that mail clients need to implement properly.
How is this relevant
Marc> You constantly apply incorrect logic. You think that "keeping Marc> them off the server entirely" equals virtual user. "keeping them Marc> off the server entirely" also includes /sbin/nologin. According Marc> to your incorrect logic’s, you support my statement because in Marc> my case users are kept off.
Again, you're not being clear here.
The idea is that if you tell me I am wrong, you proove me wrong. But you are stating something that substantiates my case. Simply said you are telling me I am right, but you do not notice it.
Marc> If your logic’s is incorrect, how can your conclusion be Marc> correct? Repeating this does not make it true, the alternative Marc> is far worse.
You're telling me my logic is broken, but I keep giving you reasons why I stand by my assertion that having virtual users is more secure,
Yes to me an indication of incorrect logics, is referring to not relevant things. Often when people can not prove their case they prove something unrelated. Something like the earth is round, that is why I am right. Sort of how you here refer to MTA's.
because it lowers the attack surface.
I wonder if you know what the definition of this is. I will tell you how I see this. The attack service is primaraly dovecot. A dovecot hack is limited if dovecot process run under uids, and processes running under specific uid's are quite well managed by the operating system.
Marc> Linux always does a better job on permissions, users, Marc> authentication than whatever 3rd party software. And if you Marc> outsource this to linux you have even more possibilities by Marc> using selinux rules.
You need to think of security happening in layers. Keeping users virtual means they have an outer layer to penetrate to get access, but they only get access to the UID of the mail/imap user.
First of all 'real' virtual users do not have a uid. What is even virtual about a user that gets an os uid? The fact that the dovecot developers are using uid's is already an argument that a virtual users environment is to risky. So they have build a 'semi' virtual user environment that utilizes the os file permissions. userdb/passwdb files even have a /etc/passwd file structure. They are just in a different location where the os does not know about it. Personally I think it is quite 'crappy' to start using uid's, that the os does not know anything about.
Which should also be locked odwn without any system privs beyond that bare minumum needed.
You can not use this argument with virtual users, this an argument for me having users in the os.
Security by layers is a good thing,
Why are you writing this? Is anyone disagreeing on this point?
and virutal users means you can support lots and lots of users more easily in alot of ways.
I just wrote, how is it relevant? The idea is that you read the arguments and learn. I think the arguments are made and people can decide what is more secure. So in future do not advice someone upon what is most secure, but link to this thread instead.
On 01-25-2022 11:35 am, Marc wrote: 2. if you do not use separate users, everything is written under the same uid.
IMO: So what? What is the difference between a linux user vs a virtual user permission wise? They are both equally unprivileged users. If dovecot can get to them, virtual or linux user, then a hacked dovecot can still get to them. You aren't saving anything.
Dovecot can also be configured to use virtual users from a database, and each virtual user be assigned a different UID for reading/writing maildir files.
- most amateurs use a crappy mysql as backend for virtual users. The likelihood of that being compromised compared to the linux os is much and much higher.
Even if SQL is compromised you aren't storing emails in SQL, just email addresses and passwords. That is why password hashing exist. If SQL is configured to only localhost connections then it is not getting compromised, unless your entire server is compromised, in which case SQL access is moot because they can just get the maildir files.
I also doubt that gmail, outlook and yahoo have separate linux users for their millions of email accounts. I have not heard of a massive email breach where hackers gained access to all gmail messages.
Saying all that, it is your server, do with it how you please. People here on the list are just telling you what is acceptable safe practice industry wide. Another drawback to using linux users is you will never be able to cluster/scale up. But if your preference is to use linux users go for it.
hi Marc,
That's correct, I have an account for each users.. Which is great perfect for now.. but if the system is growing up.. if a user change, this is a lot work.. I would prefer to get and standard mailbox.. and then let suppose that a user is changing, you just login and passwrd.... eventually, delete or keep the previous emails... and that's it !
I using sendmail, but this is not clear how to share the same passwrd file, than Dovecot.. to be honest I should be able to get a file to manage on Sendmail, login and passwrd attached to the mailbox... Nb1
On 1/25/22 09:39, Marc wrote:
So just to be clear, each user has a login on your mail server in /etc/passwd? If so, I would strongly urge you to move to using only virtual users on your mail infrastructure.
Why? Just disallow login, and that is from the perspective that a mail user should be limited mail resources.
I argue exactly the opposite. Keep as much as possible linux users. As linux has been engineered for allowing multiple user accounts, and most other virtual user providers that are used here, have not.
On 28/01/2022 13.34, Stephane Magnier wrote:
I using sendmail, but this is not clear how to share the same passwrd file, than Dovecot.. to be honest I should be able to get a file to manage on Sendmail, login and passwrd attached to the mailbox... Nb1
Not sure if this helps, but a server I run shares login info between exim and dovecot in plain text files.
In dovecot these are referenced with: passdb { driver = passwd-file args = scheme=MD5-CRYPT username_format=%n /etc/exim4/domains/%d/passwd }
userdb { driver = passwd-file args = username_format=%n /etc/exim4/domains/%d/passwd }
In exim the lookup function is a bit more complicated eg. user = ${extract{2}{:}{${lookup{$local_part}lsearch{/etc/exim4/domains/$domain/passwd}}}}
I imagine sendmail would have a similar sort of co-existence with dovecot.
P.
On January 24, 2022 1:33:46 PM AKST, John Stoffel john@stoffel.org wrote:
steph> 1) How can I says sendmail to use the same passwd file ( with MD5) than dovecot ?
Ah... just saw this. And I don't know how to configure sendmail for this. I would suggest you look on the sendmail.org site for help.
Too many professional bulk mailers on all those lists. I for one don't like the documentation runaround. There's a lot of stuff that's getting more complicated than it needs to be. I need SPF+DKIM+DMARC for basic spam control.
steph> 2) Ideally, I would like to create virtual users for the same steph> mailbox Is that possible ?
I have a setup like that myself. Nothing to do with Dovecot. It's entirely up to postfix which mailbox to deliver incoming messages to, and the user's client to address outgoing mail with a proper ID.
steph> like 2 files Users and PAsswrds pointing out the mailbox : steph> maildir :/home/mailbox/user1 ex : user1@foo.com passwrd1 steph> /home/mailbox/generic_mails and user2 passwrd2 steph> home/mailbox/generic_mails
I do this myself using postfix and dovecot and it works well. I have my users defined in an sqlite3 DB, though for a small number of users I think a flat file is simpler.
The performance of flat files really bogs down my system and causes me to lose mails if too many arrive or if the file grows too large.
The trick is to have the dovecot and postfix/sendmail using the same files for the virtual users and their passwords. There are a number of tutorials out there for doing this.
John
Without a doubt there are many useful tricks and tutorials out there. I have found several very helpful.
Maybe a future programming project idea: I want a system that will store all mail messages and user account info in, say, a postgresql transactional database, a little more manageable and reliable than ad hoc databasing with those flat files all over the place cluttering up the system.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Jan 25, 2022 at 03:50:12AM -0900, justina colmena ~biz wrote:
Maybe a future programming project idea: I want a system that will store all mail messages and user account info in, say, a postgresql transactional database, a little more manageable and reliable than ad hoc databasing with those flat files all over the place cluttering up the system.
I am in progress moving towards something like that. As of right now, perl, dovecot for IMAP, neomutt and OpenSMTPD.
Right now, .neomuttrc files *only* exist during the usage of neomutt. They have random names, cannot be written to and are immediately erased after neomutt starts (not quits). That is a very small window of threat.
I would very much like to put all of the messages into PostgreSQL also instead of file folders under the user vmail.
This is just a side project. As I have been advised, there is no need to even write a configuration file at all, but there are some issues with dbh that I need to solve with a different database module.
If someone can read files that never exist, well... At some point you have to at least consider trusting something. That or just turn it all off and get another career.
-- Chris Bennett
On Jan 30, 2022, at 10:55, Chris Bennett chris-dvcot@freedomforlife.rocks wrote:
On Tue, Jan 25, 2022 at 03:50:12AM -0900, justina colmena ~biz wrote:
Maybe a future programming project idea: I want a system that will store all mail messages and user account info in, say, a postgresql transactional database, a little more manageable and reliable than ad hoc databasing with those flat files all over the place cluttering up the system.
I am in progress moving towards something like that. As of right now, perl, dovecot for IMAP, neomutt and OpenSMTPD.
Right now, .neomuttrc files *only* exist during the usage of neomutt. They have random names, cannot be written to and are immediately erased after neomutt starts (not quits). That is a very small window of threat.
I would very much like to put all of the messages into PostgreSQL also instead of file folders under the user vmail.
This is just a side project. As I have been advised, there is no need to even write a configuration file at all, but there are some issues with dbh that I need to solve with a different database module.
If someone can read files that never exist, well... At some point you have to at least consider trusting something. That or just turn it all off and get another career.
-- Chris Bennett
At some point you gotta ask yourself why you’re trusting your database more than your OS.
And why you don’t trust the OS to handle files in a trusted way, but do for memory.
Sean
On 2022-01-31 02:30, Sean Kamath wrote:
On Jan 30, 2022, at 10:55, Chris Bennett chris-dvcot@freedomforlife.rocks wrote:
On Tue, Jan 25, 2022 at 03:50:12AM -0900, justina colmena ~biz wrote:
Maybe a future programming project idea: I want a system that will store all mail messages and user account info in, say, a postgresql transactional database, a little more manageable and reliable than ad hoc databasing with those flat files all over the place cluttering up the system.
I am in progress moving towards something like that. As of right now, perl, dovecot for IMAP, neomutt and OpenSMTPD.
Right now, .neomuttrc files *only* exist during the usage of neomutt. They have random names, cannot be written to and are immediately erased after neomutt starts (not quits). That is a very small window of threat.
I would very much like to put all of the messages into PostgreSQL also instead of file folders under the user vmail.
This is just a side project. As I have been advised, there is no need to even write a configuration file at all, but there are some issues with dbh that I need to solve with a different database module.
If someone can read files that never exist, well... At some point you have to at least consider trusting something. That or just turn it all off and get another career.
-- Chris Bennett
At some point you gotta ask yourself why you’re trusting your database more than your OS.
And why you don’t trust the OS to handle files in a trusted way, but do for memory.
dbmail exists, runs fine on sqlite3 :=)
but that joke, why try ?
how huge would that sqlite3 file be ?, i say no to one sqlite3 file, but yes if each mail user have there own sqlite3 tree with seperate sqlite3 file pr folder and user
if more huge setup is meeded, then postgresql with replication, but this is not needed with dovecot, its more solid and with performance with imap protocol, and load balanced
i would not wish for disaster with sqlite3, but it could be done, also sqlite cluster exists
dream on, its monday where noting works :=)
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive? Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive?
What do you think you are saving? Security? If someone can read files on your server, they can equally read a maildir or a .db file. K.I.S.S.
You'll get better indexing and fast full text search by storing your emails in a database rather than a flat file, hopefully after decoding any attachments. Especially for spam scoring, analysis, and classification. Much better performance deleting or moving specific messages, too.
On January 30, 2022 5:46:53 PM AKST, dovecot@ptld.com wrote:
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive? Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive?
What do you think you are saving? Security? If someone can read files on your server, they can equally read a maildir or a .db file. K.I.S.S.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Sun, Jan 30, 2022 at 06:17:49PM -0900, justina colmena ~biz wrote:
On January 30, 2022 5:46:53 PM AKST, dovecot@ptld.com wrote:
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive?
Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive? [...]
You'll get better indexing and fast full text search by storing your emails in a database rather than a flat file, hopefully after decoding any attachments. Especially for spam scoring, analysis, and classification. Much better performance deleting or moving specific messages, too.
Do you have evidence to back up these claims, specifically re: mail servers?
Like-for-like benchmarks, for instance?
Thanks,
Sam
-- A: When it messes up the order in which people normally read text. Q: When is top-posting a bad thing?
() ASCII ribbon campaign. Please avoid HTML emails & proprietary /\ file formats. (Why? See e.g. https://v.gd/jrmGbS ). Thank you.
Just ideas.
Removing or deleting a single message from near the beginning of a large flat file takes an inordinate amount of time because the remainder of the flat file has to be rewritten all the way from the point of the deleted message to the end of the file and then truncated.
On January 30, 2022 6:30:44 PM AKST, Sam Kuper sampablokuper@posteo.net wrote:
On Sun, Jan 30, 2022 at 06:17:49PM -0900, justina colmena ~biz wrote:
On January 30, 2022 5:46:53 PM AKST, dovecot@ptld.com wrote:
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive?
Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive? [...]
You'll get better indexing and fast full text search by storing your emails in a database rather than a flat file, hopefully after decoding any attachments. Especially for spam scoring, analysis, and classification. Much better performance deleting or moving specific messages, too.
Do you have evidence to back up these claims, specifically re: mail servers?
Like-for-like benchmarks, for instance?
Thanks,
Sam
-- A: When it messes up the order in which people normally read text. Q: When is top-posting a bad thing?
() ASCII ribbon campaign. Please avoid HTML emails & proprietary /\ file formats. (Why? See e.g. https://v.gd/jrmGbS ). Thank you.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Sun, Jan 30, 2022 at 07:49:56PM -0900, justina colmena ~biz wrote:
On January 30, 2022 6:30:44 PM AKST, Sam Kuper wrote:
On Sun, Jan 30, 2022 at 06:17:49PM -0900, justina colmena ~biz wrote:
On January 30, 2022 5:46:53 PM AKST, dovecot@ptld.com wrote:
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive?
Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive? [...]
You'll get better indexing and fast full text search by storing your emails in a database rather than a flat file, hopefully after decoding any attachments. Especially for spam scoring, analysis, and classification. Much better performance deleting or moving specific messages, too.
Do you have evidence to back up these claims, specifically re: mail servers?
Like-for-like benchmarks, for instance?
Just ideas.
OK, no then.
Removing or deleting a single message from near the beginning of a large flat file takes an inordinate amount of time because the remainder of the flat file has to be rewritten all the way from the point of the deleted message to the end of the file and then truncated.
You might want to look up what Maildir is before making bold but apparently unfounded claims about it.
Maildir is not a "large flat file". It is a set of conventions that amount to a database specification, in the traditional sense of the word "database": a system for storing data. (Not a relational database.)
DJB developed Maildir to gain performance and reliability improvements over mbox files. Unlike Maildirs, mbox files *are* "large flat files".
Best wishes,
Sam
-- A: When it messes up the order in which people normally read text. Q: When is top-posting a bad thing?
() ASCII ribbon campaign. Please avoid HTML emails & proprietary /\ file formats. (Why? See e.g. https://v.gd/jrmGbS ). Thank you.
On Mon, Jan 31, 2022 at 06:23:28AM +0000, Sam Kuper wrote:
On Sun, Jan 30, 2022 at 07:49:56PM -0900, justina colmena ~biz wrote:
On January 30, 2022 6:30:44 PM AKST, Sam Kuper wrote:
On Sun, Jan 30, 2022 at 06:17:49PM -0900, justina colmena ~biz wrote:
On January 30, 2022 5:46:53 PM AKST, dovecot@ptld.com wrote:
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive?
Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive? [...]
You'll get better indexing and fast full text search by storing your emails in a database rather than a flat file, hopefully after decoding any attachments. Especially for spam scoring, analysis, and classification. Much better performance deleting or moving specific messages, too.
Do you have evidence to back up these claims, specifically re: mail servers?
Like-for-like benchmarks, for instance?
Just ideas.
OK, no then.
Removing or deleting a single message from near the beginning of a large flat file takes an inordinate amount of time because the remainder of the flat file has to be rewritten all the way from the point of the deleted message to the end of the file and then truncated.
You might want to look up what Maildir is before making bold but apparently unfounded claims about it.
Maildir is not a "large flat file". It is a set of conventions that amount to a database specification, in the traditional sense of the word "database": a system for storing data. (Not a relational database.)
Many people haven't ever had to deal with the old "database" style of files instead of tables and columns. Maildir does show it's age with the little complexities it has.
DJB developed Maildir to gain performance and reliability improvements over mbox files. Unlike Maildirs, mbox files *are* "large flat files".
Corrupt your mbox file and bad things happen!
I also like being able to throw in some older backed up email when I find I need a few more to fill out that important thread from 3 years ago with Maildir.
Maildir does not have the relational database problem of needing to keep up with updates to the database software.
And nothing works very well when you suddenly discover that the company you are renting servers from decides to close up and turn everything off. While you are in another country with internet cafes only and don't even have a laptop with you! Happened to me once. 8-{
-- Chris Bennett
On 2022-01-31 07:23, Sam Kuper wrote:
DJB developed Maildir to gain performance and reliability improvements over mbox files. Unlike Maildirs, mbox files *are* "large flat files".
mbox is multiple emails in single file, maildir is single email in single file
On Mon, Jan 31, 2022 at 11:26:26AM +0100, Benny Pedersen wrote:
On 2022-01-31 07:23, Sam Kuper wrote:
DJB developed Maildir to gain performance and reliability improvements over mbox files. Unlike Maildirs, mbox files *are* "large flat files".
mbox is multiple emails in single file, maildir is single email in single file
Exactly my point!
-- A: When it messes up the order in which people normally read text. Q: When is top-posting a bad thing?
() ASCII ribbon campaign. Please avoid HTML emails & proprietary /\ file formats. (Why? See e.g. https://v.gd/jrmGbS ). Thank you.
On Mon, Jan 31, 2022 at 12:04:57PM +0100, Wojciech Puchar wrote:
mbox is multiple emails in single file, maildir is single email in single file
Exactly my point!
which is good (mbox) in mail archiving. And not much else.
Exactly.
-- A: When it messes up the order in which people normally read text. Q: When is top-posting a bad thing?
() ASCII ribbon campaign. Please avoid HTML emails & proprietary /\ file formats. (Why? See e.g. https://v.gd/jrmGbS ). Thank you.
Just ideas.
Maybe an idea to participate on a Microsoft forum? They like to use db's for email, and they are removing everything what is nice in order to push people into their cloud. So lots to change for the better there.
It's so crappy that I recently wrote Bill Gates that he should not whine so much about the environment, because if he used only half of his profits to optimize code/designs in ms products, this would result in a significant reduction in energy use. Think of what global effect that has.
FYI T-mobile (and the commercial version of dovecot?) is working on storing emails in object storage, that is the future.
On 31/01/2022 10:36 Marc marc@f1-outsourcing.eu wrote:
Just ideas.
Maybe an idea to participate on a Microsoft forum? They like to use db's for email, and they are removing everything what is nice in order to push people into their cloud. So lots to change for the better there.
It's so crappy that I recently wrote Bill Gates that he should not whine so much about the environment, because if he used only half of his profits to optimize code/designs in ms products, this would result in a significant reduction in energy use. Think of what global effect that has.
FYI T-mobile (and the commercial version of dovecot?) is working on storing emails in object storage, that is the future.
Commercial Dovecot has had the ability to store mails & indexes in Object Storage for years now, we are not "working on it" anymore.
Aki
I see. People make money outsourcing, consulting, and hooking up companies with the best solutions for email, office collaboration, CRM, etc., etc., which is great, but I didn't quite realize that look like a paid offering on the table and this isn't the right list to discuss potential free market competition...
On January 31, 2022 12:45:48 AM AKST, Aki Tuomi aki.tuomi@open-xchange.com wrote:
On 31/01/2022 10:36 Marc marc@f1-outsourcing.eu wrote:
Just ideas.
Maybe an idea to participate on a Microsoft forum? They like to use db's for email, and they are removing everything what is nice in order to push people into their cloud. So lots to change for the better there.
It's so crappy that I recently wrote Bill Gates that he should not whine so much about the environment, because if he used only half of his profits to optimize code/designs in ms products, this would result in a significant reduction in energy use. Think of what global effect that has.
FYI T-mobile (and the commercial version of dovecot?) is working on storing emails in object storage, that is the future.
Commercial Dovecot has had the ability to store mails & indexes in Object Storage for years now, we are not "working on it" anymore.
Aki
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
I see. People make money outsourcing, consulting, and hooking up companies with the best solutions for email, office collaboration, CRM, etc., etc., which is great, but I didn't quite realize that look like a paid offering on the table and this isn't the right list to discuss potential free market competition...
What are you on about?
On 31/01/2022 12:29 Benny Pedersen me@junc.eu wrote:
On 2022-01-31 10:45, Aki Tuomi wrote:
Commercial Dovecot has had the ability to store mails & indexes in Object Storage for years now, we are not "working on it" anymore.
so no one is using dovecot-pro ? (dead sources)
Ha ha.
What I mean is that it is already being used in production, not something we are trying to get working. Of course we are still developing it.
Aki
On 2022-01-31 05:49, justina colmena ~biz wrote:
Just ideas.
Removing or deleting a single message from near the beginning of a large flat file takes an inordinate amount of time because the remainder of the flat file has to be rewritten all the way from the point of the deleted message to the end of the file and then truncated.
maybe why postfix only support maildir++
all other formats is properitary formats
Removing or deleting a single message from near the beginning of a large flat file takes an inordinate amount of time because the remainder of the flat file has to be rewritten all the way from the point of the deleted message to the end of the file and then truncated.
maybe why postfix only support maildir++
all other formats is properitary formats
This 'performance statement' is not even correct under some conditions. I have tested mbox against mdbox, and mbox performed better in those tests.
On 2022-01-31 11:29, Marc wrote:
This 'performance statement' is not even correct under some conditions. I have tested mbox against mdbox, and mbox performed better in those tests.
so you are now ready to make a postfix patch ?
its called progress on make ...
On Sun, Jan 30, 2022 at 09:46:53PM -0500, dovecot@ptld.com wrote:
Storing mail in a db... at the end of the day isn't it still just a file (.db file) on the drive? Aren't you just adding bloat and complexity vs just storing the mail directly (maildir format) to a file on the drive?
What do you think you are saving? Security? If someone can read files on your server, they can equally read a maildir or a .db file. K.I.S.S.
I gain modularity for a system. The database is the foundation. I am working with:
- Dovecot
- Neomutt
- OpenSMTPD
Now, if I decide to drop or addon some new program, I can just adjust and/or add some new tables. Write a new stored procedure. Drop in a new Perl module or subroutine.
- Dovecot
- Neomutt
- OpenSMTPD
- Xyz
- Abc
- SuperDuperMail-ThingyPlus
So what I am working for is a system that is united.
Add a new user and email, CLI program, bang. All done. Change a password with a web interface. Click. All done.
I'm in no rush. This is a fun side project. I have already done this type of work successfully for other kinds of projects, so it's different, but not really outside of my past experience.
Secure today is wide open tomorrow. File, memory, etc. all get broken eventually. I'm much more worried about my own mistakes than that of others. :-*
-- Chris Bennett
participants (14)
-
Aki Tuomi
-
Benny Pedersen
-
Chris Bennett
-
dovecot＠ptld.com
-
John Stoffel
-
justina colmena ~biz
-
Marc
-
Plutocrat
-
Sam Kuper
-
Sean Kamath
-
steph.mag220＠netcourrier.com
-
Stephane Magnier
-
Wojciech Puchar
-
Zakaria