email location - files or sql
I have been running a Dovecot based mail server for over 10 years and it is time (really past) to upgrade.
I really don't want to roll my own anymore. Too much time and too many skills needed. Time to retire. But still want it 'in-house'.
What I am seeing is that many of the packages seem to roll the messages into some SQL database. My Dovecot setup uses the /home/vmail/doman/../{cur,new,etc} tree structure.
Is there any pros vs cons on this that I can read?
Is there some nice packed mailserver I can drop on a Centos-arm server that has Dovecot under the covers and I can easily rsync my current mail store to it?
Mail-in-a-box looks interesting. But I have not figured out its internals and if I have to worry about anything compiled for arm that is not part of the distro.
Thanks in advance for help.
- Robert Moskowitz:
What I am seeing is that many of the packages seem to roll the messages into some SQL database.
Do they?
My Dovecot setup uses the /home/vmail/doman/../{cur,new,etc} tree structure.
That's the classic Maildir format. Widely supported, works fine if file system nodes are not a scarce commodity. Dovecot supports other formats (see https://doc.dovecot.org/admin_manual/mailbox_formats/) as well, but I still use Maildir because it is required by Notmuch, which is my software of choice for handling all my mailing list subscriptions.
Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB.
Is there some nice packed mailserver I can drop on a Centos-arm server that has Dovecot under the covers and I can easily rsync my current mail store to it?
As far as migrating content is concerned, you're better off using some form of IMAP synchronisation. This method is not dependent on the underlying mail store format, and it preserves the IMAP flags for your existing messages.
-Ralph
On 2/23/22 19:12, Ralph Seichter wrote:
- Robert Moskowitz:
What I am seeing is that many of the packages seem to roll the messages into some SQL database. Do they?
Packages like iRedMail list email stored in the database of your choice. Of course this is just their web blurb, and I have yet to find one of these that suit my needs. More and more, I am looking at rolling my own again. Challenge has always been integrating the anti-virus. Clamav has been quite the challenge.
My Dovecot setup uses the /home/vmail/doman/../{cur,new,etc} tree structure. That's the classic Maildir format. Widely supported, works fine if file system nodes are not a scarce commodity. Dovecot supports other formats (see https://doc.dovecot.org/admin_manual/mailbox_formats/) as well, but I still use Maildir because it is required by Notmuch, which is my software of choice for handling all my mailing list subscriptions.
I liked Maildir at the time and still do. Only a 1,000+ emails a day. My wife keeps a lot (10K messages) on the server, I keep all of my various boxes on the server small. So do my other users.
Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB.
I was there for the beginning of RDB. Almost had NOMAD shoved down my throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release site of DB2, where we worked out how to do a UNION which was not supported in the original design. I have seen email systems that stuff the messages into RDB and really wonder if the hammer really fits.
Is there some nice packed mailserver I can drop on a Centos-arm server that has Dovecot under the covers and I can easily rsync my current mail store to it? As far as migrating content is concerned, you're better off using some form of IMAP synchronisation. This method is not dependent on the underlying mail store format, and it preserves the IMAP flags for your existing messages.
I did see that mentioned once, and have down to research it. Thanks for the pointer.
iRedMail does not store emails in a database. It stores account information in a database. Emails are stored on the filesystem.
Get Outlook for iOShttps://aka.ms/o0ukef
From: dovecot dovecot-bounces@dovecot.org on behalf of Robert Moskowitz rgm@htt-consult.com Sent: Thursday, February 24, 2022 9:48:11 AM To: dovecot@dovecot.org dovecot@dovecot.org Subject: Re: email location - files or sql
On 2/23/22 19:12, Ralph Seichter wrote:
- Robert Moskowitz:
What I am seeing is that many of the packages seem to roll the messages into some SQL database. Do they?
Packages like iRedMail list email stored in the database of your choice. Of course this is just their web blurb, and I have yet to find one of these that suit my needs. More and more, I am looking at rolling my own again. Challenge has always been integrating the anti-virus. Clamav has been quite the challenge.
My Dovecot setup uses the /home/vmail/doman/../{cur,new,etc} tree structure. That's the classic Maildir format. Widely supported, works fine if file system nodes are not a scarce commodity. Dovecot supports other formats (see https://doc.dovecot.org/admin_manual/mailbox_formats/) as well, but I still use Maildir because it is required by Notmuch, which is my software of choice for handling all my mailing list subscriptions.
I liked Maildir at the time and still do. Only a 1,000+ emails a day. My wife keeps a lot (10K messages) on the server, I keep all of my various boxes on the server small. So do my other users.
Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB.
I was there for the beginning of RDB. Almost had NOMAD shoved down my throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release site of DB2, where we worked out how to do a UNION which was not supported in the original design. I have seen email systems that stuff the messages into RDB and really wonder if the hammer really fits.
Is there some nice packed mailserver I can drop on a Centos-arm server that has Dovecot under the covers and I can easily rsync my current mail store to it? As far as migrating content is concerned, you're better off using some form of IMAP synchronisation. This method is not dependent on the underlying mail store format, and it preserves the IMAP flags for your existing messages.
I did see that mentioned once, and have down to research it. Thanks for the pointer.
Wondered about that. Other reasons not to go with iRedMail.
On 2/23/22 21:01, Lucas Rolff wrote:
iRedMail does not store emails in a database. It stores account information in a database. Emails are stored on the filesystem.
Get Outlook for iOS https://aka.ms/o0ukef
*From:* dovecot dovecot-bounces@dovecot.org on behalf of Robert Moskowitz rgm@htt-consult.com *Sent:* Thursday, February 24, 2022 9:48:11 AM *To:* dovecot@dovecot.org dovecot@dovecot.org *Subject:* Re: email location - files or sql
On 2/23/22 19:12, Ralph Seichter wrote:
- Robert Moskowitz:
What I am seeing is that many of the packages seem to roll the messages into some SQL database. Do they?
Packages like iRedMail list email stored in the database of your choice. Of course this is just their web blurb, and I have yet to find one of these that suit my needs. More and more, I am looking at rolling my own again. Challenge has always been integrating the anti-virus. Clamav has been quite the challenge.
My Dovecot setup uses the /home/vmail/doman/../{cur,new,etc} tree structure. That's the classic Maildir format. Widely supported, works fine if file system nodes are not a scarce commodity. Dovecot supports other formats (see https://doc.dovecot.org/admin_manual/mailbox_formats/) as well, but I still use Maildir because it is required by Notmuch, which is my software of choice for handling all my mailing list subscriptions.
I liked Maildir at the time and still do. Only a 1,000+ emails a day. My wife keeps a lot (10K messages) on the server, I keep all of my various boxes on the server small. So do my other users.
Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB.
I was there for the beginning of RDB. Almost had NOMAD shoved down my throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release site of DB2, where we worked out how to do a UNION which was not supported in the original design. I have seen email systems that stuff the messages into RDB and really wonder if the hammer really fits.
Is there some nice packed mailserver I can drop on a Centos-arm server that has Dovecot under the covers and I can easily rsync my current mail store to it? As far as migrating content is concerned, you're better off using some form of IMAP synchronisation. This method is not dependent on the underlying mail store format, and it preserves the IMAP flags for your existing messages.
I did see that mentioned once, and have down to research it. Thanks for the pointer.
On Wed, Feb 23, 2022 at 08:48:11PM -0500, Robert Moskowitz wrote:
I liked Maildir at the time and still do. Only a 1,000+ emails a day. My wife keeps a lot (10K messages) on the server, I keep all of my various boxes on the server small. So do my other users.
Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB.
I was there for the beginning of RDB. Almost had NOMAD shoved down my throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release site of DB2, where we worked out how to do a UNION which was not supported in the original design. I have seen email systems that stuff the messages into RDB and really wonder if the hammer really fits.
What is the opinion on repairs to problems that occur? Easier to fix in a DB? That does give a simple set of fixed points and dates to make repairs simple.
I backup into a tar.gz and that preserves a timepoint, but how in the heck could a fix a problem that occurs? That seems like a nightmare problem.
Does anyone have any method to do that? I'd love to know that if it exists. I love Maildir, but the backups are very large. I don't have a huge set of emails, but downloading a copy to home instead of my other server is a big task. Since I use a single ISP and once had a company shut down all servers, which makes me nervous.
-- Hmm, Chris Bennett
On 2/25/22 12:51, Chris Bennett wrote:
On Wed, Feb 23, 2022 at 08:48:11PM -0500, Robert Moskowitz wrote:
I liked Maildir at the time and still do. Only a 1,000+ emails a day. My wife keeps a lot (10K messages) on the server, I keep all of my various boxes on the server small. So do my other users.
Personally, I would not use a relational database as a mail store unless specifically required by the mail system of your choice. Email is not organised in a way that benefits from a RDB. I was there for the beginning of RDB. Almost had NOMAD shoved down my throat (UNIVAC) and did work with RIM (BCS) then R:Base. Was gamma release site of DB2, where we worked out how to do a UNION which was not supported in the original design. I have seen email systems that stuff the messages into RDB and really wonder if the hammer really fits.
What is the opinion on repairs to problems that occur? Easier to fix in a DB? That does give a simple set of fixed points and dates to make repairs simple.
I backup into a tar.gz and that preserves a timepoint, but how in the heck could a fix a problem that occurs? That seems like a nightmare problem.
Does anyone have any method to do that? I'd love to know that if it exists. I love Maildir, but the backups are very large. I don't have a huge set of emails, but downloading a copy to home instead of my other server is a big task. Since I use a single ISP and once had a company shut down all servers, which makes me nervous.
I have used a simple rsync with --delete to maintain a backup of the maildir directories. rsync is run via cron. You stop postfix, wait a bit, rsync, then restart postfix.
But recently, partly to do migration, one person pointed out that he uses Dovecot on the backup server to just use imap to replicate the mail store. I have to learn how to do this...
On 2/22/2022 9:12 AM, Robert Moskowitz wrote:
What I am seeing is that many of the packages seem to roll the messages into some SQL database. My Dovecot setup uses the /home/vmail/doman/../{cur,new,etc} tree structure.
I use dovecot and postfix in conjunction with a postfixadmin database in mysql.
The emails are stored on the filesystem using Maildir, not in the database. Configuration and authentication information for domains and users are in the database. Quotas are in the database. I have several gigabytes of email between the many accounts and would not want that stored in a huge database table file. Although I am sure something exists that stores messages in a database with dovecot, I have yet to see it.
While a database would probably be more efficient in terms of disk space utilization, it would be much less efficient to make incremental rsync-based backups like I do currently. I use rsync to copy only changes rather than the entire archive, and btrfs snapshots for a history.
Thanks, Shawn
participants (5)
-
Chris Bennett
-
Lucas Rolff
-
Ralph Seichter
-
Robert Moskowitz
-
Shawn Heisey