[Dovecot] Creating an IMAP repo for ~100 users need some advice
Hi,
I am currently in the process of setting up an IMAP repository for round 100 users....
Currently the user authentication method is being handled via a Windows Domain Controller.
The host OS for Dovecot will either be FreeBSD or CentOS.
Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP??
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails??
Can anyone give me a hand with this?
Regards,
Kaya
Kaya Saman kayasaman@gmail.com wrote:
I am currently in the process of setting up an IMAP repository for round 100 users....
Currently the user authentication method is being handled via a Windows Domain Controller.
The host OS for Dovecot will either be FreeBSD or CentOS.
Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP??
Why not join the server to the domain and simply use PAM?
Using ActiveDirectory through LDAP is a bit of a pain so I would avoid this if I where you.
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails??
Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress.
To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots.
Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore.
Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself.
Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory.
Some, like mbox or mdbox are a little bit more complex, but with the correct doveadm command you are nevertheless able to restore single mails.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 03/17/2012 07:36 PM, Sven Hartge wrote:
Kaya Samankayasaman@gmail.com wrote:
I am currently in the process of setting up an IMAP repository for round 100 users.... Currently the user authentication method is being handled via a Windows Domain Controller. The host OS for Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP?? Why not join the server to the domain and simply use PAM?
Using ActiveDirectory through LDAP is a bit of a pain so I would avoid this if I where you.
Danke Sven :-)
I don't actually have much AD/LDAP integration experience so I will try your method!
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress.
To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots.
Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues.
Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore.
To be honest I was considering rsync'ing the dir containing users mailboxes to either another storage pool or server.
Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself.
Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory.
You mention Maildir++... is this Maildir format or something new which I haven't heard about yet?
Some, like mbox or mdbox are a little bit more complex, but with the correct doveadm command you are nevertheless able to restore single mails.
Grüße, Sven.
Regards,
Kaya
Kaya Saman kayasaman@gmail.com wrote:
On 03/17/2012 07:36 PM, Sven Hartge wrote:
Kaya Samankayasaman@gmail.com wrote:
I am currently in the process of setting up an IMAP repository for round 100 users.... Currently the user authentication method is being handled via a Windows Domain Controller. The host OS for Dovecot will either be FreeBSD or CentOS. Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP??
Why not join the server to the domain and simply use PAM?
Using ActiveDirectory through LDAP is a bit of a pain so I would avoid this if I where you.
I don't actually have much AD/LDAP integration experience so I will try your method!
Question: do you need public or shared folders?
Using samba and winbindd to join a domain creates real users on your server and as far as I know configuring shared folders with real users is a bit of a pain, especially of you need shared flags (like Seen, Replied, etc.) (Someone [Timo?] please correct me.)
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails??
Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress.
To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots.
Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues.
Yes. Or using LVM on Linux.
Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore.
To be honest I was considering rsync'ing the dir containing users mailboxes to either another storage pool or server.
No need to rsync, if you use ZFS. Just create a new snapshot and you are done. Bet thing about ZFS: you get deduplication for free, so the needed space to store the backups will not grow as fast.
But you still may want to store the mails offsite/offserver for desaster recovery.
Either use doveadm backup for that purpose or use rsnapshot, again gaining you deduplication on the target server.
Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself.
Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory.
You mention Maildir++... is this Maildir format or something new which I haven't heard about yet?
Maildir++ extends the original Maildir with things like Quota and ACLs and was first implemented in Courier. http://www.courier-mta.org/imap/README.maildirquota.html
All current MTAs and POP3/IMAP servers implement this variant.
Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are).
In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands.
I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever".
Grüße, Sven.
-- Sigmentation fault. Core dumped.
Thanks so much Sven for your indepth and complete responses!
<snip> > Question: do you need public or shared folders?
I don't need anything apart from an IMAP storage solution. I don't intend to tie in Dovecot with an MTA either as I will simply be using this for storage.
Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange.
To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc.
Using samba and winbindd to join a domain creates real users on your server and as far as I know configuring shared folders with real users is a bit of a pain, especially of you need shared flags (like Seen, Replied, etc.) (Someone [Timo?] please correct me.)
Actually we might have an LDAP server already taking care of the AD<->UNIX integration..... I don't know yet it's only my first week :-)
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails?? Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress.
To have a consistent backup, your mail storage should be able to snapshot the volume the mail is stored on, so use LVM or an external storage unit capable of snapshots. Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues. Yes. Or using LVM on Linux.
Yeah.... true but I specified ZFS as I'm a fan and also am quite comfortable with Solaris/*BSD too......
Then backup the content of the snapshot using any program you like. I use Bacula for long-term offsite storage and a local rsnapshot to keep 7 days worth of mail for a quick restore. To be honest I was considering rsync'ing the dir containing users mailboxes to either another storage pool or server. No need to rsync, if you use ZFS. Just create a new snapshot and you are done. Bet thing about ZFS: you get deduplication for free, so the needed space to store the backups will not grow as fast.
Ok so that solves that! :-)
But you still may want to store the mails offsite/offserver for desaster recovery.
They are currently being stored on the parent company mail server so this will be the/off-site/ disaster recovery system in a way :-P
Either use doveadm backup for that purpose or use rsnapshot, again gaining you deduplication on the target server.
I will research this - thank you for that info :-)
Whether you are able to restore single mails or the complete storage is no property or feature of the mailbox format itself.
Some formats are simpler to handle, like Maildir++, where you just drop the file containing a mail into a directory. You mention Maildir++... is this Maildir format or something new which I haven't heard about yet? Maildir++ extends the original Maildir with things like Quota and ACLs and was first implemented in Courier. http://www.courier-mta.org/imap/README.maildirquota.html
All current MTAs and POP3/IMAP servers implement this variant.
Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are).
In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands.
I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever".
So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format with this and use mdbox instead. {Note to self: time to browse!}
Since where I'm implementing this is mainly an MS based environment they are concerned about /flat/ files.... which MS seems to typically do (although never used MS before so I wouldn't know). So there is some concern over performance, efficiency and manageability.
However, if like you say mdbox is the way to go then I will put a strong case together!
Grüße, Sven.
Regards,
Kaya
Kaya Saman kayasaman@gmail.com wrote:
<snip> > Question: do you need public or shared folders?
I don't need anything apart from an IMAP storage solution. I don't intend to tie in Dovecot with an MTA either as I will simply be using this for storage.
Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange.
To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc.
So, how do you plan to get the mails into this storage? offlineimap? imapsync? mbsync? fetchmail?
Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues. Yes. Or using LVM on Linux.
Yeah.... true but I specified ZFS as I'm a fan and also am quite comfortable with Solaris/*BSD too......
If you know ZFS and are familiar with it, then, by all means, go for it.
Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are).
In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands.
I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever".
So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format with this and use mdbox instead. {Note to self: time to browse!}
mdbox is a format invented by Timo for dovecot. But dovecot can use nearly all common mailbox formats (except MH, but no one uses that one today).
Since where I'm implementing this is mainly an MS based environment they are concerned about /flat/ files.... which MS seems to typically do (although never used MS before so I wouldn't know). So there is some concern over performance, efficiency and manageability.
Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no doubt about this.
But even with this crappy format, dovecot is able to deliver astounding performance by use of separete index files which allow it to access the storage in an efficient manner.
mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten.
mdbox solves this problem by a) storing all meta-data in the index and b) by only ever appending to a mdbox storage file, c) never truncating an existing mdbox storage file and d) using more than one mdbox storage file. Max size and TTL are configurable.
But this also means deleted mails are still inside a mdbox storage file and need to be finally removed by copying all remaining files into a new file. This process has to be manually run during low traffic hours, for example using a cronjob.
You can say, mdbox is like mbox on steroids. ;)
Flat files are not evil or bad or slow per se, but you have to use them the right way.
However, if like you say mdbox is the way to go then I will put a strong case together!
You may want to start with something familiar and convert later, which is no problem with dovecot.
Grüße, Sven
-- Sigmentation fault. Core dumped.
On 03/17/2012 10:28 PM, Sven Hartge wrote:
Kaya Samankayasaman@gmail.com wrote:
<snip> > Question: do you need public or shared folders? I don't need anything apart from an IMAP storage solution. I don't intend to tie in Dovecot with an MTA either as I will simply be using this for storage. Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange. To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc. So, how do you plan to get the mails into this storage? offlineimap? imapsync? mbsync? fetchmail?
Since everything is blocked at the Exchange end, users will have to manually transfer for now through MS Outlook.
Currently that's what they're doing to their PST's....
Hmm..... so FreeBSD coupled together with a ZFS repo for mail should take care of 'Snapshot' issues. Yes. Or using LVM on Linux. Yeah.... true but I specified ZFS as I'm a fan and also am quite comfortable with Solaris/*BSD too...... If you know ZFS and are familiar with it, then, by all means, go for it.
:-)
Depending on the amount of mail a user collects inside a folder, Maildir is not the best storage format. You may want to check into mdbox, if your users are kind of "mail hoarders" (like some of my users are).
In my opinion, Maildir has outlived its usefullnes. It was fine when users had 1,000 mails in some 10 folders, but today, users collect over 100,000 mails a year and Maildir is causing serious I/O trouble and the need to heavily fine tune your storage and filesystems to cope with those demands.
I cannot thank Timo enough for inventing mdbox, as this format breaks this viciuos cycle and, as someone else said "it ends the battle at the I/O front forever". So mdbox is a 'new' mailbox standard? ie. one can replace Maildir format with this and use mdbox instead. {Note to self: time to browse!} mdbox is a format invented by Timo for dovecot. But dovecot can use nearly all common mailbox formats (except MH, but no one uses that one today).
Ok so if you claim that mdbox is the 'best' mailbox storage solution then I'll look at implementing this.
Since where I'm implementing this is mainly an MS based environment they are concerned about /flat/ files.... which MS seems to typically do (although never used MS before so I wouldn't know). So there is some concern over performance, efficiency and manageability. Ye olde MBOX flat file format, as used in UW-imapd for ages, is a nightmare, no doubt about this.
But even with this crappy format, dovecot is able to deliver astounding performance by use of separete index files which allow it to access the storage in an efficient manner.
mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten.
mdbox solves this problem by a) storing all meta-data in the index and b) by only ever appending to a mdbox storage file, c) never truncating an existing mdbox storage file and d) using more than one mdbox storage file. Max size and TTL are configurable.
But this also means deleted mails are still inside a mdbox storage file and need to be finally removed by copying all remaining files into a new file. This process has to be manually run during low traffic hours, for example using a cronjob.
You can say, mdbox is like mbox on steroids. ;)
Flat files are not evil or bad or slow per se, but you have to use them the right way.
Thanks a lot for that info. I will research more into this but I maybe overridden at some point :-(
Need to make a strong case!
However, if like you say mdbox is the way to go then I will put a strong case together! You may want to start with something familiar and convert later, which is no problem with dovecot.
Maildir is what I'm familiar with currently and mbox format - though only use mbox as an unfortunate side product of /system mail/ accounts.
Works well with Alpine client though!
Grüße, Sven
Regards,
Kaya
Kaya Saman kayasaman@gmail.com wrote:
Flat files are not evil or bad or slow per se, but you have to use them the right way.
Thanks a lot for that info. I will research more into this but I maybe overridden at some point :-(
Need to make a strong case!
Hmm.
Just because Microsofts way of usage of flat file database sucks does not mean any usage of flat files is bad or evil or slow, if done right.
Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox
But as I wrote before, it is quite easy to convert from one format to the other: http://wiki2.dovecot.org/Migration/MailFormat
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 03/18/2012 12:04 AM, Sven Hartge wrote:
Kaya Samankayasaman@gmail.com wrote:
Flat files are not evil or bad or slow per se, but you have to use them the right way. Thanks a lot for that info. I will research more into this but I maybe overridden at some point :-( Need to make a strong case! Hmm.
Just because Microsofts way of usage of flat file database sucks does not mean any usage of flat files is bad or evil or slow, if done right.
Coming from a UNIX background I deal quite a lot with this kind of stuff so there's not problem for me. However, where I'm trying to deploy this system is a primarily MS based enterprise meaning that as the only UNIX engineer onsite and the newest addition to the team I have to convince people of working with UNIX technologies or somehow increase UNIX awareness.
As a bi-product I know nothing about MS tech. only what it told to me by my colleagues :-)
Have a look at http://wiki2.dovecot.org/MailboxFormat/dbox
I checked that out after your last email... I started Google'ing a little. :-)
Looks like it would be a good solution!
But as I wrote before, it is quite easy to convert from one format to the other: http://wiki2.dovecot.org/Migration/MailFormat
Once we get setup this may come in quite handy! Not sure what's going on currently as everyone above me is still quite set in using an SQL DB as a mail storage system???
To be honest, I run Zimbra @home for my OpenSource work and really enjoy it; in conjunction with Dovecot on FreeBSD which I run imapsync to backup **all** emails to.
It works really well...... :-)
I have messed around with Postfix, Dovecot and Horde3 in the past which also was really nice.
Grüße, Sven.
Regards,
Kaya
Kaya Saman kayasaman@gmail.com wrote:
Once we get setup this may come in quite handy! Not sure what's going on currently as everyone above me is still quite set in using an SQL DB as a mail storage system???
RDBMS where not designed for such a task. Using a relational database as a storage method for big chunks of data is very unwise, in my opinion. It degrades them to just being some sort of filing cabinet.
Now, wouldn't it be nice, if we had something like that, a filing cabinet where we can store large chunks of data and randomly read and write them in a fast manner?
Oh yes, I remember, it is called a "filesystem". Let's use some of those to store the mail data. It will be soooo awesome! ;-)
Ok, back being serious: there is nothing wrong with using a RDBMS in the way it was intented, to store user credentials, quota values, account settings, forwarding addresses, address book data, bookmarks, etc.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 03/18/2012 12:32 AM, Sven Hartge wrote:
Kaya Samankayasaman@gmail.com wrote:
Once we get setup this may come in quite handy! Not sure what's going on currently as everyone above me is still quite set in using an SQL DB as a mail storage system??? RDBMS where not designed for such a task. Using a relational database as a storage method for big chunks of data is very unwise, in my opinion. It degrades them to just being some sort of filing cabinet.
Now, wouldn't it be nice, if we had something like that, a filing cabinet where we can store large chunks of data and randomly read and write them in a fast manner?
Oh yes, I remember, it is called a "filesystem". Let's use some of those to store the mail data. It will be soooo awesome! ;-)
I think for the serious engineer there's Linux if even more serious there's UNIX and for the rest there's MS..... Actually as a medical term MS is something not that great to have; why does that also equate to IT/Computing too ;-P
Ok, back being serious: there is nothing wrong with using a RDBMS in the way it was intented, to store user credentials, quota values, account settings, forwarding addresses, address book data, bookmarks, etc.
I agree!
My humble opinion for a personal preference setup in this instance:
FreeBSD 8.2 x64 as base OS UFS2 running on root drive Create ZFS pools for storage Have users mailboxes on the ZFS pools Enable ZFS caching and snapshots Dovecot to manage IMAPv4
Get rid of MS altogether!
....Then start working a really cool implementation of UNIX/Linux only infrastructure :-)
Grüße, Sven.
Regards,
Kaya
On 3/17/2012 12:36 PM, Sven Hartge wrote:
Storing mails inside SQL? Not supported by dovecot and not very wise, IMHO. DBmail does this, but to be honest, I never heard any good feedback from admins using that product. From what I have been told, you need quite the beefy server to get a decent performance out of DBmail, compared to the needs of a "traditional" setup like with dovecot or courier-mail, but I digress.
Ugh, I've tried the product. It works pretty well, until you move more than a small handful of users and email hives to it, and you hit some hard walls pretty fast with how many inbound emails/second it can handle for even burly server configurations.
Those hard walls occur at too low a threshold for me. The product's mailing list is supportive and there are many dedicated DBMail users who step in an answer questions, but be prepared for "BUY MORE RAM" as the answer to concerns about performance. When 128GB of RAM is needed for a small organisation's email setup to perform well, I am strongly inclined to move on to the next product.
Best practices for it seem to revolve around being able to have your ENTIRE email + index content resident in RAM. Well, gosh. Why didn't I think of that before instead of wasting all of this time worrying about design and efficiency?
And if you're hoping that it will make text searches "automagically" fast, think again.
Timo's FTS_SQUAT blows it out of the water by orders of magnitude, even with mailbox sizes of around 300K emails (20GB), let alone something like Lucene or Solr.
I understand why it seems like a great idea to store email this way, but realise that the bulk of email is NOT structured or inherently relational.
=R=
On 18.3.2012, at 0.28, Sven Hartge wrote:
mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten.
Dovecot moves only minimal amount of data within mbox. A flag change writes only a few bytes to mbox file, it doesn't rewrite it the whole file. Only time when the entire mbox file is rewritten is when you expunge the first message.
Timo Sirainen tss@iki.fi wrote:
On 18.3.2012, at 0.28, Sven Hartge wrote:
mbox has big problems with concurrent writes, the bigger the mbox is, the more problems you get. This is mainly caused by the meta-data of a message (meaning flags, status, etc.) which is stored inside the mbox file itself. Flagging a message as read or replied causes the whole mbox file to be rewritten.
Dovecot moves only minimal amount of data within mbox. A flag change writes only a few bytes to mbox file, it doesn't rewrite it the whole file. Only time when the entire mbox file is rewritten is when you expunge the first message.
OK, then my memory was clouded by my exposure to the brokenness of uw-imapd and uw-popd, who rewrite an mbox file very often, resulting in abysmal performance.
Grüße, Sven.
-- Sigmentation fault. Core dumped.
On 2012-03-18 11:15 AM, Timo Sirainen tss@iki.fi wrote:
Only time when the entire mbox file is rewritten is when you expunge the first message.
Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior?
Although I have no desire to use mbox (planning on using mdbox), so it isn't important to me... ;)
--
Best regards,
Charles
On Sun, Mar 18, 2012 at 11:36:25AM -0400, Charles Marcus wrote:
Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior?
That's what uw-imap does. It creates a message with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA", which is very annoying if your users has direct access to the mbox's...
http://www.washington.edu/imap/IMAP-FAQs/index.html#6.14-jf
On 18.3.2012, at 17.36, Charles Marcus wrote:
On 2012-03-18 11:15 AM, Timo Sirainen tss@iki.fi wrote:
Only time when the entire mbox file is rewritten is when you expunge the first message.
Hmmm... wonder if there would be a way to add some kind of 'dummy' first message that dovecot would simply ignore (not show to the user), that would prevent that bevaior?
Dovecot has such a dummy first message exactly like UW-IMAP. But what I meant is that if you expunge the first message and you want to free the space used by it, there's no other choice than to rewrite all of the messages after it. (And of course the first message isn't special in any way, there's just about as much of rewriting if you delete the 2nd or 3rd or other messages from the beginning of the mbox.)
On 3/17/2012 4:24 PM, Kaya Saman wrote:
Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange.
To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc.
It's been many years since I used, or supported, MS Outlook. That said, for the 10+ years I did support it, ~1996-2006, the corporate version of Outlook, not to be confused with Outlook Express, did not store any mail in local .PST files unless specifically configured to do so. By default it keeps all mail in the user account in the Exchange server store.
Thus I would assume these Outlook clients have been manually configured to use .PST files to keep copies of mail locally, for faster access and to keep inefficient MS Exchange (MAPI) traffic off the WAN link?
Is your problem with the PST files themselves, or merely the fact they're stored on the local PC, probably in the users' roaming profiles, thus creating the problem of large data movement during logon/off?
If the problem isn't with the .PST format for storing the emails, why not simply setup a local Samba server and configure the Outlook clients to store users' PSTs on Samba shares?
Better yet, if you already have a file server for home directories, simply use a folder redirection policy to put the PST files in folders in users' home directories. This is an extremely common practice in the MS world because all Microsoft Windows apps store everything in the user profile directory by default, which again, causes big problems with roaming profiles, which many/most enterprises use.
-- Stan
On 03/18/2012 09:16 AM, Stan Hoeppner wrote:
On 3/17/2012 4:24 PM, Kaya Saman wrote:
Long story but we don't have any control over our mail server which is handled by the parent company abroad and is on MS Exchange.
To use an IMAP storage solution is the only way to get rid of pesky MS .pst files which have been causing everyone grief and havoc. It's been many years since I used, or supported, MS Outlook. That said, for the 10+ years I did support it, ~1996-2006, the corporate version of Outlook, not to be confused with Outlook Express, did not store any mail in local .PST files unless specifically configured to do so. By default it keeps all mail in the user account in the Exchange server store.
Thus I would assume these Outlook clients have been manually configured to use .PST files to keep copies of mail locally, for faster access and to keep inefficient MS Exchange (MAPI) traffic off the WAN link?
Is your problem with the PST files themselves, or merely the fact they're stored on the local PC, probably in the users' roaming profiles, thus creating the problem of large data movement during logon/off?
If the problem isn't with the .PST format for storing the emails, why not simply setup a local Samba server and configure the Outlook clients to store users' PSTs on Samba shares?
Better yet, if you already have a file server for home directories, simply use a folder redirection policy to put the PST files in folders in users' home directories. This is an extremely common practice in the MS world because all Microsoft Windows apps store everything in the user profile directory by default, which again, causes big problems with roaming profiles, which many/most enterprises use.
Thanks Stan for this,
currently our users have about 270MB space located on the Exchange server which we have no control over.
Therefor users are currently manually backing up their information to .pst.
Since all contact/calendar/other information is already stored on the server the IMAP solution is a better one. It also means that a more UNIX/Linux centric approach is being ask for/tolerated which is where I come in being the only full-bread UNIX engineer on site.
I couldn't comment on the MS side of things as I have never really used MS stuff before but my plan using ZFS and FreeBSD should be the best. Again it's going to be **scalable** storage which is perfect!
-Also easy to maintain; otherwise I don't think there would be anyone left who will be able to admin the SMB/local directory method (I can't as I don't understand MS) - additionally there isn't much space available and bandwidth either meaning we would purchase a dedicated server or build a dedicated server for this (well I would :-) ).
Regards,
Kaya
On 2012-03-18 5:16 AM, Stan Hoeppner stan@hardwarefreak.com wrote:
Is your problem with the PST files themselves, or merely the fact they're stored on the local PC, probably in the users' roaming profiles, thus creating the problem of large data movement during logon/off?
If so, using redirected folders (if you're not using them, you should be) would alleviate this problem nicely, even in a large environment.
--
Best regards,
Charles
Hi,
I am currently in the process of setting up an IMAP repository for round 100 users....
Currently the user authentication method is being handled via a Windows Domain Controller.
The host OS for Dovecot will either be FreeBSD or CentOS.
Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP??
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails??
Can anyone give me a hand with this?
Regards,
Kaya
Hi Kaya,
I can't force you to follow a specific path. All I can do, is tell you my experience on this.
Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for Webmail, OpenLDAP for LDAP (no windows software there) and CentOS v5.5 32-bit onwards. User base is about 30 users. System uptime without a glitch reached at some point (had to reboot the server for maintenance reasons) about 200 days. I am sure it would go beyond 365 days.
Hope this helps,
spyros
"I merely function as a channel that filters music through the chaos of noise"
- Vangelis
On 03/17/2012 09:51 PM, Spyros Tsiolis wrote:
Hi,
I am currently in the process of setting up an IMAP repository for round 100 users....
Currently the user authentication method is being handled via a Windows Domain Controller.
The host OS for Dovecot will either be FreeBSD or CentOS.
Would Dovecot be able to authenticate to either the DC directly or would we need to go through LDAP??
Additionally what would be the best method to store the **mail** information? - as in MySQL database or Maildir format; coinciding with this what is the best backup method in order to be able to do 'dump' backups or restore single emails??
Can anyone give me a hand with this?
Regards,
Kaya
Hi Kaya,
I can't force you to follow a specific path. All I can do, is tell you my experience on this.
Using Dovecot for IMAP, XMail for POP3/SMTP, Horde for Webmail, OpenLDAP for LDAP (no windows software there) and CentOS v5.5 32-bit onwards. User base is about 30 users. System uptime without a glitch reached at some point (had to reboot the server for maintenance reasons) about 200 days. I am sure it would go beyond 365 days.
Hope this helps,
spyros
"I merely function as a channel that filters music through the chaos of noise"
- Vangelis
Thanks for that Spyros!
Regards,
Kaya
participants (8)
-
Charles Marcus
-
Jan-Frode Myklebust
-
Kaya Saman
-
Robin
-
Spyros Tsiolis
-
Stan Hoeppner
-
Sven Hartge
-
Timo Sirainen