On Mar 20, 2013, at 6:43 AM, Timo Sirainen wrote:

On Wed, 2013-03-20 at 06:39 -0600, DormitionSkete@hotmail.com wrote:

...

I'd like to use Dovecot as our IMAP server when our users are within our LAN, but I'd also like to give them the ability to access their emails via POP3 when they are outside the LAN. I know most POP3 clients will give their users the option of not deleting the messages from the server after they are downloaded, but is there any way to restrict them from being able to do so at the server level?

In other words, I want to disallow the server from accepting the DELE command from POP3 clients.

Is that possible?

You could create a (global) ACL to not allow user to delete own mails. But some clients will probably keep redownloading the same mails over and over again then.

Thank you for the speedy reply!

Is there any chance you might consider implementing this as an option sometime? I assume the POP3 delivery code is separate from the IMAP code. You wouldn't necessarily need to return an error code to the email client. Most clients probably wouldn't know how to interpret it anyway. Just quietly ignore the DELE command.

Or would that leave us in the same position, where some clients may keep redownloading the same messages?

Also, how would I create a global ACL like you said, so I could test how our clients would react? Everybody here uses Macs, iPads or iPhones. We would not necessarily have to support a wide variety of clients.

We're using sendmail. I assume this is done in sendmail, not Dovecot? Or should I go to the sendmail group for that, if I can't find anything on the net about it with Google?

On Mar 20, 2013, at 7:22 AM, Thomas Leuxner wrote:

• DormitionSkete@hotmail.com dormitionskete@hotmail.com 2013.03.20 14:17:

...

Everybody here uses Macs, iPads or iPhones. We would not necessarily have to support a wide variety of clients.

Their Mail Clients natively support IMAP, so not sure why you would want to go with POP3 in this scenario.

Well, like I said, we have real slow upload speeds. I think POP3 would give a better user experience.

On Mar 20, 2013, at 7:51 AM, Thomas Leuxner wrote:

• DormitionSkete@hotmail.com dormitionskete@hotmail.com 2013.03.20 14:35:

...

...

Their Mail Clients natively support IMAP, so not sure why you would want to go with POP3 in this scenario.

Well, like I said, we have real slow upload speeds. I think POP3 would give a better user experience.

The clients mentioned will cache the messages locally. They will download headers first while they will retrieve the rest in the background. iOS Devices will even apply thresholds on larger messages downloading them partly and completing the rest upon request.

Regards Thomas

Really? Interesting.

Thank you.

My experience with IMAP over the internet with a couple of servers outside our monastery (while I was in it, and we have considerably better download speeds than upload) has always been that IMAP has always been incredibly slow. So, I've always just allowed users to connect to the IMAP server via webmail. It's slow, but usable.

I guess it's time to open a port in our firewall and do some testing with a couple of clients from outside. Maybe they'll work better than I've always assumed.

I appreciate the input, everybody.

Thank you.

fp

On Mar 20, 2013, at 8:59 AM, Timo Sirainen wrote:

On Wed, 2013-03-20 at 08:15 -0600, DormitionSkete@hotmail.com wrote:

...

My experience with IMAP over the internet with a couple of servers outside our monastery (while I was in it, and we have considerably better download speeds than upload) has always been that IMAP has always been incredibly slow. So, I've always just allowed users to connect to the IMAP server via webmail. It's slow, but usable.

Another idea: Get some cheap server from outside, use dsync replication to keep it synced with your internal one, and set up DNS so that users get directed to the fastest server. http://wiki2.dovecot.org/Replication

I LIKE this idea, but I have a few questions about it to see if it would be appropriate for our situation. There are a few other things to consider that I didn't mention before because they did not seem relevant earlier.

First off, I'd just like to say that we have a web server set up at a location outside of our monastery that hosts all of our websites. I'm currently in the process of building new servers to replace both it and our current email server. So, assuming this is both plausible for our situation, and within my capabilities, I should be able to work on this at my leisure, and get the initial sync of our emails done while on the same LAN.

So, the additional info and questions are the following:

1.) Our download speeds are decent enough, but in addition to having poor upload speeds, we also have very strict limits on how much we are able to download. And we use almost every bit of it every day. We cannot get more, either. We have unlimited downloads for four hours at night, however.

2.) We have very large message archives. We basically have 95% of the emails we've received for the past 16 years. So, the sync *must* only update items that have been changed. Is this how it it would work?

3.) We are currently using uw-imap with mbox. If we switch to Dovecot, using Maildir format, will the sync only update the new messages and the header files for any folders that have been changed?

4.) I thought I read somewhere in Dovecot's documentation last night that it has a 50 mb limit on folders. It can't write anything larger than that. Does this sound familiar? (Now I can't find it!) If so, is that for mbox? We currently have some mbox folders whose files are significantly larger than that. If we convert to Maildir format, where the individual messages are in their own files, could a folder contain messages totaling more than 50 MB using Dovecot?

4a. -- Oops.  I just noticed this: "NOTE2: sdbox/mdbox mailbox formats are recommended for replication. Maildir still has some issues (although probably not noticeable in normal use)."  Should I consider this a show-stopper for syncing like this?

5.) In the http://wiki2.dovecot.org/Replication page, would this be continuously synced each time a user sends, receives, deletes, or moves messages, etc.? Or would it be periodically synced?

6.) Also, that page does not make it clear if one server is like the "master" and the other the "slave". Do I do the same changes to both servers?

If, given the above additional information, it would not be an appropriate solution for us, this suggestion about syncing the two servers gave me another idea.

I was thinking, "Well, I wonder if I could just sync the Inboxes? We don't really need the folders synced. In the highly unlikely event a person would ever need something from one of his folders, he could always just log into the (slow) monastery server through web mail and get it that way."

(When we are on the road, we are generally working real hard, and we don't answer any more emails or do any other computer work than we absolutely have to.)

So, that led me to the idea to simply set up some message rules in procmail in our (slow) monastery server to copy any incoming messages to the server offsite in addition to delivering them locally. For the most part, that would be sufficient for us -- and considerably easier.

The only downsides to this are that when we reply to messages, they would not be marked as having been replied to, and we wouldn't have copies of our replies on our main server.

The not being marked as replied to is not a big deal. I know we could manually copy any sent messages from one server to the other when we returned to the monastery, if we really wanted to, but does anyone know of a better way to do it?

On Mar 20, 2013, at 3:11 PM, Robin wrote:

Your main concern sounds like performance from users who connect from outside of your enterprise network, which may happen even when your mobile devices are on site, due to the way they obtain their connectivity?

We are located deep in the Colorado Rocky Mountains. There are only a few places a person can stand in our monastery and get cell phone reception, so I don' think that is really an issue for us.

I'd greatly appreciate any advice or information about this. Both of these servers we're replacing are quite old. One is 10 1/2 years old... As I'm building the new ones, I'm trying to make things better. Email is one of the areas I think we should be able to make big improvements on.

So, any help will be greatly appreciated!

On Mar 22, 2013, at 2:47 AM, Timo Sirainen wrote:

On Wed, 2013-03-20 at 17:40 -0600, DormitionSkete@hotmail.com wrote:

...

On Mar 20, 2013, at 8:59 AM, Timo Sirainen wrote:

...

On Wed, 2013-03-20 at 08:15 -0600, DormitionSkete@hotmail.com wrote:

...

My experience with IMAP over the internet with a couple of servers outside our monastery (while I was in it, and we have considerably better download speeds than upload) has always been that IMAP has always been incredibly slow. So, I've always just allowed users to connect to the IMAP server via webmail. It's slow, but usable.

Another idea: Get some cheap server from outside, use dsync replication to keep it synced with your internal one, and set up DNS so that users get directed to the fastest server. http://wiki2.dovecot.org/Replication

I LIKE this idea, but I have a few questions about it to see if it would be appropriate for our situation. There are a few other things to consider that I didn't mention before because they did not seem relevant earlier.

First off, I'd just like to say that we have a web server set up at a location outside of our monastery that hosts all of our websites. I'm currently in the process of building new servers to replace both it and our current email server. So, assuming this is both plausible for our situation, and within my capabilities, I should be able to work on this at my leisure, and get the initial sync of our emails done while on the same LAN.

So, the additional info and questions are the following:

1.) Our download speeds are decent enough, but in addition to having poor upload speeds, we also have very strict limits on how much we are able to download. And we use almost every bit of it every day. We cannot get more, either. We have unlimited downloads for four hours at night, however.

If a delayed sync isn't a problem, you could do it only once at nights. You wouldn't need to use the replicator service at all, just run "doveadm sync -f -A -d" in a cronjob.

...

2.) We have very large message archives. We basically have 95% of the emails we've received for the past 16 years. So, the sync *must* only update items that have been changed. Is this how it it would work?

dsync can do full sync (= all messages' metadata is sent + new messages' contents), "changed sync" (= same as full sync, but only for changed folders) or incremental sync (= only new messages' metadata + contents are sent). The incremental sync is what replicator service does while it's running, but it's still currently doing a full sync at startup.

A nightly cronjob could do incremental syncing also, but it would have to run dsync separately for each user and store the sync state to some file.

The "changed sync" works well enough usually, but it has a problem if both replicas have had exactly the same amount of changes it doesn't realize that there may be differences between them and skip it.

...

3.) We are currently using uw-imap with mbox. If we switch to Dovecot, using Maildir format, will the sync only update the new messages and the header files for any folders that have been changed?

It works the same with all mailbox formats. Headers and bodies aren't synced separately, but metadata (= ~100 bytes/msg maybe) is.

...

4.) I thought I read somewhere in Dovecot's documentation last night that it has a 50 mb limit on folders. It can't write anything larger than that. Does this sound familiar? (Now I can't find it!) If so, is that for mbox? We currently have some mbox folders whose files are significantly larger than that. If we convert to Maildir format, where the individual messages are in their own files, could a folder contain messages totaling more than 50 MB using Dovecot?

Dovecot has no such limit. Postfix by default has set a file size limit for 50 MB, which effectively limits mbox sizes to 50 MB, but it can be removed with Postfix mailbox_size_limit setting.

...

4a. -- Oops. I just noticed this: "NOTE2: sdbox/mdbox mailbox formats are recommended for replication. Maildir still has some issues (although probably not noticeable in normal use)." Should I consider this a show-stopper for syncing like this?

With v2.2 I don't think there's much of a difference anymore.

...

5.) In the http://wiki2.dovecot.org/Replication page, would this be continuously synced each time a user sends, receives, deletes, or moves messages, etc.? Or would it be periodically synced?

With replicator it syncs immediately when something changes.

...

6.) Also, that page does not make it clear if one server is like the "master" and the other the "slave". Do I do the same changes to both servers?

Both servers are equal. Setup both servers exactly the same.

...

If, given the above additional information, it would not be an appropriate solution for us, this suggestion about syncing the two servers gave me another idea.

I was thinking, "Well, I wonder if I could just sync the Inboxes? We don't really need the folders synced. In the highly unlikely event a person would ever need something from one of his folders, he could always just log into the (slow) monastery server through web mail and get it that way."

If you're syncing via ssh, you can give "-m inbox" parameter to dsync_remote_cmd setting and it syncs only INBOX. But it's still unnecessarily running dsync whenever anything changes. With some hardcoding it would be easy to change that though.

...

(When we are on the road, we are generally working real hard, and we don't answer any more emails or do any other computer work than we absolutely have to.)

So, that led me to the idea to simply set up some message rules in procmail in our (slow) monastery server to copy any incoming messages to the server offsite in addition to delivering them locally. For the most part, that would be sufficient for us -- and considerably easier.

The only downsides to this are that when we reply to messages, they would not be marked as having been replied to, and we wouldn't have copies of our replies on our main server.

The not being marked as replied to is not a big deal. I know we could manually copy any sent messages from one server to the other when we returned to the monastery, if we really wanted to, but does anyone know of a better way to do it?

The users would then need to have two accounts I think, one for internal and one for outside mail. Otherwise whenever they switch between servers they need a full resync.

Thank you very, very much, Timo. I appreciate this more than you can possibly imagine.

And thank you, everyone else who contributed to this thread, as well. I appreciate all of your ideas and suggestions very much, too.

My current plan is to to move the bulk of our archives to another server -- actually, a different zone on the same Solaris machine -- and dsync just our primary email server at night. The archives are rarely used, and don't need to be accessed from outside. In fact, I'll probably leave them in uw-imap so I don't have to convert them to dovecot.

I think that should probably work the best for us, if I can pull it off. I'm a lot more of a programmer than a system admin.

Thank you all again.

On 3/20/2013 6:35 AM, DormitionSkete@hotmail.com wrote:

Well, like I said, we have real slow upload speeds. I think POP3 would give a better user experience.

About the only connectivity situation where POP3 might make for a better "user experience" is one of intermittent bursty sort that's prone to reliability issues.

IMAP provides for header-only enumations as well as partial body fetches on demand, as opposed to "all or nothing" POP3 access. With a suitable modern caching client, it will not re-download emails already viewed. I've never used any of the devices you mentioned, so I can't speak to how their mail clients are implemented.

We're using sendmail. I assume this is done in sendmail, not Dovecot?

No, sendmail is a Mail Transport Agent (MTA), which is akin to the Postal Service. All it does is convey emails from one endpoint to another as reliably as possible. What is done with the mail once it's at that endpoint is left to the "consumer" of the mail, in this case, the Mail User Agent (MUA). It can be automatically processed/filed like via procmail or LMTP, or managed via the client through POP3 or IMAP4.

Your main concern sounds like performance from users who connect from outside of your enterprise network, which may happen even when your mobile devices are on site, due to the way they obtain their connectivity? Timo's replication idea is sensible to address that problem.

Good luck! =R=