[Dovecot] Advice for new dovecot / imap proxy? setup
Hello list.
I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice.
The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. It would be desirable for multiple reasons:
- graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes
- be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future)
- be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear)
- be ready to serve traffic over IPv6 without touching our precious mailbox servers
- isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc...
Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space).
In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources).
I'd like to know if that plan sounds reasonable or if there's something stupid in it. Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow?
thanks,
Luca Lesinigo
El Miércoles, 21 de Marzo de 2012 15:43:14 Luca Lesinigo escribió:
Hello list. Hello,
I'm planning a new mail servers for our company's customers to replace the oldish Courier-IMAP based one, we already started to deploy some mail accounts on a dovecot-2.0 server as an early test. I'd like to implement the new system with dovecot-2 (I'll probably go straight to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here asking for some advice.
The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy. It would be desirable for multiple reasons: I have recently deployed a very similar setup: imap proxy, mailbox sharding... Although not exactly like yours. Comments below:
- graceful migration from the current system: we'd make the mailserver hostname point to the proxy (along with its SSL certificates) and then the proxy would route each domain to the correct IMAP non-ssl server on our LAN. No need to update customer's systems configuration and we can move one domain at a time from the old to the new server, behind the scenes This is reasonable. For example, I did this to seamless migrate lots of users from one server to another, migrating just a few of them at a time.
- be ready for similar migrations in the future (eg. right now we're still keeping the imap servers with the qmail MTA, but we'd like to switch to postfix+dovecot in the future) You can do the exact same thing in the future, of course.
- be ready for sharding mail domains on multiple IMAP servers (if/when current hardware reach its capacity or needs to be swapped out for new gear) This is fairly easy to accomplish with imap proxying.
- be ready to serve traffic over IPv6 without touching our precious mailbox servers This is doable.
- isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes.
Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail servers on our internal LAN. We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new backend mailbox server to improve our operations (currently we just run du in a cronjob once a day on the current mailserver, IMAP clients including the webmail do not know about quota and thus cannot show amount of free space). I didn't implement a lemonade profile nor quotas in my setup. However, I can confirm you that IMAP IDLE does work with imap proxy.
In addition to that, customer's will hit the SMTP server running on that 'proxy' system and this is good to keep its configuration separated from the SMTP server of the actual mail servers (which has a different configuration and is restricted to get connections only from our MX systems and not from outside sources). No problem with that, but this is related to the MTA configuration, not dovecot.
I'd like to know if that plan sounds reasonable or if there's something stupid in it. Also, is the proxy going to support all kind of IMAP stuff of the backend server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients thanks to linux inotify, etc...) or will it limit me somehow? You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail.
On 21.3.2012, at 16.43, Luca Lesinigo wrote:
The issue I'm investigating right now is how to manage a single IMAP / POP / SMTP / webmail "entry point" for multiple mail servers... in other words an IMAP proxy.
Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying.
We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc)
Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either.
Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto:
Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this....
We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :)
Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto:
- isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it.
However, I can confirm you that IMAP IDLE does work with imap proxy.
That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go.
You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things.
You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external).
Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients).
-- Luca Lesinigo
Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto:
Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main
On 03/23/2012 02:12 PM, Luca Lesinigo wrote: potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems.
We'd like to support all the recent IMAP goodies to make modern users happy (IMAP IDLE, LEMONADE, etc) Dovecot doesn't support the full LEMONADE yet, but I don't know if there are any LEMONADE clients either. Oh well I included it in the list because I read about it somewhere, possibly on the dovecot site. But what I really meant was simply "support the latest goodies" :)
Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto:
- isolate the mailbox servers from direct external access and just run IMAP on them, let other systems run ssl, pop3, smtp, webmail, etc... I don't think I understand you here. You will need to run POP3 on the mailbox servers if you want to give POP3 access to the mailboxes. Don't ask me why, but I was thinking that a dovecot proxy could talk just imap to the backends and use that to serve both POP3 and IMAP to clients. And it's possibly what happens with the imapc backend, but I need to do some RTFM about it.
The same proxy_maybe (dummy proxy) setup works great for POP3 too. Very simple to set up, works like a charm. Nothing much to think about.
However, I can confirm you that IMAP IDLE does work with imap proxy. That's great, I really want to provide the best possible "push-like" experience to modern clients, and as far as I know IMAP IDLE on the protocol side plus some notification mechanism (as opposed to regular polling) on the backend side is the way to go.
It will work as well as it was working with your existing courier server. But it will work great for accounts migrated to native dovecot.
You have my comments above, I think it is doable. In my opinion, the IMAP proxy part is the easiest one. MTA configuration to distribute the mails among the different mailbox servers can be trickier. Actually that part is already there. Mail enters my systems via some MX servers (with the usual antispam and so on) and it's finally delivered via SMTP to the correct mail server via postfix recipient maps (that's because I already receive on my MXes mail for domains not hosted on my mail server, the common scenario is where I route a domain's mail to the customer's exchange server). But right now the mail server also receives direct SMTP connections from the clients in addition to incoming mail from my MXes and I'd really prefer to separate the two things.
It's a very good idea to have completely separate machines for outgoing mail. Once you have imap-only boxes, you can eliminate the need for an MTA by using the dovecot LMTP server. Your postfix transport map can send mail to either smtp:imap.yourdomain.com:25 or lmtp:imap.yourdomain.com:2525 on a per account basis, and you can get rid of the MTA in due time.
You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, thus the dovecot proxy instance will handle the sharding for delivering and for reading mail. On the proxy system I plan to run postfix to implement authenticated SMTP (it would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to support that :| ), but all mail will be reinjected through our MX servers to be scanned before final delivery (either local or external).
Since you're sending everything back to the MX, you might as well have your MX use LMTP, looking up the correct protocol and host from the database, and spend the next couple of years telling your customers to change their mail client configuration to use a dedicated outgoing mail server. It's worth the trouble.
Thanks people for the suggestions, my next stop is getting to know imapc and its details, and how the various other parts will fit with that (eg. giving pop3 service to clients).
-- Luca Lesinigo
On 23.3.2012, at 20.24, Gedalya wrote:
On 03/23/2012 02:12 PM, Luca Lesinigo wrote:
Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto:
Are you thinking about actual "dummy" proxying (which is normally what Dovecot proxying is about) or about the "imapc" backend (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend servers, there's really no reason to use imapc proxying. I actually didn't know about the two different modes. I guess I would need imapc to support the older Courier-IMAP server until I migrated everything away from it, and that I could use "dummy" proxying for the newer dovecot backends. I don't know if the two can be used at the same time (eg. imapc to the older backend and dummy to the newer) and/or if there is any drawback in running everything on imapc (old and new dovecot server). I'll be investigating this.... I'm using the dummy proxying with a very different backend, certainly not dovecot, and it works great. For your needs (as I understand them) It's a much simpler and robust solution than imapc. Try it out. The main potential source of trouble is possible differences in the CAPABILITY string, but it hasn't caused me any actual problems.
Right, a lot of people have done migration from Courier -> Dovecot using the dummy proxying. Since v2.0 the proxying automatically handles any CAPABILITY string issues.
participants (4)
-
Gedalya
-
Luca Lesinigo
-
Miguel Tormo
-
Timo Sirainen