I'm top posting because I can't make heads or tails of this thread. Does this thread mean that Dovecot will no longer be Free Software?

It appears that only Dovecot Director will be taken proprietary, but if all of Dovecot is in jeopardy, I need to switch to another local IMAP server program. Any suggestions will be welcome.

Thanks,

SteveT

Aki Tuomi said on Thu, 20 Oct 2022 13:02:38 +0300 (EEST)

...

On 20/10/2022 12:24 EEST Steff Majeur afamiliartaste@proton.me wrote:

I recently stumbled upon the following commit on the Dovecot core Github repository: https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005...

Apparently, Dovecot Director is going to be removed in the next major version of Dovecot and the commercial Dovecot cluster architecture will be its successor: https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25...

Yes, this is going to happen.

...

This would be a huge blow for many organizations around the world that are currently using Dovecot with Director in a shared storage environment.

Can anyone of the Dovecot developers maybe enlighten us about the future of Dovecot?

• Will there still be the Director feature in the next community release of Dovecot?

Next 2.3 CE release will have a director.

...

• If not, will there be a community feature that is on par with the current Director feature?

There will be more information about this closer to new major release, that we are working on. Director is still present in https://github.com/dovecot/core/tree/release-2.3

...

• For how long will Dovecot version 2.3 still be supported (security fixes, bug fixes)? Is there any EOL plan?

This will be informed later, but as general rule, once we make a new major release, 2.3 will go into maintenance mode, and will receive only select bug fixes and CVE fixes.

...

Thanks for any clarification! Steff

Aki

SteveT

Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm

Most small/medium servers do not need director. You can use replicator get a pri/bu pair.

What is small?

On 20/10/2022 22:00 EEST Steve Litt slitt@troubleshooters.com wrote:

Aki Tuomi said on Thu, 20 Oct 2022 21:41:53 +0300 (EEST)

...

Most small/medium servers do not need director. You can use replicator get a pri/bu pair.

I've never needed to use replicator. I don't even know what a pri/bu pair is. I just have fetchmail feed to procmail which delivers messages into my Dovecot maildir, and then access the Dovecot IMAP server with an email client. Hopefully I'll be able to continue doing it this way.

...

Only the director part is being removed, rest of Dovecot remains. For the next major release we are also removing certain deprecated parts that have a replacement in elsewhere of the code.

Is there a document on the deprecations and their replacements? I'd like to read it.

...

The mail server functionality is going to remain 100% open source and free.

The preceding sentence is a huge relief for me. Thanks!

SteveT

https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/

This is subject to change, as we have not actually released this version yet.

Aki

My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away.

In other scenarios, the “pri\bu” means primary and backup DC instances which should be fine for many folks who just have a single server.

Again this is my understanding so please feel free to correct me where I’m off-base…

On 20 Oct 2022, at 12:00, Steve Litt wrote:

Aki Tuomi said on Thu, 20 Oct 2022 21:41:53 +0300 (EEST)

...

Most small/medium servers do not need director. You can use replicator get a pri/bu pair.

I've never needed to use replicator. I don't even know what a pri/bu pair is. I just have fetchmail feed to procmail which delivers messages into my Dovecot maildir, and then access the Dovecot IMAP server with an email client. Hopefully I'll be able to continue doing it this way.

...

Only the director part is being removed, rest of Dovecot remains. For the next major release we are also removing certain deprecated parts that have a replacement in elsewhere of the code.

Is there a document on the deprecations and their replacements? I'd like to read it.

...

The mail server functionality is going to remain 100% open source and free.

The preceding sentence is a huge relief for me. Thanks!

SteveT

Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm

Please post your solution.

Sent from my iPhone - please excuse brevity and typos

On Oct 20, 2022, at 10:21 PM, Zhang Huangbin zhb@iredmail.org wrote:



...

On Oct 21, 2022, at 4:19 AM, Antonio Leding tech@leding.net wrote:

My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away.

Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers.

I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case.

It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster.

---

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

You still need in some sense one coherent file system to store and retrieve the mail messages. Although a load-balance cluster would still be quite useful for rejecting the bulk of unauthorized connections.

I am sure in many cases a small/medium server can in fact sit and function quite adequately behind a large enterprise load balancing firewall and proxy, given the typical quantities of spam "out there" and the large number of bad connections typically attempted on any given system.

On Thursday, October 20, 2022 9:19:59 PM AKDT, Zhang Huangbin wrote:

...

On Oct 21, 2022, at 4:19 AM, Antonio Leding tech@leding.net wrote:

My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away.

Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers.

I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case.

It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster.

---

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

On 2022-10-21 06:19, Zhang Huangbin wrote:

...

On Oct 21, 2022, at 4:19 AM, Antonio Leding tech@leding.net wrote:

My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away.

Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers.

I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case.

It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster.

---

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

Hi,

I was wondering if one can achieve the same implementation with haproxy without dovecot director? Load balancing all requests to pop3, imap, managesieve and lmtp services from specified frontend servers i.e. webmail to specified backend servers and using NFS mount filesystem/syncing data across all servers to access emails with high availability?

Not sure whats the big deal director is offering? Is it just a native functionality providing a feature to find which backend server have X emails available and chooses to load from e.g. its content i.e. like checks which first server that doesnt return http 404 response equivalent in IMAP/POP3/LMTP/ManageSieve?

Sometime ago I used Varnish caching directors to implement high availability using 404 response status in http web server, and it seems great if we can have this feature in dovecot too, as it offers high availability with delayed-syncing/partial-syncing across unknown selected servers, I managed to use Varnish too in dovecot proxy service i.e. the webmail, yet it requires NFS mount or high available file system all servers can have through immediate access to e.g. maildir?

Any helpful input that would clear the picture for me in regards dovecot director, would be ver much appreciated.

With thanks.

Zakaria.

On Oct 21, 2022, at 5:51 PM, Zhang Huangbin zhb@iredmail.org wrote:

If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue.

To be clear: Accessing same mailbox from different IMAP servers __at the same time__.

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

On 2022-10-21 10:51, Zhang Huangbin wrote:

...

On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote:

I was wondering if one can achieve the same implementation with haproxy without dovecot director?

The most important part of Director is it makes sure same mail user always proxied to same backend IMAP server.

If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue.

HAProxy can proxy mail user from same client IP to same backend IMAP server, but not same mail user from different IPs.

Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/):

"Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a temporary user -> mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem."

---

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

Aha makes sense, although I was not able to see how can index files be corrupted when its if will going to be updated, its in same manner as from different connection, e.g. opening email account from different app clients, with different connections, does not corrupt the index files?

Also, Is it the issue Director resolving as well its with maintaining the logged in dovecot connection to same backend? Anyhow, thanks for your valuable efforts in clearing this :)

I wondered if there is any other solution to avoid corrupting index files? Perhaps if dovecot offer database indexing as well as login sessions, it seems that this would eliminate Director requirement, and offer better high availability, as for now userdb/authdb is only available per my knowledge, and using database cluster resolves the issue with user and auth queries during simultaneous connections to a different backends.

Otherwise, it seems in large enterprise deployment with high availability a Director implementation will be needed, hopefully we will find an alternative solution by the time Dovecot 3 is released.

I might need to get my head around building dovecot with customised modules and review the code which was removed and return it back, if anyone is planning to this, and well off ahead of me, please let me know, we might be able to help one another.

With thanks.

Zakaria.

To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature.

Aki

On 21/10/2022 14:14 EEST Amol Kulkarni amolk112k@gmail.com wrote:

Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ?

On Fri, 21 Oct 2022 at 16:21, hi@zakaria.website wrote:

...

On 2022-10-21 10:51, Zhang Huangbin wrote:

...

...

On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote:

I was wondering if one can achieve the same implementation with haproxy without dovecot director?

The most important part of Director is it makes sure same mail user always proxied to same backend IMAP server.

If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue.

HAProxy can proxy mail user from same client IP to same backend IMAP server, but not same mail user from different IPs.

Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/):

"Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a temporary user -> mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem."

---

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

Aha makes sense, although I was not able to see how can index files be corrupted when its if will going to be updated, its in same manner as from different connection, e.g. opening email account from different app clients, with different connections, does not corrupt the index files?

Also, Is it the issue Director resolving as well its with maintaining the logged in dovecot connection to same backend? Anyhow, thanks for your valuable efforts in clearing this :)

I wondered if there is any other solution to avoid corrupting index files? Perhaps if dovecot offer database indexing as well as login sessions, it seems that this would eliminate Director requirement, and offer better high availability, as for now userdb/authdb is only available per my knowledge, and using database cluster resolves the issue with user and auth queries during simultaneous connections to a different backends.

Otherwise, it seems in large enterprise deployment with high availability a Director implementation will be needed, hopefully we will find an alternative solution by the time Dovecot 3 is released.

I might need to get my head around building dovecot with customised modules and review the code which was removed and return it back, if anyone is planning to this, and well off ahead of me, please let me know, we might be able to help one another.

With thanks.

Zakaria.

To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. To be realy clear, you are not removing the proxy feature in dovecot that can be used to proxy users to different backend server on which the users mailboxes are stored?

Thanks Oliver

Aki

On 21/10/2022 14:14 EEST Amol Kulkarni amolk112k@gmail.com wrote:

Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ?

On Fri, 21 Oct 2022 at 16:21, hi@zakaria.website wrote:

...

...

...

On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote:

I was wondering if one can achieve the same implementation with
haproxy without dovecot director?

The most important part of Director is it makes sure same mail user > always proxied to same backend IMAP server.

If mailbox is in Maildir format (and stored on shared storage

On 2022-10-21 10:51, Zhang Huangbin wrote: like > NFS), accessing it from different server may corrupt Dovecot index > files and mailbox becomes unaccessible. Director perfectly avoids this > issue.

...

HAProxy can proxy mail user from same client IP to same backend

IMAP > server, but not same mail user from different IPs.

...

Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/):

"Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a
temporary user -> mail server mapping. As long as user has

simultaneous > connections, the user is always redirected to the same server. Each > proxy server is running its own director process, and the directors are > communicating the state to each others. Directors are mainly useful for > setups where all of the mail storage is seen by all servers, such as > with NFS or a cluster filesystem."

...

---

Zhang Huangbin, founder of:

• iRedMail: Open source email server solution: https://www.iredmail.org/
• Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io

Aha makes sense, although I was not able to see how can index files be corrupted when its if will going to be updated, its in same manner as from different connection, e.g. opening email account from different app clients, with different connections, does not corrupt the index files?

Also, Is it the issue Director resolving as well its with maintaining the logged in dovecot connection to same backend? Anyhow, thanks for your valuable efforts in clearing this :)

I wondered if there is any other solution to avoid corrupting index
files? Perhaps if dovecot offer database indexing as well as login
sessions, it seems that this would eliminate Director requirement, and offer better high availability, as for now userdb/authdb is only available per my knowledge, and using database cluster resolves the issue with user and auth queries during simultaneous connections to a different backends.

Otherwise, it seems in large enterprise deployment with high
availability a Director implementation will be needed, hopefully we will find an alternative solution by the time Dovecot 3 is released.

I might need to get my head around building dovecot with customised
modules and review the code which was removed and return it back, if
anyone is planning to this, and well off ahead of me, please let me
know, we might be able to help one another.

With thanks.

Zakaria.

On 26/10/2022 11:41 EEST Narcis Garcia debianlists@actiu.net wrote:

El 26/10/22 a les 10:29, MK ha escrit:

...

...

To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. To be realy clear, you are not removing the proxy feature in dovecot that can be used to proxy users to different backend server on which the users mailboxes are stored?

Thanks Oliver

Does this removal include or exclude IMAP backends?

--

Narcis Garcia

No. The only thing removed is the director component. Proxying works, IMAP backends are not removed. Director is responsible for mapping users to particular hosts.

This change will affect mostly people with more than 1-2 backends, with 2 backends you can still have primary/backup setup. Dovecot will still happily proxy connections to your backends.

AKi

On 26/10/2022 12:42 EEST Narcis Garcia debianlists@actiu.net wrote:

El 26/10/22 a les 10:51, Aki Tuomi ha escrit:

...

...

On 26/10/2022 11:41 EEST Narcis Garcia debianlists@actiu.net wrote:

El 26/10/22 a les 10:29, MK ha escrit:

...

...

To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. To be realy clear, you are not removing the proxy feature in dovecot that can be used to proxy users to different backend server on which the users mailboxes are stored?

Thanks Oliver

Does this removal include or exclude IMAP backends?

--

Narcis Garcia

No. The only thing removed is the director component. Proxying works, IMAP backends are not removed. Director is responsible for mapping users to particular hosts.

This change will affect mostly people with more than 1-2 backends, with 2 backends you can still have primary/backup setup. Dovecot will still happily proxy connections to your backends.

AKi

Mmhh what about this for same FQDN? one@example.net -> local Dovecot mailbox two@example.net -> local Dovecot mailbox three@example.net -> Specific IMAP backend four@example.net -> local Dovecot mailbox

--

Narcis Garcia

Still gonna work.

Aki

Maciej Milaszewski schreef op 2022-10-26 11:52:

Hi What is the planned replacement like

doveadm director status move / kick / flush add /up / del

In 3.0 ?

This question has been answered in the thread.

Will there be a fork dovecot ?

If we, the community, start one, yes.

-- With kind regards,

William Edwards

Director never worked especially well, and for most use cases it's just unnecessarily complex. I think usually it could be replaced with:

• Database (sql/ldap/whatever) containing user -> backend table.
• Configure Dovecot proxy to use this database as passdb.
• For HA change dovemon to update the database if backend is down to move users elsewhere
• When backend comes up, move users into it. Set delay_until extra field for user in passdb to 5 seconds into future and kick the user in its old backend (e.g. via doveadm HTTP API).

All this can be done with existing Dovecot. Should be much easier to build a project doing this than forking director.

TL;DR: 🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩

Sure, this affects medium/large/Enterprise folks (that's where I was using Director -- though currently retired, so no existing self-interest in this email).

This will also affect *any* installation with a whopping two dovecot servers with mdbox backends talking to a single linux NFS server as well. That's not exactly "Enterprise". Replication is great, but it is not a replacement for Director (nor is any sort of load balancing, regardless of the confused comments in this thread about nginx).

I think the real issue here is that Dovecot is removing *existing, long-standing, critical functionality* from the open source version. That is a huge, huge red flag.

I'm also a little bewildered by the comment "Director never worked especially well". Worked great for me, at scale, for years. Complex? Yup, but that was the price of mdbox (worth it). And if you're setting up a proxy cluster (instead of a full Director cluster) in front of your IMAP servers, you've already tackled 90% of the complexity anyway (i.e. using Director isn't the hard part).

This *feels" to me like a parent company looking to remove features from the open source version in order to add feature differentiation to the paid version.

I've loved the Dovecot project for over a decade and a half. And incidentally I have a very warm spot in my heart for Timo and Aki, thanks to Dovecot and especially this mailing list.

I've also loved the PowerDNS project for a decade and a half, so this removal of *existing functionality* is doubly worrisome. I'd like both projects to be monetisable and profitable enough to their parent so that they continue on for a very, very long time.

But removing long-standing features is a bad look. Please reconsider this decision.

On Thu, Oct 27, 2022 at 4:04 AM Jan Bramkamp crest@rlwinm.de wrote:

...

Director never worked especially well, and for most use cases it's just unnecessarily complex. I think usually it could be replaced with:

• Database (sql/ldap/whatever) containing user -> backend table.
• Configure Dovecot proxy to use this database as passdb.
• For HA change dovemon to update the database if backend is down to move users elsewhere
• When backend comes up, move users into it. Set delay_until extra field for user in passdb to 5 seconds into future and kick the user in its

On 27.10.22 04:24, Timo Sirainen wrote: old backend (e.g. via doveadm HTTP API).

...

All this can be done with existing Dovecot. Should be much easier to

build a project doing this than forking director. Thank you for putting what is about to be lost to the community edition into an operational perspectiv: no reason to panic. Nobody is taking replicated active-passive pairs from small to medium scale operators. Neither are the hooks required for more fancy load balancing and steering on the chopping block.

On 2022-11-01 16:58, Mark Moseley wrote:

TL;DR: 🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩🚩

I think the real issue here is that Dovecot is removing *existing, long-standing, critical functionality* from the open source version. That is a huge, huge red flag.

It certainly looks like a poor decision, driven by corporate interests. Makes me wonder which other feature will be moved to the commercial edition once the dust has settled.

It really hurts the great reputation Dovecot has built over all these years. I've got my first Dovecot installation back in ~2006 and ever since I've been advocating it as the best IMAP server. So sad to see this feature removal now.

Ciao

• Frank

On 2022-11-01 16:58, Mark Moseley wrote:

This *feels" to me like a parent company looking to remove features from the open source version in order to add feature differentiation to the paid version.

I've loved the Dovecot project for over a decade and a half. And incidentally I have a very warm spot in my heart for Timo and Aki, thanks to Dovecot and especially this mailing list.

I've also loved the PowerDNS project for a decade and a half, so this removal of _existing functionality_ is doubly worrisome. I'd like both projects to be monetisable and profitable enough to their parent so that they continue on for a very, very long time.

But removing long-standing features is a bad look. Please reconsider this decision.

Big +1

Tom

On 2022-11-02 09:11, Aki Tuomi wrote:

You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does.

It's not the same thing.

It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director.

It seems to be critical enough to offer a replacement for paying customers, while at the same time leaving the community edition with no valid replacement.

Ciao

• Frank

On 11/2/22 03:54, Aki Tuomi wrote:

...

On 02/11/2022 11:55 EET Frank Wall fw@moov.de wrote:

On 2022-11-02 09:11, Aki Tuomi wrote:

...

You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does. It's not the same thing.

...

It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director. It seems to be critical enough to offer a replacement for paying customers, while at the same time leaving the community edition with no valid replacement.

Ciao

• Frank Can you tell me what kind of functionality you are unable to achieve with the passdb solution?

Aki

Can you tell us what you are gaining (other than monitarily) by removing a completely functionally working feature that numerous people are using?

Adding new paid features is one thing (i.e. nginx), taking away a feature to replace it with a paid feature is something completely different.

-- Brad

It would certainly be a shame if that sort of thing started happening with Dovecot. Since day one, the Dovecot community has always been very pleasant, friendly, and drama-free. If forks start happening due to profiteering, that will irrevocably change the Dovecot community, with feelings of broken trust.

That would be a shame.

No one decries the commercial side of Dovecot wanting to make money. Timo and others have worked very hard on this project for many years. I was a very early adopter of Dovecot, a refugee from (the awful) Cyrus IMAP server, and I watched it grow up to be a highly useful and widely respected package. Creating a commercial version to reward the developers and fund future development is fine; I applaud it.

But it really smells like the current move with Director is crossing a line.

Those in charge of making this decision would do well to pay very close attention here.

         -Dave

On 11/2/22 12:46, Jan Hugo Prins wrote:

I think the only thing they will gain is a community that is angry and will in the end leave the product / fork the complete product.

Jan Hugo

On November 2, 2022 5:39:53 PM GMT+01:00, Brad Schuetz brad@omnis.com wrote:

On 11/2/22 03:54, Aki Tuomi wrote:

        On 02/11/2022 11:55 EET Frank Wall <fw@moov.de> wrote:

        On 2022-11-02 09:11, Aki Tuomi wrote:

            You can also see the email sent by others which shows
            how you can do
            this without replication, using proxy and passdb to
            direct users to
            right backend. Which is basically what director does.

        It's not the same thing.

            It is not critical functionality. You can feasibly run a
            two-node
            dovecot system on NFS without having director.

        It seems to be critical enough to offer a replacement for paying
        customers, while at the same time leaving the community edition
        with no valid replacement.


        Ciao
        - Frank

    Can you tell me what kind of functionality you are unable to
    achieve with the passdb solution?

    Aki


Can you tell us what you are gaining (other than monitarily) by removing a completely functionally working feature that numerous people are using?

Adding new paid features is one thing (i.e. nginx), taking away a feature to replace it with a paid feature is something completely different.

-- 
Brad

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- Dave McGuire, AK4HZ New Kensington, PA

If the community has enough resources to fork the whole project, it would probably be far more efficient and easier to just fork the Director component.

I’m not familiar enough with dovecot sources to tell if this is possible, but if the community really wants to keep Director alive, maybe it should start investigating if building it as an out of tree component is possible ?

Le 2 nov. 2022 à 17:46, Jan Hugo Prins jhp@jhprins.org a écrit :

I think the only thing they will gain is a community that is angry and will in the end leave the product / fork the complete product.

Jan Hugo

On November 2, 2022 5:39:53 PM GMT+01:00, Brad Schuetz brad@omnis.com wrote: On 11/2/22 03:54, Aki Tuomi wrote: On 02/11/2022 11:55 EET Frank Wall fw@moov.de wrote:

On 2022-11-02 09:11, Aki Tuomi wrote: You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does. It's not the same thing.

It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director. It seems to be critical enough to offer a replacement for paying customers, while at the same time leaving the community edition with no valid replacement.

Ciao

• Frank Can you tell me what kind of functionality you are unable to achieve with the passdb solution?

Aki

Can you tell us what you are gaining (other than monitarily) by removing a completely functionally working feature that numerous people are using?

Adding new paid features is one thing (i.e. nginx), taking away a feature to replace it with a paid feature is something completely different.

-- Brad

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Nginx is an excellent suggestion for the purpose. However I do not like German client certificates. That is far too much "proof" of identification 18/21++ on a public network with nowhere to hide and those of us who are not German citizens and do not have the advantage of a friendly local police jurisdiction with massive international clout and an assumed legitimacy for all the online surveillance, policing, and copping with unfounded sex charges etc. being pressed online.

Not that I care much for alcohol, but the analogy that comes to mind with such "proof" of identity presented across the internet as a public certificate is that of "public drunkenness," versus, say, "drinking privately in one's quarters," i.e., making an encrypted connection, and only then within the encrypted channel establishing identity and authorization with a username and password or other means of authentication.

On Friday, October 21, 2022 3:29:36 AM AKDT, spi wrote:

Am 21.10.22 um 13:14 schrieb Amol Kulkarni:

...

Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ?

Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you can define a backend server the session is routed to. Currently I use that approach to authenticate users by client certificates and route them to the appriopriate backend (well, I only have one ;-).

-- Cheers spi

Op 21 okt. 2022 om 19:42 heeft Brendan Braybrook brendan@tucows.com het volgende geschreven:

On 2022-10-21 04:29, spi wrote:

...

...

Am 21.10.22 um 13:14 schrieb Amol Kulkarni: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you can define a backend server the session is routed to. Currently I use that approach to authenticate users by client certificates and route them to the appriopriate backend (well, I only have one ;-).

we've recently switched to director, but we used to use nginx for this as well (we started using nginx before director existed). if you load balance the nginx proxies themselves, you can easily handle hundreds of thousands of concurrent imap connections with them.

in debian/ubuntu, i don't think the nginx packages include the mail proxy bits. iirc, we had to compile nginx ourselves with the mail proxy bits included.

the nginx config is pretty simple, you have to pre-specifiy the capabilities for each protocol and set up some sort of way for nginx to auth and get which backend node to send to as spi notes (in this example, it's an http call):

mail { auth_http localhost:8080/cgi-bin/auth; proxy_pass_error_message on;

pop3_capabilities "TOP" "UIDL" "RESP-CODES" "PIPELINING" "AUTH-RESP-CODE" "USER" "SASL PLAIN" "SASL PLAIN LOGIN"; server { listen 110; protocol pop3; proxy on; }

imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS" "IDLE"; server { listen 143; protocol imap; proxy on; } }

localhost:8080/cgi-bin/auth then just auths the user/pass that nginx gets from the incoming request and returns success and the next hop for nginx to proxy to.

the only real difficulty is that you then need to write your own state system into your cgi auth script to ensure that users get sent to the same backend imap server if they already have an existing connection and have some way to safely fail over to other backend imap servers should one go down. (it's nice to have director handle this state stuff for you)

Although Director does not do health checks and down servers automatically. I was working on an open source program for that (as an alternative to Dovemon), but that plan is canceled with this announcement :)

...

I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case.

It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster.

Curious, trying to understand..

Why would not a true load balancer not be an attractive option for those that need to load balance services across multiple front ends?

It is the model we use with most of our ISP's and scales very well.

The choice of load balancer is important, but with HA load balancers, you are assured that you don't have a single point of failure, and you can spread loads more granularly, eg POP, IMAP and other services.

Not to mention, you can use the same load balancer from many other traffic shaping solutions.

the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted)

in more usual HTTP case, you'd probably use some sort of cookie based session affinity to keep connections from a particular user going to the same backend http server.

but in the IMAP/POP case most load balancers don't really know anything about the connection and are just blindly forwarding them to the backend nodes. director (or the custom nginx LB setups) get to handle part of the IMAP/POP transaction and get a bit of context (knowing which user the connection is for) to then make additional decisions about which backend imap node to send the connection through to (preventing the index corruption problem).

you could use IP based affinity on pop/imap connections for a context-unaware load balancer, but if you end up with a lot of NAT users your connections will end up being unbalanced across the backend servers. and connections from something like a webmail server will all end up going to the same backend server (since they'd all come from the same IP address).

you could also just have a dumb load balancer sitting in front and just randomly sending the connections to any backend imap server, but each backend imap server would have to maintain its own copy of the indexes. workable, but not particularly efficient, especially if you have large a large number of backend imap servers (though, with a small setup with only 2 or 3 backend imap servers for redundancy instead of performance, probably acceptable)

you'd still want some sort of load balanced director or nginx pool as well, in order to handle redundancy at that level. but that's a much easier task, as you don't have to worry about the session context at that point. (we have hardware load balancers in front of the director nodes)

On 2022-10-21 13:25, dovecot@ptld.com wrote:

...

the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted)

As someone else asked on this thread, what prevents two clients, both being directed to the same server, from fighting over index files? Wouldn't file locks over NFS prevent this problem? And if so, doesn't that also prevent two dovecot installations from fighting over index files?

i believe the dovecot processes have some sort of interprocess communication when they are running on the same host that they use to negotiate writes to the index files. i don't really know the details, other than that the index files get corrupted very quickly if multiple hosts are accessing them at once. the index files are fine if a users' multiple imap sessions are on a single host.

iirc, dovecot does use file locks when moving/deleting maildir+ message files. but that's not really the issue - it's all about the index files.

those index files just weren't designed to have parallel access from multiple machines.

What is a way to test your system to know if dovecot is using the default fcntl file locks over NFS4 and they actually work? Or is it better/safer to use dotlock on NFS4 without director?

nfs locks do work, as long as your nfs server supports them well. dotlocks don't require any nfs server support, but they are slower. but, for the most part if you are redirecting users sessions to the same server it doesn't matter. we've used both locking types, though dotlocks were more reliable on some nfs servers.

as long as you aren't using DBOX for mail storage, having the indexes get corrupted isn't the end of the world - dovecot will just regenerate them (though you might have to remove the broken files and kill the users' sessions to force this).

there's some dovecot documentation with suggestions: https://doc.dovecot.org/configuration_manual/nfs/ and some older docs: https://wiki1.dovecot.org/NFS

On 2022-10-21 11:38, Heiko Schlittermann wrote:

...

Apparently, Dovecot Director is going to be removed in the next major version of Dovecot and the commercial Dovecot cluster architecture will be its successor:

We - the communitiy - are free to continue development of the director.

So, who's going to fork dovecot (director)?

Ciao

• Frank

On 2022-10-24, Alessio Cecchi alessio@skye.it wrote:

Director is not only used by large companies but also in small installations consisting of 2 servers and cannot be immediately replaced with Nginx as it has to manage the user/backend association for POP, IMAP, LMTP, Managesieve.

For the small multi-server installations I've done I have used ldap (though another db would work) where a primary server is defined for each user. The MTA does a lookup and uses the relevant host as destination for LMTP delivery. For client connections, users can connect to any server; Dovecot config uses proxy_maybe so if they hit the primary server for their mailbox then it's served directly, and otherwise it's proxied. (And in my case I care more about availability than splitting disk storage, so I replicate in Dovecot). This doesn't use Director.

Isn't Director only really useful in the case where you have 2 or more servers *and shared mailbox storage*, and you don't have a way to define a "primary" server for the mailbox? I don't really see how it's useful for simpler configs.