Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0 (https://doc.dovecot.org/3.0/settings/core/#core_setting-mail_attachment_fs):
Changed in version 2.4.0 (CE): SIS is deprecated and writing of SIS files is disabled. Reading is supported for now, any missing SIS attachments are replaced with files filled with spaces.
Changed in version 3.0.0 (Pro): SIS is deprecated and writing of SIS files is disabled. Reading is supported for now, any missing SIS attachments are replaced with files filled with spaces.
And https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/ says:
Saving new mails’ attachments via fs-sis is disabled, but reading SIS attachments is still supported. Missing SIS attachments are replaced with files filled with spaces.
What is not entirely clear from this comment is whether this only applied to “sis posix” or also to “sis-queue posix”. Does anybody whether sis-queue is also affected by this?
I saw that the notice was added to the dovecot/documentation Git repository in April 2023, but I couldn’t find any further information there or the main branch of the dovecot/core repository. Maybe, the code for 2.4 / 3.0 has not made it into that branch or repository yet.
SIS sounds like a neat concept and I would like to use it, but obviously I do not want to use a soon-to-be-deprecated feature in a completely new deployment. So, does anybody know an alternative solution for attachment deduplication (except file-system level deduplication)?
Thanks, Sebastian
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching <sebastian@marsching.com> wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you, Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot <dovecot@dovecot.org> wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching <sebastian@marsching.com> wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
FUD ?
I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server"
------- Original Message ------- On Friday, October 13th, 2023 at 19:37, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you, Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebastian@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
On 10/14/23 03:26, Laura Smith via dovecot wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server"
Yes.
Aki, it would be much appreciated if you can deliver your point in the form of a clarification of what the product manager actually said in that video, in very clear language.
Thanks!
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot <dovecot@dovecot.org> wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server"
------- Original Message ------- On Friday, October 13th, 2023 at 19:37, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you, Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebastian@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
16.10.2023 10:30, Michael Slusarz via dovecot пишет:
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
Well, so Laura is absolutely right ...
To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot <dovecot@dovecot.org> wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server"
------- Original Message ------- On Friday, October 13th, 2023 at 19:37, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you, Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebastian@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
On 2023-10-16, Dmitry Melekhov <dm@belkam.com> wrote:
16.10.2023 10:30, Michael Slusarz via dovecot пишет:
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
Well, so Laura is absolutely right ...
"Things like dsync will be GONE in the community version."
That's not right, dsync is still there. Replicator is not, so dsync can't be triggered automatically by dovecot after changes to the mailbox (which is very handy for a simple "nearly live" Dovecot-aware backup mechanism when used on what is mostly treated as a single server setup, but with a replica on another machine).
There are other ways to run dsync, though not as convenient.
I'm still trying to decide how best to do this after replicator is gone. If the community version had obox (s3-compatible) storage I think I'd probably use that and e.g. replicated minio as backend, this seems like it would be more robust. As it is, I'll probably scan mailboxes for timestamp changes every few minutes and fire off dsync as needed.
What is being removed is the replicator plugin (that used dsync).
That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
Well, so Laura is absolutely right ...
"Things like dsync will be GONE in the community version."
That's not right, dsync is still there. Replicator is not, so dsync can't be triggered automatically by dovecot after changes to the mailbox (which is very handy for a simple "nearly live" Dovecot-aware backup mechanism when used on what is mostly treated as a single server setup, but with a replica on another machine).
There are other ways to run dsync, though not as convenient.
I'm still trying to decide how best to do this after replicator is gone. If the community version had obox (s3-compatible) storage I think I'd probably use that and e.g. replicated minio as backend, this seems like it would be more robust. As it is, I'll probably scan mailboxes for timestamp changes every few minutes and fire off dsync as needed.
Is s3 not to slow for this?
Is s3 not to slow for this?
I think the clue is in the name "s3-compatible".
Clearly calling out to "real" (AWS) S3 would be a non-starter.
But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ?
Do you know of anything that does this reliably?
I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly.
Is s3 not to slow for this?
I think the clue is in the name "s3-compatible".
Clearly calling out to "real" (AWS) S3 would be a non-starter.
But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ?
Do you know of anything that does this reliably?
I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency. The actual system is not perfect, but simple and robust and there is room for improvements. Dovecot is a very solid base. The replicator removing is sad, but aligned with the business pressure open-xchange business: Their customers are only big players all-ready wasting lots of money on "enterprise grade" architecture and big and expensive cloud services. It is perfectly aligned with their customer view of efficiency... At the same time, this customers have a very distorted view of the cost of pure service. They want a price at the lowest common denominator of human cost over the planet... I think that open-xchange only try to off cut all possible maintenance cost from the "product". It it too much ? Will it pay off ? Who live will see... From over more
Le 16/10/2023 à 19:44, Marc a écrit : than thirty years of experience of observing the open source world, projects that achieve "world domination" are NEVER the ones that sacrifices the technical goal for business reasons/pressure even if it take decades to be recognized for it true value.
Now that Exchange has a robust multinode replication and distribution system dovecot will loose it own. Ironic isn't it :-)
Emmanuel.
Day 16. 10. 2023 21:30, Emmanuel Fusté <manu.fuste@gmail.com> wrote:
Le 16/10/2023 à 19:44, Marc a écrit :
Is s3 not to slow for this?
I think the clue is in the name "s3-compatible".
Clearly calling out to "real" (AWS) S3 would be a non-starter.
But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ?
Do you know of anything that does this reliably?
I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency.
S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage.
Is s3 not to slow for this?
I think the clue is in the name "s3-compatible".
Clearly calling out to "real" (AWS) S3 would be a non-starter.
But a local installation of something like CEPH, MinIO or whatever on
the
same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably?
I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency.
S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage.
The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication.
Le 17 oct. 2023 à 13:12, Marc <Marc@f1-outsourcing.eu> a écrit :
Is s3 not to slow for this?
I think the clue is in the name "s3-compatible".
Clearly calling out to "real" (AWS) S3 would be a non-starter.
But a local installation of something like CEPH, MinIO or whatever on the same LAN ? I'd think that should be workable, no ? Do you know of anything that does this reliably?
I tested a few years ago with ceph[1] but at that point there was some issues where it had a 2x write applification (on top of the 3x) if I remember correctly. All of this is if not dead end will be a lots of complexity and inefficiency and a lot of waste of money. Only the application know how to things efficiently and with consistency.
S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage.
The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication.
[1] https://docs.ceph.com/en/reef/dev/deduplication/ <https://docs.ceph.com/en/reef/dev/deduplication/>
Moreover, following Filip remark about block deduplication, having any kind of deduplication that is not optimized for the email case (where attachments are always embed in slightly different documents) would make it ineffective. Is it really worse bothering deploying a whole Ceph cluster for that ?
Le 17 oct. 2023 à 13:12, Marc <Marc@f1-outsourcing.eu> a écrit :
Is s3 not to slow for this?
I think the clue is in the name "s3-
compatible".
Clearly calling out to "real" (AWS) S3
would be a non-starter.
But a local installation of something
like CEPH, MinIO or whatever on
the
same LAN ? I'd think that should be
workable, no ?
Do you know of anything that does this reliably?
I tested a few years ago with ceph[1] but at that
point there was some
issues where it had a 2x write applification (on top of the 3x) if I
remember correctly.
All of this is if not dead end will be a lots of complexity
and
inefficiency and a lot of waste of money. Only the application know
how to
things efficiently and with consistency.
S3-compatible storage is very good for multi-server installations
where you
need redundancy, availability. S3 is basically HTTP server so you can
code
your own logic on stored emails, balancers, caches, deduplication,
compression, encryption it does't need to be off-the-shelf storage.The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication.
[1] https://docs.ceph.com/en/reef/dev/deduplication/
Moreover, following Filip remark about block deduplication, having any kind of deduplication that is not optimized for the email case (where attachments are always embed in slightly different documents) would make it ineffective. Is it really worse bothering deploying a whole Ceph cluster for that ?
The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication.
[1] https://docs.ceph.com/en/reef/dev/deduplication/
Moreover, following Filip remark about block deduplication, having any kind of deduplication that is not optimized for the email case (where attachments are always embed in slightly different documents) would make it ineffective.
Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Is it really worse bothering deploying a whole Ceph cluster for that ?
No you should not get ceph just for this. But ceph brings you nice redundancy, distributed storage. I am totally fan of it.
Le 17 oct. 2023 à 16:34, Marc <Marc@f1-outsourcing.eu> a écrit :
The problem is a bit what everyone understands as s3. I associate this indeed also with an http endpoint on object storage. But the ceph plugin skips this http and talks directly to object store. I don't think you would like to operate on this http level. If I look at this page of ceph[1], it also looks like you do not want to get yourself involved in deduplication.
[1] https://docs.ceph.com/en/reef/dev/deduplication/
Moreover, following Filip remark about block deduplication, having any kind of deduplication that is not optimized for the email case (where attachments are always embed in slightly different documents) would make it ineffective.
Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Interesting. How do you tell dovecot to do that ?
Is it really worse bothering deploying a whole Ceph cluster for that ?
No you should not get ceph just for this. But ceph brings you nice redundancy, distributed storage. I am totally fan of it.
Me too. I’m using it extensively to store multi terabytes of data, but it may be overkill if you don’t need all of this.
Le 17 oct. 2023 à 16:34, Marc <Marc@f1-outsourcing.eu> a écrit :
The problem is a bit what everyone understands as s3. I associate
this indeed also with an http endpoint on object storage. But the
ceph
plugin skips this http and talks directly to object store. I don't
think
you would like to operate on this http level. If I look at this page
of
ceph[1], it also looks like you do not want to get yourself involved
in
deduplication.
[1]
https://docs.ceph.com/en/reef/dev/deduplication/
Moreover, following Filip remark about block deduplication, having
any kind
of deduplication that is not optimized for the email case (where
attachments are always embed in slightly different documents) would
make it
ineffective.Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Interesting. How do you tell dovecot to do that ?
Is it really worse bothering deploying a whole Ceph cluster
for that ?
No you should not get ceph just for this. But ceph brings you nice
redundancy, distributed storage. I am totally fan of it.Me too. I’m using it extensively to store multi terabytes of data, but it may be overkill if you don’t need all of this.
Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Interesting. How do you tell dovecot to do that ?
I thought I read about something like this,
mail_location = .... ATTACHMENTS=/attachment
but now you have made me read the docs[1] I can't really find it.
@Aki maybe if this SIS is phased out, it is good to offer a solution that stores the attachments separately? I think that would allow current SIS users to implement something alternative.
[1] https://doc.dovecot.org/configuration_manual/mail_location/#
Le 18 oct. 2023 à 09:35, Marc <Marc@f1-outsourcing.eu> a écrit :
Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Interesting. How do you tell dovecot to do that ?
I thought I read about something like this,
mail_location = .... ATTACHMENTS=/attachment
but now you have made me read the docs[1] I can't really find it.
@Aki maybe if this SIS is phased out, it is good to offer a solution that stores the attachments separately? I think that would allow current SIS users to implement something alternative.
Thanks for the pointer. Thanks to it, I found it in the documentation. It was supposed to be defined like this in v2.0.0, but it is now a core setting (and is only available for sd/mdbox storage):
mail_attachment_dir • Default: <empty> • Values: String The directory in which to store mail attachments.
With sdbox and mdbox, mail attachments can be saved to external files, which also allows single-instance storage of them.
If no value is specified, attachment saving to external files is disabled.
On 2023-10-18 3:35 a.m., Marc wrote:
Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Interesting. How do you tell dovecot to do that ?
I thought I read about something like this,
mail_location = .... ATTACHMENTS=/attachment
but now you have made me read the docs[1] I can't really find it.
@Aki maybe if this SIS is phased out, it is good to offer a solution that stores the attachments separately? I think that would allow current SIS users to implement something alternative.
https://doc.dovecot.org/settings/core/#core_setting-mail_attachment_dir
I hope this option will not be obsoleted.
On 2023-10-18 3:35 a.m., Marc wrote: Dovecot has this option to store attachments separately not? So I am not sure this is then still a problem.
Interesting. How do you tell dovecot to do that ?I thought I read about something like this,
mail_location = .... ATTACHMENTS=/attachment
but now you have made me read the docs[1] I can't really find it.
@Aki maybe if this SIS is phased out, it is good to offer a solution that stores the attachments separately? I think that would allow current SIS users to implement something alternative.
https://doc.dovecot.org/settings/core/#core_setting-mail_attachment_dir
I hope this option will not be obsoleted.
17.10.2023 12:22, Filip Hanes via dovecot пишет:
S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage.
is S3 better then cephfs?
17.10.2023 12:22, Filip Hanes via dovecot пишет: S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the- shelf storage.
is S3 better then cephfs?
17.10.2023 12:22, Filip Hanes via dovecot пишет:
S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the-shelf storage.
is S3 better then cephfs?
The drawbacks of cephfs is you need to have the mds. If you scale the mds you could have some issues. I think even in newer ceph releases you need to start pin them on pools / directories. I still have issues with cephfs mounts locking up on a hyperconverged setup so I am not using it in production, but I am still on a older version.
The flip side to using the mds, is that it is caching a lot of meta data so in theory you could have a better performance with cephfs than writing directly to rados. Writing to rados directly seems to me the most stable.
What I thought was super strange about the s3/radowsgw layer is that if you rename a file, the file is actually copied to a new name. It is not renamed. I am not sure if this is a standard and still like this, but s3 is just developed for a different use. So it depends on how you use s3/radosgw, object storage directly or cephfs.
Dňa 17. 10. 2023 o 13:20, Dmitry Melekhov <dm@belkam.com> napísal:
17.10.2023 12:22, Filip Hanes via dovecot пишет: S3-compatible storage is very good for multi-server installations where you need redundancy, availability. S3 is basically HTTP server so you can code your own logic on stored emails, balancers, caches, deduplication, compression, encryption it does't need to be off-the- shelf storage.
is S3 better then cephfs?
I understand S3 as protocol for object storage. When you use popular protocol, you can change implementation later. You can put nginx for caching, compression, balancing. Later you can code some proxy analysing bodies or whatever. Its pluggable. Yes each proxy adds latency. Large setups are better made modular (microservices) for incremental upgrades than squeezing performance and trying to find all-in-one solution.
Ceph has S3 implemented on top of its cluster storage. We are using ceph S3, cephfs and rados too, in different use cases. Other S3 implementation is Minio on top of any posix filesystem - you can choose which fills your needs. Seaweed has S3 endpoint with interesting storage solution, IMHO good for email storage - in theory. I have not used it.
------- Original Message ------- On Tuesday, October 17th, 2023 at 15:27, Filip Hanes via dovecot <dovecot@dovecot.org> wrote:
Other S3 implementation is Minio on top of any posix filesystem - you can choose which fills your needs.
Minio is great in general, the only thing I would say it its a little bit weird to setup if you're in a VM environment. It was really based around physical hosts, so you need to replicate that on VMs (i.e. 3 x virtual disks per VM so that the error encoding stuff works just like it would on physical hosts).
But certainly compared to Ceph its a lot easier on the sysadmin side !
Well, so Laura is absolutely right ...
"Things like dsync will be GONE in the community version."
That's not right, dsync is still there. Replicator is not, so dsync can't be triggered automatically by dovecot after changes to the mailbox
Well, to be fair :
I said what I said based on the video. And the video seemed pretty clear cut to me ?
Its not there in the form that many (most ?) people would use it for (i.e. with Replicator).
Then Aki came along and said "there is no hidden cache of code going into 3.0 that will not be open source". When the video kind of makes it clear 3.0 Pro with all its new features (e.g. multi-server) is very much going to be a closed-source job. And that the present open-source version is, just like they say in the video, is going to be "supported for single-server use only".
Therefore the waters are still very much muddy overall. The dsync question might well have now been clarified somewhat. But the rest of "how much 3.0 Pro will we see in open source" ? If we're being generous we would say muddy waters, but my gut feeling is the video made clear their direction of travel in that the present Open Source version will continue as-is with updates and support, bu won't be getting any of the fancy new features and functionality that 3.0 Pro is.
Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator { unix_listener replicator-doveadm { mode = 0666 } }
plugin { mail_replica = tcp:example.com:12345 }
If that is the case, well then I have to find another way to keep mails in sync between 2 mailservers. Hope the community will find a new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway.
To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot <dovecot@dovecot.org> wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server"
------- Original Message ------- On Friday, October 13th, 2023 at 19:37, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you, Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebastian@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Hello to everyone!
Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
For server synchronization (non "realtime") we are using "imapsync" ( https://imapsync.lamiral.info/ )
regards!
On 16/10/23 08:11, Taavi Ansper via dovecot wrote:
Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator { unix_listener replicator-doveadm { mode = 0666 } }
plugin { mail_replica = tcp:example.com:12345 }
If that is the case, well then I have to find another way to keep mails in sync between 2 mailservers. Hope the community will find a new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the video, although I could have phrased my explanation better. "dsync" refers to the tool/utility (part of doveadm) that does mail synchronization between a source account to a destination account. As Aki said, this is not going anywhere. This is a necessary tool for any kind of migrations, for example. dsync is under active maintenance, as we heavily use this tool internally. What is being removed is the replicator plugin (that used dsync). That's what is being referred to in the video. Replicator hasn't been actively maintained for years now so this was dead code anyway. To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale. Neither replicator nor sis is code that is moving from open to closed source. These plugins aren't used in Pro. They are unmaintained so they are being removed, as happens with any kind of old code. michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot <dovecot@dovecot.org> wrote:
FUD ?
I knew someone would accuse me of that which is why I linked to the video from the horse's mouth, I transcribe what the speaker said:
"there will be an open source version, but that open source version will be maintained for single server use only. we are actually taking out anything any actually kinda' involves multiple servers, dsync replication and err some other stuff. so dovecot will be a fully-featured single node server"
------- Original Message ------- On Friday, October 13th, 2023 at 19:37, Aki Tuomi <aki.tuomi@open-xchange.com> wrote:
Dear Laura, please don't spread FUD that you made up.
Dsync is not going anywhere, and we are not close-sourcing Dovecot Core. There is not a trove of code going into Dovecot 3.0 that "never sees the daylight".
Thank you, Aki
On 13/10/2023 21:10 EEST Laura Smith via dovecot dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot Community user, don't waste your time reading the Dovecot Pro release notes.
To expand:
I think you have to understand that lots of things that are going into Dovecot 3 (Pro) will never see the light of day in the community edition.
In addition, Dovecot have publicly quite plainly announced in public that they are actively removing all multi-server related functionality from Dovecot Community.
I don't think the community has quite yet grasped it. Things like dsync will be GONE in the community version.
If you don't believe me, look at this video, about 15 minutes in: https://youtu.be/s-JYrjCKshA?feature=shared&t=912
------- Original Message ------- On Friday, October 13th, 2023 at 17:15, Sebastian Marsching sebastian@marsching.com wrote:
Hi,
I am currently in the process of planning a new deployment of Dovecot. I was planning to use mdbox or sdbox with “mail_attachment_fs = sis posix”, but I stumbled across the following notice in the documentation for Dovecot 3.0
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-leave@dovecot.org
Com os melhores cumprimentos, <https://www.ipl.pt/> Pedro Ribeiro *Departamento de Sistemas de Informação e Comunicações - IPLNet <https://www.net.ipl.pt>* Serviços da Presidência Telf. +351 210 464 700 / Ext. 80101
Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem. For server synchronization (non "realtime") we are using "imapsync" ( https:// imapsync.lamiral.info/ ) regards! On 16/10/23 08:11, Taavi Ansper via dovecot wrote: Hi
So in my 99-dsync.conf
This would not work in newer releases?
service replicator {
unix_listener replicator-doveadm {
mode = 0666
}
}
plugin {
mail_replica = tcp:example.com:12345
}
If that is the case, well then I have to find another way to keep
mails in sync between 2 mailservers. Hope the community will find a
new solution!
On 16.10.23 09:30, Michael Slusarz via dovecot wrote:
Aki is correct and is consistent with what I said in the
video, although I could have phrased my explanation better.
"dsync" refers to the tool/utility (part of doveadm) that
does mail synchronization between a source account to a
destination account. As Aki said, this is not going
anywhere. This is a necessary tool for any kind of
migrations, for example. dsync is under active
maintenance, as we heavily use this tool internally.
What is being removed is the replicator plugin (that used
dsync). That's what is being referred to in the video.
Replicator hasn't been actively maintained for years now so
this was dead code anyway.
To answer the OP: sis is also being removed and should
not be used by any new installation. Code remains to read
data written by the old plug-in so that these installations
don't require a migration between 2.3 and 2.4. This is
another plugin that hasn't be actively maintained in years,
and has all kinds of limitations that prevent it from
running at scale.
Neither replicator nor sis is code that is moving from
open to closed source. These plugins aren't used in Pro.
They are unmaintained so they are being removed, as happens
with any kind of old code.
michael
On 10/13/2023 1:26 PM MDT Laura Smith via dovecot
<dovecot@dovecot.org> wrote:
FUD ?
I knew someone would accuse me of that which is
why I linked to the video from the horse's mouth,
I transcribe what the speaker said:
"there will be an open source version, but that
open source version will be maintained for single
server use only. we are actually taking out
anything any actually kinda' involves multiple
servers, dsync replication and err some other
stuff. so dovecot will be a fully-featured single
node server"
------- Original Message -------
On Friday, October 13th, 2023 at 19:37, Aki Tuomi
<aki.tuomi@open-xchange.com> wrote:
Dear Laura, please don't spread FUD
that you made up.
Dsync is not going anywhere, and we are
not close-sourcing Dovecot Core. There
is not a trove of code going into
Dovecot 3.0 that "never sees the
daylight".
Thank you,
Aki
On 13/10/2023 21:10 EEST
Laura Smith via dovecot
dovecot@dovecot.org wrote:
TL;DR If you are a Dovecot
Community user, don't waste
your time reading the Dovecot
Pro release notes.
To expand:
I think you have to
understand that lots of
things that are going into
Dovecot 3 (Pro) will never
see the light of day in the
community edition.
In addition, Dovecot have
publicly quite plainly
announced in public that they
are actively removing all
multi-server related
functionality from Dovecot
Community.
I don't think the community
has quite yet grasped it.
Things like dsync will be
GONE in the community
version.
If you don't believe me, look
at this video, about 15
minutes in:
https://youtu.be/s-
JYrjCKshA?feature=shared&t=912
------- Original Message ----
---
On Friday, October 13th, 2023
at 17:15, Sebastian Marsching
sebastian@marsching.com
wrote:
Hi,
I am currently in
the process of
planning a new
deployment of
Dovecot. I was
planning to use
mdbox or sdbox with
“mail_attachment_fs
= sis posix”, but I
stumbled across the
following notice in
the documentation
for Dovecot 3.0
_______________________________________________
dovecot mailing
list -
-
dovecot@dovecot.org
To unsubscribe send
an email to
dovecot-
leave@dovecot.org
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-
leave@dovecot.org
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-leave@dovecot.org
_______________________________________________
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-leave@dovecot.org-- Com os melhores cumprimentos, [https://www.net.ipl.pt/ipl_img/logo_ipl_email.png] Pedro Ribeiro Departamento_de_Sistemas_de_Informação_e_Comunicações_-_IPLNet Serviços da Presidência Telf. +351 210 464 700 / Ext. 80101
Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication.
On Mon, 16 Oct 2023, Marc wrote:
Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication.
In an office where people insist on mailing documents to everyone, and using email as a document storage system, yes, it is very useful.
--
======================================================================== Chris Candreva -- chris@westnet.com -- http://www.westnet.com/~chris
Hello everyone!
I'm reviving the topic just to add that after reconstructing our storage with SIS disabled the occupied space increased from *5.3TB* to *9.6TB*, almost doubling!
It's a feature promoting storage efficiency, I think it demands some ponderation the advantages of keeping or improving the module.
I just have to thank all the people involved in this software for the great work and contribution to the community, with or without SIS in the future!
regards.
On 16/10/2023 14:00, Chris Candreva wrote:
On Mon, 16 Oct 2023, Marc wrote:
Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication. In an office where people insist on mailing documents to everyone, and using email as a document storage system, yes, it is very useful.
-- Com os melhores cumprimentos, <https://www.ipl.pt/> Pedro Ribeiro *Departamento de Sistemas de Informação e Comunicações - IPLNet <https://www.net.ipl.pt>* Serviços da Presidência Telf. +351 210 464 700 / Ext. 80101
Hello everyone! I'm reviving the topic just to add that after reconstructing our storage with SIS disabled the occupied space increased from 5.3TB to 9.6TB, almost doubling! It's a feature promoting storage efficiency, I think it demands some ponderation the advantages of keeping or improving the module. I just have to thank all the people involved in this software for the great work and contribution to the community, with or without SIS in the future! regards. On 16/10/2023 14:00, Chris Candreva wrote: On Mon, 16 Oct 2023, Marc wrote: Is this feature really useful? I can imagine if you are twitter or ig and everyone is posting the same video this could be usefull. Are there any stats on this available, so you know what to expect implementing deduplication. In an office where people insist on mailing documents to everyone, and using email as a document storage system, yes, it is very useful.
-- Com os melhores cumprimentos, [https://www.net.ipl.pt/ipl_img/logo_ipl_email.png] Pedro Ribeiro Departamento_de_Sistemas_de_Informação_e_Comunicações_-_IPLNet Serviços da Presidência Telf. +351 210 464 700 / Ext. 80101
- Pedro Ribeiro via dovecot <dovecot@dovecot.org>:
Hello everyone! I'm reviving the topic just to add that after reconstructing our storage with SIS disabled the occupied space increased from 5.3TB to 9.6TB, almost doubling! It's a feature promoting storage efficiency, I think it demands some ponderation the advantages of keeping or improving the module.
Amen to that. I think the deduplication is a useful, storage efficient way -- especially in large environments with lots of users (who get the same mails :) )
-- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebrandt@charite.de | https://www.charite.de
Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
did you really mean deduplicated or distributed?
I think this duduplicating. Storage systems are offering such solutions. I think ceph has something like this, although I am not sure for rbd disk images. I think it makes more sense to have something like this done by a fs or storage solution.
Le 16 oct. 2023 à 15:51, Marc <Marc@f1-outsourcing.eu> a écrit :
Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
did you really mean deduplicated or distributed?
I think this duduplicating. Storage systems are offering such solutions. I think ceph has something like this, although I am not sure for rbd disk images. I think it makes more sense to have something like this done by a fs or storage solution.
If you are using Ubuntu, OpenZFS is readily available, and support deduplication natively. Else it is also available on other platforms, but may require more setup.
Day 17. 10. 2023 7:46, Jean-Daniel Dupas <jddupas@xooloo.com> wrote:
If you are using Ubuntu, OpenZFS is readily available, and support deduplication natively. Else it is also available on other platforms, but may require more setup.
Filesystems does not have deduplication effective for emails. They mostly use block based deduplication, which does not work when the same attachment in second body is shifted even one byte, like when any header value is longer/shorter.
Deduplication using rolling hash (rsync) is better but still different clients can encode same part in different ways.
Even dovecot SIS is not very sophisticated in this, but it is better than block based deduplication.
Good deduplication can lower storage to almost 1/3 of full email bodies.
-- Filip Hanes
------- Original Message ------- On Tuesday, October 17th, 2023 at 06:46, Jean-Daniel Dupas <jddupas@xooloo.com> wrote:
If you are using Ubuntu, OpenZFS is readily available, and support deduplication natively.
I thought nobody sane actually used ZFS dedup because it eats RAM for breakfast, lunch and dinner ?
On 16.10.23 13:17, Pedro Ribeiro via dovecot wrote:
Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
A block level de-duplicating filesystem can only deduplicate data that is aligned to block boundaries. E-mail attachments tend to move around in to a different alignment in each copy stored into a different mailbox. Unless the storage format is designed to split off the attachments into files there is not much to be gained by block level dedup. So for the foreseeable future I'll have to stay off Dovecot 3.x or add four to five times more storage to both my IMAP servers since my users love to send big documents to multiple recipients.
Is this an attempt to figuring out the pain tolerance of existing users before they fork the project or pay up the Danegeld?
On 17/10/2023 03:26 EEST Jan Bramkamp <crest@rlwinm.de> wrote:
On 16.10.23 13:17, Pedro Ribeiro via dovecot wrote:
Hello to everyone! Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
A block level de-duplicating filesystem can only deduplicate data that is aligned to block boundaries. E-mail attachments tend to move around in to a different alignment in each copy stored into a different mailbox. Unless the storage format is designed to split off the attachments into files there is not much to be gained by block level dedup. So for the foreseeable future I'll have to stay off Dovecot 3.x or add four to five times more storage to both my IMAP servers since my users love to send big documents to multiple recipients.
Is this an attempt to figuring out the pain tolerance of existing users before they fork the project or pay up the Danegeld?
SIS won't be available even if you paid up the Danegeld. You can use mail_attachment_fs with posix driver.
Aki
Pedro Ribeiro wrote:
Ooops, we are using SIS, guess the solution for a similar optimization will be a native deduplicated filesystem.
A non-deduplicated filesystem is fine considering the current hash-based folder structure. Just:
- Switch to a hash with no known collision method (i.e. not sha1)
- Run a crontab that causes all files with the same hash in filename to be hard linked together.
This kind of "skip byte-by-byte" thinking was there since the initial implementation of SIS1, but was never added for some reason.
I currently do a much, much reduced version of that: I just run a
crontab to get jdupes to do its byte-by-byte comparison over the
attachment directory, daily. Duplicates get hard linked.
PS: I swear the documentation for sis-queue is wrong.
Sincerely, Mingye Wang
If that is the case, well then I have to find another way to keep mails in sync between 2 mailservers. Hope the community will find a new solution!
I have been keeping one eye on Stalwart (https://stalw.art/) for a while now.
I haven't tested it as yet, but I'm very much tempted to get a test instance up and running.
If that is the case, well then I have to find another way to keep mails
in sync between 2 mailservers. Hope the community will find a new solution!
I have been keeping one eye on Stalwart (https://stalw.art/) for a while now.
I haven't tested it as yet, but I'm very much tempted to get a test instance up and running.
Interesting, nice they use this rust, I am curious how they define this scaling. What I don't get is why are they messing with smtp. I always get a bad feeling when a company is trying to do everything.
Interesting, nice they use this rust, I am curious how they define this scaling. What I don't get is why are they messing with smtp. I always get a bad feeling when a company is trying to do everything.
Good they are using rust and even better they've had an independent security audit (https://www.stalw.art/blog/security-audit).
On the scaling side, maybe see the storage page ? (https://www.stalw.art/docs/storage/overview). The metadata is stored in a database which can be replicated. And the mails themselves can be stored in filesystem or "S3-compatible" storage, and so there are scaling options there too ? But clearly some experimentation is required to see how it works in practice.
Are they messing with SMTP ? As I understand it its an IMAP/JMAP server. And (like Dovecot) it has LMTP for getting mail into it from e.g. Postfix ? From my reading of the docs it looks like SMTP is only there if you don't want to use LMTP to get mail into it ?
On 2023-10-16 2:30 a.m., Michael Slusarz via dovecot wrote:
To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
So if we are using "mail_attachment_fs = sis posix" in 2.4 it will become "mail_attachment_fs = posix" ?
i.e. separating attachments still supported but without SIS?
Thanks for any clarification.
On 2023-10-16 2:30 a.m., Michael Slusarz via dovecot wrote: To answer the OP: sis is also being removed and should not be used by any new installation. Code remains to read data written by the old plug-in so that these installations don't require a migration between 2.3 and 2.4. This is another plugin that hasn't be actively maintained in years, and has all kinds of limitations that prevent it from running at scale.
So if we are using "mail_attachment_fs = sis posix" in 2.4 it will become "mail_attachment_fs = posix" ?
i.e. separating attachments still supported but without SIS?
Thanks for any clarification.
participants (19)
-
Aki Tuomi
-
Bjoern Franke
-
Chris Candreva
-
Dmitry Melekhov
-
Emmanuel Fusté
-
Filip Hanes
-
Gedalya
-
Jan Bramkamp
-
Jean-Daniel Dupas
-
Laura Smith
-
Marc
-
Michael Slusarz
-
Mingye Wang
-
Oscar del Rio
-
Pedro Ribeiro
-
Ralf Hildebrandt
-
Sebastian Marsching
-
Stuart Henderson
-
Taavi Ansper