Re: Users with enough rope to hang themselves
Rupert Gallagher writes:
I keep finding myself in a corner with a user. He uses mail extensively, which is fine, he has a huge archive of own professional correspondence, which is fine, but he uses mail folders as if they were regular system folders, with very long paths, and keeps renaming them and moving them around, daily, breaking the mail index
Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate indices)?
and ultimately wasting his own time looking around for lost mail. His Inbox holds a gargantuan of subfolders, causing both the client and the server to overwork each time he opens the mail. His Archive is a maze of subfolders with repeating names. I advised him almost daily across 20 year on how to stay organised, but he keeps abusing the service.
Semantically, he may be inept/disorganized/unappreciative, but I wouldn't raise this to abuse. However, the damages are often the same. Maybe the fix is not technical but social by making it clear you're done trying to fix his mistakes and he's on his own. Just sayin'.
I want to help him by limiting what he can do with folders. This is the agenda:
- the Archive is the only place where he can create folders;
I'm guessing https://doc.dovecot.org/configuration_manual/acl/
- folder names have a maximum length of 20 characters.
No clue here: maybe artful remapping of namespaces?
https://doc.dovecot.org/configuration_manual/mail_location/#custom-namespace...
Joseph Tam jtam.home@gmail.com
I keep finding myself in a corner with a user. He uses mail extensively, which is fine, he has a huge archive of own professional correspondence, which is fine, but he uses mail folders as if they were regular system folders, with very long paths, and keeps renaming them and moving them around, daily, breaking the mail index
I am offering users auto archiving of inbox and sent messages folders, both will be combined into a folder 2024 etc. Most users even appreciate this.
A bit weird is the moving of folders, most people have some logics behind it, and stay with it. Maybe he is using some shitty client on the phone? You have no idea what kind of crap 'rookie I only care about how it looks' developers can produce. I can remember trying to offer assitance to developers of such app. My user was using their app, and it generated like 800% more load than the average user constantly(can't remember exactly).
and ultimately wasting his own time looking around for lost mail. His Inbox holds a gargantuan of subfolders, causing both the client and the server to overwork each time he opens the mail. His Archive is a maze
I don't get why you care? I am little surprised to read that this could be an issue. Is this maybe related to how you store messages? I am using this mdbox. I think nothing much changes there if they create/move folders etc. I have everything in 4mb files.
of subfolders with repeating names. I advised him almost daily across 20 year on how to stay organised, but he keeps abusing the service.
I have users that are doing the same, and offline clients are very capable of searching and retrieving messages. Never heard anything about this being a problem.
Semantically, he may be inept/disorganized/unappreciative, but I wouldn't raise this to abuse. However, the damages are often the same. Maybe the fix is not technical but social by making it clear you're done trying to fix his mistakes and he's on his own. Just sayin'.
I want to help him by limiting what he can do with folders. This is the agenda:
- the Archive is the only place where he can create folders;
I'm guessing https://doc.dovecot.org/configuration_manual/acl/
- folder names have a maximum length of 20 characters.
No clue here: maybe artful remapping of namespaces?
What about switching this user to different storage? I think that is doable and not that much work for one user. I am really surprised about reading this, since I have never experienced anything like this. I can't really believe I have been so clueless for decades ;) Can this really be an issue?
I keep finding myself in a corner with a user. He uses mail extensively, which is fine, he has a huge archive of own professional correspondence, which is fine, but he uses mail folders as if they were regular system folders, with very long paths, and keeps renaming them and moving them around, daily, breaking the mail index
Hi Rupert,
I share your frustration. I have had similar users, who used mailbox as a sort of task/ticket manager. Each task was a separate folder, (with all related emails), and they renamed/moved the folder when the task status has changed.
It was OK as long as they used POP3 + Outlook and everything was in local PST folder. Until the PST grew over some limit (4 GB or such) that outlook was not able to handle.
So we migrated them to IMAP (cyrus), and ran into a problem with forbidden characters in folder names. Most prominent was "." - used for date format. But they kept on trying other special chars, always hitting something forbidden. (too late I realized I should have switched the folder separator from . to / before moving from POP3 to IMAP)
To make things worse, Outlook did not complain about forbidden characters, it just made folder "local only" and stopped syncing it to the IMAP server. Users gladly edited the folder name and removed the "local only" from the name so it was impossible to trace back which folders were synced and which not.
It was a constant pain ... Finally resolved by migrating them to MS-365. Maybe dovecot would be more forgiving, but we did not dare to try.
-- Best regards Vladislav Kurz
It was a constant pain ... Finally resolved by migrating them to MS-365. Maybe dovecot would be more forgiving, but we did not dare to try.
That is why these fuckers of Apple and Microsoft are doing it, and these morons at EU market abuse commissions don't get it, don't read complaints. etc. Also autodiscovery for external (as in, not Microsoft/Apple) mail is being frustrated. It is either on purpose frustrating or they have incompetent designers/developers.
-------- Original Message -------- On Apr 4, 2024, 14:02, Marc < Marc@f1-outsourcing.eu> wrote:
Also autodiscovery for external (as in, not Microsoft/Apple) mail is being frustrated.
Apple Mail on iPhones is currently ignoring autodiscovery and forcing their own smtp server, breaking DMARC.
-------- Original Message -------- On Apr 4, 2024, 14:02, Marc < Marc@f1-outsourcing.eu> wrote:
Also autodiscovery for external (as in, not Microsoft/Apple) mail is being frustrated.
Apple Mail on iPhones is currently ignoring autodiscovery and forcing their own smtp server, breaking DMARC.
On 04/04/2024 03:15 EEST Joseph Tam jtam.home@gmail.com wrote:
Rupert Gallagher writes:
I keep finding myself in a corner with a user. He uses mail extensively, which is fine, he has a huge archive of own professional correspondence, which is fine, but he uses mail folders as if they were regular system folders, with very long paths, and keeps renaming them and moving them around, daily, breaking the mail index
Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate indices)?
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid problems.
Aki
breaking the mail index
Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate indices)?
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid problems.
So you can just add this to an existing configuration, without the need to do anything? Like eg reindex or so? I always had the impression this affects how messages are stored.
mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx becomes: mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx:LAYOUT:index
Does this really speed things up?
[1] https://doc.dovecot.org/configuration_manual/mail_location/
On 05/04/2024 12:36 EEST Marc marc@f1-outsourcing.eu wrote:
breaking the mail index
Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate indices)?
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid problems.
So you can just add this to an existing configuration, without the need to do anything? Like eg reindex or so? I always had the impression this affects how messages are stored.
mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx becomes: mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx:LAYOUT:index
Does this really speed things up?
You can't put it on existing setup just like that, you need to do storage format migration, or copy all the folders into their GUID folders when you do this.
It will speed things up and will also allow you to use almost anything as mailbox name.
Aki
breaking the mail index
Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate
indices)?
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid problems.
So you can just add this to an existing configuration, without the need to do anything? Like eg reindex or so? I always had the impression this affects how messages are stored.
mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx becomes: mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx:LAYOUT:index
Does this really speed things up?
You can't put it on existing setup just like that, you need to do storage format migration, or copy all the folders into their GUID folders when you do this.
It will speed things up and will also allow you to use almost anything as mailbox name.
I was in the process of setting up new server, it already had a few test mailboxes on it.
why do you mention this specific version 2.3.21? el9 still is at 2.3.16.
So if I add the LAYOUT:index to mail_location for my mdbox setup, it only renames Drafts,INBOX,Junk etc to some uuid (I guess)?
├── mdbox │ ├── dbox-alt-root │ ├── mailboxes │ │ ├── Drafts │ │ │ └── dbox-Mails │ │ ├── INBOX │ │ │ ├── dbox-Mails │ │ │ └── test │ │ │ └── dbox-Mails │ │ ├── Junk │ │ │ └── dbox-Mails │ │ ├── NotSpam │ │ │ └── dbox-Mails │ │ ├── Sent │ │ │ └── dbox-Mails │ │ └── Trash │ │ └── dbox-Mails │ └── storage │ ├── m.1 │ ├── m.2 │ ├── m.3 │ ├── m.4 │ ├── m.5 │ └── m.6
This server is offline any way. What would be the easiest way to do this migration? Something I can do directly on this server?
On 05/04/2024 13:26 EEST Marc marc@f1-outsourcing.eu wrote:
breaking the mail index
Tangentially query: is Dveocot smart enough to optimize mailbox renaming to do index renaming (i.e. does not try to copy or recreate
indices)?
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid problems.
So you can just add this to an existing configuration, without the need to do anything? Like eg reindex or so? I always had the impression this affects how messages are stored.
mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx becomes: mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx:LAYOUT:index
Does this really speed things up?
You can't put it on existing setup just like that, you need to do storage format migration, or copy all the folders into their GUID folders when you do this.
It will speed things up and will also allow you to use almost anything as mailbox name.
I was in the process of setting up new server, it already had a few test mailboxes on it.
why do you mention this specific version 2.3.21? el9 still is at 2.3.16.
Because we have fixed issues with index layout since 2.3.16. It will work with 2.3.16 too but there are issues you'd probably not want to deal with, most importantly:
- LAYOUT=index List index rebuild was missing.
- LAYOUT=index: Duplicate GUIDs were not detected.which were added in 2.3.17 and futher fixed in later versions. It's a chore to dig up specific versions, so I recommend using the latest.
So if I add the LAYOUT:index to mail_location for my mdbox setup, it only renames Drafts,INBOX,Junk etc to some uuid (I guess)?
├── mdbox │ ├── dbox-alt-root │ ├── mailboxes │ │ ├── Drafts │ │ │ └── dbox-Mails │ │ ├── INBOX │ │ │ ├── dbox-Mails │ │ │ └── test │ │ │ └── dbox-Mails │ │ ├── Junk │ │ │ └── dbox-Mails │ │ ├── NotSpam │ │ │ └── dbox-Mails │ │ ├── Sent │ │ │ └── dbox-Mails │ │ └── Trash │ │ └── dbox-Mails │ └── storage │ ├── m.1 │ ├── m.2 │ ├── m.3 │ ├── m.4 │ ├── m.5 │ └── m.6
This server is offline any way. What would be the easiest way to do this migration? Something I can do directly on this server?
If there are only few accounts, you can configure new mail location with LAYOUT=index and use doveadm sync / backup to migrate mails there, then move things over the old stuff - or copy old stuff elsewhere and do the same to current location as new.
Aki
> breaking the mail index
Tangentially query: is Dveocot smart enough to optimize
renaming
to do index renaming (i.e. does not try to copy or recreate indices)?
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid
So you can just add this to an existing configuration, without the need to do anything? Like eg reindex or so? I always had the impression
mailbox problems. this affects how messages are stored.
mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx becomes: mail_location =
mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx:LAYOUT:index
Does this really speed things up?
You can't put it on existing setup just like that, you need to do storage format migration, or copy all the folders into their GUID folders when you do this.
It will speed things up and will also allow you to use almost anything as mailbox name.
I was in the process of setting up new server, it already had a few test mailboxes on it.
why do you mention this specific version 2.3.21? el9 still is at 2.3.16.
Because we have fixed issues with index layout since 2.3.16. It will work with 2.3.16 too but there are issues you'd probably not want to deal with, most importantly:
- LAYOUT=index List index rebuild was missing.
- LAYOUT=index: Duplicate GUIDs were not detected.
which were added in 2.3.17 and futher fixed in later versions. It's a chore to dig up specific versions, so I recommend using the latest.
So if I add the LAYOUT:index to mail_location for my mdbox setup, it only renames Drafts,INBOX,Junk etc to some uuid (I guess)?
├── mdbox │ ├── dbox-alt-root │ ├── mailboxes │ │ ├── Drafts │ │ │ └── dbox-Mails │ │ ├── INBOX │ │ │ ├── dbox-Mails │ │ │ └── test │ │ │ └── dbox-Mails │ │ ├── Junk │ │ │ └── dbox-Mails │ │ ├── NotSpam │ │ │ └── dbox-Mails │ │ ├── Sent │ │ │ └── dbox-Mails │ │ └── Trash │ │ └── dbox-Mails │ └── storage │ ├── m.1 │ ├── m.2 │ ├── m.3 │ ├── m.4 │ ├── m.5 │ └── m.6
This server is offline any way. What would be the easiest way to do this migration? Something I can do directly on this server?
If there are only few accounts, you can configure new mail location with LAYOUT=index and use doveadm sync / backup to migrate mails there, then move things over the old stuff - or copy old stuff elsewhere and do the same to current location as new.
I tested with this
doveadm sync -u testacc 'mdbox:/home/testing/testacc/mdbox:INDEX=/home/testing/testacc/index:CONTROL=/home/testing/testacc/mail/control:LAYOUT=index'
Which gives me the expectec result. But I have also an archive namespace and an alt namespace that is going to be quite a lot of moving around on this production server. Not to mention having all these messages copied. Is this really the most efficient way to migrate to such id folders?
├── index │ ├── dovecot.list.index │ ├── dovecot.list.index.log │ ├── dovecot.mailbox.log │ ├── mailboxes │ │ ├── 36179613fa3330637557000052412a8e │ │ │ ├── dovecot.index.cache │ │ │ └── dovecot.index.log │ │ ├── 67b77b07fa3330635457000052412a8e │ │ │ └── dovecot.index.log │ │ ├── 7908061cf23330634d57000052412a8e │ │ │ ├── dovecot.index.cache │ │ │ └── dovecot.index.log │ │ ├── 80628213e4f562613b27000052412a8e │ │ │ ├── dovecot.index │ │ │ ├── dovecot.index.backup │ │ │ ├── dovecot.index.cache │ │ │ └── dovecot.index.log │ │ ├── a2585d3ae5714e636229030052412a8e │ │ │ └── dovecot.index.log │ │ ├── b45e8a3819f662614d27000052412a8e │ │ │ ├── dovecot.index.cache │ │ │ └── dovecot.index.log │ │ └── c07dd6138d5d3063615b000052412a8e │ │ └── dovecot.index.log │ └── storage │ ├── dovecot.map.index │ └── dovecot.map.index.log ├── mail │ └── control │ └── subscriptions └── mdbox ├── mailboxes │ ├── 36179613fa3330637557000052412a8e │ ├── 67b77b07fa3330635457000052412a8e │ ├── 7908061cf23330634d57000052412a8e │ ├── 80628213e4f562613b27000052412a8e │ ├── a2585d3ae5714e636229030052412a8e
On 05/04/2024 15:53 EEST Marc via dovecot dovecot@dovecot.org wrote:
> > breaking the mail index > > Tangentially query: is Dveocot smart enough to optimize
renaming > to do index renaming (i.e. does not try to copy or recreate indices)? >
Dovecot is, if you use LAYOUT=index. This will use only mailbox GUID on disk, and the folder name is only in indexes (both list index and mailbox index for recovery purposes).
I would recommend using 2.3.21 with LAYOUT=index to avoid
So you can just add this to an existing configuration, without the need to do anything? Like eg reindex or so? I always had the impression
mailbox problems. this affects how messages are stored.
mail_location = mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx becomes: mail_location =
mdbox:xxxxxx:INDEX=xxxxxx:CONTROL=/xxxxxxx:LAYOUT:index
Does this really speed things up?
You can't put it on existing setup just like that, you need to do storage format migration, or copy all the folders into their GUID folders when you do this.
It will speed things up and will also allow you to use almost anything as mailbox name.
I was in the process of setting up new server, it already had a few test mailboxes on it.
why do you mention this specific version 2.3.21? el9 still is at 2.3.16.
Because we have fixed issues with index layout since 2.3.16. It will work with 2.3.16 too but there are issues you'd probably not want to deal with, most importantly:
- LAYOUT=index List index rebuild was missing.
- LAYOUT=index: Duplicate GUIDs were not detected.
which were added in 2.3.17 and futher fixed in later versions. It's a chore to dig up specific versions, so I recommend using the latest.
So if I add the LAYOUT:index to mail_location for my mdbox setup, it only renames Drafts,INBOX,Junk etc to some uuid (I guess)?
├── mdbox │ ├── dbox-alt-root │ ├── mailboxes │ │ ├── Drafts │ │ │ └── dbox-Mails │ │ ├── INBOX │ │ │ ├── dbox-Mails │ │ │ └── test │ │ │ └── dbox-Mails │ │ ├── Junk │ │ │ └── dbox-Mails │ │ ├── NotSpam │ │ │ └── dbox-Mails │ │ ├── Sent │ │ │ └── dbox-Mails │ │ └── Trash │ │ └── dbox-Mails │ └── storage │ ├── m.1 │ ├── m.2 │ ├── m.3 │ ├── m.4 │ ├── m.5 │ └── m.6
This server is offline any way. What would be the easiest way to do this migration? Something I can do directly on this server?
If there are only few accounts, you can configure new mail location with LAYOUT=index and use doveadm sync / backup to migrate mails there, then move things over the old stuff - or copy old stuff elsewhere and do the same to current location as new.
I tested with this
doveadm sync -u testacc 'mdbox:/home/testing/testacc/mdbox:INDEX=/home/testing/testacc/index:CONTROL=/home/testing/testacc/mail/control:LAYOUT=index'
Which gives me the expectec result. But I have also an archive namespace and an alt namespace that is going to be quite a lot of moving around on this production server. Not to mention having all these messages copied. Is this really the most efficient way to migrate to such id folders?
You said you are in process of migrating to new server. And yes, this is the safest way to do this change.
You probably likely want to have a migration.conf file with **new** locations, and use doveadm -c /path/to/migration.conf sync -1R tcp:localhost:1234 to make it bit less horrible to migrate.
Aki
doveadm sync -u testacc
'mdbox:/home/testing/testacc/mdbox:INDEX=/home/testing/testacc/index:CONT ROL=/home/testing/testacc/mail/control:LAYOUT=index'
Which gives me the expectec result. But I have also an archive
namespace and an alt namespace that is going to be quite a lot of moving around on this production server. Not to mention having all these messages copied. Is this really the most efficient way to migrate to such id folders?
You said you are in process of migrating to new server. And yes, this is the safest way to do this change.
You probably likely want to have a migration.conf file with **new** locations, and use doveadm -c /path/to/migration.conf sync -1R tcp:localhost:1234 to make it bit less horrible to migrate.
I guess this migration.conf can be a dump of doveconf -a not?
Should I copy these sieve dirs manually or is there also sync for that?
On 05/04/2024 17:49 EEST Marc via dovecot dovecot@dovecot.org wrote: doveadm sync -u testacc 'mdbox:/home/testing/testacc/mdbox:INDEX=/home/testing/ testacc/index:CONT ROL=/home/testing/testacc/mail/control:LAYOUT=index' > Which gives me the expectec result. But I have also an archive namespace and an alt namespace that is going to be quite a lot of moving around on this production server. Not to mention having all these messages copied. Is this really the most efficient way to migrate to such id folders? You said you are in process of migrating to new server. And yes, this is the safest way to do this change. You probably likely want to have a migration.conf file with **new** locations, and use doveadm -c /path/to/migration.conf sync -1R tcp:localhost:1234 to make it bit less horrible to migrate. I guess this migration.conf can be a dump of doveconf -a not? Should I copy these sieve dirs manually or is there also sync for that? Sieve is synced. See https://doc.dovecot.org/admin_manual/migrating_mailboxes/ Aki
participants (5)
-
Aki Tuomi
-
Joseph Tam
-
Marc
-
Rupert Gallagher
-
Vladislav Kurz