Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Timo Sirainen]

On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot <dovecot at dovecot.org> wrote:
> 
> Dovecot Team,
> 
> I'd like to report a number of bugs, that are to my view all critical.

It would help to get your doveconf -n, example command lines causing the problems and the error messages it outputs or what the wrong behavior looks like in filesystem. It's now rather difficult to guess what exactly you tried and what happened.

Also what kind of output does readpst make? I'm not sure why you're using DIRNAMEs here.

> doveadm-sync -1/general
> 
> 1) If DIRNAMEs are not different between command line and mail_location doveadm sync will fail, saying that the source and destination directories are the same

This sounds very strange. I'm not sure what exactly you did, and I couldn't reproduce with a small test.

> 2) The -n / -N flags do not work, and a sync will fail strangely if location is specified in the namespace definition

Again, sounds strange.

> 3) Adds mbox to path name under mailbox directory (where syncing from an mbox source)

Probably with different parameters you could avoid it.

> 4) Not having the mailboxes at source named the same as those at destination causes errors and partial sync 
> 
> 5) Not having the target mailboxes formatted to receive the sync (/<mailboxroot>/DIRNAME/) will cause sync errors.

I don't understand these. Target mailboxes are supposed to be empty initially, and after the initial sync they should be in the expected format. Why would they be different?

> doveadm-sync
> 
> 1) With large synchronizations UIDs are corrupted where multiple syncs are executed and the program can no longer synchronize

What exactly is the error message?

> dovecot
> 
> 1) Panics and fails to expand ~ to user home: observed cases are where multiple namespaces are being used

Panic message and more details would also help.

> With regards to the last error that I requested help on i.e. \Noselect. This has been resolved more-or-less by the workarounds that I have implemented for the bugs reported above.
> 
> I have seen a number of threads whilst researching the \Noselect issue where people have been very confused. My finding was that \Noselect is a function of the IMAP specification server-side implementation RFC3501 ( <http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/1/nJ1qwIFgSyW4nfuZp-VMgA/aHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzM1MDEjc2VjdGlvbi02LjMuNg#section-6.3.6>https://tools.ietf.org/html/rfc3501#section-6.3. <http://h5lm.mjt.lu/lnk/AMwAAEPRQ68AAAAAAAAAAKAFynQAASKA_AUAAAAAAAf-tABdIgvzBSx7xYU6SQW2MDXORHSy0wAHm5I/2/cturGvUAbhIkMPkkWYpZGw/aHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL3JmYzM1MDEjc2VjdGlvbi02LjMu#section-6.3.>6). And for me the server was returning directories with \Noselect because the mailboxes were malformed on account of dovadm-sync errors. In order to fix this I formed a bash command to transverse the mailbox hierarchy and create the missing folders critical to the sdbox format, namely DIRNAME.

Nowadays we have also an option to disable creation of \Noselect folders, because they confuse people. Using mail_location = ...:NO-NOSELECT - It won't affect existing folders immediately though.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190710/bd71ec6f/attachment-0001.html>