Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

[Arnold Opio Oree]

Dovecot Team,

I'd like to report a number of bugs, that are to my view all critical.

System: Replicated on multiple Debian 10 (Buster) systems
Dovecot Version(s): 2.3.4.1

doveadm-sync -1/general

1) If DIRNAMEs are not different between command line and mail_location
doveadm sync will fail, saying that the source and destination
directories are the same

2) The -n / -N flags do not work, and a sync will fail strangely if
location is specified in the namespace definition

3) Adds mbox to path name under mailbox directory (where syncing from
an mbox source)

4) Not having the mailboxes at source named the same as those at
destination causes errors and partial sync 

5) Not having the target mailboxes  formatted to receive the sync
(/<mailboxroot>/DIRNAME/) will cause sync errors.

doveadm-sync

1) With large synchronizations UIDs are corrupted where multiple syncs
are executed and the program can no longer synchronize

dovecot

1) Panics and fails to expand ~ to user home: observed cases are where
multiple namespaces are being used

Please let me know if you need me to elaborate or to provide any
further information that you may need to replicate the bugs, or if I
can help in any other way.

With regards to the last error that I requested help on i.e.
\Noselect.  This has been resolved more-or-less by the workarounds that
I have implemented for the bugs reported above.

I have seen a number of threads whilst researching the \Noselect issue
where people have been very confused. My finding was that \Noselect is
a function of the IMAP specification server-side implementation RFC3501
(https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
server was returning directories with \Noselect because the mailboxes
were malformed on account of dovadm-sync errors. In order to fix this I
formed a bash command to transverse the mailbox hierarchy and create
the missing folders critical to the sdbox format, namely DIRNAME.

Kind regards,

Arnold Opio Oree

-----Original Message-----
From: Arnold Opio Oree via dovecot <dovecot at dovecot.org>
Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree <
arnold.oree at parallaxict.com>
To: dovecot at dovecot.org
Cc: rs at sys4.de, aki.tuomi at open-xchange.com
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive.
Date: Thu, 04 Jul 2019 14:52:28 +0100


Hi all,

The guidance provided so far has been really helpful, and has helped a
great deal to bringing down wasted energy on finding and executing a
viable path. I am now at the final due action to complete our Dovecot
application to our use-case, but am stuck on an issue that I cannot
find any easily accessible documentation on.

Generally this is what has been done:

1. Uploaded the enterprise data PST to the target groupware server.
2. Prepared the server by changing the mailbox format to sdbox and the
the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/
3. Converted the pst (on-server) to a recursive mbox hierarchy using
readpst
4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and
to copy it into the enterprise archive user's mailboxes
4.i. The biggest issue I faced at this point was doveadm-sync saying
that the source and destination pointed to the same location, whereas
they clearly did not. 
4.i.a. I resolved this by removing the location= setting from the
target namespace, and allowing it to default to mail_location =
setting, and then using a completely different DIRNAME for the import
doveadm-sync execution (which was the desired final DIRNAME); I then
once the sync had been successful, changed the mail_location DIRNAME so
that it pointed to the imported mail DIRNAME; and hence the imported
email data was in the live mailboxes
4.i.b. doveadm-import failed several times, and was throwing quite
inexplicable errors, so I moved onto doveadm-sync
4.i.c. I also had to make sure that the source and destination folder
names matched, otherwise doveadm-syc threw very many errors and only
partially imported the data
4.i.d. An issue which I decided just to live with is that an mbox
DIRNAME was added to each mailbox as well as the DIRNAME specified so
the path to mail is mbox/dbox-Mails. My thought is that with the data
live on an IMAP server it will be possible to do a dysync through TCP
to correct this problem.

The final issue that I am facing now, is that when readpst finds empty
folders in the source pst hierarchy, it does not create an mbox file in
the mbox hierarchy folder space. This causes doveadm-sync to not create
the target data required for its mailbox structure i.e. DIRNAME sub-
folder and index file (with our configuration). At this point either
doveadm-sync or the dovecot process makes these empty folders not
selectable.

The question now is how would I go about making all of these folders
selectable, e.g. with an internal or external command line tool to
change flags / create necessary sdbox mailbox constituent data?

Many thanks,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldoree at parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.

-----Original Message-----
From: Arnold Opio Oree via dovecot <dovecot at dovecot.org>
Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree <
arnold.oree at parallaxict.com>
To: Robert Schetterer <rs at sys4.de>, dovecot at dovecot.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive.
Date: Thu, 27 Jun 2019 12:05:35 +0100

> Also you may run into client limits.... i ve seen this with outlook, 
> apple mail, thunderbird via imap in the past

Thanks for this note Robert, it was not really an aspect that I was
considering. 

We are operating our groupware services user access through both
Evolution Groupware and KDE Kontact / KMail on Debian Linux
workstations. Hopefully if there is a client issue it should be local
to only one groupware client. 

I will be sure to study / investigate in this - client - area should
any issues that are not traceable to the server-side arise.

Many thanks,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldoree at parallaxdt.com


http://www.parallaxdt.com


tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.


-----Original Message-----
From: Robert Schetterer via dovecot <
dovecot at dovecot.org
>
Reply-To: Robert Schetterer <
rs at sys4.de
>
To: 
dovecot at dovecot.org

Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive.
Date: Thu, 27 Jun 2019 12:53:49 +0200

Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot:
> On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote:
> > Hello to you all,
> > 
> > I'd like to ask about my intended application of Dovecot to create
> > a folder-hierarchy for storing our enterprise emails, which are
> > treated as live data rather than archives for compliance or
> > occasional / reactive retrieval.
> > 
> > The data is presently not that large (a few gigabytes), but it is
> > expected to grow rapidly. Up to this stage the data has been
> > contained
> > in a Microsoft Exchange mailbox (2013), and then in an offline PST.
> > The move to the offline PST was by necessity, as the large number
> > of
> > folders, and depth of hierarchy to my best understanding caused the
> > exchange server / outlook / evolution mail clients to begin to
> > malfunction. To cope with this the archive was broken up and the
> > bulk stored in the offline PST and the most active components
> > stored in
> > online Exchange mailboxes.
> > 
> > I have some understanding of the fs mbox format, and also the
> > mitigations to be made for certain characters / strings. My main
> > concern is whether Dovecot is likely to be able to cope well with a
> > large number of folders / depth of hierarcy.
> > 
> > I will really appreciate any help / advice you can give.
> > 
> > Best regards,
> > 
> > Arnold Opio Oree
> > 
> 
> Hi!
> 
> Dovecot 2.2.34/2.3 supports unlimited folder depth, the only limiting
> factor is that the total name may not exceed 4096 bytes. Also
> individual
> folder names may not exceed 255 bytes.
> 
> Prior to that the limit is 255 per folder up to 16 levels.
> 
> I can't recommend using 'mbox' storage format, please consider using
> maildir or sdbox instead.
> 
> Aki
> 






-- 
Dear [Recipient],

[Content]

Regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldoree at parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies
78A Hatfeild Mead
London
SM4 5PF

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.
Chief Executive Officer
Parallax Digital Technologies

arnoldoree at parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190707/6dc7d717/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: divide.jpg
Type: image/jpeg
Size: 4590 bytes
Desc: not available
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190707/6dc7d717/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pdtsignaturelogo.jpg
Type: image/jpeg
Size: 35023 bytes
Desc: not available
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190707/6dc7d717/attachment-0003.jpg>