Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!

Mon Jul 8 15:21:51 EEST 2019

Hello Aki,

Thanks for looking into these.

I will as requested attempt the relevant procedures under Dovecot
2.3.6.

To make the test fair, I will need to fork the relevant production
groupware stack (which is now stable and in operation, with our
enterprise (email) data successfully migrated from Microsoft Exchange)
to a new staging server; given that the current staging server is now
of a materially different configuration to the production server (where
the most controlled observations of these bugs were made).

Kindly give me some time, as I have an urgent internal openstack
deployment project to kick-off, that has been delayed by an overrun of
this internal groupware stack deployment project.

Best regards,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldoree at parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.

-----Original Message-----
From: Aki Tuomi via dovecot <dovecot at dovecot.org>
Reply-To: Aki Tuomi <aki.tuomi at open-xchange.com>
To: arnoldoree at parallaxict.com, Arnold Opio Oree <
arnold.oree at parallaxict.com>, Arnold Opio Oree via dovecot <
dovecot at dovecot.org>
Cc: debian-release at lists.debian.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive - BUG REPORTS!
Date: Mon, 8 Jul 2019 08:48:08 +0300 (EEST)

Hi!

Thank you for reporting these. We will look into them. In the mean
time, can you see if any of these are fixed in 2.3.6?

Aki
> On 07/07/2019 18:12 Arnold Opio Oree via dovecot <dovecot at dovecot.org
> > wrote:
> 
> 
> Dovecot Team,
> 
> I'd like to report a number of bugs, that are to my view all
> critical.
> 
> System: Replicated on multiple Debian 10 (Buster) systems
> Dovecot Version(s): 2.3.4.1
> 
> doveadm-sync -1/general
> 
> 1) If DIRNAMEs are not different between command line and
> mail_location doveadm sync will fail, saying that the source and
> destination directories are the same
> 
> 2) The -n / -N flags do not work, and a sync will fail strangely if
> location is specified in the namespace definition
> 
> 3) Adds mbox to path name under mailbox directory (where syncing from
> an mbox source)
> 
> 4) Not having the mailboxes at source named the same as those at
> destination causes errors and partial sync 
> 
> 5) Not having the target mailboxes formatted to receive the sync
> (/<mailboxroot>/DIRNAME/) will cause sync errors.
> 
> doveadm-sync
> 
> 1) With large synchronizations UIDs are corrupted where multiple
> syncs are executed and the program can no longer synchronize
> 
> dovecot
> 
> 1) Panics and fails to expand ~ to user home: observed cases are
> where multiple namespaces are being used
> 
> Please let me know if you need me to elaborate or to provide any
> further information that you may need to replicate the bugs, or if I
> can help in any other way.
> 
> With regards to the last error that I requested help on i.e.
> \Noselect. This has been resolved more-or-less by the workarounds
> that I have implemented for the bugs reported above.
> 
> I have seen a number of threads whilst researching the \Noselect
> issue where people have been very confused. My finding was that
> \Noselect is a function of the IMAP specification server-side
> implementation RFC3501 ( 
> https://tools.ietf.org/html/rfc3501#section-6.3.6). And for me the
> server was returning directories with \Noselect because the mailboxes
> were malformed on account of dovadm-sync errors. In order to fix this
> I formed a bash command to transverse the mailbox hierarchy and
> create the missing folders critical to the sdbox format, namely
> DIRNAME.
> 
> Kind regards,
> 
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
> 
> arnoldoree at parallaxdt.com
> 
> http://www.parallaxdt.com
> 
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
> 
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
> 
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately,
> disregard its contents and disclose them to no other persons.
> 
> -----Original Message-----
> From: Arnold Opio Oree via dovecot < dovecot at dovecot.org>
> Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree < 
> arnold.oree at parallaxict.com>
> To: dovecot at dovecot.org
> Cc: rs at sys4.de, aki.tuomi at open-xchange.com
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive.
> Date: Thu, 04 Jul 2019 14:52:28 +0100
> 
> 
> Hi all,
> 
> The guidance provided so far has been really helpful, and has helped
> a great deal to bringing down wasted energy on finding and executing
> a viable path. I am now at the final due action to complete our
> Dovecot application to our use-case, but am stuck on an issue that I
> cannot find any easily accessible documentation on.
> 
> Generally this is what has been done:
> 
> 1. Uploaded the enterprise data PST to the target groupware server.
> 2. Prepared the server by changing the mailbox format to sdbox and
> the the Dovecot mail location to
> mail_location=/var/vmail/domain/user/mail/
> 3. Converted the pst (on-server) to a recursive mbox hierarchy using
> readpst
> 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox
> and to copy it into the enterprise archive user's mailboxes
> 4.i. The biggest issue I faced at this point was doveadm-sync saying
> that the source and destination pointed to the same location, whereas
> they clearly did not. 
> 4.i.a. I resolved this by removing the location= setting from the
> target namespace, and allowing it to default to mail_location =
> setting, and then using a completely different DIRNAME for the import
> doveadm-sync execution (which was the desired final DIRNAME); I then
> once the sync had been successful, changed the mail_location DIRNAME
> so that it pointed to the imported mail DIRNAME; and hence the
> imported email data was in the live mailboxes
> 4.i.b. doveadm-import failed several times, and was throwing quite
> inexplicable errors, so I moved onto doveadm-sync
> 4.i.c. I also had to make sure that the source and destination folder
> names matched, otherwise doveadm-syc threw very many errors and only
> partially imported the data
> 4.i.d. An issue which I decided just to live with is that an mbox
> DIRNAME was added to each mailbox as well as the DIRNAME specified so
> the path to mail is mbox/dbox-Mails. My thought is that with the data
> live on an IMAP server it will be possible to do a dysync through TCP
> to correct this problem.
> 
> The final issue that I am facing now, is that when readpst finds
> empty folders in the source pst hierarchy, it does not create an mbox
> file in the mbox hierarchy folder space. This causes doveadm-sync to
> not create the target data required for its mailbox structure i.e.
> DIRNAME sub-folder and index file (with our configuration). At this
> point either doveadm-sync or the dovecot process makes these empty
> folders not selectable.
> 
> The question now is how would I go about making all of these folders
> selectable, e.g. with an internal or external command line tool to
> change flags / create necessary sdbox mailbox constituent data?
> 
> Many thanks,
> 
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
> 
> arnoldoree at parallaxdt.com
> 
> http://www.parallaxdt.com
> 
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
> 
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
> 
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately,
> disregard its contents and disclose them to no other persons.
> 
> -----Original Message-----
> From: Arnold Opio Oree via dovecot < dovecot at dovecot.org>
> Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree < 
> arnold.oree at parallaxict.com>
> To: Robert Schetterer < rs at sys4.de>, dovecot at dovecot.org
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive.
> Date: Thu, 27 Jun 2019 12:05:35 +0100
> 
> > Also you may run into client limits.... i ve seen this with
> > outlook, 
> > apple mail, thunderbird via imap in the past
> 
> Thanks for this note Robert, it was not really an aspect that I was
> considering. 
> 
> We are operating our groupware services user access through both
> Evolution Groupware and KDE Kontact / KMail on Debian Linux
> workstations. Hopefully if there is a client issue it should be local
> to only one groupware client. 
> 
> I will be sure to study / investigate in this - client - area should
> any issues that are not traceable to the server-side arise.
> 
> Many thanks,
> 
> Arnold Opio Oree
> Chief Executive Officer
> Parallax Digital Technologies
> 
> arnoldoree at parallaxdt.com
> 
> 
> http://www.parallaxdt.com
> 
> 
> tel : +44 (0) 333 577 8587
> fax : +44 (0) 20 8711 2477
> 
> Parallax Digital Technologies is a trading name of Parallax Global
> Limited. U.K. Co. No. 08836288
> 
> The contents of this e-mail are confidential. If you are not the
> intended recipient you are to delete this e-mail immediately,
> disregard
> its contents and disclose them to no other persons.
> 
> 
> -----Original Message-----
> From: Robert Schetterer via dovecot <
> dovecot at dovecot.org
> >
> Reply-To: Robert Schetterer <
> rs at sys4.de
> >
> To: 
> dovecot at dovecot.org
> 
> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
> archive.
> Date: Thu, 27 Jun 2019 12:53:49 +0200
> 
> Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot:
> > On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote:
> > > Hello to you all,
> > > 
> > > I'd like to ask about my intended application of Dovecot to
> > > create
> > > a folder-hierarchy for storing our enterprise emails, which are
> > > treated as live data rather than archives for compliance or
> > > occasional / reactive retrieval.
> > > 
> > > The data is presently not that large (a few gigabytes), but it is
> > > expected to grow rapidly. Up to this stage the data has been
> > > contained
> > > in a Microsoft Exchange mailbox (2013), and then in an offline
> > > PST.
> > > The move to the offline PST was by necessity, as the large number
> > > of
> > > folders, and depth of hierarchy to my best understanding caused
> > > the
> > > exchange server / outlook / evolution mail clients to begin to
> > > malfunction. To cope with this the archive was broken up and the
> > > bulk stored in the offline PST and the most active components
> > > stored in
> > > online Exchange mailboxes.
> > > 
> > > I have some understanding of the fs mbox format, and also the
> > > mitigations to be made for certain characters / strings. My main
> > > concern is whether Dovecot is likely to be able to cope well with
> > > a
> > > large number of folders / depth of hierarcy.
> > > 
> > > I will really appreciate any help / advice you can give.
> > > 
> > > Best regards,
> > > 
> > > Arnold Opio Oree
> > > 
> > 
> > Hi!
> > 
> > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only
> > limiting
> > factor is that the total name may not exceed 4096 bytes. Also
> > individual
> > folder names may not exceed 255 bytes.
> > 
> > Prior to that the limit is 255 per folder up to 16 levels.
> > 
> > I can't recommend using 'mbox' storage format, please consider
> > using
> > maildir or sdbox instead.
> > 
> > Aki
> > 
> 
> 
> 
> 
> 
> 
> 
> 

---
Aki Tuomi
-- 