Applying Dovecot for a large / deep folder-hierarchy archive.

[Arnold Opio Oree]

Hi all,

The guidance provided so far has been really helpful, and has helped a
great deal to bringing down wasted energy on finding and executing a
viable path. I am now at the final due action to complete our Dovecot
application to our use-case, but am stuck on an issue that I cannot
find any easily accessible documentation on.

Generally this is what has been done:

1. Uploaded the enterprise data PST to the target groupware server.
2. Prepared the server by changing the mailbox format to sdbox and the
the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/
3. Converted the pst (on-server) to a recursive mbox hierarchy using
readpst
4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and
to copy it into the enterprise archive user's mailboxes
4.i. The biggest issue I faced at this point was doveadm-sync saying
that the source and destination pointed to the same location, whereas
they clearly did not. 
4.i.a. I resolved this by removing the location= setting from the
target namespace, and allowing it to default to mail_location =
setting, and then using a completely different DIRNAME for the import
doveadm-sync execution (which was the desired final DIRNAME); I then
once the sync had been successful, changed the mail_location DIRNAME so
that it pointed to the imported mail DIRNAME; and hence the imported
email data was in the live mailboxes
4.i.b. doveadm-import failed several times, and was throwing quite
inexplicable errors, so I moved onto doveadm-sync
4.i.c. I also had to make sure that the source and destination folder
names matched, otherwise doveadm-syc threw very many errors and only
partially imported the data
4.i.d. An issue which I decided just to live with is that an mbox
DIRNAME was added to each mailbox as well as the DIRNAME specified so
the path to mail is mbox/dbox-Mails. My thought is that with the data
live on an IMAP server it will be possible to do a dysync through TCP
to correct this problem.

The final issue that I am facing now, is that when readpst finds empty
folders in the source pst hierarchy, it does not create an mbox file in
the mbox hierarchy folder space. This causes doveadm-sync to not create
the target data required for its mailbox structure i.e. DIRNAME sub-
folder and index file (with our configuration). At this point either
doveadm-sync or the dovecot process makes these empty folders not
selectable.

The question now is how would I go about making all of these folders
selectable, e.g. with an internal or external command line tool to
change flags / create necessary sdbox mailbox constituent data?


-----Original Message-----
From: Arnold Opio Oree via dovecot <dovecot at dovecot.org>
Reply-To: arnoldoree at parallaxict.com, Arnold Opio Oree <
arnold.oree at parallaxict.com>
To: Robert Schetterer <rs at sys4.de>, dovecot at dovecot.org
Subject: Re: Applying Dovecot for a large / deep folder-hierarchy
archive.
Date: Thu, 27 Jun 2019 12:05:35 +0100

> Also you may run into client limits.... i ve seen this with outlook,
> apple mail, thunderbird via imap in the past

Thanks for this note Robert, it was not really an aspect that I
wasconsidering. 
We are operating our groupware services user access through
bothEvolution Groupware and KDE Kontact / KMail on Debian
Linuxworkstations. Hopefully if there is a client issue it should be
localto only one groupware client. 
I will be sure to study / investigate in this - client - area should
any issues that are not traceable to the server-side arise.
Many thanks,
Arnold Opio OreeChief Executive OfficerParallax Digital Technologies
arnoldoree at parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587fax : +44 (0) 20 8711 2477
Parallax Digital Technologies is a trading name of Parallax
GlobalLimited. U.K. Co. No. 08836288
The contents of this e-mail are confidential. If you are not
theintended recipient you are to delete this e-mail immediately,
disregardits contents and disclose them to no other persons.

-----Original Message-----From: Robert Schetterer via dovecot <
dovecot at dovecot.org>Reply-To: Robert Schetterer <rs at sys4.de>To: 
dovecot at dovecot.org
Subject: Re: Applying Dovecot for a large / deep folder-
hierarchyarchive.Date: Thu, 27 Jun 2019 12:53:49 +0200
Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot:
> On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote:
> > Hello to you all,
> > I'd like to ask about my intended application of Dovecot to createa
> > folder-hierarchy for storing our enterprise emails, which
> > aretreated as live data rather than archives for compliance
> > oroccasional / reactive retrieval.
> > The data is presently not that large (a few gigabytes), but it
> > isexpected to grow rapidly. Up to this stage the data has
> > beencontainedin a Microsoft Exchange mailbox (2013), and then in an
> > offline PST.The move to the offline PST was by necessity, as the
> > large numberoffolders, and depth of hierarchy to my best
> > understanding caused theexchange server / outlook / evolution mail
> > clients to begin tomalfunction. To cope with this the archive was
> > broken up and thebulk stored in the offline PST and the most active
> > componentsstored inonline Exchange mailboxes.
> > I have some understanding of the fs mbox format, and also
> > themitigations to be made for certain characters / strings. My
> > mainconcern is whether Dovecot is likely to be able to cope well
> > with alarge number of folders / depth of hierarcy.
> > I will really appreciate any help / advice you can give.
> > Best regards,
> > Arnold Opio Oree
> 
> Hi!
> Dovecot 2.2.34/2.3 supports unlimited folder depth, the only
> limitingfactor is that the total name may not exceed 4096 bytes.
> Alsoindividualfolder names may not exceed 255 bytes.
> Prior to that the limit is 255 per folder up to 16 levels.
> I can't recommend using 'mbox' storage format, please consider
> usingmaildir or sdbox instead.
> Aki


Many thanks,

Arnold Opio Oree
Chief Executive Officer
Parallax Digital Technologies

arnoldoree at parallaxdt.com

http://www.parallaxdt.com

tel : +44 (0) 333 577 8587
fax : +44 (0) 20 8711 2477

Parallax Digital Technologies is a trading name of Parallax Global
Limited. U.K. Co. No. 08836288

The contents of this e-mail are confidential. If you are not the
intended recipient you are to delete this e-mail immediately, disregard
its contents and disclose them to no other persons.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20190704/d600dbe2/attachment.html>