How to move/reorganise existing e-mails to yearly subfolders

[Marc Roos]

What is the problem with having huge online mailboxes? Just choose a 
good european provider that has encryption all the way through to their 
storage platform.

I had exactly the same idea about migrating. You have to think twice 
about moving emails around of users. They do not like it ;)  I have 
created an 'archive' environment on a distributed filesystem, and it 
takes me quite a lot of persuading to have people (or allow me) to move 
messages from common Sent and Inbox mailboxes to the Archive namespace 
(was not able to use the alternative storage option that dbox allows). 
Side note is that if you do archive these emails, most users do not even 
notice you have done this.

I ended up creating a script and a webmail plugin for users to enable 
autoarchiving, which creates something like this. 

[@~]# mailbox-ls.sh testtest size
listing mailboxes of testtest:
Archive messages=0
Archive/2011 messages=0
Archive/2012 messages=0
Archive/2013 messages=0
Archive/2014 messages=0
Archive/2015 messages=0
Archive/2016 messages=0
Archive/2017 messages=0
Archive/2018 messages=0
Archive/2019 messages=3500
Archive/Archive messages=1
Deleted Messages messages=16
Drafts messages=2
INBOX messages=1286
INBOX/test2 messages=11
Junk messages=2
Sent messages=0
Trash messages=132

A cron job checks then if the script has run for the user this year, if 
not it starts archiving in the down hours otherwise it runs again in 
next year's 2nd quarter. What ever you choose, move messages with 
"doveadm move". I would not trust anyone else's programming with my 
users email, you should also not. Read the man pages on tools that work 
via imap, if they change headers, users are going to download all their 
messages again. I was thinking of splitting up folders like eg 
inbox/sales to Archive/2016/sales, Archive/2017/sales. But when I 
migrate to mdbox this is not necessary anymore.

I do not like the sound of "Postfix BCC feature", I use sendmail and I 
can duplicate messages with that, without altering anything in them. You 
do not want anything that changes your data. If your provider uses this 
mdbox format (maybe others support this also) then messages a user 
deletes, are not even removed from the server until "doveadm purge" is 
given ;)


 

-----Original Message-----
From: R. Diez [mailto:rdiezmail-2006 at yahoo.de] 
Sent: Monday, October 19, 2020 3:49 PM
To: dovecot at dovecot.org
Subject: How to move/reorganise existing e-mails to yearly subfolders

Hi all:

I am new to e-mail servers and I am evaluating Dovecot. Not really the 
best combination. 8-)

I am trying to find a balance between legal data retention requirements 
and online mailbox size. I do not want huge online mailboxes, as doing 
offline, rotating data backups could then take forever (among other 
reasons). I would rather avoid online (cloud) backups (data protection 
etc.).

If I set a mailbox size limit, users will have to delete old mails by 
themselves. Or I could somehow script the deletion of attachments from 
old e-mails, as attachments are usually the main cause of huge 
mailboxes. Incidentally, can anyone point me to an easy way to achieve 
this? Preferably over with IMAP, otherwise with Dovecot tools.

With regards to legal data retention (which I am no expert about 
either), I thought I could use some Postfix BCC feature I heard about in 
order to copy all incoming and outgoing e-mails to a single "data 
retention" mailbox. Or maybe several of them. I could then archive 
e-mails from that mailbox on a yearly basis.

I would like to automatically organise e-mails inside that mailbox into 
subfolders like this:

2019/alice
2019/bob
2020/alice
2020/bob

That is: [year]/[username]

With such a folder structure, it is easier to see what is going on.

Is there a tool that can reorganise existing e-mails into such a folder 
structure?

I found some tools on the Internet to backup and export mails from IMAP 
to IMAP or maildir destinations. But I could not find a tool that just 
reorganises (moves) e-mails in such a manner inside an existing mailbox, 
maybe with a user-defined pattern for the destination folders.

I guess moving e-mails around on the same mailbox would be much faster 
than exporting and reimporting them in some clever way.

I could always write a Perl script, but that takes time. Such a tool may 
already exist. Or perhaps somebody could mention a similar, good-written 
script I can use as a starting point. I am sure there are many small 
gotchas to avoid. At the moment, I am only confident with Perl and Java. 
Maybe JavaScript.

It would be best to reorganise the e-mails over IMAP. This way, I am 
independent of the e-mail server. But a Dovecot-specific solution would 
also be helpful.

I could use such a reorganisation tool not just for archiving or data 
retention purposes, but to reorganise other mailboxes too, like my 
personal mailbox.

I would rather have a script. Clicking around in Thunderbird does not 
scale.

I have seen that you can use a "sieve" in Dovecot to achieve this. But I 
guess that would only apply to new e-mails. And that would probably 
apply just to incoming mails, and not to outgoing ones. If I migrate 
from the existing e-mail servers, the e-mails will probably be 
mass-imported without going through any "sieve", right?

Besides, I already have a few huge mailboxes with many years of e-mails, 
stored in a different e-mail server, and it would be nice to be able to 
reorganise them as they are. That way, I could archive the existing 
older e-mails before migrating to Dovecot, which would reduce the disk 
size and the export/import time for the migration.

Many thanks in advance,
   rdiez