Gnu sieve vs Dovecot sieve-filter - sieve-filter extremely slow at lda (writing emails to local mbox files)

[Zenaan Harkness]

(I did subscribe to this mailing list, albeit with zen at
freedbms.net, so either way I'm getting all your emails - thank you
-so- much for replying...)

MUA is mutt, reading email in a terminal (sorry, forgot to mention this before).

For many years now my email folder (mbox files) collection has grown
to many GiB, mostly mailing lists.

If I am to change email storage format, it should be mutt compatible;
looking at https://wiki2.dovecot.org/MailboxFormat I see that only
DJB's Maildir is compatible with both Dovecot ("a reliable choice"
says the wiki), and mutt.

I can imagine that sdbox or mdbox could be made "mutt compatible" so
to speak, by running some sort of local IMAP server, and accessing my
email from mutt that way; this is undesirable to my mind because this
would require:

 1) a new learning curve wrt mutt and reading email on IMAP servers
 2) a new learning curve to set up a local IMAP server (securely)
 3) the inability to use mutt without a local IMAP server to read my local email

but such a setup would also have some quite desirable benefits:

 1) once set up, multiple MUAs could be used, and I'd have a beginning
grasp on setting up an IMAP server and front ends (this is something
on my bucket list, to assist my local church with)
 2) simpler remote "online" access to my local "offline" email store
(e.g. using my mobile phone when on the road) by setting up a webmail
server (much simpler (read "possible" to use on a mobile phone) than
using a vpn and mutt...), thus freeing me up from the behemoth web
email providers...

Next, I do not know how to "pipe the messages to the dovecot lda".
After downloading from my POP3 provider into a local mbox file (this
is my step 1), then I sort the emails (this is my step 2): the
following should be on a single line:

/usr/bin/sieve-filter -veW -c
$HOME/etc/email/sieve-dovecot-config.conf -o
mail_location=mbox:~/mail:INBOX=~/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions
~/etc/email/sieve.rc email-incoming-unsorted

As you can see from the above command, sieve-filter is given the name
of the mbox ("mail folder") to sort, as its very last argument on the
command line - so in this instance, sieve-filter really has no excuse,
and should be not be re-reading the sieve rules script for each email
- now perhaps that's not happening, I only made an assumption because
of a CPU hitting 100% for a minute or two just to process a few 100
emails...

What could also be happening (again, an assumption), is that
sieve-filter is written to assume dovecot index files to be in
existence.

I disabled those with the "INDEX=" clause you see in the command
above, which obviously has been given no value.

The reason I figured out how to disable the creation of the indexes in
the .imap directories, is that for my setup, Gnu sieve has proven that
I should not need such indexes - with mbox files, just append each
email to the end of the target "mailbox folder" mbox file, and we're
done! This literally should not cost 100% CPU, even for one
millisecond! But more importantly, because my working email folder is
~30GiB, without disabling this index creation step, sieve-filter
forced the creation of indexes, which "took so long I gave up and hit
CTRL-C, which did not work, so I kill -9'ed the sieve-filter and
whatever other process was not stopping".

Last year someone on debian-user recommended I upgrade to using
Dovecot/Pigeonhole's sieve-filter (rather than Gnu sieve) due to the
issues with Gnu sieve.

I am starting to think that I should perhaps try to figure out if it's
possible to (re)process the emails Gnu sieve has a problem with, to
massage them into a shape that Gnu sieve accepts - then my immediate
problem would certainly be solved...

Thank you all again..
Zenaan