(I did subscribe to this mailing list, albeit with zen at freedbms.net, so either way I'm getting all your emails - thank you -so- much for replying...)
MUA is mutt, reading email in a terminal (sorry, forgot to mention this before).
For many years now my email folder (mbox files) collection has grown to many GiB, mostly mailing lists.
If I am to change email storage format, it should be mutt compatible; looking at https://wiki2.dovecot.org/MailboxFormat I see that only DJB's Maildir is compatible with both Dovecot ("a reliable choice" says the wiki), and mutt.
I can imagine that sdbox or mdbox could be made "mutt compatible" so to speak, by running some sort of local IMAP server, and accessing my email from mutt that way; this is undesirable to my mind because this would require:
- a new learning curve wrt mutt and reading email on IMAP servers
- a new learning curve to set up a local IMAP server (securely)
- the inability to use mutt without a local IMAP server to read my local email
but such a setup would also have some quite desirable benefits:
- once set up, multiple MUAs could be used, and I'd have a beginning grasp on setting up an IMAP server and front ends (this is something on my bucket list, to assist my local church with)
- simpler remote "online" access to my local "offline" email store (e.g. using my mobile phone when on the road) by setting up a webmail server (much simpler (read "possible" to use on a mobile phone) than using a vpn and mutt...), thus freeing me up from the behemoth web email providers...
Next, I do not know how to "pipe the messages to the dovecot lda". After downloading from my POP3 provider into a local mbox file (this is my step 1), then I sort the emails (this is my step 2): the following should be on a single line:
/usr/bin/sieve-filter -veW -c $HOME/etc/email/sieve-dovecot-config.conf -o mail_location=mbox:~/mail:INBOX=~/mail/Inbox:INDEX=:UTF-8:VOLATILEDIR=/tmp/dovecot-volatile/%2.256Nu/%u:SUBSCRIPTIONS=dovecot_subscriptions ~/etc/email/sieve.rc email-incoming-unsorted
As you can see from the above command, sieve-filter is given the name of the mbox ("mail folder") to sort, as its very last argument on the command line - so in this instance, sieve-filter really has no excuse, and should be not be re-reading the sieve rules script for each email
- now perhaps that's not happening, I only made an assumption because of a CPU hitting 100% for a minute or two just to process a few 100 emails...
What could also be happening (again, an assumption), is that sieve-filter is written to assume dovecot index files to be in existence.
I disabled those with the "INDEX=" clause you see in the command above, which obviously has been given no value.
The reason I figured out how to disable the creation of the indexes in the .imap directories, is that for my setup, Gnu sieve has proven that I should not need such indexes - with mbox files, just append each email to the end of the target "mailbox folder" mbox file, and we're done! This literally should not cost 100% CPU, even for one millisecond! But more importantly, because my working email folder is ~30GiB, without disabling this index creation step, sieve-filter forced the creation of indexes, which "took so long I gave up and hit CTRL-C, which did not work, so I kill -9'ed the sieve-filter and whatever other process was not stopping".
Last year someone on debian-user recommended I upgrade to using Dovecot/Pigeonhole's sieve-filter (rather than Gnu sieve) due to the issues with Gnu sieve.
I am starting to think that I should perhaps try to figure out if it's possible to (re)process the emails Gnu sieve has a problem with, to massage them into a shape that Gnu sieve accepts - then my immediate problem would certainly be solved...
Thank you all again.. Zenaan