Remove "Duplicate" emails (and documentation update)

[@lbutlr]

On 2018-02-23 (16:47 MST), Joseph Tam <jtam.home at gmail.com> wrote:
> 
> On Fri, 23 Feb 2018, @lbutlr wrote:
> 
>> $ doveadm -f table fetch -u kremels 'hdr.message-id guid uid
>> hdr.x-listname' mailbox "Archive" | sort| awk 'cnt[$1]++{if
>> (cnt[$1]==2) print prev[$1]; print} {prev[$1]=$0}' |grep -E "[0-9] +$"
>> |awk '{print "doveadm expunge -u kremels MAILBOX-GUID "$2" UID "$3}?
> 
> I was unaware of the syntax "hdr.{header}" -- all the reference materials
> I've seen only refers to "hdr" which returns the entire header block.

the error message from doveadm if you specify an invalid field is:

Available fetch fields: hdr.<name> body.<section> binary.<section> user mailbox mailbox-guid seq uid guid flags modseq hdr body body.snippet text text.utf8 size.physical size.virtual date.received date.sent date.saved date.received.unixtime date.sent.unixtime date.saved.unixtime imap.envelope imap.body imap.bodystructure pop3.uidl pop3.order refcount storageid

>> First, even after expunging a message and running doveadm index -u
>> kremels ?Archive?, subsequent runs still show the same duplicate
>> messages.
> 
> I suspect client side caching.

No, there is no client side involved. I am executing all of these these commands on the mail server. I expunge the messages, I index (or even force-resync) and the next search shows the same messages even though they are not in the Maildir anymore.

> If {count} is what you expected, then dovecot has the correct information
> and it's likely some client-side caching issue.

I would have needed to check the count before doing this, and I did not.

>> Second, what I really want to do is run this over ALL the mailboxes,
>> except for Junk and Sent but if that is possible I can?t find the right
>> syntax.
> 
> You mean to remove duplicates from any 2 mailboxes, or remove duplicates
> in mailboxes also found in Archive?

I want to find any duplicates (based on msg ID) across all mailboxes, except Sent

> 	doveadm -f table fetch -u kremels \
> 		'hdr.message-id guid uid' \
> 		NOT mailbox Archive \
> 		NOT mailbox Junk \
> 		NOT mailbox Sent \
> 		| sort -b >list1

Aha! Didn't know you could use NOT mailbox. That probably solves my issue on that score.

-- 
"It's unacceptable to think" - George W Bush 15/Sep/2006