Quoting "Timo Sirainen" <tss@iki.fi>:
Try instead with:
doveadm fetch guid all | sort | uniq | wc
When you removed indexes Dovecot no longer knew about copies of messages.
Well, well, well. This is interesting. Back with the indices created by dsync:
$ doveadm fetch guid all | grep guid: | sort | uniq -c | sort -n | tail 17 guid: 1b28b22d4b2ee2885b5b81221c41201d 17 guid: 730c692395661dd62f82088804b85652 17 guid: 865e1537fddba6698e010d0b9dbddd02 17 guid: d271b6ba8af0e7fa39c16ea8ed13abcf 17 guid: d2cd391e837cf51cc85991bde814dc54 17 guid: ebce8373da6ffb134b58aca7906d61f1 18 guid: 1222b6c222ecb53fdbbec407400cba36 18 guid: 65695586efc69adc2d7294216ea88e55 19 guid: 4288f61ebbdcd44870c670439a97693b 20 guid: 080ec72aa49e2a01c8e249fe127605f6
This would explain why rebuilding the indices reduced the number of messages. However, those guid assignments seem really weird, because:
$ doveadm fetch hdr guid 080ec72aa49e2a01c8e249fe127605f6 | grep -i
'^Message-ID: '
Message-ID: <4B1ACA53.7040503@rkit.pp.ru>
Message-ID: <29bf512f0912051251u74d246afxafdfb9e5ea24342c@mail.gmail.com>
Message-ID: <5e0214850912051300r3ebd0e44n61a4d6e020c94f4c@mail.gmail.com>
Message-ID: <4B1ACD40.3040507@btinternet.com>
Message-Id: <200912052220.00317.daniel.is.fischer@web.de>
Message-Id: <200912052225.28597.daniel.is.fischer@web.de>
Message-ID: <20091205212848.GA23711@seas.upenn.edu>
Message-Id: <200912051336.13792.hgolden@socal.rr.com>
Message-Id: <200912052243.03144.daniel.is.fischer@web.de>
Message-Id: <0B59A706-8C41-47B9-A858-5ACE297581E1@cs.uu.nl>
Message-ID: <20091205215707.GA6161@protagoras.phil.berkeley.edu>
Message-ID: <471726.55822.qm@web113106.mail.gq1.yahoo.com>
Message-ID: <4B1AD7FB.8050704@btinternet.com>
Message-ID: <5fdc56d70912051400h663a25a9w4f9b2e065a5b395e@mail.gmail.com>
Message-Id: <1B613EE3-B4F8-4F6E-8A36-74BACF0C86FC@yandex.ru>
Message-ID: <4B1ADA0E.5070207@btinternet.com>
Message-Id: <36C40624-B050-4A8C-8CAF-F15D84467180@phys.washington.edu>
Message-ID: <SNT119-W313697775F905AE968566CC6920@phx.gbl>
Message-id:
<alpine.DEB.2.00.0912052309170.31599@anubis.informatik.uni-halle.de>
Message-ID: <29bf512f0912051423safd7842ka39c8b8b6dee1ac0@mail.gmail.com>
So all these completely unrelated messages have somehow received the same guid? And that guid is stored even in the storage files themselves so they cannot be cleaned up even with force-resync? Something is _seriously_ wrong.
The complexity and opaqueness of the mdbox format is a worrisome. It would ease my mind quite a bit if there were a simple tool that would just dump out the plain message contents that are stored inside the storage files, without involving any of dovecot's index machinery. Then I would at least know that whatever happens, as long as the storage files stay intact, I can always migrate my mails into some other format.
Lauri