Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder

[Charles Marcus]

Just to close this out... as I said, this is on a hosted (cheapo)
dreamhost account, and their support was - well, crap.

I was trying to provide suggestions for commands they could run, but I
finally gave up.

They were able to get me a 'tarbell' (thought it was a typo when I first
saw it, but they used the exact same word many times over multiple
emails, so... ) of the folder, so I was able to restore it to my own
dovecot server, and I discovered it was actually crashing dovecot after
a few minutes after a select (clicked on folder).

Interestingly, I accidentally fixed it by simply decompressing it on my
Windows box - all of those hardlinks gave errors, and when it was
finished, they were not in the resulting uncompressed folder. Restoring
that to my dovecot server and deleting the index files resulted in a
successful restoration of the mails.


On 2/9/2016 2:02 AM, Steffen Kaiser <skdovecot at smail.inf.fh-brs.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Mon, 8 Feb 2016, Charles Marcus wrote:
>
>> My problem is, one of my most used folders, which was working fine up
>> until a week or so ago, stopped loading the messages, and after some
>> frustrating troubleshooting via email with people who don't listen very
>> well, I finally got a tarball of this folder, and they are using maildir.
>>
>> There are about 24,000 messages in there (non-zero-byte files). This
>> number sounds about right. All other folders (including INBOX, Sent,
>> etc) are still working fine.
>>
>> The problem, though, is there are over 815,000 zero-byte-files in the
>> cur directory, all showing as hardlinks (looks like maybe a whole bunch
>> of duplicates for each of the real message files in the cur directory).
> "zero-byte-files ... showing as hardlinks"
>
> You mean this:
>
> hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1
>
> ?
>
> This is a pseudo-notation of tar to indicate hardlinks. This is no 
> zero-byte file. yes, these entries are duplicates of other messages.
>
> Note, https://en.wikipedia.org/wiki/Hard_link
> if two files are hardlinked together, there is no "to" or "from". You 
> cannot tell, which existed before. They just indicate that those directory 
> entries point to the same physical file with the same access rights and 
> times and data.
> Extract the tar file to a Unix-like, inode-based filesystem supporting 
> hardlinks to see.
>
>> There are also 43 non-zero-byte message files in the new directory, and
>> 1,515 of these zero-byte hardlinks (to message files in the new directory).
>> There are also no non-zero-byte message files in the tmp directory, but
>> there are 52 of the hardlinks, linked to something in the new directory.
> if there is such entry in the tmp directory, it indicates a failed 
> delivery attempt. If one entry in "tmp" is hardlinked to one entry in 
> "new" of the same mailbox, it may indicate that the message was to spool 
> into another mailbox (via hardlink, too), which failed fatally.
>
> Is it possible that those messages are messages from your hoster and the 
> message was to spool to many user mailboxes?
>
>> I've never seen any of these kinds of zero-byte files before on the one
>> server I managed for a long time (not shared, just used for a single
>> domain).
> See above.
>
>> Anyone ever seen this before?
> What does "stopped loading messages" mean?
>
> The MUA cannot download messages?
>
> Check if the server returns OK selecting the mailbox and if the numbers 
> match, see 
> http://wiki2.dovecot.org/TestInstallation
>
> You could use
>
> a select INBOX
> b copy 1 "mailbox-name"
>
> to copy a new message there and re-select the broken mailbox and compare 
> the numbers.
>
> Also you could test, if the server crashes on a message in the mailbox, 
> try
>
> c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)]
> c FETCH 1:* FLAGS
> c FETCH 1:* BODY[TEXT]
>
>> Would running:
>>
>> doveadm index -u myuser *
> only, if the index is corrupt.
>
>> or
>>
>> doveadm force-resync -u myuser *
> you can run doveadm, but cannot doveconf on the server?
>
>> be appropriate commands to try to repair the damage (whatever it is)?
>>
>> Any other commands I could suggest running?
>>
>> Thanks. I know I haven't given much to go on.
> - -- 
> Steffen Kaiser
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb
> hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco
> ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY
> Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm
> xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL
> 4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg==
> =oCQf
> -----END PGP SIGNATURE-----