Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder

[Steffen Kaiser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 8 Feb 2016, Charles Marcus wrote:

> My problem is, one of my most used folders, which was working fine up
> until a week or so ago, stopped loading the messages, and after some
> frustrating troubleshooting via email with people who don't listen very
> well, I finally got a tarball of this folder, and they are using maildir.
>
> There are about 24,000 messages in there (non-zero-byte files). This
> number sounds about right. All other folders (including INBOX, Sent,
> etc) are still working fine.
>
> The problem, though, is there are over 815,000 zero-byte-files in the
> cur directory, all showing as hardlinks (looks like maybe a whole bunch
> of duplicates for each of the real message files in the cur directory).

"zero-byte-files ... showing as hardlinks"

You mean this:

hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1

?

This is a pseudo-notation of tar to indicate hardlinks. This is no 
zero-byte file. yes, these entries are duplicates of other messages.

Note, https://en.wikipedia.org/wiki/Hard_link
if two files are hardlinked together, there is no "to" or "from". You 
cannot tell, which existed before. They just indicate that those directory 
entries point to the same physical file with the same access rights and 
times and data.
Extract the tar file to a Unix-like, inode-based filesystem supporting 
hardlinks to see.

> There are also 43 non-zero-byte message files in the new directory, and
> 1,515 of these zero-byte hardlinks (to message files in the new directory).

> There are also no non-zero-byte message files in the tmp directory, but
> there are 52 of the hardlinks, linked to something in the new directory.

if there is such entry in the tmp directory, it indicates a failed 
delivery attempt. If one entry in "tmp" is hardlinked to one entry in 
"new" of the same mailbox, it may indicate that the message was to spool 
into another mailbox (via hardlink, too), which failed fatally.

Is it possible that those messages are messages from your hoster and the 
message was to spool to many user mailboxes?

> I've never seen any of these kinds of zero-byte files before on the one
> server I managed for a long time (not shared, just used for a single
> domain).

See above.

> Anyone ever seen this before?

What does "stopped loading messages" mean?

The MUA cannot download messages?

Check if the server returns OK selecting the mailbox and if the numbers 
match, see 
http://wiki2.dovecot.org/TestInstallation

You could use

a select INBOX
b copy 1 "mailbox-name"

to copy a new message there and re-select the broken mailbox and compare 
the numbers.

Also you could test, if the server crashes on a message in the mailbox, 
try

c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)]
c FETCH 1:* FLAGS
c FETCH 1:* BODY[TEXT]

> Would running:
>
> doveadm index -u myuser *

only, if the index is corrupt.

> or
>
> doveadm force-resync -u myuser *

you can run doveadm, but cannot doveconf on the server?

>
> be appropriate commands to try to repair the damage (whatever it is)?
>
> Any other commands I could suggest running?
>
> Thanks. I know I haven't given much to go on.

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb
hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco
ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY
Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm
xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL
4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg==
=oCQf
-----END PGP SIGNATURE-----