Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder
Hello,
I have an el-cheapo shared hosting account on Dreamhost, and have had it for a very long time.
For the most part everything usually works fairly well, considering I do keep a lot of folders, and mail, on some of my accounts.
They are running dovecot, but still don't have a response as to the version, or doveconf -n output yet.
My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir.
There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine.
The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory).
There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory).
There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory.
I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain).
Anyone ever seen this before?
Would running:
doveadm index -u myuser *
or
doveadm force-resync -u myuser *
be appropriate commands to try to repair the damage (whatever it is)?
Any other commands I could suggest running?
Thanks. I know I haven't given much to go on.
Charles /*
*/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 8 Feb 2016, Charles Marcus wrote:
My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir.
There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine.
The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory).
"zero-byte-files ... showing as hardlinks"
You mean this:
hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1
?
This is a pseudo-notation of tar to indicate hardlinks. This is no zero-byte file. yes, these entries are duplicates of other messages.
Note, https://en.wikipedia.org/wiki/Hard_link if two files are hardlinked together, there is no "to" or "from". You cannot tell, which existed before. They just indicate that those directory entries point to the same physical file with the same access rights and times and data. Extract the tar file to a Unix-like, inode-based filesystem supporting hardlinks to see.
There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory).
There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory.
if there is such entry in the tmp directory, it indicates a failed delivery attempt. If one entry in "tmp" is hardlinked to one entry in "new" of the same mailbox, it may indicate that the message was to spool into another mailbox (via hardlink, too), which failed fatally.
Is it possible that those messages are messages from your hoster and the message was to spool to many user mailboxes?
I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain).
See above.
Anyone ever seen this before?
What does "stopped loading messages" mean?
The MUA cannot download messages?
Check if the server returns OK selecting the mailbox and if the numbers match, see http://wiki2.dovecot.org/TestInstallation
You could use
a select INBOX b copy 1 "mailbox-name"
to copy a new message there and re-select the broken mailbox and compare the numbers.
Also you could test, if the server crashes on a message in the mailbox, try
c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)] c FETCH 1:* FLAGS c FETCH 1:* BODY[TEXT]
Would running:
doveadm index -u myuser *
only, if the index is corrupt.
or
doveadm force-resync -u myuser *
you can run doveadm, but cannot doveconf on the server?
be appropriate commands to try to repair the damage (whatever it is)?
Any other commands I could suggest running?
Thanks. I know I haven't given much to go on.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL 4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg== =oCQf -----END PGP SIGNATURE-----
Just to close this out... as I said, this is on a hosted (cheapo) dreamhost account, and their support was - well, crap.
I was trying to provide suggestions for commands they could run, but I finally gave up.
They were able to get me a 'tarbell' (thought it was a typo when I first saw it, but they used the exact same word many times over multiple emails, so... ) of the folder, so I was able to restore it to my own dovecot server, and I discovered it was actually crashing dovecot after a few minutes after a select (clicked on folder).
Interestingly, I accidentally fixed it by simply decompressing it on my Windows box - all of those hardlinks gave errors, and when it was finished, they were not in the resulting uncompressed folder. Restoring that to my dovecot server and deleting the index files resulted in a successful restoration of the mails.
On 2/9/2016 2:02 AM, Steffen Kaiser skdovecot@smail.inf.fh-brs.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 8 Feb 2016, Charles Marcus wrote:
My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir.
There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine.
The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory). "zero-byte-files ... showing as hardlinks"
You mean this:
hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1
?
This is a pseudo-notation of tar to indicate hardlinks. This is no zero-byte file. yes, these entries are duplicates of other messages.
Note, https://en.wikipedia.org/wiki/Hard_link if two files are hardlinked together, there is no "to" or "from". You cannot tell, which existed before. They just indicate that those directory entries point to the same physical file with the same access rights and times and data. Extract the tar file to a Unix-like, inode-based filesystem supporting hardlinks to see.
There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory). There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory. if there is such entry in the tmp directory, it indicates a failed delivery attempt. If one entry in "tmp" is hardlinked to one entry in "new" of the same mailbox, it may indicate that the message was to spool into another mailbox (via hardlink, too), which failed fatally.
Is it possible that those messages are messages from your hoster and the message was to spool to many user mailboxes?
I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain). See above.
Anyone ever seen this before? What does "stopped loading messages" mean?
The MUA cannot download messages?
Check if the server returns OK selecting the mailbox and if the numbers match, see http://wiki2.dovecot.org/TestInstallation
You could use
a select INBOX b copy 1 "mailbox-name"
to copy a new message there and re-select the broken mailbox and compare the numbers.
Also you could test, if the server crashes on a message in the mailbox, try
c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)] c FETCH 1:* FLAGS c FETCH 1:* BODY[TEXT]
Would running:
doveadm index -u myuser * only, if the index is corrupt.
or
doveadm force-resync -u myuser * you can run doveadm, but cannot doveconf on the server?
be appropriate commands to try to repair the damage (whatever it is)?
Any other commands I could suggest running?
Thanks. I know I haven't given much to go on.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL 4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg== =oCQf -----END PGP SIGNATURE-----
participants (2)
-
Charles Marcus
-
Steffen Kaiser