[Dovecot] failed: Message has been copied too many times
Hello,
I'm running dovecot 2.1.16 in a ubuntu 12.04 server, with lazy_expunge, SiS and mdbox format.
The problem I'm having is that the index for one the mailboxes of one of my users is growing too much. This is not the first time of this problem. In previous cases, is because a message is duplicated thousand of times (I haven't found any reason for this). In these cases, moving messages to other folder and moving again to this same folder fixes the problem. So I have tried this, and I have moved all messages from this folder to another one.
But this time, after this, indexes directory is still using a lot of space. And the folder seems to have more serious problems:
amateo_adm@myotis51:~$ sudo doveadm search -u vlo mailbox BORRADOS.INBOX.MNCS doveadm(vlo): Error: Syncing mailbox BORRADOS.INBOX.MNCS failed: Message has been copied too many times (59306 + -1)
I have tried to delete the folder, with the same result:amateo_adm@myotis51:~$ sudo doveadm mailbox delete -u vlo BORRADOS.INBOX.MNCS doveadm(vlo): Error: Can't delete mailbox BORRADOS.INBOX.MNCS: Message has been copied too many times (59306 + -1)
And to purge user's mailbox, but it didn't fix the problem.PS: If this could help, BORRADOS. is the namespace used by lazy_expunge plugin.
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
El 24/10/13 12:37, Angel L. Mateo escribió:
Hello,
I'm running dovecot 2.1.16 in a ubuntu 12.04 server, withlazy_expunge, SiS and mdbox format.
The problem I'm having is that the index for one the mailboxes ofone of my users is growing too much. This is not the first time of this problem. In previous cases, is because a message is duplicated thousand of times (I haven't found any reason for this). In these cases, moving messages to other folder and moving again to this same folder fixes the problem. So I have tried this, and I have moved all messages from this folder to another one.
But this time, after this, indexes directory is still using a lotof space. And the folder seems to have more serious problems:
amateo_adm@myotis51:~$ sudo doveadm search -u vlo mailbox BORRADOS.INBOX.MNCS doveadm(vlo): Error: Syncing mailbox BORRADOS.INBOX.MNCS failed: Message has been copied too many times (59306 + -1)
I have tried to delete the folder, with the same result:amateo_adm@myotis51:~$ sudo doveadm mailbox delete -u vlo BORRADOS.INBOX.MNCS doveadm(vlo): Error: Can't delete mailbox BORRADOS.INBOX.MNCS: Message has been copied too many times (59306 + -1)
And to purge user's mailbox, but it didn't fix the problem.PS: If this could help, BORRADOS. is the namespace used by lazy_expunge plugin.
One more thing, when I said that the index has grown too much is because index directory for this mailbox folder is about 850MB, and the mailbox originally had about 3000 messages. This is the list of files in the index directory:
root@myotis51:/mail/indexes/vl/vlo/expunged/mailboxes/INBOX/MNCS# ls -lh total 853M -rw------- 1 vmail vmail 127M oct 24 11:16 dovecot.index -rw------- 1 vmail vmail 127M oct 24 11:02 dovecot.index.backup -rw------- 1 vmail vmail 186M oct 24 12:14 dovecot.index.cache -rw------- 1 vmail vmail 180M oct 24 12:36 dovecot.index.log -rw------- 1 vmail vmail 138M oct 24 11:02 dovecot.index.log.2
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
On 24.10.2013, at 13.41, Angel L. Mateo <amateo@um.es> wrote:
amateo_adm@myotis51:~$ sudo doveadm search -u vlo mailbox BORRADOS.INBOX.MNCS doveadm(vlo): Error: Syncing mailbox BORRADOS.INBOX.MNCS failed: Message has been copied too many times (59306 + -1)
Looks like the index is corrupted. It should never have gotten that high. Have you tried doveadm force-resync -u vlo INBOX?
One more thing, when I said that the index has grown too much is because index directory for this mailbox folder is about 850MB, and the mailbox originally had about 3000 messages. This is the list of files in the index directory:
root@myotis51:/mail/indexes/vl/vlo/expunged/mailboxes/INBOX/MNCS# ls -lh total 853M -rw------- 1 vmail vmail 127M oct 24 11:16 dovecot.index -rw------- 1 vmail vmail 127M oct 24 11:02 dovecot.index.backup -rw------- 1 vmail vmail 186M oct 24 12:14 dovecot.index.cache -rw------- 1 vmail vmail 180M oct 24 12:36 dovecot.index.log -rw------- 1 vmail vmail 138M oct 24 11:02 dovecot.index.log.2
These files are way too large for a mailbox with just 3000 mails.
El 26/10/13 17:10, Timo Sirainen escribió:
On 24.10.2013, at 13.41, Angel L. Mateo <amateo@um.es> wrote:
amateo_adm@myotis51:~$ sudo doveadm search -u vlo mailbox BORRADOS.INBOX.MNCS doveadm(vlo): Error: Syncing mailbox BORRADOS.INBOX.MNCS failed: Message has been copied too many times (59306 + -1)
Looks like the index is corrupted. It should never have gotten that high. Have you tried doveadm force-resync -u vlo INBOX?
I have just tried it now, for the same problem with other user. But it didn't fix anything.
The problem is that for some reason, dovecot is duplicating user mails. In the problem I'm having today, a user has 3833 distinct messages in a mailbox, but 122 of them are duplicated 19866 each, so a search in the folder shows 2228712 messages. A force-resync of the folder doesn't fix anything.
What worried me the most is that this problem is appearing very often in my system (once a week at least).
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
El 28/10/13 10:46, Angel L. Mateo escribió:
The problem is that for some reason, dovecot is duplicating usermails. In the problem I'm having today, a user has 3833 distinct messages in a mailbox, but 122 of them are duplicated 19866 each, so a search in the folder shows 2228712 messages. A force-resync of the folder doesn't fix anything.
What worried me the most is that this problem is appearing veryoften in my system (once a week at least).
Are you sure it's dovecot who is duplicating emails? I have seen this before (not so many copies, but the same effect) and I've always thought it was a thunderbird thing (in every case I've seen the client was thunderbird, and I think your users also use it).
I've always solved with the thunderbird function to remove duplicates.
HTH
On 2013-10-31 11:17 AM, Joseba Torre <joseba.torre@ehu.es> wrote:
Are you sure it's dovecot who is duplicating emails? I have seen this before (not so many copies, but the same effect) and I've always thought it was a thunderbird thing (in every case I've seen the client was thunderbird, and I think your users also use it).
I have seen this happen when the user tries to mass delete thousands of emails using 'Move to Trash', then is impatient when it slows to a crawl, and they try to close Thunderbird and end up 'force quitting' it...
We had one user that ended up with 5 copies of thousands of messages from doing this repeatedly
I've always solved with the thunderbird function to remove duplicates.
Exactly what I used to clean up the mess...
--
Best regards,
*/Charles/*
El 31/10/13 16:17, Joseba Torre escribió:
Are you sure it's dovecot who is duplicating emails? I have seen this
No, I'm not sure.before (not so many copies, but the same effect) and I've always thought it was a thunderbird thing (in every case I've seen the client was thunderbird, and I think your users also use it).
It could be the reason. My users are using thunderbird and this problem always happened with users with lot of automatic mails in a mailbox, so the could be trying to delete them (although I told to some of them and they told me that they didn't, but you know...)
I've always solved with the thunderbird function to remove duplicates.
Where is this function? Is it a complement?-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
El 05/11/13 09:10, Angel L. Mateo escribió:
El 31/10/13 16:17, Joseba Torre escribió:
Are you sure it's dovecot who is duplicating emails? I have seen this
No, I'm not sure.before (not so many copies, but the same effect) and I've always thought it was a thunderbird thing (in every case I've seen the client was thunderbird, and I think your users also use it).
It could be the reason. My users are using thunderbird and thisproblem always happened with users with lot of automatic mails in a mailbox, so the could be trying to delete them (although I told to some of them and they told me that they didn't, but you know...)
In my personal case, it happens every now and then without any external activity, and definetly without any manual deleting (may be it's related to automatic filtering?)
I've always solved with the thunderbird function to remove duplicates.
Where is this function? Is it a complement?
I don't think so. Just right click on any folder in the folder pane, and in spanish there is a "Borrar duplicados" option. But I have installed a "Remove Duplicate Messages (Alternate)" extension, maybe this option comes from this.
Aaaaaaagur.
El 05/11/13 14:59, Joseba Torre escribió:
El 05/11/13 09:10, Angel L. Mateo escribió:
El 31/10/13 16:17, Joseba Torre escribió:
Are you sure it's dovecot who is duplicating emails? I have seen this
No, I'm not sure.before (not so many copies, but the same effect) and I've always thought it was a thunderbird thing (in every case I've seen the client was thunderbird, and I think your users also use it).
It could be the reason. My users are using thunderbird and thisproblem always happened with users with lot of automatic mails in a mailbox, so the could be trying to delete them (although I told to some of them and they told me that they didn't, but you know...)
In my personal case, it happens every now and then without any external activity, and definetly without any manual deleting (may be it's related to automatic filtering?)
After a few more cases I'm almost surely that the problem is with thunderbird filters (when a filter has to move a lot of messages from inbox folder to another one).
-- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868887590 Fax: 868888337
On 28.10.2013, at 11.46, Angel L. Mateo <amateo@um.es> wrote:
El 26/10/13 17:10, Timo Sirainen escribió:
On 24.10.2013, at 13.41, Angel L. Mateo <amateo@um.es> wrote:
amateo_adm@myotis51:~$ sudo doveadm search -u vlo mailbox BORRADOS.INBOX.MNCS doveadm(vlo): Error: Syncing mailbox BORRADOS.INBOX.MNCS failed: Message has been copied too many times (59306 + -1)
Looks like the index is corrupted. It should never have gotten that high. Have you tried doveadm force-resync -u vlo INBOX?
I have just tried it now, for the same problem with other user. But it didn't fix anything.
The problem is that for some reason, dovecot is duplicating user mails. In the problem I'm having today, a user has 3833 distinct messages in a mailbox, but 122 of them are duplicated 19866 each, so a search in the folder shows 2228712 messages. A force-resync of the folder doesn't fix anything.
What worried me the most is that this problem is appearing very often in my system (once a week at least).
In normal operation it wouldn’t be possible to get the refcount that high. The IMAP command would simply fail instead of growing the refcount obove 32768. The only possibilities I can think of are:
a) something causing corruption b) multiple IMAP sessions at exactly the same time copying the same mail
Anyway, it’s definitely a bug that the refcount can’t be shrunk. Fixed: http://hg.dovecot.org/dovecot-2.2/rev/8cdd4124de0b
participants (4)
-
Angel L. Mateo
-
Charles Marcus
-
Joseba Torre
-
Timo Sirainen