[Dovecot] some mdbox mails get lost each day
Hi,
I have a strange problem here. I had installed Dovecot-2.1.6 on a customers setup. They have 102 mail accounts on the server. The started using this dovecot version on the 20th June. One mailbox suddenly started expunging mails. Not from the current day, but one week back in the past. First I thought about an Outlook issue. Some filter or archieving thing, but we did a test and on one day, the curstomer did not start Outlook first; instead he checked his mails with the SOGo-webinterface. We saw that mails got deleted again.
I checked for sieve filters. None.
I upgraded dovecot to 2.1.7, as I have not done any newer packages yet for Ubuntu (using my own PPA). Same problem one day later.
I tested with telnet locally. Same result as in the client. I also did doveadm dump -t log and index and saw that there had been expunged mails.
Yesterday I asked the customer to backup all his mails in Outlook. After that I renamed the whole mdbox folder of this account on the server and sent one test mail to recreate the mailbox. After that her restored his mail in Outlook. This morning her contacted me and said that all mails are ok right now and no mails have been erased anymore.
Coud this been a corrupted index?
I also chekced the storage/-folder and see that all mails are still on disk. Now he asked me if there is any way to get all the lost messaged back? It would be okay for him, if all mails since June are sent again, as he can clean his mailbox afterwards. So does somebody have an idea on how to solve this? Maybe scripting something in python and using local sendmail command?
Or is there some doveadm voodoo to repair the index or unexpunge all mails?
Thanks in advance
-Christian Rößner
Rößner-Network-Solutions Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 7882340-0 F: +49 6631 7882340-9 M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com
On 9.8.2012, at 11.14, Christian Rößner wrote:
I have a strange problem here. I had installed Dovecot-2.1.6 on a customers setup. They have 102 mail accounts on the server. The started using this dovecot version on the 20th June. One mailbox suddenly started expunging mails. Not from the current day, but one week back in the past. First I thought about an Outlook issue. Some filter or archieving thing, but we did a test and on one day, the curstomer did not start Outlook first; instead he checked his mails with the SOGo-webinterface. We saw that mails got deleted again.
A wrongly used doveadm expunge in a cronjob? mail_log plugin is very helpful when trying to figure out what deletes messages. http://wiki2.dovecot.org/Plugins/MailLog
Yesterday I asked the customer to backup all his mails in Outlook. After that I renamed the whole mdbox folder of this account on the server and sent one test mail to recreate the mailbox. After that her restored his mail in Outlook. This morning her contacted me and said that all mails are ok right now and no mails have been erased anymore.
Coud this been a corrupted index?
Corrupted indexes would log errors in log file. Do you see any errors/warning?
I also chekced the storage/-folder and see that all mails are still on disk. Now he asked me if there is any way to get all the lost messaged back? It would be okay for him, if all mails since June are sent again, as he can clean his mailbox afterwards. So does somebody have an idea on how to solve this? Maybe scripting something in python and using local sendmail command?
Or is there some doveadm voodoo to repair the index or unexpunge all mails?
cp mdbox ~/mdbox-backup rm -rf ~/mdbox-backup/storage/dovecot.map* ~/mdbox-backup/mailboxes/ doveadm import mdbox:~/mdbox-backup all-mails all
Now all the mails in m.* files are under all-mails folder.
Hi,
cp mdbox ~/mdbox-backup
done
rm -rf ~/mdbox-backup/storage/dovecot.map* ~/mdbox-backup/mailboxes/
done
doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
doveadm -Dv import -u foo.bar@example.com mdbox:/srv/mail/foo.bar@example.com-backup all-mails ALL doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_mail_log_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(root): Debug: Effective uid=0, gid=0, home=/root doveadm(root): Debug: fs: root=/srv/mail/foo.bar@example.com-backup, index=, control=, inbox=, alt= doveadm(root): Debug: Namespace : Using permissions from /srv/mail/foo.bar@example.com-backup: mode=0700 gid=-1 doveadm(foo.bar@example.com): Debug: Effective uid=5000, gid=5000, home=/srv/mail/foo.bar@example.com doveadm(foo.bar@example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:/srv/mail/foo.bar@example.com doveadm(foo.bar@example.com): Debug: fs: root=/srv/mail/foo.bar@example.com, index=, control=, inbox=, alt= doveadm(foo.bar@example.com): Debug: Namespace : Using permissions from /srv/mail/foo.bar@example.com: mode=0700 gid=-1
The mail box is under:
/srv/mail/foo.bar@example.com
The backup is under:
/srv/mail/foo.bar@example.com-backup
It does nothing. The command directly returns. No copied mails. So what am I missing here? Current version here is Dovecot 2.1.7.
Thanks in advance
-Christian Rößner
Rößner-Network-Solutions Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 7882340-0 F: +49 6631 7882340-9 M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com
On 9.8.2012, at 16.10, Christian Rößner wrote:
doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
I thought I'd make things easier by skipping the index rebuild, because I thought doveadm import does it automatically, but I guess it doesn't notice the problem:
doveadm -o mail=mdbox:~/mdbox-backup force-resync INBOX
Hi,
doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
I thought I'd make things easier by skipping the index rebuild, because I thought doveadm import does it automatically, but I guess it doesn't notice the problem:
doveadm -o mail=mdbox:~/mdbox-backup force-resync INBOX
done
And this created several folders under mailboxes again, but still the import command does not import anything. What am I missing here?
-Christian Rößner
Rößner-Network-Solutions Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 7882340-0 F: +49 6631 7882340-9 M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com
Hi,
doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
I thought I'd make things easier by skipping the index rebuild, because I thought doveadm import does it automatically, but I guess it doesn't notice the problem:
doveadm -o mail=mdbox:~/mdbox-backup force-resync INBOX
done
And this created several folders under mailboxes again, but still the import command does not import anything. What am I missing here?
I tested today with Dovecot 2.1.9. Still not getting mails imported :-(
How can I see, if the force-resync stuff was successful? Is there any doveadm dump -t index/log variable name that tells me, there do exist XYZ number of mails in the backup mail folder?
Many thanks in advance
-Christian Rößner
Rößner-Network-Solutions Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 7882340-0 F: +49 6631 7882340-9 M: +49 176 93118939 USt-IdNr.: DE225643613 http://www.roessner-network-solutions.com
Hi,
doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
I thought I'd make things easier by skipping the index rebuild, because I thought doveadm import does it automatically, but I guess it doesn't notice the problem:
doveadm -o mail=mdbox:~/mdbox-backup force-resync INBOX
done
And this created several folders under mailboxes again, but still the import command does not import anything. What am I missing here?
I tested today with Dovecot 2.1.9. Still not getting mails imported :-(
How can I see, if the force-resync stuff was successful? Is there any doveadm dump -t index/log variable name that tells me, there do exist XYZ number of mails in the backup mail folder?
Many thanks in advance
sorry for bumping, but I am stuck at the moment.
-Christian Rößner
Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 78823400, F: +49 6631 78823409, M: +49 176 93118939 USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com
On 16.8.2012, at 11.15, Christian Rößner wrote:
doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
I thought I'd make things easier by skipping the index rebuild, because I thought doveadm import does it automatically, but I guess it doesn't notice the problem:
doveadm -o mail=mdbox:~/mdbox-backup force-resync INBOX
done
And this created several folders under mailboxes again, but still the import command does not import anything. What am I missing here?
I tested today with Dovecot 2.1.9. Still not getting mails imported :-(
How can I see, if the force-resync stuff was successful? Is there any doveadm dump -t index/log variable name that tells me, there do exist XYZ number of mails in the backup mail folder?
Many thanks in advance
sorry for bumping, but I am stuck at the moment.
- Can you read mails from the resynced mdbox? For example:
doveadm -o mail=mdbox:/anothermdbox fetch uid all
- Is the problem just the doveadm import? Can you import anything to anything at all? What does it say with -v parameter?
Am 16.08.2012 um 10:20 schrieb Timo Sirainen:
On 16.8.2012, at 11.15, Christian Rößner wrote:
> doveadm import mdbox:~/mdbox-backup all-mails all
problem
I tried so, but this does not work here:
I thought I'd make things easier by skipping the index rebuild, because I thought doveadm import does it automatically, but I guess it doesn't notice the problem:
doveadm -o mail=mdbox:~/mdbox-backup force-resync INBOX
done
And this created several folders under mailboxes again, but still the import command does not import anything. What am I missing here?
I tested today with Dovecot 2.1.9. Still not getting mails imported :-(
How can I see, if the force-resync stuff was successful? Is there any doveadm dump -t index/log variable name that tells me, there do exist XYZ number of mails in the backup mail folder?
Many thanks in advance
sorry for bumping, but I am stuck at the moment.
- Can you read mails from the resynced mdbox? For example:
doveadm -o mail=mdbox:/anothermdbox fetch uid all
doveadm -o mail=mdbox:/srv/mail/foo.bar@example.com-backup fetch uid all | grep uid | wc -l 949
There seem to be mails. At least I get several lines with uid: <number>
- Is the problem just the doveadm import? Can you import anything to anything at all? What does it say with -v parameter?
It seems, it is just the import thing.
doveadm -Dv import -u foo.bar@example.com mdbox:/srv/mail/foo.bar@example.com-backup all-mails ALL doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so doveadm(root): Debug: Module loaded: /usr/lib/dovecot/modules/lib20_mail_log_plugin.so doveadm(root): Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm doveadm(root): Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_lookup (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_zlib_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_zlib_plugin.so: undefined symbol: i_stream_create_deflate (this is usually intentional, so just ignore this message) doveadm(root): Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_list_backend (this is usually intentional, so just ignore this message) doveadm(root): Debug: Effective uid=0, gid=0, home=/root doveadm(root): Debug: fs: root=/srv/mail/foo.bar@example.com-backup, index=, control=, inbox=, alt= doveadm(root): Debug: Namespace : Using permissions from /srv/mail/foo.bar@example.com-backup: mode=0700 gid=-1 doveadm(foo.bar@example.com): Debug: Effective uid=5000, gid=5000, home=/srv/mail/foo.bar@example.com doveadm(foo.bar@example.com): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:/srv/mail/foo.bar@example.com doveadm(foo.bar@example.com): Debug: fs: root=/srv/mail/foo.bar@example.com, index=, control=, inbox=, alt= doveadm(foo.bar@example.com): Debug: Namespace : Using permissions from /srv/mail/foo.bar@example.com: mode=0700 gid=-1
Thanks in advance
-Christian Rößner
Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 78823400, F: +49 6631 78823409, M: +49 176 93118939 USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com
On 16.8.2012, at 14.13, Christian Rößner wrote:
- Can you read mails from the resynced mdbox? For example:
doveadm -o mail=mdbox:/anothermdbox fetch uid all
doveadm -o mail=mdbox:/srv/mail/foo.bar@example.com-backup fetch uid all | grep uid | wc -l 949
There seem to be mails. At least I get several lines with uid: <number>
- Is the problem just the doveadm import? Can you import anything to anything at all? What does it say with -v parameter?
It seems, it is just the import thing.
doveadm -Dv import -u foo.bar@example.com mdbox:/srv/mail/foo.bar@example.com-backup all-mails ALL
I find this strange. Does the mdbox size on filesystem get increased at all? (You sure it's not just that they're not visible in your IMAP client?)
Hi,
doveadm -Dv import -u foo.bar@example.com mdbox:/srv/mail/foo.bar@example.com-backup all-mails ALL
I find this strange. Does the mdbox size on filesystem get increased at all? (You sure it's not just that they're not visible in your IMAP client?)
I find that strange, either :-)
I have done:
tree . ├── Drafts │ └── dbox-Mails │ └── dovecot.index.log ├── Gel&APY-schte Elemente │ └── dbox-Mails │ ├── dovecot.index.cache │ └── dovecot.index.log ├── Gesendete Elemente │ └── dbox-Mails │ ├── dovecot.index │ ├── dovecot.index.backup │ ├── dovecot.index.cache │ └── dovecot.index.log ├── INBOX │ └── dbox-Mails │ ├── dovecot.index │ ├── dovecot.index.backup │ ├── dovecot.index.cache │ └── dovecot.index.log ├── Junk │ └── dbox-Mails │ └── dovecot.index.log ├── Junk-E-Mail │ └── dbox-Mails │ └── dovecot.index.log ├── Sent │ └── dbox-Mails │ └── dovecot.index.log └── Trash └── dbox-Mails └── dovecot.index.log
16 directories, 15 files
So at least I should see a folder called all-mail, don't I?
If nothing helps, I create a new user called foo.bar@example.com-backup and ask the customer to include that account as well just for importing stuff, he still needs.
On the perspective of being a developer, too, I am interested why this does not work. So if you like to dig deeper into this, I gonna help you as much as I can.
-Christian Rößner
Bachelor of Science Informatik Erlenwiese 14, 36304 Alsfeld T: +49 6631 78823400, F: +49 6631 78823409, M: +49 176 93118939 USt-IdNr.: DE225643613, http://www.roessner-network-solutions.com
participants (2)
-
Christian Rößner
-
Timo Sirainen