[Dovecot] Turning off "Fixed Duplicates" feature
Is there a way to disable this feature? Seems to be causing more harm then good right now. Without getting into too much details it is fixing the duplcates but it does not remove the old file, and the new file it creates is not marked as read like the old one was. I understand that this problem can be created by multiple rsyncs and files changing in between, but is there any way that I can just turn this feature off all together? I do not see anything in the wiki regrading settings to disable.
-- Jordan Tardif DreamHost
On Thu, 2009-11-12 at 15:42 -0800, Jordan Tardif wrote:
Is there a way to disable this feature? Seems to be causing more harm then good right now. Without getting into too much details it is fixing the duplcates but it does not remove the old file, and the new file it creates is not marked as read like the old one was. I understand that this problem can be created by multiple rsyncs and files changing in between, but is there any way that I can just turn this feature off all together? I do not see anything in the wiki regrading settings to disable.
There's no way to disable it, because it shouldn't be happening in the first place. If your rsyncs are causing it, maybe you should be using e.g. maildirlock utility to make sure Dovecot's not changing the maildir while rsync is writing.
Or you could of course always just comment out that code.
On Thu, 2009-11-12 at 15:42 -0800, Jordan Tardif wrote:
Is there a way to disable this feature? Seems to be causing more harm then good right now. Without getting into too much details it is fixing the duplcates but it does not remove the old file, and the new file it creates is not marked as read like the old one was. I understand that this problem can be created by multiple rsyncs and files changing in between, but is there any way that I can just turn this feature off all together? I do not see anything in the wiki regrading settings to disable.
There's no way to disable it, because it shouldn't be happening in the first place. If your rsyncs are causing it, maybe you should be using e.g. maildirlock utility to make sure Dovecot's not changing the maildir while rsync is writing.
Or you could of course always just comment out that code.
Heres a litle bit of background on what we are doing and maybe if you have time you can let me know what you think the best way of doing this would be. When we move users to dovecot servers, we first create a new user on the destination machine, then do a rsync from the old user (that contains all the courier* files) to the new one, and then run the migration script found in the dovecot wiki to convert the courier* files to dovecot*. 6 Hours later we rsync from the old user to the new one again to make sure that any emails that have been delived to the old user while dns updates make it to the new users account. Then we run the migration script again with --overwrite so it creates the new courier* files. Would you also reccomend removing any dovecot.index.cache files at this time as well? Can you think of another way to do this move so that dovecot would not have this problem with duplicates?
-- Jordan Tardif DreamHost
On Thu, 2009-11-12 at 16:04 -0800, Jordan Tardif wrote:
Heres a litle bit of background on what we are doing and maybe if you have time you can let me know what you think the best way of doing this would be. When we move users to dovecot servers, we first create a new user on the destination machine, then do a rsync from the old user (that contains all the courier* files) to the new one, and then run the migration script found in the dovecot wiki to convert the courier* files to dovecot*. 6 Hours later we rsync from the old user to the new one again to make sure that any emails that have been delived to the old user while dns updates make it to the new users account. Then we run the migration script again with --overwrite so it creates the new courier* files. Would you also reccomend removing any dovecot.index.cache files at this time as well? Can you think of another way to do this move so that dovecot would not have this problem with duplicates?
When do the imap/pop3 sessions actually start using the new machine? Does that depend on DNS also? Running migration script multiple times isn't a good idea. It can change IMAP UIDs and cause trouble with clients that use a local cache. It could even lose messages since client doesn't realize that there's a new message with an existing one's UID..
On Thu, 2009-11-12 at 16:04 -0800, Jordan Tardif wrote:
Heres a litle bit of background on what we are doing and maybe if you have time you can let me know what you think the best way of doing this would be. When we move users to dovecot servers, we first create a new user on the destination machine, then do a rsync from the old user (that contains all the courier* files) to the new one, and then run the migration script found in the dovecot wiki to convert the courier* files to dovecot*. 6 Hours later we rsync from the old user to the new one again to make sure that any emails that have been delived to the old user while dns updates make it to the new users account. Then we run the migration script again with --overwrite so it creates the new courier* files. Would you also reccomend removing any dovecot.index.cache files at this time as well? Can you think of another way to do this move so that dovecot would not have this problem with duplicates?
When do the imap/pop3 sessions actually start using the new machine? Does that depend on DNS also? Running migration script multiple times isn't a good idea. It can change IMAP UIDs and cause trouble with clients that use a local cache. It could even lose messages since client doesn't realize that there's a new message with an existing one's UID..
Yes that depends on DNS updating. So is there not way to avoid this Duplication problem if you need to do an rsync 2x?
-- Jordan Tardif DreamHost
On Thu, 2009-11-12 at 16:12 -0800, Jordan Tardif wrote:
Yes that depends on DNS updating. So is there not way to avoid this Duplication problem if you need to do an rsync 2x?
Not really. The move should preferably be atomic.. With Dovecot v2.0 there's dsync utility, which would pretty much do what you want. But that doesn't really help you yet.
I guess the easiest solution would be to:
- Remove the duplicate detection/fixing code from Dovecot code.
- Create a script that resolves duplicates by deleting one of them.
On Thu, 12 Nov 2009, Timo Sirainen wrote:
On Thu, 2009-11-12 at 16:12 -0800, Jordan Tardif wrote:
Yes that depends on DNS updating. So is there not way to avoid this Duplication problem if you need to do an rsync 2x?
Not really. The move should preferably be atomic.. With Dovecot v2.0 there's dsync utility, which would pretty much do what you want. But that doesn't really help you yet.
I guess the easiest solution would be to:
- Remove the duplicate detection/fixing code from Dovecot code.
- Create a script that resolves duplicates by deleting one of them.
So if I do #1 , do you see this causing any other problems with dovecot?
-- Jordan Tardif DreamHost
On Thu, 2009-11-12 at 16:22 -0800, Jordan Tardif wrote:
I guess the easiest solution would be to:
- Remove the duplicate detection/fixing code from Dovecot code.
- Create a script that resolves duplicates by deleting one of them.
So if I do #1 , do you see this causing any other problems with dovecot?
I actually meant that you should do both. But no, 1) shouldn't cause problems.
Or actually looking at the code, it already tries to detect duplicate hard links and unlinks one of them. You could change the code from:
if (st1.st_ino == st2.st_ino &&
CMP_DEV_T(st1.st_dev, st2.st_dev)) {
/* Files are the same. this means either a race condition
between stat() calls, or that the files were link()ed. */
if (st1.st_nlink > 1 && st2.st_nlink == st1.st_nlink &&
st1.st_ctime == st2.st_ctime &&
st1.st_ctime < ioloop_time - DUPE_LINKS_DELETE_SECS) {to just something like:
if (st1.st_size == st2.st_size) {and then some day when your migrations are done switch it back.
I actually meant that you should do both. But no, 1) shouldn't cause problems.
Or actually looking at the code, it already tries to detect duplicate hard links and unlinks one of them. You could change the code from:
if (st1.st_ino == st2.st_ino && CMP_DEV_T(st1.st_dev, st2.st_dev)) { /* Files are the same. this means either a race condition between stat() calls, or that the files were link()ed. */ if (st1.st_nlink > 1 && st2.st_nlink == st1.st_nlink && st1.st_ctime == st2.st_ctime && st1.st_ctime < ioloop_time - DUPE_LINKS_DELETE_SECS) {
to just something like:
if (st1.st_size == st2.st_size) {
and then some day when your migrations are done switch it back.
This looks like it has done the trick! Thank you very much! Looking forward to dovecot 2.0
-- Jordan Tardif DreamHost
participants (2)
-
Jordan Tardif
-
Timo Sirainen