[Dovecot] Same filenames on different servers?
Hi,
I'm really fuzzy about some problem on how Dovecot used to name files to store messages in Maildir.
I have two identical servers (let's call them "First" and "Second"), and have Dovecot and Exim set up on both of them (Maildirs are used to store messages and boxes). First server is the server to be used by people around, and the Second server is backup one, the only thing it can do while the First is alive is to get (via rsync) First's mail spool.
I set up rsync to run every 5 minutes, and that's quite well sync spools from First to Second. So if something wrong happens to First server we simple disconnect it from the network and put Second server in place of First.
The problem is that files in Maildirs on First server are named like that:
1262610335.H798006P5447.first.domain.my,S=1962:2, 1262610402.H529761P5596.first.domain.my,S=2975:2, 1262610410.H787724P5657.first.domain.my,S=75691:2, ...
and the files on Second server should be named like
1262610335.H798006P5447.second.domain.my,S=1962:2, 1262610402.H529761P5596.second.domain.my,S=2975:2, 1262610410.H787724P5657.second.domain.my,S=75691:2, ...
As soon as I enter the Second's mailbox with IMAP client Dovecot used to rename files the way they should be. If after that I rsync spools I'll end up with double sized spool as there'll be both First and Second files:
1262610335.H798006P5447.first.domain.my,S=1962:2, 1262610335.H798006P5447.second.domain.my,S=1962:2, 1262610402.H529761P5596.first.domain.my,S=2975:2, 1262610402.H529761P5596.second.domain.my,S=2975:2, ...
Long description for simple problem, yes :)
I also tried to set:
maildir_copy_preserve_filename = yes
but still no luck.
My setup is FreeBSD 7.2 with all versions in 1.1.x and 1.2.x branches (now it is 1.2.8).
Thank you in advance for you advice, yours, Alexander
On 4.1.2010, at 15.32, Alexander Chekalin wrote:
The problem is that files in Maildirs on First server are named like that:
1262610335.H798006P5447.first.domain.my,S=1962:2, 1262610402.H529761P5596.first.domain.my,S=2975:2, 1262610410.H787724P5657.first.domain.my,S=75691:2, ...
and the files on Second server should be named like
1262610335.H798006P5447.second.domain.my,S=1962:2, 1262610402.H529761P5596.second.domain.my,S=2975:2, 1262610410.H787724P5657.second.domain.my,S=75691:2,
No, they shouldn't. The maildir filename must be globally unique. Otherwise it doesn't matter what it is. Just keep them as they are.
The main problem with rsync is going to be if two servers have the same filename, but one has renamed its flags. So there are two identical base filenames but with different flags. Then Dovecot will log a warning about duplicate message and rename one of the files.
Oh, I finally find out the problem. If Dovecot see in the IMAP mailbox two message files named equally (say in cur/ and new/ dirs) it renames one of them into new name dispute this is the same message.
I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server).
It there any workaround?
Yours, Alexander Chekalin
04.01.2010 18:41, Timo Sirainen wrote:
On 4.1.2010, at 15.32, Alexander Chekalin wrote:
The problem is that files in Maildirs on First server are named like that:
1262610335.H798006P5447.first.domain.my,S=1962:2, 1262610402.H529761P5596.first.domain.my,S=2975:2, 1262610410.H787724P5657.first.domain.my,S=75691:2, ...
and the files on Second server should be named like
1262610335.H798006P5447.second.domain.my,S=1962:2, 1262610402.H529761P5596.second.domain.my,S=2975:2, 1262610410.H787724P5657.second.domain.my,S=75691:2,
No, they shouldn't. The maildir filename must be globally unique. Otherwise it doesn't matter what it is. Just keep them as they are.
The main problem with rsync is going to be if two servers have the same filename, but one has renamed its flags. So there are two identical base filenames but with different flags. Then Dovecot will log a warning about duplicate message and rename one of the files.
On 2010-01-11, Alexander Chekalin (achekalin@lazurit.com) wrote:
I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server).
It there any workaround?
I would only use rsync for migrating to a new box, or for backups only...
It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync?
Or, please describe what you are trying to accomplish?
Yes, I'm trying to make some backup solution.
The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on.
And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/, and Dovecot for some reason renames one of it).
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
Yours, Alexander
11.01.2010 15:07, Charles Marcus wrote:
On 2010-01-11, Alexander Chekalin (achekalin@lazurit.com) wrote:
I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server).
It there any workaround?
I would only use rsync for migrating to a new box, or for backups only...
It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync?
Or, please describe what you are trying to accomplish?
Hi,
This is a general backup issue (files changing during backup run) and nothing Dovecot specific. You should look into something that gives you a consistent view of the data, f.i. LVM snapshots (when you're running linux).
Regards, Tom
Alexander Chekalin wrote:
Yes, I'm trying to make some backup solution.
The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on.
And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/, and Dovecot for some reason renames one of it).
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
Yours, Alexander
11.01.2010 15:07, Charles Marcus wrote:
On 2010-01-11, Alexander Chekalin (achekalin@lazurit.com) wrote:
I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server).
It there any workaround?
I would only use rsync for migrating to a new box, or for backups only...
It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync?
Or, please describe what you are trying to accomplish?
In fact I just try to understand if Dovecot's behaviour is right and my expectations are wrong, or vise versa. I mean when server see two identical messages (files) both in cur and in new dirs it can compare these files not only by name and rename if these names are the same, but maybe check content of the messages.
11.01.2010 16:11, Tom Hendrikx пишет:
Hi,
This is a general backup issue (files changing during backup run) and nothing Dovecot specific. You should look into something that gives you a consistent view of the data, f.i. LVM snapshots (when you're running linux).
Regards, Tom
Alexander Chekalin wrote:
Yes, I'm trying to make some backup solution.
The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on.
And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/, and Dovecot for some reason renames one of it).
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
Yours, Alexander
11.01.2010 15:07, Charles Marcus wrote:
On 2010-01-11, Alexander Chekalin (achekalin@lazurit.com) wrote:
I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server).
It there any workaround?
I would only use rsync for migrating to a new box, or for backups only...
It soundslike you're using the wrong tool for the job - maybe you should be using something like imapsync?
Or, please describe what you are trying to accomplish?
On 11.1.2010, at 16.41, Alexander Chekalin wrote:
In fact I just try to understand if Dovecot's behaviour is right and my expectations are wrong, or vise versa. I mean when server see two identical messages (files) both in cur and in new dirs it can compare these files not only by name and rename if these names are the same, but maybe check content of the messages.
Feel free to write such code :) There is already a bit of checking to drop hard linked duplicates.
Timo,
I'd really love to but I'm really not sure in my code skills to patch such a sw as Dovecot.
11.01.2010 16:42, Timo Sirainen пишет:
On 11.1.2010, at 16.41, Alexander Chekalin wrote:
In fact I just try to understand if Dovecot's behaviour is right and my expectations are wrong, or vise versa. I mean when server see two identical messages (files) both in cur and in new dirs it can compare these files not only by name and rename if these names are the same, but maybe check content of the messages.
Feel free to write such code :) There is already a bit of checking to drop hard linked duplicates.
Hi,
The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on.
Use Heartbeat and DRBD for replication - all mail is synched between the servers.
If first crashes, heartbeat runs "second" as new server for the users.
As dovecot on "second" is only started if "first" is down, there are no issues about inconsistencies.
best regards,
Anton
I see the only option I have is to use filesystem sync and not rsync. Frankly I'd rather use rsync and not play with GEOM (I'm on FreeBSD) on servers right now.
The problem as I've said is not sync itself but rather Dovecot's way of treating files. You can test it yourself: go to your our cur/ dir, and copy any message to new/ dir, and then login to the mailbox with IMAP client. As soon as these messages are identical I'd expect Dovecot to remove one of it but Dovecot simple rename second message file so you'll see two identical messages in your INBOX in mail client.
That's what is strange for me. I can fight it on IMAP level (compare messages and find duplicates) but this way won't look nice.
Your, Alexander
11.01.2010 16:25, Anton Dollmaier пишет:
Hi,
The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on.
Use Heartbeat and DRBD for replication - all mail is synched between the servers.
If first crashes, heartbeat runs "second" as new server for the users.
As dovecot on "second" is only started if "first" is down, there are no issues about inconsistencies.
best regards,
Anton
On 11.1.2010, at 16.06, Alexander Chekalin wrote:
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
With v2.0 dsync would be great for this..
I'd love to use it (it's going to be magic solution for me).
But no I haven't looked at v2 yet. Is dsync is ready to use now or this is just an alpha/beta concept code?
11.01.2010 16:34, Timo Sirainen пишет:
On 11.1.2010, at 16.06, Alexander Chekalin wrote:
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
With v2.0 dsync would be great for this..
v2.0 is still in beta. I should release second beta sometimes soon, but I'm a bit too busy / internetless currently. I'm anyway using the latest hg for my own mails, so it's not completely broken, but I wouldn't necessarily recomment it for larger installations..
On 11.1.2010, at 16.45, Alexander Chekalin wrote:
I'd love to use it (it's going to be magic solution for me).
But no I haven't looked at v2 yet. Is dsync is ready to use now or this is just an alpha/beta concept code?
11.01.2010 16:34, Timo Sirainen пишет:
On 11.1.2010, at 16.06, Alexander Chekalin wrote:
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
With v2.0 dsync would be great for this..
On 1/11/2010 9:34 AM, Timo Sirainen wrote:
On 11.1.2010, at 16.06, Alexander Chekalin wrote:
Doing this using imapsync won't work. It is slow (hours vs minutes with rsync) so I can't run it every 5 minutes, and, moreover, it crashes on some messages, so can not rely on it in production.
With v2.0 dsync would be great for this..
How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...?
M.A.
No of course it won't. But dsync seems to be a solution that won't deal with storage level so it will resolve conflicts easily.
Personally I prefer rsync :)
How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...?
On 11.1.2010, at 16.46, Mario Antonio wrote:
With v2.0 dsync would be great for this..
How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...?
If there are no changes, dsync needs to do a lot less disk IO, since it just needs to open index files.
Currently you'd have to write a script to iterate through all users, but I don't see it as a huge issue. Scripting it shouldn't be too difficult, and writing code to dsync to directly support it would probably require less than 100 lines.
So I'd better wait for dsync? Or I can get it (without iterate code) now somehow?
11.01.2010 16:54, Timo Sirainen пишет:
On 11.1.2010, at 16.46, Mario Antonio wrote:
With v2.0 dsync would be great for this..
How could dsync beat rsync? With just one rsync command you can replicate the whole mail store (a root folder) Can you do the same with Dsync? or Do you have to write a script that feeds Domains/Users ...?
If there are no changes, dsync needs to do a lot less disk IO, since it just needs to open index files.
Currently you'd have to write a script to iterate through all users, but I don't see it as a huge issue. Scripting it shouldn't be too difficult, and writing code to dsync to directly support it would probably require less than 100 lines.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 11 Jan 2010, Alexander Chekalin wrote:
Yes, I'm trying to make some backup solution.
The idea was to have two identical (well, nearly identical) servers with the same software and data on them, and while the first do its job as SMTP and POP3/IMAP4 server, the second server just get copy of first's current state (that is, copy user db and mail spool). In case of something wrong with 1st server I'll just put its IP on second one and will go on.
So, where does the duplicate name comes from? This paragraphe sounds as no user accesses the secondary server by default.
And the problem is that rsynced mailboxes sometimes have duplicates that may confuse users. It happens when box changes between rsyncs by means of Dovecot's ordinary operations (when msg file move from new/ to cur/ on 1st server and after rsync to 2nd this may produce two files with same filename in new/ and cur/
When Dovecot on the second server is not running - well, I guess that's true as long as nobody logs in - , you can do a
rsync --delete -razH primaryServer secondaryServer
so the secondary server has a complete copy of the Maildir of the primary server (well, if you have separated INDEXes or CONTROL from there, another rsync is required).
If you omit the "--delete", you should have plenty of duplicate messages in cur/, because any change of keywords or status will change the filename, too. Also, any deleted message would be still there. So I still wonder, where your duplicates come from.
Your user won't complain that a message is back "NEW", once the secondary took over, will they?
I make "rsync backups" as well, but restore them as "new" folders, e.g. .Archive.Sub.Folder becomes .RESTORE.Archive.Sub.Folder, in order to avoid the problem you describe.
and Dovecot for some reason renames one of it).
Because Maildir is designed that way that no duplicate filename stem (part before colon) must exist. So any server may handle this situation on its own.
Regards,
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBS0tAg7+Vh58GPL/cAQIYhgf+OpvlwRZS1JcNXnCeBY7tsNunWk6zIHQD cU1I5LS8it/zLLBt2ORYkm0+u7rRxMYcigEgNsHB1m71f2BeYthlczPzYSm+A17b OmPwb9oZ6ZCJxk8jXzF6GZ39PgT6UoPHqiN0oo6RH2Xso6FYrp0/KibU96K9+uO0 3dXBbl10TdxCfPYvLgO46ounaBVBW3D/xDzKIUeW4DTWApnPuB/gjXs68lYKoZrb F/2pGQAwDGF1uMyOq1dY4Mefe05uzgA4TR/IBzYHMtBhG+YLpKqxg/sGFTTMjTPS DfMtHc0fFbyiIpJOZ7YwwRpq2hNd4tXm7Yq3PmG7eU+guxQboeBkJw== =I4Da -----END PGP SIGNATURE-----
On Mon, Jan 11, 2010 at 02:18:07PM +0200, Alexander Chekalin wrote:
Oh, I finally find out the problem. If Dovecot see in the IMAP mailbox two message files named equally (say in cur/ and new/ dirs) it renames one of them into new name dispute this is the same message.
I'm not sure if this is intended behaviour. It is quite easy to see such situation when doing periodical rsync (SMTP put message to new/ dir, then rsync copy it to second server, then message on the first server move to cur/, and second rsync results in two files on second server).
It there any workaround?
You could use imapsync instead: http://www.linux-france.org/prj/imapsync/
It synchronizes two IMAP mailboxes (including message flags etc) independent of the underlying implementation.
Geert-- Geert Hendrickx -=- ghen@telenet.be -=- PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!
participants (8)
-
Alexander Chekalin
-
Anton Dollmaier
-
Charles Marcus
-
Geert Hendrickx
-
Mario Antonio
-
Steffen Kaiser
-
Timo Sirainen
-
Tom Hendrikx