[Dovecot] Courier->Dovecot Migration Issue
For testing purposes, on a small set of users, we've run the courier-dovecot-migrate.pl script. We are converting from Courier 0.43 to Dovecot 1.2.9. For the most part, things seem to be ok. However, an iPhone IMAP client user is reporting a problem where some of her messages don't show up. Even stranger is the fact that these same messages do appear when using Outlook 2k7 for instance. The common denominator for when the messages are missing seems to be whether or not the message appeared in both the Courier IMAP and Courier POP3 uid files. The missing messages appear in the CourierIMAP file, but NOT the Courier POP file. Here is a snippet from the resulting Dovecot-uidlist file:
3 V1141844265 N12392
9338 P00cf2eb6_fadc_4992_ad3a_57cc83a16eab W5146 :00cf2eb6_fadc_4992_ad3a_57cc83a16eab:2,S
9788 P0b18a2d6_97dd_4712_a09c_23c7a3d59467 W4055 :0b18a2d6_97dd_4712_a09c_23c7a3d59467:2,S
9942 P0b3cfd0b_1fc7_47b3_805f_b369a3bb6c65 W4359 :0b3cfd0b_1fc7_47b3_805f_b369a3bb6c65:2,S
10100 Pe178f1ed_d648_47e8_807a_3e2d655d7f84 W12028 :e178f1ed_d648_47e8_807a_3e2d655d7f84:2,S
10101 Pe3f28a53_a4d4_4f3d_babe_5df72bc35b47 W13974 :e3f28a53_a4d4_4f3d_babe_5df72bc35b47:2,S
10102 Pf3c3024b_2850_43ef_a397_d5cae5c8f537 W7177 :f3c3024b_2850_43ef_a397_d5cae5c8f537:2,S
10104 P6e808a68_0a7b_4528_aa67_4cc16361c56b W10817 :6e808a68_0a7b_4528_aa67_4cc16361c56b:2,S
10105 P65c8218d_7b2a_43b7_82a4_2743025bd954 W3579 :65c8218d_7b2a_43b7_82a4_2743025bd954:2,S
10106 P963cc43e_0355_481e_99d5_5468708bc28f W6410 :963cc43e_0355_481e_99d5_5468708bc28f:2,S
10107 P1405aefc_b98b_4a35_a7dd_a69199f724d0 W4703 :1405aefc_b98b_4a35_a7dd_a69199f724d0:2,S
10108 P67908fad_ec0a_4786_b0f1_c7cfaa23d09a W127288 :67908fad_ec0a_4786_b0f1_c7cfaa23d09a:2,S
10109 :9176f3b2_1927_4657_baa8_bfe4c39e89d4:2,S
10110 :79047185_aef7_48ec_aff2_6b71bd566acd:2,S
10111 :efe275a0_a7a8_4f47_b638_45a18bc0b108:2,S
The last three entries exist in the Courier IMAP file, but NOT the Courier POP file. The resulting dovecot-uidlist format is different, but to be honest, I haven't been able to find the exact specification for the format of the dovecot-uidlist file (if it exists).
What is really strange to me is: If there was a problem with the migration script and resulting dovecot-uidlist file, why would these messages show up fine in some client (ex. Outlook) and not others (ex. iPhone).
Regards,
Tony
Tony Rutherford wrote:
For testing purposes, on a small set of users, we've run the courier-dovecot-migrate.pl script. We are converting from Courier 0.43 to Dovecot 1.2.9. For the most part, things seem to be ok. However, an iPhone IMAP client user is reporting a problem where some of her messages don't show up. Even stranger is the fact that these same messages do appear when using Outlook 2k7 for instance. The common denominator for when the messages are missing seems to be whether or not the message appeared in both the Courier IMAP and Courier POP3 uid files. The missing messages appear in the CourierIMAP file, but NOT the Courier POP file. Here is a snippet from the resulting Dovecot-uidlist file:
3 V1141844265 N12392
9338 P00cf2eb6_fadc_4992_ad3a_57cc83a16eab W5146 :00cf2eb6_fadc_4992_ad3a_57cc83a16eab:2,S
9788 P0b18a2d6_97dd_4712_a09c_23c7a3d59467 W4055 :0b18a2d6_97dd_4712_a09c_23c7a3d59467:2,S
9942 P0b3cfd0b_1fc7_47b3_805f_b369a3bb6c65 W4359 :0b3cfd0b_1fc7_47b3_805f_b369a3bb6c65:2,S
10100 Pe178f1ed_d648_47e8_807a_3e2d655d7f84 W12028 :e178f1ed_d648_47e8_807a_3e2d655d7f84:2,S
10101 Pe3f28a53_a4d4_4f3d_babe_5df72bc35b47 W13974 :e3f28a53_a4d4_4f3d_babe_5df72bc35b47:2,S
10102 Pf3c3024b_2850_43ef_a397_d5cae5c8f537 W7177 :f3c3024b_2850_43ef_a397_d5cae5c8f537:2,S
10104 P6e808a68_0a7b_4528_aa67_4cc16361c56b W10817 :6e808a68_0a7b_4528_aa67_4cc16361c56b:2,S
10105 P65c8218d_7b2a_43b7_82a4_2743025bd954 W3579 :65c8218d_7b2a_43b7_82a4_2743025bd954:2,S
10106 P963cc43e_0355_481e_99d5_5468708bc28f W6410 :963cc43e_0355_481e_99d5_5468708bc28f:2,S
10107 P1405aefc_b98b_4a35_a7dd_a69199f724d0 W4703 :1405aefc_b98b_4a35_a7dd_a69199f724d0:2,S
10108 P67908fad_ec0a_4786_b0f1_c7cfaa23d09a W127288 :67908fad_ec0a_4786_b0f1_c7cfaa23d09a:2,S
10109 :9176f3b2_1927_4657_baa8_bfe4c39e89d4:2,S
10110 :79047185_aef7_48ec_aff2_6b71bd566acd:2,S
10111 :efe275a0_a7a8_4f47_b638_45a18bc0b108:2,S
The last three entries exist in the Courier IMAP file, but NOT the Courier POP file. The resulting dovecot-uidlist format is different, but to be honest, I haven't been able to find the exact specification for the format of the dovecot-uidlist file (if it exists).
What is really strange to me is: If there was a problem with the migration script and resulting dovecot-uidlist file, why would these messages show up fine in some client (ex. Outlook) and not others (ex. iPhone).
Regards,
Tony
During the migration, it appears that the emphasis is placed on maintaining the Pop3 message sequence...at the cost of possibly changing IMAP UIDs. The iPhone appears to go very strictly by the actual order of the IMAP uids...and since they're now out of sync...the messages appear to be out of order on the iPhone client. I understand the importance of ensuring that pop clients don't redownload messages...so tough to argue.
Here's a question though. If the dovecot-uidlist file is deleted (for whatever reason), it gets rebuilt by Dovecot. But, how does it get rebuilt? It does not appear to generate uids based on the date of messages...I believe that to be true. Is there any flag/option in Dovecot to build the uidlist file based on message date (uids ordered by date)? The problem is that if they're not ordered by date, again, when the iPhone gets the first 200 messages, they may not necessarily be the latest 200 messages. Also...this behaviour does not appear in other clients we've tested...TB and Outlook2007.
Tony
On Wed, 2010-02-17 at 11:31 -0500, Tony Rutherford wrote:
The last three entries exist in the Courier IMAP file, but NOT the Courier POP file. The resulting dovecot-uidlist format is different, but to be honest, I haven't been able to find the exact specification for the format of the dovecot-uidlist file (if it exists). .- During the migration, it appears that the emphasis is placed on maintaining the Pop3 message sequence...at the cost of possibly changing IMAP UIDs.
Well, with v1.1+ it's possible to preserve both POP3 and IMAP UIDs (each line is <IMAP uid> P<POP3 UIDL> :<filename>). I guess the script doesn't merge IMAP and POP3 messages well enough. Feel free to fix the script ;)
Here's a question though. If the dovecot-uidlist file is deleted (for whatever reason), it gets rebuilt by Dovecot. But, how does it get rebuilt? It does not appear to generate uids based on the date of messages...I believe that to be true. Is there any flag/option in Dovecot to build the uidlist file based on message date (uids ordered by date)?
The added mails are ordered by their filename. The filename typically begins with timestamp of when the message was received, so it's practically in the same order as mails were received.
Timo Sirainen wrote:
On Wed, 2010-02-17 at 11:31 -0500, Tony Rutherford wrote:
The last three entries exist in the Courier IMAP file, but NOT the Courier POP file. The resulting dovecot-uidlist format is different, but to be honest, I haven't been able to find the exact specification for the format of the dovecot-uidlist file (if it exists).
.-
During the migration, it appears that the emphasis is placed on maintaining the Pop3 message sequence...at the cost of possibly changing IMAP UIDs.
Well, with v1.1+ it's possible to preserve both POP3 and IMAP UIDs (each line is <IMAP uid> P<POP3 UIDL> :<filename>). I guess the script doesn't merge IMAP and POP3 messages well enough. Feel free to fix the script ;)
Here's a question though. If the dovecot-uidlist file is deleted (for whatever reason), it gets rebuilt by Dovecot. But, how does it get rebuilt? It does not appear to generate uids based on the date of messages...I believe that to be true. Is there any flag/option in Dovecot to build the uidlist file based on message date (uids ordered by date)?
The added mails are ordered by their filename. The filename typically begins with timestamp of when the message was received, so it's practically in the same order as mails were received.
Yes, unless you're unfortunate enough to be supporting a non-standard legacy message name format based on uuidgen! I did see the function where it attempts to compare based on the "standard" filename format...but again, unfortunately I have to deal with non-standard as well as the standard format :(
Thanks, Tony
On Thu, 2010-02-18 at 11:26 -0500, Tony Rutherford wrote:
Well, with v1.1+ it's possible to preserve both POP3 and IMAP UIDs (each line is <IMAP uid> P<POP3 UIDL> :<filename>). I guess the script doesn't merge IMAP and POP3 messages well enough. Feel free to fix the script ;)
Yes, unless you're unfortunate enough to be supporting a non-standard legacy message name format based on uuidgen!
I've no idea what uuidgen is, but..
I did see the function where it attempts to compare based on the"standard" filename format...but again, unfortunately I have to deal with non-standard as well as the standard format :(
No, it works with everything with Dovecot v1.1+. Like it says above, you can have in dovecot-uidlist:
1 Phello-world :12345.blah.host:2,
Now when opening such mailbox with POP3 and issuing UIDL command, you actually get:
1 hello-world
Timo Sirainen wrote:
On Thu, 2010-02-18 at 11:26 -0500, Tony Rutherford wrote:
Well, with v1.1+ it's possible to preserve both POP3 and IMAP UIDs (each line is <IMAP uid> P<POP3 UIDL> :<filename>). I guess the script doesn't merge IMAP and POP3 messages well enough. Feel free to fix the script ;)
Yes, unless you're unfortunate enough to be supporting a non-standard legacy message name format based on uuidgen!
I've no idea what uuidgen is, but..
I did see the function where it attempts to compare based on the"standard" filename format...but again, unfortunately I have to deal with non-standard as well as the standard format :(
No, it works with everything with Dovecot v1.1+. Like it says above, you can have in dovecot-uidlist:
1 Phello-world :12345.blah.host:2,
Now when opening such mailbox with POP3 and issuing UIDL command, you actually get:
1 hello-world
uuidgen generates a random unique identifier with no way of gathering a date/time stamp from it. Here's an example filename: "ffad8168_ce4b_4d25_a54d_290ed9b3a7f6:2,S" It's not ideal for many, many reasons...
Tony
On Thu, 2010-02-18 at 11:26 -0500, Tony Rutherford wrote:
Here's a question though. If the dovecot-uidlist file is deleted (for whatever reason), it gets rebuilt by Dovecot. But, how does it get rebuilt? It does not appear to generate uids based on the date of messages...I believe that to be true. Is there any flag/option in Dovecot to build the uidlist file based on message date (uids ordered by date)?
The added mails are ordered by their filename. The filename typically begins with timestamp of when the message was received, so it's practically in the same order as mails were received.
Yes, unless you're unfortunate enough to be supporting a non-standard legacy message name format based on uuidgen! I did see the function where it attempts to compare based on the "standard" filename format...but again, unfortunately I have to deal with non-standard as well as the standard format :(
Oh, this was for the above question. I was mixing it up with the other one. Right, yeah, doesn't look like it's all that great then. But you're kind of in the minority. :) And this shouldn't be a real problem typically in any case. Only in the migration case, and the migration script could be fixed for that.
participants (2)
-
Timo Sirainen
-
Tony Rutherford