[Dovecot] errors in dovecot-uidlist
We have migrated a mailsystem to postfix/dovecot, using the courier migrate script, and we have run into a problem with dovecot-uidlist.
The messages are not listed in the correct order .. This is an example dovecot-uidlist -file:
1 1177585200 49 22 1179084774.P7191Q0M321163.webmail1,S=8245:2,S 24 1179377823.P15932Q0M835219.smtp1,S=3462:2,S 25 1179668346.P23942Q0M941365.webmail1,S=4820357:2,S 31 1179864561.P22903Q0M433768.smtp2,S=3306:2,S 34 1180344355.P25491Q0M17232.smtp2,S=5111:2,S 35 1180454170.P14932Q0M841649.smtp2,S=2024:2, 36 1168867750.H82058P20971.smtp1:2,SR 37 1170339716.H537368P4032.smtp1:2,S 38 1171291025.H352867P8785.smtp2:2,SR 39 1171357049.H141820P2105.smtp1:2,SR 40 1171460195.H933928P14324.smtp1:2,SR 41 1171616121.H812220P27503.smtp2:2,SR 42 1171872919.H950768P16201.smtp1:2,S 43 1173694618.H330094P23347.smtp1:2,S 44 1175238705.H589518P693.smtp1:2,SR 45 1176800600.H716261P10514.smtp1:2,SR 46 1177615732.H355340P10418.webmail1:2,SR 47 1180471524.P30985Q0M109430.smtp2,S=16621 48 1180597849.P8169Q0M990486.smtp1,S=33326
As you can see, the unix timestamps are descending between uid 35 and uid 36 .. and this causes problems if we intend to make webmail access the messages through imap. (it expects to receive the messages sorted by when they were received)
Is there any way I can sort the list accourding to the unix timestamps, and still make it transparent to the pop3/imap clients?
I would prefer a solution that does not involve users downloading allready downloaded messages via pop3.
-- Erland Nylend
On Mon, 2007-06-04 at 12:47 +0200, Erland Nylend wrote:
As you can see, the unix timestamps are descending between uid 35 and uid 36 .. and this causes problems if we intend to make webmail access the messages through imap. (it expects to receive the messages sorted by when they were received)
Is there any way I can sort the list accourding to the unix timestamps, and still make it transparent to the pop3/imap clients?
I would prefer a solution that does not involve users downloading allready downloaded messages via pop3.
There's no easy way. The UID numbers were apparently converted from courierpop3dsizelist file. So either the UID numbers need to be lost, or the list isn't sorted. Dovecot doesn't yet support having separate UIDs for POP3 and IMAP.
One thing you could do is insert X-UIDL headers to the maildir files and set pop3_reuse_xuidl=yes, but that would change the file sizes and possibly break some things.
But IMAP supports sorting messages by received date (internaldate). Maybe your webmail can be configured to do that?
On 2007-06-05, 21:29, Timo Sirainen wrote:
Is there any way I can sort the list accourding to the unix timestamps, and still make it transparent to the pop3/imap clients?
I would prefer a solution that does not involve users downloading allready downloaded messages via pop3.
There's no easy way. The UID numbers were apparently converted from courierpop3dsizelist file. So either the UID numbers need to be lost, or the list isn't sorted. Dovecot doesn't yet support having separate UIDs for POP3 and IMAP.
I have been debugging this a bit further, and I believe I have been able to reproduce how the problem started.
We run the migrate script on a (courier) account containing messages.
The script fails, and produces a dovecot-uidlist -file containing only "1 -1 0" (you made a patch when the problem was reported: http://dovecot.org/list/dovecot-cvs/2007-May/008728.html)
We delete the erroneous dovecot-uidlist -files, and so we are left with an account without a dovecot-uidlist file, but it still contains messages in cur
Messages come in, and deliver creates a dovecot-uidlist with uids "1,2 ... N"
The user runs imap/pop3, and dovecot notices that there are messages in cur, and adds them to the uidlist, after uids "1,2,3 ... N"
.. so, we end up with an unordered list.
Suggestion: make deliver create the index files, and a complete dovecot-uidlist when a message is sent to an account without index files.
-- Erland Nylend
On Thu, 2007-06-14 at 13:01 +0200, Erland Nylend wrote:
We delete the erroneous dovecot-uidlist -files, and so we are left with an account without a dovecot-uidlist file, but it still contains messages in cur
Messages come in, and deliver creates a dovecot-uidlist with uids "1,2 ... N"
The user runs imap/pop3, and dovecot notices that there are messages in cur, and adds them to the uidlist, after uids "1,2,3 ... N"
.. so, we end up with an unordered list.
Ah. I had never thought of that. But it's a problem only when you're migrating from another server..
Suggestion: make deliver create the index files, and a complete dovecot-uidlist when a message is sent to an account without index files.
This would slow down deliver unneededly. A better way might be to just not update dovecot-uidlist at all (which doesn't belong to "indexes" btw). This could probably be somewhat easily patched for v1.0, but for v1.1 which supports UIDPLUS extension Dovecot must always get an UID for newly saved messages.
And that still wouldn't solve the underlying problem that if you mix external maildir deliveries and Dovecot-delievered/saved mails you end up getting them in wrong order.
I don't really see any good solutions to this problem. Perhaps it's not a real problem except with migrations. For that the migration script itself could add all the files to dovecot-uidlist file.
participants (2)
-
Erland Nylend
-
Timo Sirainen