[dovecot] message order reversed on copying
Hi,
I noticed an annoying behavior of dovecot: the message order (sequence numbering) is reversed on COPYing in the maildir.
I suspect that the following code in maildir-save.c is working like "pushing".
mf->next = ctx->files;
...
ctx->files = mf;Later ctx->files is used like this.
for (mf = ctx->files; mf != NULL; mf = mf->next) {-- fuyuki
On Sat, 2003-04-05 at 10:15, Kimura Fuyuki wrote:
I noticed an annoying behavior of dovecot: the message order (sequence numbering) is reversed on COPYing in the maildir.
I suspect that the following code in maildir-save.c is working like "pushing".
Order for new mails is currently the same as the order in which readdir() returns the mails. I guess some filesystems list the files in the order they were written to the directory, but many don't. It anyway can't be relied on, so I won't even try.
Better fix for this comes once I add support for UIDPLUS IMAP extension. That requires knowing the new UID for mail in destination mailbox, so I have to update the indexes immediately, which also saves the mail order permanently.
On 6 Apr 2003, Timo Sirainen wrote:
On Sat, 2003-04-05 at 10:15, Kimura Fuyuki wrote:
I noticed an annoying behavior of dovecot: the message order (sequence numbering) is reversed on COPYing in the maildir.
I suspect that the following code in maildir-save.c is working like "pushing".
Order for new mails is currently the same as the order in which readdir() returns the mails. I guess some filesystems list the files in the order they were written to the directory, but many don't. It anyway can't be relied on, so I won't even try.
Better fix for this comes once I add support for UIDPLUS IMAP extension. That requires knowing the new UID for mail in destination mailbox, so I have to update the indexes immediately, which also saves the mail order permanently.
You shouldn't need to do that. All you need to do is to sort the new message files (by name or by mtime - by name will be faster), before processing them. I remember fixing this same problem in the maildir patches for UW-imap, and this is pretty much all it took:
if (!(dir = opendir (tmp)))
return;
while (d = readdir (dir)) {
if (d->d_name[0] == '.')
continue; /* skip .files */
nfiles = scandir (tmp, &names, NULL, maildir_namesort);
for (i = 0; i < nfiles; i++)
{
struct direct *d = names[i];
sprintf (file,"%s/../new/%s",LOCAL->dir,d->d_name); /* make sure this is a normal file */
- if (stat (file,&sbuf) == 0 && S_ISREG (sbuf.st_mode)) {
- if (d->d_name[0] != '.' && stat (file, &sbuf) == 0 && S_ISREG (sbuf.st_mode))
- {
As a workaround, Kimura, you can delete dovecot's .imap.* index files, and the new UIDs will be sorted again - until next time.
-- Charlie
On Sun, 2003-04-06 at 22:12, Charlie Brady wrote:
Better fix for this comes once I add support for UIDPLUS IMAP extension. That requires knowing the new UID for mail in destination mailbox, so I have to update the indexes immediately, which also saves the mail order permanently.
You shouldn't need to do that. All you need to do is to sort the new message files (by name or by mtime - by name will be faster), before processing them.
Yes, well, that does it too :)
- nfiles = scandir (tmp, &names, NULL, maildir_namesort);
scandir() isn't portable though. I'll just read them into array and qsort() them. I pretty much have to do that anyway with the new scanning code which uses (NFS-safe) .uidlist file to find out if the message already has UID.
As a workaround, Kimura, you can delete dovecot's .imap.* index files, and the new UIDs will be sorted again - until next time.
Which changes UIDVALIDITY and forces clients to discard their cache. Not too good idea. Pretty much the same as just disabling the index files.
On 6 Apr 2003, Timo Sirainen wrote:
On Sun, 2003-04-06 at 22:12, Charlie Brady wrote:
You shouldn't need to do that. All you need to do is to sort the new message files (by name or by mtime - by name will be faster), before processing them.
Yes, well, that does it too :)
- nfiles = scandir (tmp, &names, NULL, maildir_namesort);
scandir() isn't portable though.
No? Which OS doesn't have it?
As a workaround, Kimura, you can delete dovecot's .imap.* index files, and the new UIDs will be sorted again - until next time.
Which changes UIDVALIDITY and forces clients to discard their cache.
Yes, I could have mentioned that. Still, it's the only way I know to restore the sorting order once it's jumbled.
Not too good idea. Pretty much the same as just disabling the index files.
No, because after you fix the problem the indexes will be usable again. But until then, you are right :-)
-- Charlie
On Mon, 2003-04-07 at 01:15, Charlie Brady wrote:
- nfiles = scandir (tmp, &names, NULL, maildir_namesort); scandir() isn't portable though. No? Which OS doesn't have it?
It's not in any standard -> it's not portable. Maybe it's portable enough, but I try to avoid those whenever possible.
Which changes UIDVALIDITY and forces clients to discard their cache.
Yes, I could have mentioned that. Still, it's the only way I know to restore the sorting order once it's jumbled.
How about making your client sort the messages by received-date? :)
Timo Sirainen wrote:
Yes, I could have mentioned that. Still, it's the only way I know to restore the sorting order once it's jumbled.
How about making your client sort the messages by received-date? :)
I generally tend to copy in received order, even if the dates are "jumbled". Then they're TRULY in the correct order.
--Ian.
At Sun, 6 Apr 2003 15:12:11 -0400 (EDT), Charlie Brady <charlieb-dovecot@e-smith.com> wrote:
As a workaround, Kimura, you can delete dovecot's .imap.* index files, and the new UIDs will be sorted again - until next time.
I'm now running dovecot with maildir_copy_with_hardlinks and it works well enough since it doesn't do the deliberate reversing, though I'll be happier if the dovecot implements UIDPLUS or merges your fix. :)
-- fuyuki
On Mon, 2003-04-07 at 02:50, Kimura Fuyuki wrote:
As a workaround, Kimura, you can delete dovecot's .imap.* index files, and the new UIDs will be sorted again - until next time.
I'm now running dovecot with maildir_copy_with_hardlinks and it works well enough since it doesn't do the deliberate reversing, though I'll be happier if the dovecot implements UIDPLUS or merges your fix. :)
My maildir syncing rewrite is almost finished now, just a few tweaks left and I'll cvs commit that. It sorts the new files, supports permanent UIDs in NFS-safe .uidlist file, takes less memory, checks that maildir doesn't contain base filename duplicates, and behaves better with other clients.
Finally. I started that code a month ago..
participants (4)
-
Charlie Brady
-
Ian R. Justman
-
Kimura Fuyuki
-
Timo Sirainen