[Dovecot] infinite loop deleting ~160'000 mails (was Re: Unexpected behaviour when deleteing a big mailbox)

[Timo Sirainen]

On Fri, 2007-03-30 at 15:39 +0200, Steffen Kaiser wrote:
> dovecot: Mar 30 08:49:27 Error: IMAP(dvtest2) [12578]: Transaction log 
> file /var/cache/dovecot/31022/index/.box1/dovecot.index.log: marked 
> corrupted
> dovecot: Mar 30 08:49:28 Error: IMAP(dvtest2) [12581]: Corrupted 
> transaction log file 
> /var/cache/dovecot/31022/index/.box1/dovecot.index.log: Append with UID 
> 168554, but next_uid = 168555
> dovecot: Mar 30 08:49:28 Error: IMAP(dvtest2) [12488]: Transaction log 
> file /var/cache/dovecot/31022/index/.box1/dovecot.index.log: marked 
> corrupted

These and the duplicate link errors are because of
maildir_copy_preserve_filename=yes. With =no it shouldn't log any
errors.

> box1 (the copybox) contains 168778 messages.
> When I first selected the box1, there were some 
> "Fixed a duplicate link" Warnings, then Thunderbird tried to download all 
> the message headers. It never returned to something useful state.
> straceing the imap process I saw that Dovecot reads all messages fully, 
> header & body, then sends the client:
> 
> * 49136 FETCH (UID 49235 RFC822.SIZE 7426 FLAGS (\Recent) 
> BODY[HEADER.FIELDS (FROM TO CC SUBJECT DATE MESSAGE-ID PRIORITY X-PRIORITY 
> REFERENCES NEWSGROUPS IN-REPLY-TO CONTENT-TYPE)] {668}
> And the headers.

For all the messages I guess? So was there a Dovecot bug related to
this, or was it just that Thunderbird doesn't like that many messages?

> When I tried to delete all the messages through the "Trashbox", the 
> copy-to-Trash process fails with (see end of mail).
> Then the process starts again with uid=1, again the same error is issued 
> at the same uid.
> After the first "Out-of-mem" condition, I see "delete uid 1" in the log, 
> but the messages are still there and the COPY is still performed again.

Hmm. So copying 180k messages eats 250MB of memory? That sounds a bit
much, but I guess it's possible.. Maildir code takes around 80-100
bytes/message, index file code takes 8 bytes/message + more for
keywords. But even assuming about the worst case scenarios, it shouldn't
take more than 30MB. Wonder where the rest is going.

> Dunno, I simply suggest that the admin may limit the number
> of items a COPY command may be performed for.  I mean, one can
> move/copy/delete-to-trash mails in chunks of 50'000 messages - nobody
> sane has that many message in one mailbox, hasn't one?

Right, so this practically never happens with real users. I don't want
to add any more settings that are mostly useless.

You can anyway make it work by growing mail_process_size.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070330/415056cb/attachment-0001.pgp