mdbox / selecting folder accesses hundrets of others

[Florian Lohoff]

On Mon, May 10, 2021 at 11:29:05AM +0200, Timo Sirainen wrote:
> > So every folder will at least get 2 stat calls - Okay - may be because of the 
> > mailbox list command - lets try deleting a folder.
> 
> Sounds like dovecot.list.index.
> 

[ ... ]

> > So i am in a deadend.
> > 
> > 2.2.27 eats mdboxes for breakfast and cant repair them
> > 2.3.4+ is unusuable with this mailbox.
> > 
> > All these operations were in the sub-second range with 2.2.27 ... Guess
> > i need to find something else to store mails ... 
> 
> v2.2 has mailbox_list_index=no by default, while v2.3 has changed it to "yes". So you could try changing it back to "no". Or maybe mailbox_list_index_very_dirty_syncs=yes would help also.
> 
> > Reopening the same folder again is quick and seamless. So it seems this is some
> > cache recreation code looking into other folders.
> 
> Do you mean that whenever any of the 220k folders is opened, it's first slow but then fast afterwards (even if other folders are opened)? So it's only when a new folder is opened it's being slow? That does sound like some kind of a bug.
> 
> BTW. doveconf -n output could also be helpful. This behavior might also be due to some setting.

Gotcha - Thats it - I disabled mailbox_list_index (and
mailbox_list_index_very_dirty_syncs) and now it churns through
the half a million mails queued up since the last week and even 
web/imap accesses are now happy as they were.

So somthing is pretty broken here. No operation on this mailbox should
ever request a mailbox list. All accesses are to explicitly named
folders. Be it deliver or imap list commands.

It seems even when simply opening a folder there are corner cases
where imapd/doveadm trys to rebuild the dovecot.list.index. It does
not say so in the log for what reason it does so.

Doing that on large mailboxes like this takes ages and locks the
mailbox for writing. So this machine was rebuilding the
dovecot.list.index over and over again. Delivery and other accesses
were halted to a crawl as rebuilding the index took multiple
minutes.

Flo
-- 
Florian Lohoff                                                     f at zz.de
  Any sufficiently advanced technology is indistinguishable from magic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://dovecot.org/pipermail/dovecot/attachments/20210510/ec0a70eb/attachment.sig>