On 16 Aug 2017, at 23.59, Joseph Tam jtam.home@gmail.com wrote:

Timo Sirainen wrote:

...

There are various changes in this release that can be used to significantly reduce disk IO with:

1. NFS storage especially, but I guess also other remote filesystems and even some with local disks
2. When mail storage and INDEX storage are separated

Thanks for these changes! Big win for my setup. My servers are not overly stressed, but how much performance gain (using any metric you choose) can one expect from these changes?

Depends a lot of things, but for the one specific NFS installation we reduced NFS IOPS by something like 70% And actually they haven't finished setting up the local LISTINDEX changes yet, so maybe even more.

...

• mail_location can now include VOLATILEDIR=<path> parameter. This is used for creating lock files and in future potentially other files that don't need to exist permanently. The path could point to tmpfs for example. This is especially useful to avoid creating lock files to NFS or other remote filesystems. For example: mail_location=sdbox:~/sdbox:VOLATILEDIR=/tmp/volatile/%2.256Nu/%u

Is "/tmp/volatile" auto-created, or must be pre-created?

Autocreated.

...

• mail_location's LISTINDEX=<path> can now contain a full path. This allows storing mailbox list index to a different storage than the rest of the indexes, for example to tmpfs.

Is this in any way related to VOLATILEDIR? Can they be set to the same value without problems?

Yeah, can be set to the same directory. Maybe even a good idea.

...

• mail_location can now include NO-NOSELECT parameter. This automatically deletes any \NoSelect mailboxes that have no children. These mailboxes are sometimes confusing to users.

Sorry for my IMAP ignorance, but how can this situation come about?

a CREATE noselectbox/ --> With this change it's the same as CREATE "noselectbox" = will become selectable.

or

a CREATE noselectbox/childbox b DELETE noselectbox/childbox --> noselectbox is left behind. With this change it will be auto-deleted.

...

• mail_location can now include ITERINDEX parameter. This tells Dovecot to perform mailbox listing from the INDEX path instead of from the mail root path. It's mainly useful when the INDEX storage is on a faster storage.
• If mailbox_list_index_very_dirty_syncs=yes, the list index is no longer refreshed against filesystem when listing mailboxes. This allows the mailbox listing to be done entirely by only reading the mailbox list index.
• Added mailbox_list_index_include_inbox setting to control whether INBOX's STATUS information should be cached in the mailbox list index. The default is "no", but it may be useful to change it to "yes", especially if LISTINDEX points to tmpfs.

So as I understand it, the optimzation comes about from segregating mail data information into 3 parts: raw mail, indices, and volatile components, putting them into increasingly better performing storage media.

How do these I/O optimizations affect the client's view of a mailbox if their mailbox is subject to modification outside the dovecot system (e.g. procmail, mail readers directly modifies mailbox files)? Is there a trade-off between metadata consistency and performance, or it's a win-win all around?

I think already even with mailbox_list_index=yes it won't notice external changes done to the mailboxes when doing STATUS calls, so better to avoid that. It does get fixed automatically once the folder is selected though.