[Dovecot] New mailbox format

[Timo Sirainen]

On 23.9.2005, at 18:04, John Peacock wrote:

> Timo Sirainen wrote:
>> While the maildir format itself is simple, it's actually really
>> difficult to handle correctly when the maildir is changing under us.
>> Files can be renamed at any time so you'll have to be prepared to look
>> for the file's new name at any time. Filesystems also don't work the 
>> way
>> maildir assumes they do, so you have to work around their limitations
>> too.
>
> I thought that was the point of dovecot-lda, so that you know that 
> dovecot is the only process touching the message files and can update 
> the indexes as things change, rather than after the fact.

Dovecot still can't know that it's the only one, but I had been 
thinking about adding an option to make Dovecot assume it's the only 
program modifying the maildir. That should avoid scanning the cur/ 
directory constantly. Except that probably still requires some changes 
since up-to-date maildir filenames aren't stored anywhere, but maybe 
they could be guessed by looking at the base filename and mail's 
current flags from indexes and generating the most-likely filename 
based on them (or did I already have that code..? :)

>> This could possibly be also automatically set per mailbox. If user
>> always expunges all mails at a time (POP3) or never expunges (mailing
>> list archives), there would be only a single file.
>
> If you are thinking that way, it might be useful to mark mailboxes as 
> either:
>
> 	a) sequential access (most POP3 and archives);
> 	b) random access (IMAP).
>
> and provide different tuning based on usage expectations, rather than 
> trying to actively tune.  Then the system admin could choose which 
> fits their needs best; we only have a few people using POP3 and 
> everyone else is IMAP, and I frankly don't care what kind of 
> performance the POP3 people have.

I haven't really thought about this yet, but a global setting wouldn't 
usually be enough for all users and I'm not sure how it can be set 
per-user or per-mailbox in any useful way.

>> I think SQL database as a mail store would have much worse performance
>> than with any filesystem based mail store.
>
> Except that with clustering, you could have multiple DB servers 
> getting hit by multiple dovecot servers.  SQL could potentially scale 
> to much larger environments than filesystem support.

How did you think it would work?

One way to do it would be to distribute users to different SQL servers. 
That's just as possible with filesystems.

If you thought about some kind of database that is distributed across 
multiple computers.. Well, that would probably work, but I haven't 
heard much of them being in any actual use. Probably because they all 
cost too much. I don't think PostgreSQL or MySQL had support for these 
kind of clusters yet?

There are also clustered filesystems that are distributed across 
multiple computers. Dovecot already works with them.

> Plus, think of all the neat fulltext indexing possibilities.

Unfortunately that doesn't help with IMAP at all, and might make it 
actually more difficult and slower to implement the kind of indexing 
that would benefit IMAP's SEARCH command.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20050923/87acfbbd/PGP.pgp