[Dovecot] v3.0 architecture

[pod]

Timo Sirainen <tss at iki.fi> writes:

> I do kind of like that idea, but I don't really se how it would be
> practical here, especially if high performance is wanted.

I don't really see why a priori it would be any less performant than any
other particular RPC mechanism.

> 1. I'm not implementing Dovecot to Plan9, so the "filesystem" would  still
> have to be wrapped in some kind of a protocol. I suppose you  could get
> them visible to filesystem using FUSE, but that would still  be
> Linux-only.

I probably didn't explain well enough.  One doesn't need to be
implementing for or on a Plan9 system and there's no need for there to be
any involvement with the OS or kernel's notion of filesystem.

9P is a _wire_ protocol for expressing filesystem hierarchy and I/O on
files and dirs within that filesystem.  I would like to say "think of it
as NFS simplified" but even that will, I suspect, for lots of people draw
with it far too much irrelevant baggage.  It is a perfectly tractable
proposition to implement both 9P servers and clients, e.g. wmii [1],
solely with the assistance of a userland library, e.g. libixp [2].

> 2. Latency over network is pretty high, so a nice clean filesystem  layout
> wouldn't probably be possible without sacrificing performance.  And a
> non-clean layout probably would just make it horrible to use in  all ways.

Agreed, careful design of the layout is rather important.  But, I suggest,
it requires no more or less care than goes into the design of a more
traditional RPC mechanism - how are errors signalled, can more than one
RPC be in flight at any one time, how is data marshalling done, etc.

Using a synthetic filesystem at least provides a layer of abstraction that
might help.  I don't claim it makes it easier - it just provides a layer
in which some of these questions are already answered.  It's an alternate
way to factorise the problem.

> Actually I think even the current lib-storage API won't be low-enough
> latency. Most IMAP commands should be able to be done by sending a  single
> request and then reading responses. Well, I'm not going to  start coding
> this anytime soon.

I'm afraid I am sufficiently unfamiliar with the lib-storage API to
comment on how straightforwardly one might map it to a 9P-using world.

> Anyway, I'm still more concerned about how to implement the client  side
> so that a single process can asynchronously process commands and  handle
> multiple connections, without the code looking awful difficult  to
> understand mess. I think it might be possible with C, but I'm not  aware
> of any existing code that does it as cleanly as synchronous code  looks
> like.

Very valid concerns.  I don't think I am able to offer further insight
though :-(

Please forgive me if I come across as overly evangelistic.  I do not
intend to.  It is an area of personal interest to me and it felt like it
mapped onto your problem nicely.  Thanks for listening.

Footnotes: 
[1]  http://wmii.suckless.org/
[2]  http://libs.suckless.org/libixp