[Dovecot] Design: Asynchronous I/O for single/multi-dbox

[Timo Sirainen]

On Tue, 2010-03-16 at 13:19 -0400, Frank Cusack wrote:
> On 3/16/10 6:44 PM +0200 Timo Sirainen wrote:
> > Threads are useful when you need to do CPU intensive work in parallel,
> > and threads need access to shared data structures.
> 
> And when you don't want to block on I/O.  Threads are almost always
> easier than AIO, and especially easy (ie, no scary complexity issues
> eg deadlock) if you aren't sharing data structures.

But there are some shared data structures, so I'd need to add locking
for them. And threads won't help with the biggest benefit of the whole
AIO change: "fast" access to high-latency storage. To access such
storage without constant waiting on IO, multiple read requests must be
done in parallel during the execution of a single command. So even if I
added support for threads, I'd need to process a single command in
parallel, which eventually leads to more or less the same kind of AIO
processing I'm thinking about now anyway, only with tons of extra
locking.

Also Dovecot's file locking rules are simple and short-lived. In v2.0
it's already possible to have a single process serve multiple
connections. And Apple's Dovecot does that already with v1.1. I haven't
seen/heard of any deadlocking problems with either. And actually my long
term plan is to make single-dbox (including index files) completely
lockless.

> You mentioned locking overhead earlier, which is a red herring, but
> even on Linux if you are running an SMP kernel you have the locking
> "overhead" even if your process is single threaded.

Yes, but once threads are enabled there are even more locking overhead
from libc. (Yeah, probably small overhead.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20100316/71c77262/attachment.bin