[Dovecot] Roadmap to future

[Ed W]

Hi

My 2p

> dbox plans, could be implemented for v1.2:
>   

Interesting to see this as production ready product.  Could be very 
interesting if it adds  performance and new features. 
> v2.0 master/config rewrite
> --------------------------
>
> It's mostly working already. One of the larger problems left is how to
> handle plugin settings. Currently Dovecot just passes everything in plugin
> {} section to mail processes without doing any checks. 

Isn't one of the simplest things still to build a "hash" or "dictionary" 
(pick your term) of all the things in the config file and then let the 
relevant chunk of code figure out what to do with it?

You need some kind of API to get out the bunch of possible options (eg 
see Xine/mplayer and how it lists all the default keymappings so that 
you can easily build a custom set).  But after that you should really be 
able to stick in anything you like and the code just parses it and uses 
sensible defaults...

If config is shared then it might be nice to break out the config parser 
into a separate lib so that it's easily shared around - in particular 
default configs

What about it the default configs (and hence config structure) are 
provided with each plugin in a simple text file? Main app has a way to 
scoop all these together if someone needs a base template config file?

> Index file optimizations
>   

Nice to reduce admin knowhow wherever possible.  ie reduce the number of 
corner cases where files can grow uncontrollably.  This is important 
when converting someone from MS Exchange to dovecot...


> Proxying
> --------
>
>  - These could be implemented to v1.2.
>  - Log in normally (no proxying) if destination IP is the server itself.
>   

Personally I think the two main use cases here are:

1) Set of frontend servers feeding to dedicated backend servers.  Simple 
passthrough case
2) Smaller setup has mixed frontend/backend servers (also useful to some 
extent when upgrading from some old setup with another brand of IMAP 
server).  Basically the frontend server should notice if it's also the 
final destination and skip the proxying step, otherwise it behaves just 
like step 1) above

Idea 2) above is problematic in the case of mixed backend server types, 
eg I was upgrading from Courier to Dovecot.  I don't think it's 
currently within scope to do anything too clever here - there are other 
proxy servers which can be looked at if necessary for really complex 
setups.  I fixed it by using the CAPABILITIES option and limiting to the 
min set available on both servers - can't really see a better fix anyway 
because clients often ask for CAPA before even logging in...

So lets keep this simple and see if anyone shouts for anything more 
complicated before implementing it...  Webmail optimisation is a far 
more common request than some of the advanced stuff that you were 
suggesting I should think?

Adding DNS lookups to the current proxy process would likely help some 
LDAP users I think?


> Replication
>   

I'm really up for this.  However, your plans all seem to involve online 
sync - I need a very low bandwidth dialup sync...

Much more interested in features like

- only syncing certain mailboxes, or prioritising the order at least
- Perhaps only syncing the main mime structure and the delayed 
attachments sync (really nice if could be done cleanly!!)
- Completely multi-master, eg customer walks to remote location, plugs 
in and works seamlessly - walks back to main office and continues 
working there.  Server figures out how to combine changes later
- Really, really low bandwidth usage

You mention some problems like unique ID's.  This is actually pretty 
simple - just use some GUID type process.  All you need is something 
guaranteed unique on each server, combinations of time, something unique 
per server and some randomness get you there.  Discount anything which 
is some kind of counter because it will break eventually.

>  - Support for manual resync of all mailboxes, in case something was done
>    outside Dovecot.
>   

Definitely want this as robust as possible in case of corner cases where 
some change gets past the replication routine

>  - Conflict resolution. 

It's better that you get a duplicated message than something is lost.  
Most usage patterns for most email are: read/delete, or read/file.  
Optimise for those

>  - Synchronous expunges also? Maybe optional. The main reason for this is that
>    it wouldn't be nice to break IMAP state by having expunged messages come
>    back.
>   

Expunges need to be async and tracked.  Very likely we will see people 
delete messages from two servers (because they moved server and didn't 
see the delete sync across yet. 

Also need to watch for clients copying up deleted messages...?

> 4. Multi-master replication:
>
>  - Almost everything is already in place. The main problem here is IMAP UID
>    allocation. They must be allocated incrementally, so each mailbox must
>    have a globally increasing UID counter. The usage goes like:
>   

Don't see why?  Use some kind of GUID and ensure that there are never 
any collisions right from step one?

Study Lotus Notes on this.  Absolutely fantastic sync.  You can create 
any kind of loops you want, shake it all around bring servers up in any 
order, sync clients to random servers and it all seems to work pretty 
well.  This is what we really want for this

> 5. Automation:
>
>  - Automatically move/replicate users between servers when needed. This will
>    probably be implemented separately long after 4.
>   

Don't see why this is a lower priority?  Can I suggest that you *start* 
with this and go back to do the normal replication afterwards?

The moving between servers will have wider usage I should think?  It's 
also clearly a synchronous operation and can be implemented initially as 
a completely special case, changed later to use any replication code. 

I think this code will initially be external to the main server, and I 
think a huge amount will probably be learned about the requirements and 
problems that will occur by implementing this stuff first.  It doesn't 
even need to be that efficient initially...

Actually this same code seems more likely to develop into a more general 
demon:

- Adjust filters per user
- Adjust permissions on shared folders
- Other admin operations per mailbox (forced purge, cleanup Trash, etc)


> The replication processes would be ugly to implement for v1.x master, so
> this pretty much depends on v2.0 master rewrite.
>   

OK, but rewrites are bad, incremental changes are always preferred...



Can I also add to your TODO list:

- Lemonade Profile!
- Enhancements to the Expire plugin making it easier to enforce a quota, 
eg trimming Trash initially, then Sent Items, then old stuff in other 
folders, etc, etc

In particular push email is all the rage right now and not having push 
features and not having a "Blackberry" plugin is really contributing to 
the rise of Exchange and other options right now.  Please lets get open 
IMAP servers back on the map by giving users push type features.  OK, 
it's not easy to figure out what to do right now, but it's a bit chicken 
and egg - until there is some rudimentry support in IMAP servers then 
the networks won't react and help, lets get something in no matter how 
rubbish and see how it develops from there (make it flexible so that 
people with crazy ideas can extend it...)

Good luck

Ed W