Hi
My 2p
dbox plans, could be implemented for v1.2:
Interesting to see this as production ready product. Could be very interesting if it adds performance and new features.
v2.0 master/config rewrite
It's mostly working already. One of the larger problems left is how to handle plugin settings. Currently Dovecot just passes everything in plugin {} section to mail processes without doing any checks.
Isn't one of the simplest things still to build a "hash" or "dictionary" (pick your term) of all the things in the config file and then let the relevant chunk of code figure out what to do with it?
You need some kind of API to get out the bunch of possible options (eg see Xine/mplayer and how it lists all the default keymappings so that you can easily build a custom set). But after that you should really be able to stick in anything you like and the code just parses it and uses sensible defaults...
If config is shared then it might be nice to break out the config parser into a separate lib so that it's easily shared around - in particular default configs
What about it the default configs (and hence config structure) are provided with each plugin in a simple text file? Main app has a way to scoop all these together if someone needs a base template config file?
Index file optimizations
Nice to reduce admin knowhow wherever possible. ie reduce the number of corner cases where files can grow uncontrollably. This is important when converting someone from MS Exchange to dovecot...
Proxying
- These could be implemented to v1.2.
- Log in normally (no proxying) if destination IP is the server itself.
Personally I think the two main use cases here are:
- Set of frontend servers feeding to dedicated backend servers. Simple passthrough case
- Smaller setup has mixed frontend/backend servers (also useful to some extent when upgrading from some old setup with another brand of IMAP server). Basically the frontend server should notice if it's also the final destination and skip the proxying step, otherwise it behaves just like step 1) above
Idea 2) above is problematic in the case of mixed backend server types, eg I was upgrading from Courier to Dovecot. I don't think it's currently within scope to do anything too clever here - there are other proxy servers which can be looked at if necessary for really complex setups. I fixed it by using the CAPABILITIES option and limiting to the min set available on both servers - can't really see a better fix anyway because clients often ask for CAPA before even logging in...
So lets keep this simple and see if anyone shouts for anything more complicated before implementing it... Webmail optimisation is a far more common request than some of the advanced stuff that you were suggesting I should think?
Adding DNS lookups to the current proxy process would likely help some LDAP users I think?
Replication
I'm really up for this. However, your plans all seem to involve online sync - I need a very low bandwidth dialup sync...
Much more interested in features like
- only syncing certain mailboxes, or prioritising the order at least
- Perhaps only syncing the main mime structure and the delayed attachments sync (really nice if could be done cleanly!!)
- Completely multi-master, eg customer walks to remote location, plugs in and works seamlessly - walks back to main office and continues working there. Server figures out how to combine changes later
- Really, really low bandwidth usage
You mention some problems like unique ID's. This is actually pretty simple - just use some GUID type process. All you need is something guaranteed unique on each server, combinations of time, something unique per server and some randomness get you there. Discount anything which is some kind of counter because it will break eventually.
- Support for manual resync of all mailboxes, in case something was done outside Dovecot.
Definitely want this as robust as possible in case of corner cases where some change gets past the replication routine
- Conflict resolution.
It's better that you get a duplicated message than something is lost.
Most usage patterns for most email are: read/delete, or read/file.
Optimise for those
- Synchronous expunges also? Maybe optional. The main reason for this is that it wouldn't be nice to break IMAP state by having expunged messages come back.
Expunges need to be async and tracked. Very likely we will see people delete messages from two servers (because they moved server and didn't see the delete sync across yet.
Also need to watch for clients copying up deleted messages...?
- Multi-master replication:
- Almost everything is already in place. The main problem here is IMAP UID allocation. They must be allocated incrementally, so each mailbox must have a globally increasing UID counter. The usage goes like:
Don't see why? Use some kind of GUID and ensure that there are never any collisions right from step one?
Study Lotus Notes on this. Absolutely fantastic sync. You can create any kind of loops you want, shake it all around bring servers up in any order, sync clients to random servers and it all seems to work pretty well. This is what we really want for this
- Automation:
- Automatically move/replicate users between servers when needed. This will probably be implemented separately long after 4.
Don't see why this is a lower priority? Can I suggest that you *start* with this and go back to do the normal replication afterwards?
The moving between servers will have wider usage I should think? It's also clearly a synchronous operation and can be implemented initially as a completely special case, changed later to use any replication code.
I think this code will initially be external to the main server, and I think a huge amount will probably be learned about the requirements and problems that will occur by implementing this stuff first. It doesn't even need to be that efficient initially...
Actually this same code seems more likely to develop into a more general demon:
- Adjust filters per user
- Adjust permissions on shared folders
- Other admin operations per mailbox (forced purge, cleanup Trash, etc)
The replication processes would be ugly to implement for v1.x master, so this pretty much depends on v2.0 master rewrite.
OK, but rewrites are bad, incremental changes are always preferred...
Can I also add to your TODO list:
- Lemonade Profile!
- Enhancements to the Expire plugin making it easier to enforce a quota, eg trimming Trash initially, then Sent Items, then old stuff in other folders, etc, etc
In particular push email is all the rage right now and not having push features and not having a "Blackberry" plugin is really contributing to the rise of Exchange and other options right now. Please lets get open IMAP servers back on the map by giving users push type features. OK, it's not easy to figure out what to do right now, but it's a bit chicken and egg - until there is some rudimentry support in IMAP servers then the networks won't react and help, lets get something in no matter how rubbish and see how it develops from there (make it flexible so that people with crazy ideas can extend it...)
Good luck
Ed W