[Dovecot] scalability and arhive ideas

[fernando at dfcom.com.br]

Hi,

I´m reading the past topics related to archive and scalability of dovecot,
they are all very interesting. Here, I´m using two dovecot proxies in
front of five storages pairs, and we split the domain´s accounts among
those servers. So, we can share the i/o load and if one server goes down
only few accounts of the domain stops (not all of them).

But, we began to have space problems - and the solution would be insert
more and more storage servers. So I was searching for some archive
solutions (hard links - at S.O level, or some dovecot extension). A friend
told me that he knows an ISP that share even the mailbox of the users
among many servers -

this is very weird and (at same time) very interesting approach. Instead
of put all messages into one maildir and this maildir into one server,
this "maildir" (?) is spplited among many servers - so, if one servers
fails the account is still acessible and they move old/big messages to a
new "cheap" storage - archiving transparently.

Well, maybe it´s an stupid idea, but couldn´t dovecot imap/pop proxy do
the same ? I mean, imagine the following scenario:

1- Proxy does an user account sql - today the return is (among other
data), the final server IP, but it could be the storage servers for this
account
2- Proxy does parallel connections  (instead of today´s one connection
with the final server). Retrieve the messages and caches locally where the
message is. If the cache is lost, no problem, it just connects again and
re-cache them.
3- When a message is deleted/flagged, etc - it has the cache (allowing it
to know where send the command to), when a message inserted (as sent
folder) - the system can have the 'default storage' where the messages are
delivered and saved to -  the other storages are just for archiving (or
even a round-robin delivery !? - hard to control).

Problems:

1- Have the cache at proxy level.
2- Quota calculation (how the delivery process can check this ?)
3- Maybe mixing proxy connection and this new approach is not easy way.
4- the more servers, the more time waiting their answers (imagine one of
them answering slowly).

Advantages:

1- No local mounts at storage level - (as nfs and other network mounted
partitions).
2- Servers independency - if space is at a critical value, insert a pair
of servers, move messages there, insert the servers in the sql answer
(server1, server2, server3, ... , serverN).
3- Scalability - the more servers, the more i/o load you can share.

Well, sorry for the long post...but I hope to collaborate with some ideas
(even if they are stupid :( )

Regards,
Fernando Bertasso Figaro