[Dovecot] NoSQL Storage Backend
Hello,
I found a posting on the mailing list from 2008 "Adding new storage backend to Dovecot" while searching for POP3 and IMAP servers, that allow to use a NoSQL backend rather than a traditional file system.
The problem with traditional filesystems is, that they are very limited. You cannot easily extend it, even with RAID 10 all messages may be lost due to a RAID controller failure and just-in-time backups of a heavy loaded system with consistently changing messages folders are virtually impossible.
Modern object stores like Scality, CouchDB or MongoDB can be very handy, as they easily allow to keep an arbitrary amount of copies of each message on different servers, you can simply add a storage node and new storage is immediately available and mailboxes aren't bound to a certain server any longer, but messages are evenly distributed across all storage nodes, thus distributing load as well.
Since the original posting was dated 2008 I was in good hope, that someone worked on that already, but at least in dovecot-2.0.9/src/lib-storage I couldn't find anything relating to NoSQL message stores.
Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team?
Kind regards Marten
Am 08.02.2011 13:22, schrieb Marten Lehmann:
Hello,
I found a posting on the mailing list from 2008 "Adding new storage backend to Dovecot" while searching for POP3 and IMAP servers, that allow to use a NoSQL backend rather than a traditional file system.
There has been a recent mail from Timo regarding the future development plans for 2.1 where he mentioned a key/value storage backend. http://dovecot.markmail.org/message/juecu7wm2nwkluji?q=plans+2.1
Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team?
I am also very interested on seeing this implemented, maybe also on a paid basis.... I think the core team is Timo :-)
Regards, Oliver
On 8.2.2011, at 14.22, Marten Lehmann wrote:
I found a posting on the mailing list from 2008 "Adding new storage backend to Dovecot" while searching for POP3 and IMAP servers, that allow to use a NoSQL backend rather than a traditional file system.
I don't remember any mails about NoSQL backend from 2008, but at the end of 2009 it became something I started planning on doing. It's listed in http://wiki2.dovecot.org/Roadmap
Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team?
It should be pretty easy to develop if you don't care about performance, but since you likely do, it gets more difficult. The latencies for db lookups are higher than local reads, so it would be good to be able to do multiple lookups in parallel. I've recently started imapc backend, which has similar problems and I was planning on solving those problems with it, so it will be easier then for nosql backend. Also it would be good to design a storage backend for nosql that didn't require any kind of locking.
One guy recently started looking into developing something for this, but I haven't heard from him for some weeks. I could ask if he's gotten anywhere.. As for me, I'm not entirely sure about my plans for this spring. There's another company wanting to pay me, but I don't know how much of my time they'll really need.
On Tue, Feb 8, 2011 at 1:22 PM, Marten Lehmann lehmann@cnm.de wrote:
Modern object stores like Scality, CouchDB or MongoDB can be very handy, as they easily allow to keep an arbitrary amount of copies of each message on different servers, you can simply add a storage node and new storage is immediately available and mailboxes aren't bound to a certain server any longer, but messages are evenly distributed across all storage nodes, thus distributing load as well.
The Scality webpages mentions they´ve developed storage connector for dovecot:
http://www.scality.com/storage-solutions/
whatever that means..
-jf
On 2011-02-08 1:21 PM, Jan-Frode Myklebust wrote:
The Scality webpages mentions they´ve developed storage connector for dovecot:
http://www.scality.com/storage-solutions/
whatever that means..
Not so great - they're linking to dovecot.COM instead of dovecot.ORG...
--
Best regards,
Charles
Hi, full disclaimer, I am the CTO of Scality. We just fixed the link to dovecot.com to dovecot.org on our website, thanks for pointing that out.
Indeed we have developed a connector to our object storage for dovecot 2.x, this is aimed at very large scale deployments, millions on users and petabytes of storage. Our next goal is to also distribute the metadata on our RING platform, think of it as a nosql database put tailor made for Dovecot. This would enable stateless operations on a layer of dovecot IMAP/POP frontends, although from a performance/caching point of view, a layer of statefull load balancers would redirect the same user to one dovecot box, until it dies and then transparently fails over to another box, without having to move any data around.
As Timo pointed out, focusing on keeping a lightning fast user experience makes the exercise more difficult as a naive DB layer implementation would add a lot of latency. Locking is our enemy...
Best regards
Giorgio
tanstaafl_bh wrote:
On 2011-02-08 1:21 PM, Jan-Frode Myklebust wrote:
The Scality webpages mentions they´ve developed storage connector for dovecot: http://www.scality.com/storage-solutions/
--
Best regards,
Charles
-- View this message in context: http://old.nabble.com/NoSQL-Storage-Backend-tp30872825p30891304.html Sent from the Dovecot mailing list archive at Nabble.com.
The Scality webpages mentions they´ve developed storage connector for dovecot:
http://www.scality.com/storage-solutions/
whatever that means..
Not so great - they're linking to dovecot.COM instead of dovecot.ORG...
I asked them about the dovecot support two days ago but still no reply...
Hey Marten,
As a disclaimer, I work with Scality.
Thank you guys for letting us know about the typo in the URL for dovecot on the website. I have tried to reply to the email that sent us the information to thank them, but it was bogus ;(
We have also seen your request for information. Our Account Manager from Germany will be in touch with you very soon.
Best,
-Marc @mastachand http://linkd.in/heve30
On Feb 10, 2011, at 10:26 AM, Marten Lehmann wrote:
The Scality webpages mentions they´ve developed storage connector for dovecot:
http://www.scality.com/storage-solutions/
whatever that means..
Not so great - they're linking to dovecot.COM instead of dovecot.ORG...
I asked them about the dovecot support two days ago but still no reply...
On Thu, 10 Feb 2011 10:48:56 -0800, Marc Villemade marc.villemade@scality.com wrote:
Hey Marten,
As a disclaimer, I work with Scality.
Thank you guys for letting us know about the typo in the URL for dovecot on the website. I have tried to reply to the email that sent us the information to thank them, but it was bogus ;(
We have also seen your request for information. Our Account Manager from Germany will be in touch with you very soon.
Best,
-Marc @mastachand http://linkd.in/heve30
On Feb 10, 2011, at 10:26 AM, Marten Lehmann wrote:
The Scality webpages mentions they´ve developed storage connector for dovecot:
http://www.scality.com/storage-solutions/
whatever that means..
Not so great - they're linking to dovecot.COM instead of dovecot.ORG...
I asked them about the dovecot support two days ago but still no reply...
Marc,
That must have been us (jesusjone@fu.com?), sorry we just wanted to get the message over without any sales pitches or something similar coming back. Glad to hear it's fixed.
Michael
On Feb 10, 2011, at 11:02 AM, list@airstreamcomm.net list@airstreamcomm.net wrote:
On Thu, 10 Feb 2011 10:48:56 -0800, Marc Villemade marc.villemade@scality.com wrote:
Hey Marten,
As a disclaimer, I work with Scality.
Thank you guys for letting us know about the typo in the URL for dovecot [.....]
Marc,
That must have been us (jesusjone@fu.com?), sorry we just wanted to get the message over without any sales pitches or something similar coming back. Glad to hear it's fixed.
Michael
Hey There Michael,
Thanks for letting us know, and no worries about the email, i wasn't expecting an answer with such an email addy, but felt like trying anyway to thank you, just in case :-D
Best,
-Marc @mastachand http://linkd.in/heve30
On Tue, 8 Feb 2011, Marten Lehmann wrote:
The problem with traditional filesystems is, that they are very limited. You cannot easily extend it, even with RAID 10 all messages may be lost due to a RAID controller failure and just-in-time backups of a heavy loaded system with consistently changing messages folders are virtually impossible.
Hi Marten. There are many options that allow you to snapshop an active filesystem and guarantee a consistent backup. Eg LVM on Linux or many other OSes. WAFL on a NetApp, etc.
In my experience a lot of high end mail systems store mail on the filesystem in Maildir format.
Modern object stores like Scality, CouchDB or MongoDB can be very handy, as they easily allow to keep an arbitrary amount of copies of each message on
This is certainly an interesting idea.
different servers, you can simply add a storage node and new storage is immediately available and mailboxes aren't bound to a certain server any longer, but messages are evenly distributed across all storage nodes, thus distributing load as well.
Being able to easily rebalance load would be useful. I've known of several well known companies with a lot of mail users where mail was stored on filesystems across many systems and significant imbalances developed over time. There was no easy way to rebalance them without causing an outage for at least some users.
Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team?
In my experience OSS developers are quite open to being paid to add features :)
Cheers,
Rob
-- Email: robert@timetraveller.org Linux counter ID #16440 IRC: Solver (OFTC & Freenode) Web: http://www.practicalsysadmin.com Contributing member of Software in the Public Interest (http://spi-inc.org/) Open Source: The revolution that silently changed the world
The way I do MySQL backups is that I do a slave on another server. When I need to backup I stop the slave - back it up - then start it again.
On 2/8/2011 12:56 PM, Robert Brockway wrote:
On Tue, 8 Feb 2011, Marten Lehmann wrote:
The problem with traditional filesystems is, that they are very limited. You cannot easily extend it, even with RAID 10 all messages may be lost due to a RAID controller failure and just-in-time backups of a heavy loaded system with consistently changing messages folders are virtually impossible.
Hi Marten. There are many options that allow you to snapshop an active filesystem and guarantee a consistent backup. Eg LVM on Linux or many other OSes. WAFL on a NetApp, etc.
In my experience a lot of high end mail systems store mail on the filesystem in Maildir format.
Modern object stores like Scality, CouchDB or MongoDB can be very handy, as they easily allow to keep an arbitrary amount of copies of each message on
This is certainly an interesting idea.
different servers, you can simply add a storage node and new storage is immediately available and mailboxes aren't bound to a certain server any longer, but messages are evenly distributed across all storage nodes, thus distributing load as well.
Being able to easily rebalance load would be useful. I've known of several well known companies with a lot of mail users where mail was stored on filesystems across many systems and significant imbalances developed over time. There was no easy way to rebalance them without causing an outage for at least some users.
Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would it be possible to get this developed on a paid basis by the dovecot core team?
In my experience OSS developers are quite open to being paid to add features :)
Cheers,
Rob
participants (10)
-
Charles Marcus
-
Giorgio Regni
-
Jan-Frode Myklebust
-
list＠airstreamcomm.net
-
Marc Perkel
-
Marc Villemade
-
Marten Lehmann
-
Oliver Eales
-
Robert Brockway
-
Timo Sirainen