[Dovecot] Maildir feature I'd like to see - SSD for newer messages
SSD drives are very fast but expensive. So I have a crude idea that I'd like to see. May not be practical but would like to get some thoughts on it.
Some new SSDs use SATA 3 (6gb/sec) with 355/mb/sec read speeds and 215MB/sec write. Put these in raid 0 and it screams! Can you imagine how fast that would be?
What would be nice is if new email were on faster drives with old email being migrated to larger mechanical cheaper storage. Perhaps messages over a month old? From dovecot's perspective it would sort of all look the same but maybe one a week a script would run migrating older messages to slower media.
It would also make backups easier. The faster newer messages could be archived hourly. On the weekend after the archiving script was run the older message could be backed up once a week.
I'm not sure what it would take to make dovecot seamlessly access email from two different locations or if this is practical. Just wanted to throw the idea out there to see if something sticks.
On Thu, 23 Dec 2010 11:27:45 -0800, Marc Perkel marc@perkel.com wrote:
SSD drives are very fast but expensive. So I have a crude idea that I'd like to see. May not be practical but would like to get some thoughts on it.
You are asking about automatic storage tiering. You can get what you want in a transparant way, independient of Dovecot. Some storage vendor (search for Fully Automated Storage Tiering - FAST from EMC or Compellent, recently bought by Dell) get what you are asking for.
If the budget is low, you can achive a "poor´s man" storage tiering with some shell scripting, cron and soft links; Or you can look at http://code.google.com/p/fscops/ for a more mature implementation. Or just use ZFS and "hybrid storage pools"
Merry Xmas
Javier
On 12/23/2010 11:37 AM, Javier de Miguel Rodríguez wrote:
On Thu, 23 Dec 2010 11:27:45 -0800, Marc Perkel marc@perkel.com wrote:
SSD drives are very fast but expensive. So I have a crude idea that I'd like to see. May not be practical but would like to get some thoughts on it.
You are asking about automatic storage tiering. You can get what you want in a transparant way, independient of Dovecot. Some storage vendor (search for Fully Automated Storage Tiering - FAST from EMC or Compellent, recently bought by Dell) get what you are asking for.
If the budget is low, you can achive a "poor´s man" storage tiering with some shell scripting, cron and soft links; Or you can look at http://code.google.com/p/fscops/ for a more mature implementation. Or just use ZFS and "hybrid storage pools"
I wonder if there's any way to do this. Say there are two different storage devices mounted as follows:
/slow-hard-drive/email-directory /fast-ssd-drive/email-directory
Now suppose there were a way to join these drives so that you could access them as a single directory:
/joined-drive/email-directory
The Dovecot could see the joined drive but my backup/archiving scripts could see the drives independently and move old data from the fast drives to the slow drives.
I think back in the old DOS days you could do that. Is there a way in Linux to do the same thing? Join two directories so as to make them look like one?
On 12/23/2010 3:13 PM, Marc Perkel wrote:
I wonder if there's any way to do this. Say there are two different storage devices mounted as follows:
/slow-hard-drive/email-directory /fast-ssd-drive/email-directory
Now suppose there were a way to join these drives so that you could access them as a single directory:
/joined-drive/email-directory
The Dovecot could see the joined drive but my backup/archiving scripts could see the drives independently and move old data from the fast drives to the slow drives.
I think back in the old DOS days you could do that. Is there a way in Linux to do the same thing? Join two directories so as to make them look like one?
This is getting rather far from Dovecot, but I believe you're looking for unionfs: http://en.wikipedia.org/wiki/UnionFS
"Tom Talpey" tmtalpey@gmail.com wrote on 23.12.2010 21:27:02:
On 12/23/2010 3:13 PM, Marc Perkel wrote:
I wonder if there's any way to do this. Say there are two different storage devices mounted as follows:
/slow-hard-drive/email-directory /fast-ssd-drive/email-directory
You guys do know that Dovecot 2 supports alt storage right?
Regards, Miha
On 12/23/2010 1:14 PM, Miha Vrhovnik wrote:
"Tom Talpey"tmtalpey@gmail.com wrote on 23.12.2010 21:27:02:
On 12/23/2010 3:13 PM, Marc Perkel wrote:
I wonder if there's any way to do this. Say there are two different storage devices mounted as follows:
/slow-hard-drive/email-directory /fast-ssd-drive/email-directory
You guys do know that Dovecot 2 supports alt storage right?
No - what's alt storage?
Marc Perkel put forth on 12/23/2010 3:18 PM:
On 12/23/2010 1:14 PM, Miha Vrhovnik wrote:
You guys do know that Dovecot 2 supports alt storage right?
No - what's alt storage?
http://wiki2.dovecot.org/MailboxFormat/dbox#Alternate_storage
Only works with [m]dbox format. Command used to move the messages:
http://wiki2.dovecot.org/Tools/Doveadm/Altmove
This is pretty much exactly what the original OP is looking for. But it requires migrating mail to [m]dbox storage format.
-- Stan
Quoting Marc Perkel marc@perkel.com:
Some new SSDs use SATA 3 (6gb/sec) with 355/mb/sec read speeds and
215MB/sec write. Put these in raid 0 and it screams! Can you imagine
how fast that would be?
I'd never raid-0 anything important...
What would be nice is if new email were on faster drives with old
email being migrated to larger mechanical cheaper storage. Perhaps
messages over a month old? From dovecot's perspective it would sort
of all look the same but maybe one a week a script would run
migrating older messages to slower media.
I used to do something "similar" in that the user's inbox was on fast disk, and all their other folders (assumes IMAP for the most part) were on slower disks. A cronjob would run once a month that locks the inbox, selects and mail older than 6 months, and moves it to a folder called "old-mail" -- thus migrating any 6+ month old mail from fast to slow storage...
I suppose you could adapt that, maybe with a shorter time period (1 month would seem okay, not sure about anything shorter).
I know this isn't exactly what you want or asked for, but it is an idea based on a past implementation which worked well.
I'm not sure what it would take to make dovecot seamlessly access
email from two different locations or if this is practical. Just
wanted to throw the idea out there to see if something sticks.
Well, different folders make it a snap... If you don't want to re-folder, then it may not be so easy (I'll let someone else answer that).
-- Eric Rostetter The Department of Physics The University of Texas at Austin
Go Longhorns!
participants (6)
-
Eric Rostetter
-
Javier de Miguel Rodríguez
-
Marc Perkel
-
Miha Vrhovnik
-
Stan Hoeppner
-
Tom Talpey