[Dovecot] Overly long email of miscellaneous Dovecot migration questions
Hello to the list! I've been asked to spec out the feasibility and, if feasible, plan a migration from Courier to Dovecot for both POP3 and IMAP for about 4 million mailboxes. I've been trying to absorb all the dovecot-related info I could over the past couple of weeks from the docs and from the list and I'm sure that I've muddled at least some of it, so apologies in advance if I make grossly incorrect assumptions.
BTW, sorry for the giant email but I've got a bunch of questions. This is the sort of overly wordy email that makes my eyes glaze over in mailing lists. But I've got things configured to where (I think) everything will 'just work' if we migrate (barring running POP3 mailbox conversions), so I'm not looking for configuration help, just gotchas and assumption-checking mostly. Me breaking 4 million mailboxes isn't really a good move for my job security ;)
Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at.
This is a pretty fantastic piece of software. I love the neat little details that make admins' lives easier, e.g. how easy it is to use multiple SSL certs in the same server. That made me smile. I hadn't looked at dovecot in a number of years, so I was really impressed at the full feature set. It's been a while since I've played with a new server application that I've enjoyed this much.
Background: All of our mail is stored on NFS and will be for the foreseeable future, all in maildir format. Courier POP3/IMAP runs on load balanced Linux servers, so clients will be hitting multiple servers, though stickiness based on client IP address is currently done. Webmail as well is done on top of a local Courier IMAP server accessing the same NFS servers but on an entirely different pool of servers. All authentication is done against SQL. Courier has been very, very good to us over the past 9 years, but our NFS servers (12 Netapps) are unfortunately being beaten to death. We've got lots of users with thousands of emails in their inbox (i.e. in ./Maildir/cur/ alone) and I've seen plenty of mailboxes with 10's of thousands in their inbox and I've even seen some really scary ones approaching 100k emails just in cur/. Thanks to an endless cycle of industry raising mailbox limits, we actually have to support this (though not so much the 100k message ones, at least).
Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir. I know the cool thing to do would be to move the index files to a separate NFS server running entirely on SSDs for the fastest accesses possible. Building a separate index NFS cluster obviously adds complexity and performance with the index files in the maildir seems pretty great already. But I also know that real life often makes assumptions made during benchmarks look very foolish. My inclination is to try things out with the indexes in the maildir directory (and then start looking at moving to a dedicated NFS server for the indexes in a subsequent phase) but I don't want to regret that choice later. Anybody with large-scale NFS maildir stores have any advice either way?
Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things.
Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?: Since migrating POP mailboxes is going to be a mighty task, our gameplan would probably be to start migrating *just* IMAP services at first and likely on just a few servers at a time. Being load balanced, there are several scenarios where a user could theoretically access a mailbox on different servers and therefore end up hitting dovecot at one point in time and then later hitting courier or vice versa. Or they could use IMAP from one location and POP3 (during the span of time before we migrated our POP3 services) from another. I've not yet seen any side effects from switching mail servers like that (though I'm aware of what would happen if both dovecot and courier *POP3* were in service at the same time -- redownloaded mail, bleh). Has anyone seen otherwise? I'm not worried as much by the 0.1% of IMAP users who might be using advanced features but rather the other 99.9% who are using it pretty vanilla-ishly.
Union mailboxes: I'm pretty sure in a fairly recent thread that Timo said that something like a 'union' mailbox (at least with maildir) wasn't possible. I tried messing with multiple 'private' namespaces (i.e. a namespace called "ARCHIVE" with a "location" different than the INBOX location, ideally placed on slower but denser NFS servers) but even with 'hidden=no' and 'list=yes', only the main INBOX folders would show up, so I'm guessing that's not going to work. That would be a killer feature, to be able to serve an alternate namespace that would show up in a mail client's subscribable list that could be on slower storage than the main inbox (though I'm not sure a mail client can even handle multiple namespaces).
Any problems with keeping only quota limits in db and not current quota numbers? Our limits come out of a SQL table but the current counts just live in the maildir file. Trying to update quota counts in SQL for 4 million mailboxes is a non-starter for us at this point.
From what I've done, dovecot seems pretty happy to maintain the quota using the 'maildir' quota plugin but let it be overridden by the userdb lookup. So I think I'm fine on that count, but I'd love to hear from anyone knowing of extreme gotchas doing it that way.
Any problem with leaving the namespace in "Courier compatibility" mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4 million mailboxes, FAQs all over the place, and support reps trained in a particular way of doing things in IMAP, it'd be hellish to try to change the prefix (I know I could leave the courier namespace around with 'hidden=yes' but retraining support staff is perhaps better left to phase #2). Do I lose anything besides tidiness by not changing it to "prefix =" as if I was deploying dovecot from scratch? Does it hurt performance in any significant way? Benchmarking doesn't look any different, so I'm guessing not.
One thing that threw me and might be good for a FAQ (unless it was just me misconfiguring things) was when I started playing with putting the index files in an alternate location. I was utterly perplexed why it'd create the directories for the indexes but they'd be empty. Based on their location and names when they're in the maildir, I was just looking for the same dovecot.index* files right in the alternate directory. It wasn't till I started strace'ing that I noticed that the index files were indeed getting created but in a subdirectory called .INBOX (and with me just doing 'ls').
The courier-dovecot-migrate.pl does a fine job at converting our POP3 mailboxes. We'll definitely be doing a mass run of it across all of our mailboxes. But with this many mailboxes, there's no way to get around the time lag between the script getting run and new mail showing up. Which means that there'll be some number of new messages that will re-download in POP3 when the switchover to Dovecot POP3 is done. To keep from getting too many complaints from customers, I'm thinking that I might have to write a wrapper script to do something like compare the mtime of ./Maildir/new/ against dovecot-uidlist and overwrite if ./Maildir/new/ is newer or perhaps see if the mtime of courierpop3dsizelist is newer than dovecot-uidlist and only if that's the case execute the conversion script with --overwrite. Considering we get between 500-2000 POP3 logins/sec, running any script after a POP3 login terrifies me--even a 6 line bash script. Anybody have any opinions to share on that? With a pooled architecture, it's near impossible to only have some mailboxes hit Courier and some hit Dovecot and keep slowly incrementing. At best my available units of increments to migrate are a few thousand at a time and more realistically 10-20k at a time. If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life.
A big thanks to anyone who even actually reads this entire tedious email and a tremendous thanks to anyone who actually replies to any of it!
On 17.3.2010, at 1.01, Mark Moseley wrote:
- Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at.
Yes, hopefully it's coming soon :)
- Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir.
Have you read the thread starting from http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month or so? That provides a good view of potential problems with NFS.
- Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things.
v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
- Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?
Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or both IMAPs is just going to cause trouble because of conflicting UIDs. You might be able to make both Dovecot and Courier use the same POP3 UIDLs, but I wouldn't really trust it.
One possibility would be to just run the migration script on login, so users would migrate to Dovecot as they log in.
- Union mailboxes: I'm pretty sure in a fairly recent thread that Timo said that something like a 'union' mailbox (at least with maildir) wasn't possible.
I also had a thought where you could do that if you wrote some scripts and such that copied the mails to the other storage and replaced the original file with a symlink. But that of course has some potential race conditions and other problems with either side of the symlink disappearing but the other not.
Single-dbox would really be the best solution for this in future. It's currently somewhat broken in v2.0 tree, but it probably won't be too long until I'm going to start doing a migration from Maildir to dbox for a similar NFS system than yours.
I tried messing with multiple 'private' namespaces (i.e. a namespace called "ARCHIVE" with a "location" different than the INBOX location, ideally placed on slower but denser NFS servers) but even with 'hidden=no' and 'list=yes', only the main INBOX folders would show up, so I'm guessing that's not going to work.
You can create more namespaces, so I guess that was some kind of a configuration problem.
That would be a killer feature, to be able to serve an alternate namespace that would show up in a mail client's subscribable list that could be on slower storage than the main inbox (though I'm not sure a mail client can even handle multiple namespaces).
The clients don't need to be aware of namespaces. And you should be able to do that already. But do you think users would actually move their mails to there?
- Any problems with keeping only quota limits in db and not current quota numbers? Our limits come out of a SQL table but the current counts just live in the maildir file.
That's how most people do it.
- Any problem with leaving the namespace in "Courier compatibility" mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4 million mailboxes, FAQs all over the place, and support reps trained in a particular way of doing things in IMAP, it'd be hellish to try to change the prefix (I know I could leave the courier namespace around with 'hidden=yes' but retraining support staff is perhaps better left to phase #2). Do I lose anything besides tidiness by not changing it to "prefix =" as if I was deploying dovecot from scratch? Does it hurt performance in any significant way? Benchmarking doesn't look any different, so I'm guessing not.
Keeping the INBOX. prefix for the initial migration is a good idea. Once everything has worked perfectly for a few months, moving to hidden=yes could be a good idea too. There are some clients that don't like the INBOX. prefix.
- One thing that threw me and might be good for a FAQ (unless it was just me misconfiguring things) was when I started playing with putting the index files in an alternate location. I was utterly perplexed why it'd create the directories for the indexes but they'd be empty. Based on their location and names when they're in the maildir, I was just looking for the same dovecot.index* files right in the alternate directory. It wasn't till I started strace'ing that I noticed that the index files were indeed getting created but in a subdirectory called .INBOX (and with me just doing 'ls').
I try to avoid having a FAQ. Instead I try to change Dovecot so that the question itself goes away. I haven't heard of people having this problem before .. but one possible solution could be to just create a file to the directory containing some text that makes it clear. Then again that is slightly bloaty too..
If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life.
Check what the new Courier POP3 UIDLs look like. If all of them are maildir base filename, setting pop3_uidl_format=%f should make this conversion easy. Just run it through once to get old UIDLs converted and then you don't need to worry about it, because both Courier and Dovecot give the same UIDLs. But I never really understood how Courier assigns new UIDLs, sometimes it seems to use the filename and sometimes not..
First off, thanks for the reply. I appreciate it greatly!
On Tue, Mar 16, 2010 at 4:36 PM, Timo Sirainen tss@iki.fi wrote:
On 17.3.2010, at 1.01, Mark Moseley wrote:
- Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at.
Yes, hopefully it's coming soon :)
- Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir.
Have you read the thread starting from http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month or so? That provides a good view of potential problems with NFS.
I'd read through that thread and subsequently forgot the implications. Presumably the same pitfalls apply to 2.0? From the wiki and from the thread, it sounds like this just affects index files. One thing I didn't see in the thread (though it'd be easy to miss in a thread that long) is whether performance suffered horribly with 'noac' on a *separate* index mount (the mentions I can find where they tried 'noac', it was on the entire mail store, not a separate index mount) -- though doing 'noac' on anything seems like a recipe for disaster. Another extreme option might be to just keep indexes on local disk only. Obviously users hitting another server wouldn't benefit from the other server and it'd incur updates to the indexes but they seem somewhat minimal -- in sane cases at least where the number of newer-than-the-index messages is reasonable. With aggressive load balancer stickiness at maybe a /24 netblock level, I'd be able to keep people localized for a little while (with no way around people accessing the same mailbox from home and work). Am I underestimating the costs of incremental index updates on local disks? In testing, tt seemed like dovecot was pretty conservative in disk accesses/reads when updating. I'm hoping the 95%-of-the-time worst case will be someone hitting the same mailbox from two locations, so they'd be hitting at most two servers. With local index caches, I'm mildly sure I could live with that (esp if I can talk my way into an SSD on each one). Am I being too naive?
- Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things.
v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
- Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?
Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or both IMAPs is just going to cause trouble because of conflicting UIDs. You might be able to make both Dovecot and Courier use the same POP3 UIDLs, but I wouldn't really trust it.
One possibility would be to just run the migration script on login, so users would migrate to Dovecot as they log in.
Good to know. I'll definitely limit everything IMAP to just dovecot then (POP3 was a given). Looking back, I forgot about the fact that folder subscriptions could get out of whack too, in a mixed IMAP environment, if someone updated their subscriptions on one platform.
- Union mailboxes: I'm pretty sure in a fairly recent thread that Timo said that something like a 'union' mailbox (at least with maildir) wasn't possible.
I also had a thought where you could do that if you wrote some scripts and such that copied the mails to the other storage and replaced the original file with a symlink. But that of course has some potential race conditions and other problems with either side of the symlink disappearing but the other not.
Single-dbox would really be the best solution for this in future. It's currently somewhat broken in v2.0 tree, but it probably won't be too long until I'm going to start doing a migration from Maildir to dbox for a similar NFS system than yours.
My main concern with doing anything with symlinks is how easy it'd be for users to inadvertently undo, by moving messages around or deleting/recreating folders.
I tried messing with multiple 'private' namespaces (i.e. a namespace called "ARCHIVE" with a "location" different than the INBOX location, ideally placed on slower but denser NFS servers) but even with 'hidden=no' and 'list=yes', only the main INBOX folders would show up, so I'm guessing that's not going to work.
You can create more namespaces, so I guess that was some kind of a configuration problem.
I'll play around with that more. So it should be possible to have more than one private 'hidden=no; list=yes' namespace that'd show up in a mail client's subscribable folders list?
That would be a killer feature, to be able to serve an alternate namespace that would show up in a mail client's subscribable list that could be on slower storage than the main inbox (though I'm not sure a mail client can even handle multiple namespaces).
The clients don't need to be aware of namespaces. And you should be able to do that already. But do you think users would actually move their mails to there?
Good to know. As far as getting people to use it, I'll let marketing worry about that one :) Though one thought would be to set quotas on the ARCHIVE namespace to be much higher than the regular INBOX one (with the assumption of us returning regular INBOX quotas to a reasonable level). Or alternatively, we could move unused mail there and it'd still be accessible with the same mail account (though obviously with user intervention to subscribe to the new folder). Or we could start sticking Spam there. Offhand, before I start sifting through source code, is Dovecot on a 32-bit system limited to a 32-bit -- i.e. 2 gig -- limit on quota size like Courier is?
- Any problems with keeping only quota limits in db and not current quota numbers? Our limits come out of a SQL table but the current counts just live in the maildir file.
That's how most people do it.
Excellent, thanks
- Any problem with leaving the namespace in "Courier compatibility" mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4 million mailboxes, FAQs all over the place, and support reps trained in a particular way of doing things in IMAP, it'd be hellish to try to change the prefix (I know I could leave the courier namespace around with 'hidden=yes' but retraining support staff is perhaps better left to phase #2). Do I lose anything besides tidiness by not changing it to "prefix =" as if I was deploying dovecot from scratch? Does it hurt performance in any significant way? Benchmarking doesn't look any different, so I'm guessing not.
Keeping the INBOX. prefix for the initial migration is a good idea. Once everything has worked perfectly for a few months, moving to hidden=yes could be a good idea too. There are some clients that don't like the INBOX. prefix.
Good to know. Yeah, it'd be a good phase 2 thing. There'll already be so many moving parts that it'd be good to do the optional ones later on.
- One thing that threw me and might be good for a FAQ (unless it was just me misconfiguring things) was when I started playing with putting the index files in an alternate location. I was utterly perplexed why it'd create the directories for the indexes but they'd be empty. Based on their location and names when they're in the maildir, I was just looking for the same dovecot.index* files right in the alternate directory. It wasn't till I started strace'ing that I noticed that the index files were indeed getting created but in a subdirectory called .INBOX (and with me just doing 'ls').
I try to avoid having a FAQ. Instead I try to change Dovecot so that the question itself goes away. I haven't heard of people having this problem before .. but one possible solution could be to just create a file to the directory containing some text that makes it clear. Then again that is slightly bloaty too..
It's definitely a thing to just chalk up to me being a newbie and probably isn't worth intervention. As far as a marker file on the same level as .INBOX, I know for my part I'd rather save the 4 million extra inodes instead of having that placeholder.
If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life.
Check what the new Courier POP3 UIDLs look like. If all of them are maildir base filename, setting pop3_uidl_format=%f should make this conversion easy. Just run it through once to get old UIDLs converted and then you don't need to worry about it, because both Courier and Dovecot give the same UIDLs. But I never really understood how Courier assigns new UIDLs, sometimes it seems to use the filename and sometimes not..
So do you mean keep pop3_uidl_format=%f basically forever? That'd work for me, but are there any big implications, performance-wise or other, of not using the default "%08Xu%08Xv"? Does that require a corresponding change to the conversion script? So I'd have Dovecot use the Courier-style UIDLs, run the conversion to get the old UIDLs (i.e. anything not using the Courier "%f", which is a bit) into dovecot-uidlist, and then not worry about any new mail, since clients will see the same UIDLs (and dovecot will just update dovecot-uidlist with those UIDLs the first time they log into Dovecot's POP3)?
A good portion of these are generated by our Courier 4.x, with lots more still kicking around from Courier 3.x. The main wrinkle is that we've acquired many many mailboxes from other companies running different servers, so the filenames can be a mixed bag. I'll hope for the best :)
On Wed, 2010-03-17 at 12:15 -0700, Mark Moseley wrote:
- Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir.
Have you read the thread starting from http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month or so? That provides a good view of potential problems with NFS.
I'd read through that thread and subsequently forgot the implications. Presumably the same pitfalls apply to 2.0?
Yes.
From the wiki and from the thread, it sounds like this just affects index files. One thing I didn't see in the thread (though it'd be easy to miss in a thread that long) is whether performance suffered horribly with 'noac' on a *separate* index mount (the mentions I can find where they tried 'noac', it was on the entire mail store, not a separate index mount) -- though doing 'noac' on anything seems like a recipe for disaster.
noac increases the NFS traffic for indexes by something like 10x, regardless of how it's mounted.
Another extreme option might be to just keep indexes on local disk only. Obviously users hitting another server wouldn't benefit from the other server and it'd incur updates to the indexes but they seem somewhat minimal -- in sane cases at least where the number of newer-than-the-index messages is reasonable. With aggressive load balancer stickiness at maybe a /24 netblock level, I'd be able to keep people localized for a little while (with no way around people accessing the same mailbox from home and work). Am I underestimating the costs of incremental index updates on local disks? In testing, tt seemed like dovecot was pretty conservative in disk accesses/reads when updating. I'm hoping the 95%-of-the-time worst case will be someone hitting the same mailbox from two locations, so they'd be hitting at most two servers. With local index caches, I'm mildly sure I could live with that (esp if I can talk my way into an SSD on each one). Am I being too naive?
I don't know if anyone has run local indexes in larger setups, so I can't really give any good answers. The worst case of rebuilding the whole index isn't anyway any worse than what Courier has to do every time.
I'll play around with that more. So it should be possible to have more than one private 'hidden=no; list=yes' namespace that'd show up in a mail client's subscribable folders list?
Yes. And probably hidden=yes is better, since it doesn't really need to show up as a separate namespace for clients (even though most would ignore it anyway, but some might show it a bit weirdly).
Offhand, before I start sifting through source code, is Dovecot on a 32-bit system limited to a 32-bit -- i.e. 2 gig -- limit on quota size like Courier is?
Quota is always 64bit. And Dovecot usually tries to enable 64 bit file offsets for 32 bit systems too, and then there aren't any 32 bit file size limits anywhere.
If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life.
Check what the new Courier POP3 UIDLs look like. If all of them are maildir base filename, setting pop3_uidl_format=%f should make this conversion easy. Just run it through once to get old UIDLs converted and then you don't need to worry about it, because both Courier and Dovecot give the same UIDLs. But I never really understood how Courier assigns new UIDLs, sometimes it seems to use the filename and sometimes not..
So do you mean keep pop3_uidl_format=%f basically forever? That'd work for me, but are there any big implications, performance-wise or other, of not using the default "%08Xu%08Xv"?
%f is at least more reliable, because it's not fully guaranteed that UIDVALIDITY+UID never changes (although in practice they don't). If they do, then POP3 UIDL changes and clients redownload mail. The only disadvantage is the somewhat larger UIDLs causing a bit more and network bandwidth usage.
It's also possible to set pop3_save_uidl=yes, and that's probably a good idea also, so that whenever UIDL is sent to client, it's also saved to dovecot-uidlist. That allows you to later change pop3_uidl_format without changing existing UIDLs.
Does that require a corresponding change to the conversion script?
No.
So I'd have Dovecot use the Courier-style UIDLs, run the conversion to get the old UIDLs (i.e. anything not using the Courier "%f", which is a bit) into dovecot-uidlist, and then not worry about any new mail, since clients will see the same UIDLs (and dovecot will just update dovecot-uidlist with those UIDLs the first time they log into Dovecot's POP3)?
Yeah.
On Wed, Mar 17, 2010 at 1:09 PM, Timo Sirainen tss@iki.fi wrote:
On Wed, 2010-03-17 at 12:15 -0700, Mark Moseley wrote:
From the wiki and from the thread, it sounds like this just affects index files. One thing I didn't see in the thread (though it'd be easy to miss in a thread that long) is whether performance suffered horribly with 'noac' on a *separate* index mount (the mentions I can find where they tried 'noac', it was on the entire mail store, not a separate index mount) -- though doing 'noac' on anything seems like a recipe for disaster.
noac increases the NFS traffic for indexes by something like 10x, regardless of how it's mounted.
Yeah, this would be something of a last act of desperation. Hopefully if it comes to that, having a separate indexes-only NFS mount that's the *only* thing mounted with noac would make it barely tolerable. Is it safe to say that with 'noac' but using 'sync' wouldn't help? And I'm assuming it's file attributes being cached that leads to corruptions, so would leaving acdirmin/max at their default values but lowering acregmax to 1 (not sure if 0 disables it completely) prevent corruption in all but the most extremely concurrent cases? Again this would be a last ditch thing, esp considering how hot our Filers run already -- adding another chunk of cpu% would put the nail in the coffin.
It sounds like a local cache would be far preferable. Since it's *just* IMAP, I'm guessing it won't be too huge. I'm thinking I dedicate a local partition, mount it *with* atime, and then cron something to delete all index files that haven't been accessed in a certain amount of time (a week, 2 weeks, a month, whatever). I'd use something like this: mail_location=....:INDEX=/path/to/index/partition/Indexes/%2Mu/%2.2Mu/%u to keep from getting too many entries in a single directory/subdirectory.
Another extreme option might be to just keep indexes on local disk only. Obviously users hitting another server wouldn't benefit from the other server and it'd incur updates to the indexes but they seem somewhat minimal -- in sane cases at least where the number of newer-than-the-index messages is reasonable. With aggressive load balancer stickiness at maybe a /24 netblock level, I'd be able to keep people localized for a little while (with no way around people accessing the same mailbox from home and work). Am I underestimating the costs of incremental index updates on local disks? In testing, tt seemed like dovecot was pretty conservative in disk accesses/reads when updating. I'm hoping the 95%-of-the-time worst case will be someone hitting the same mailbox from two locations, so they'd be hitting at most two servers. With local index caches, I'm mildly sure I could live with that (esp if I can talk my way into an SSD on each one). Am I being too naive?
I don't know if anyone has run local indexes in larger setups, so I can't really give any good answers. The worst case of rebuilding the whole index isn't anyway any worse than what Courier has to do every time.
I'll check it out. And if we go that route and I'm still sane at the end, I'll be sure to report back :)
I'll play around with that more. So it should be possible to have more than one private 'hidden=no; list=yes' namespace that'd show up in a mail client's subscribable folders list?
Yes. And probably hidden=yes is better, since it doesn't really need to show up as a separate namespace for clients (even though most would ignore it anyway, but some might show it a bit weirdly).
Offhand, before I start sifting through source code, is Dovecot on a 32-bit system limited to a 32-bit -- i.e. 2 gig -- limit on quota size like Courier is?
Quota is always 64bit. And Dovecot usually tries to enable 64 bit file offsets for 32 bit systems too, and then there aren't any 32 bit file size limits anywhere.
Excellent. Not that I really want to give out +2gb mailboxes but now I won't have to explain for the 30th time to marketing folks why I can't raise someone over the 2gig quota.
If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life.
Check what the new Courier POP3 UIDLs look like. If all of them are maildir base filename, setting pop3_uidl_format=%f should make this conversion easy. Just run it through once to get old UIDLs converted and then you don't need to worry about it, because both Courier and Dovecot give the same UIDLs. But I never really understood how Courier assigns new UIDLs, sometimes it seems to use the filename and sometimes not..
So do you mean keep pop3_uidl_format=%f basically forever? That'd work for me, but are there any big implications, performance-wise or other, of not using the default "%08Xu%08Xv"?
%f is at least more reliable, because it's not fully guaranteed that UIDVALIDITY+UID never changes (although in practice they don't). If they do, then POP3 UIDL changes and clients redownload mail. The only disadvantage is the somewhat larger UIDLs causing a bit more and network bandwidth usage.
It's also possible to set pop3_save_uidl=yes, and that's probably a good idea also, so that whenever UIDL is sent to client, it's also saved to dovecot-uidlist. That allows you to later change pop3_uidl_format without changing existing UIDLs.
Ah, I think I didn't quite get it the first time around. I was thinking that just setting pop3_uidl_format would be enough. But I see now that I've got to have a sequence base in dovecot-uidlist created by the conversion script first. And only if I've run the conversion script relatively recently (and the mailbox doesn't get any of the seen-by-Courier and unseen-by-Dovecot emails removed (i.e. via imap/webmail) so the ordering is identical will things not get redownloaded. It seems like it'd be rare for someone to delete a message from their POP3 mailbox via IMAP/webmail, so I'm going to cross my fingers that this takes care of 99% of the cases.
BTW, my Courier version is pretty recent and it seems to be giving back just this format: pop3_uidl_format = UID%u-%v
Just to be sure I'm interpreting this right (I say that a lot), if I use 'pop3_save_uidl=yes' from the beginning, after everyone is 100% migrated to dovecot on both POP3/IMAP, I should be able to change back to the dovecot-style UIDLs without triggering a re-download for any POP3 leave-mail-on-server people?
Does that require a corresponding change to the conversion script?
No.
So I'd have Dovecot use the Courier-style UIDLs, run the conversion to get the old UIDLs (i.e. anything not using the Courier "%f", which is a bit) into dovecot-uidlist, and then not worry about any new mail, since clients will see the same UIDLs (and dovecot will just update dovecot-uidlist with those UIDLs the first time they log into Dovecot's POP3)?
Yeah.
On Wed, Mar 17, 2010 at 3:12 PM, Mark Moseley moseleymark@gmail.com wrote:
On Wed, Mar 17, 2010 at 1:09 PM, Timo Sirainen tss@iki.fi wrote: <snip>
I don't know if anyone has run local indexes in larger setups, so I can't really give any good answers. The worst case of rebuilding the whole index isn't anyway any worse than what Courier has to do every time.
I'll check it out. And if we go that route and I'm still sane at the end, I'll be sure to report back :)
Update on this one point:
We've been running local indexes for a while now and it's worked wonderfully. The load balancers in front of 2 groups of 30 IMAP servers are doing a sticky-source persistence of a day. Our NFS reads/sec dropped by literally 8x by moving to Dovecot, even with the local indexes--looking at the graphs, it's a seriously jaw-dropping plummet. I've not heard a single complaint of corruption. Next step is to get the indexes on SSDs just to squeeze out a few extra jiffies.
All in all, it's been remarkably successful (and almost bizarrely gotcha-free -- though I haven't tackled POP3 yet). Our formerly irate IMAP users are all sweetness and light now. Thanks Timo!
On 3/16/2010 7:36 PM, Timo Sirainen wrote:
On 17.3.2010, at 1.01, Mark Moseley wrote:
- Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at.
Yes, hopefully it's coming soon :)
- Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir.
Have you read the thread starting from http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month or so? That provides a good view of potential problems with NFS.
- Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things.
v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
- Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?
Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or both IMAPs is just going to cause trouble because of conflicting UIDs. You might be able to make both Dovecot and Courier use the same POP3 UIDLs, but I wouldn't really trust it.
One possibility would be to just run the migration script on login, so users would migrate to Dovecot as they log in.
I would add a big warning as well to running Courier and Dovecot concurrently. If users are strictly segregated to using one or the other (consistently), then strictly running Courier and Dovecot on the same machine is not an issue (on different ports or whatever). But, I would NOT recommend having it such that a user could hit Courier one time and Dovecot another...I think that is begging for big trouble. As Timo mentions, most of the trouble will be with syncing the UIDs. Even running POP Courier and Dovecot IMAP might be a problem...especially if your users can use both POP and IMAP....Courier has 2 uidl files, (at least the version we were migrating from)..one for POP and the other for IMAP, and they get combined to one in Dovecot (dovecot-uidlist). Then if a user comes in via Dovecot...and updates the dovecot-uidlist...by definition the Courier files would be out of date. I would really steer away from that approach.
- Union mailboxes: I'm pretty sure in a fairly recent thread that Timo said that something like a 'union' mailbox (at least with maildir) wasn't possible.
I also had a thought where you could do that if you wrote some scripts and such that copied the mails to the other storage and replaced the original file with a symlink. But that of course has some potential race conditions and other problems with either side of the symlink disappearing but the other not.
Single-dbox would really be the best solution for this in future. It's currently somewhat broken in v2.0 tree, but it probably won't be too long until I'm going to start doing a migration from Maildir to dbox for a similar NFS system than yours.
I tried messing with multiple 'private' namespaces (i.e. a namespace called "ARCHIVE" with a "location" different than the INBOX location, ideally placed on slower but denser NFS servers) but even with 'hidden=no' and 'list=yes', only the main INBOX folders would show up, so I'm guessing that's not going to work.
You can create more namespaces, so I guess that was some kind of a configuration problem.
That would be a killer feature, to be able to serve an alternate namespace that would show up in a mail client's subscribable list that could be on slower storage than the main inbox (though I'm not sure a mail client can even handle multiple namespaces).
The clients don't need to be aware of namespaces. And you should be able to do that already. But do you think users would actually move their mails to there?
- Any problems with keeping only quota limits in db and not current quota numbers? Our limits come out of a SQL table but the current counts just live in the maildir file.
That's how most people do it.
- Any problem with leaving the namespace in "Courier compatibility" mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4 million mailboxes, FAQs all over the place, and support reps trained in a particular way of doing things in IMAP, it'd be hellish to try to change the prefix (I know I could leave the courier namespace around with 'hidden=yes' but retraining support staff is perhaps better left to phase #2). Do I lose anything besides tidiness by not changing it to "prefix =" as if I was deploying dovecot from scratch? Does it hurt performance in any significant way? Benchmarking doesn't look any different, so I'm guessing not.
Keeping the INBOX. prefix for the initial migration is a good idea. Once everything has worked perfectly for a few months, moving to hidden=yes could be a good idea too. There are some clients that don't like the INBOX. prefix.
- One thing that threw me and might be good for a FAQ (unless it was just me misconfiguring things) was when I started playing with putting the index files in an alternate location. I was utterly perplexed why it'd create the directories for the indexes but they'd be empty. Based on their location and names when they're in the maildir, I was just looking for the same dovecot.index* files right in the alternate directory. It wasn't till I started strace'ing that I noticed that the index files were indeed getting created but in a subdirectory called .INBOX (and with me just doing 'ls').
I try to avoid having a FAQ. Instead I try to change Dovecot so that the question itself goes away. I haven't heard of people having this problem before .. but one possible solution could be to just create a file to the directory containing some text that makes it clear. Then again that is slightly bloaty too..
If we're going to have to live with users complaining about a one-time redownload of just post-conversion mail, I'll need to get started convincing the higher-ups that that's life.
Check what the new Courier POP3 UIDLs look like. If all of them are maildir base filename, setting pop3_uidl_format=%f should make this conversion easy. Just run it through once to get old UIDLs converted and then you don't need to worry about it, because both Courier and Dovecot give the same UIDLs. But I never really understood how Courier assigns new UIDLs, sometimes it seems to use the filename and sometimes not..
On Wed, Mar 17, 2010 at 12:43 PM, Tony Rutherford tony@bluetie.com wrote:
On 3/16/2010 7:36 PM, Timo Sirainen wrote:
On 17.3.2010, at 1.01, Mark Moseley wrote:
- Since Dovecot 2.0 seems like it's just around the corner, that's all I've been testing, and indeed all I've even looked at.
Yes, hopefully it's coming soon :)
- Our #1 main motivation for looking Dovecot is relief for our currently overtaxed NFS servers, mostly in the form of the index files. Benchmarking dovecot looks great, even with the index files in the maildir.
Have you read the thread starting from http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month or so? That provides a good view of potential problems with NFS.
- Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things.
v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
- Any problems running Courier POP3 and Dovecot IMAP for a while, possibly Courier IMAP and Dovecot IMAP concurrently?
Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or both IMAPs is just going to cause trouble because of conflicting UIDs. You might be able to make both Dovecot and Courier use the same POP3 UIDLs, but I wouldn't really trust it.
One possibility would be to just run the migration script on login, so users would migrate to Dovecot as they log in.
I would add a big warning as well to running Courier and Dovecot concurrently. If users are strictly segregated to using one or the other (consistently), then strictly running Courier and Dovecot on the same machine is not an issue (on different ports or whatever). But, I would NOT recommend having it such that a user could hit Courier one time and Dovecot another...I think that is begging for big trouble. As Timo mentions, most of the trouble will be with syncing the UIDs. Even running POP Courier and Dovecot IMAP might be a problem...especially if your users can use both POP and IMAP....Courier has 2 uidl files, (at least the version we were migrating from)..one for POP and the other for IMAP, and they get combined to one in Dovecot (dovecot-uidlist). Then if a user comes in via Dovecot...and updates the dovecot-uidlist...by definition the Courier files would be out of date. I would really steer away from that approach.
I'll definitely keep that in mind. I should be able to keep things pretty segregated in terms of POP3 alone or IMAP alone but my big worry is that Courier POP3+Dovecot IMAP scenario. I'm thinking especially of people using POP3 in one place and webmail in another location. I'm guessing that people using POP3 in one location but direct IMAP in another are fairly rare. Whatever the case, it sounds like I need to do some log analysis to see what % of people access in both ways. Thanks for the advice!
On Wed, 2010-03-17 at 13:27 -0700, Mark Moseley wrote:
I'll definitely keep that in mind. I should be able to keep things pretty segregated in terms of POP3 alone or IMAP alone but my big worry is that Courier POP3+Dovecot IMAP scenario.
I know a large installation that was (is?) using Dovecot IMAP + Courier POP3 for years. That works fine. Or there were some problems, but I think they were mainly related to some external tools that assumed things worked the way Dovecot did them, but Courier messed up that expectation.. I can't remember specifics.
Anyway afterward when migrating POP3 to Dovecot, the migration script should also work fine as long as you delete courierimapuidlist files before running it, so that it'll only try to preserve POP3 UIDLs. Of course, testing it well is still a good idea..
Oops, forgot to ask one other thing....
On Tue, Mar 16, 2010 at 4:36 PM, Timo Sirainen tss@iki.fi wrote:
On 17.3.2010, at 1.01, Mark Moseley wrote: <snip>
- Exim: We currently deliver all of our mail via Exim on separate servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail servers delivering to maildirs only do Exim. From what I've seen in the docs and various threads, from what I can gather, the best thing to do in that case would be to use Exim's built-in maildir handling, instead of using 'deliver'. That would be my preference anyway, but I wanted to make sure I didn't misinterpret things.
v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
Since Exim wouldn't be touching the index files, is it safe to leave exim as-is and let it handle the deliveries to the maildirs natively? Exim's already got access to everything it needs including the quota. I just want to make sure I won't horribly corrupt anything leaving it as-is -- but I'm also desirous to not have to mess with how deliveries are currently being done.
On Wed, 2010-03-17 at 13:19 -0700, Mark Moseley wrote:
Since Exim wouldn't be touching the index files, is it safe to leave exim as-is and let it handle the deliveries to the maildirs natively? Exim's already got access to everything it needs including the quota. I just want to make sure I won't horribly corrupt anything leaving it as-is -- but I'm also desirous to not have to mess with how deliveries are currently being done.
That's fine.
On 3/17/2010 1:21 PM, Timo Sirainen wrote:
On Wed, 2010-03-17 at 13:19 -0700, Mark Moseley wrote:
Since Exim wouldn't be touching the index files, is it safe to leave exim as-is and let it handle the deliveries to the maildirs natively? Exim's already got access to everything it needs including the quota. I just want to make sure I won't horribly corrupt anything leaving it as-is -- but I'm also desirous to not have to mess with how deliveries are currently being done.
That's fine.
ok, but wouldn't that negate the point of the dovecot deliver being able to immediately update the index so that it wouldn't have to be rebuilt? If you had exim to the delivery straight to the Maildir, then the index wouldn't get updated, right?
On Tue, 2010-05-25 at 23:27 -0700, Tim Traver wrote:
Since Exim wouldn't be touching the index files, is it safe to leave exim as-is and let it handle the deliveries to the maildirs natively? Exim's already got access to everything it needs including the quota. I just want to make sure I won't horribly corrupt anything leaving it as-is -- but I'm also desirous to not have to mess with how deliveries are currently being done.
That's fine.
ok, but wouldn't that negate the point of the dovecot deliver being able to immediately update the index so that it wouldn't have to be rebuilt?
Index is never "rebuilt", it's "updated". If you don't use deliver, the main noticeable difference comes if user has a lot of new mails, and client fetches all of their headers. Since they're not in cache file yet, there's a delay until they're all read and parsed.
On Wed, May 26, 2010 at 3:11 AM, Timo Sirainen tss@iki.fi wrote:
On Tue, 2010-05-25 at 23:27 -0700, Tim Traver wrote:
Since Exim wouldn't be touching the index files, is it safe to leave exim as-is and let it handle the deliveries to the maildirs natively? Exim's already got access to everything it needs including the quota. I just want to make sure I won't horribly corrupt anything leaving it as-is -- but I'm also desirous to not have to mess with how deliveries are currently being done.
That's fine.
ok, but wouldn't that negate the point of the dovecot deliver being able to immediately update the index so that it wouldn't have to be rebuilt?
Index is never "rebuilt", it's "updated". If you don't use deliver, the main noticeable difference comes if user has a lot of new mails, and client fetches all of their headers. Since they're not in cache file yet, there's a delay until they're all read and parsed.
Yeah, we're doing exim deliveries straight to the maildir, so there's no MTA effect on indexes. I'm sure that there's some non-negligible performance hit doing things this way. The upshot though is that it's so *very* much better performing that with courier+NFS that everyone is happy: netapps aren't creaking under ops/sec load, so users' IMAP and webmail are much much faster and execs are happy that they don't need to immediately buy more netapps to deal with heavy load.
I've attached a slightly cropped rrd graph of NFS read bytes/sec on 6 mail netapps from one of our datacenters. See if you can spot where we started moving IMAP to dovecot over the course of about a month :)
On 2010-05-26 1:21 PM, Mark Moseley wrote:
I've attached a slightly cropped rrd graph of NFS read bytes/sec on 6 mail netapps from one of our datacenters. See if you can spot where we started moving IMAP to dovecot over the course of about a month :)
Wow, that's impressive... we had similar experience here, but ours was just a single server, no NFS or anything, much lower load than yours... but employees just as happy... :)
--
Best regards,
Charles
Mark,
-----Original Message-----
I've attached a slightly cropped rrd graph of NFS read bytes/sec on 6 mail netapps from one of our datacenters. See if you can spot where we started moving IMAP to dovecot over the course of about a month :)
While we're sharing graphs, I've got one you might like - dovecot on NFS mounted with noac vs attribute caching enabled. noac fixes some distributed locking issues quite well, but absolutely trashes the servers.
-Brad
participants (6)
-
Brad Davidson
-
Charles Marcus
-
Mark Moseley
-
Tim Traver
-
Timo Sirainen
-
Tony Rutherford