[Dovecot] Migrating to mdbox
Hi all,
I think it would be a good idea for me to move from mbox to mdbox, as I have 57,000 messages in my inbox, and clients hang for a minute whenever they request a CHECK, or Dovecot feels the need to run one itself.
Looking at past threads:
http://www.mail-archive.com/dovecot@dovecot.org/msg35212.html http://comments.gmane.org/gmane.mail.imap.dovecot/53936
it seems that the safest way is to run:
sudo dsync -v -u chris backup mdbox:/home/chris/mdbox
eliminate any errors that it reports, test the new mdbox access temporarily, and then switch my config in Dovecot. Ideally I'd like to do that just for me, without disrupting other users of this server (that I help to admin).
This error occurs every time I run dsync, and I understand it, so I've elided it:
dsync(chris): Error: open() failed with mbox file /home/chris/mail/admin: Permission denied dsync(chris): Error: Failed to sync mailbox admin: Internal error occurred. Refer to server log for more information. [2011-03-09 15:50:10] dsync(chris): Info: spam: highest_modseq changed: 2 != 3
I don't understand the following errors from dsync (2.0.9):
chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox dsync(chris): Error: Next message unexpectedly corrupted in mbox file /home/chris/mail/spam at 404513741 dsync(chris): Warning: Transaction log file /home/chris/mail/.imap/spam/dovecot.index.log was locked for 252 seconds dsync(chris): Info: spam: only in source dsync(chris): Info: temp: only in source dsync(chris): Info: Sent: only in source dsync(chris): Info: addressbook: only in source dsync(chris): Info: sent-mail-sep-2008: only in source dsync(chris): Info: sent-mail-oct-2010: only in source dsync(chris): Info: INBOX.Sent: only in source dsync(chris): Info: sent-mail: only in source dsync(chris): Info: INBOX.Trash: only in source
chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox dsync(chris): Info: spam: only in source dsync(chris): Info: spam: only in dest dsync(chris): Error: Failed to sync mailbox spam: Mailbox doesn't exist: spam dsync(chris): Error: msg iteration failed: Couldn't open mailbox 407c272bc369304dbc210000d6a93087 dsync(chris): Error: Trying to open a non-listed mailbox with guid=407c272bc369304dbc210000d6a93087 dsync(chris): Error: msg iteration failed: Couldn't open mailbox 407c272bc369304dbc210000d6a93087 dsync(chris): Error: Trying to open a non-listed mailbox with guid=407c272bc369304dbc210000d6a93087 dsync(chris): Error: Failed to sync mailbox spam: Mailbox doesn't exist: spam
Note: how is it possible for the same folder to be "only in source" and "only in dest"?
chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox dsync(chris): Info: spam: highest_modseq changed: 2 != 3 dsync(chris): Info: sent-mail: uidnext changed: 378 != 375
chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox dsync(chris): Info: spam: highest_modseq changed: 2 != 3
Now the latter is all I get whenever I run dsync. Is it broken?
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
Hi Stan,
On Thu, 10 Mar 2011, Stan Hoeppner wrote:
Because CHECK takes forever, I can understand why it's necessary on mbox files, I don't see that it should be necessary on an mdbox store, and so I'm hoping that mdbox will be much faster.
If you have 57k messages in your INBOX as you say, then you have a much bigger problem to solve than your choice of mailbox format.
Well forgive me for wanting to be easily able to find any email I received in the last year from Pine (which makes switching folders a little painful).
Seriously, this is 2011, it shouldn't be impossible to index and randomly access 60,000 records containing a few GB of data in reasonable time.
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
Hi Stan,
On Thu, 10 Mar 2011, Chris Wilson wrote:
Apparently your server doesn't want me to reply:
A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed:
stan@hardwarefreak.com SMTP error from remote mail server after RCPT TO:<stan@hardwarefreak.com>: host greer.hardwarefreak.com [65.41.216.221]: 554 5.7.1 <one-mail.aptivate.org[87.106.150.205]>: Client host rejected: Access denied
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
On 9.3.2011, at 18.02, Chris Wilson wrote:
mbox_very_dirty_syncs=yes should get rid of this problem even without switching to mdbox.
So that is intentional?..
mbox bug. Same would happen if you tried to read that same mail via imap. It fixed itself automatically though.
dsync(chris): Warning: Transaction log file /home/chris/mail/.imap/spam/dovecot.index.log was locked for 252 seconds
Normal with a large mbox.
Just saying that destination doesn't have these mailboxes yet.
I think they have different GUIDs, or maybe Dovecot failed to get one of their GUID. Dsync works with mailbox GUIDs, not names. Anyway, I don't really know why it would suddenly lose one of them.
Nope. The modseq syncing isn't really perfect. So everything should be synced now.
Hi Timo,
On Thu, 10 Mar 2011, Timo Sirainen wrote:
That option sounds scary :) I thought it would be safer to switch to a format where messages are read-only, especially as I currently have some strange symptoms (blank or nearly blank messages, messages cut in two, etc.) with mbox and I know it's not good in general, or good at handling large mailboxes in particular.
Not exactly, it's a symlink to /var/spool/mail/admin which for some reason I don't have permission to open at the moment.
Okay good, that probably explains why when I tried to debug it (suspecting a bad Content-Length header in the previous messages) I couldn't see any problem.
OK, good.
Perhaps it's worth printing the guids in that case? Is the guid stored in the first message in the mailbox?
Note that the mdbox for spam was created (maybe not completely) by a previous run of dsync, and I really can't see how the guid of the spam mbox could have changed (it definitely wasn't recreated yesterday).
I'm a bit surprised by how much less disk space it's taking:
chris@one-mail(~)$ du -sk /var/spool/mail/chris mail mdbox 1010516 /var/spool/mail/chris 2083872 mail 1016664 mdbox
It doesn't seem to have backed up most of my mailboxes:
chris@one-mail(~)$ ls mdbox/mailboxes/
addressbook INBOX.Sent INBOX.Trash Sent sent-mail sent-mail-oct-2010 sent-mail-sep-2008 spam temp
chris@one-mail(~)$ ls mail
addressbook archive INBOX.Drafts mail postponed-msgs sent-mail sent-mail-nov-2010 spam vcn admin badbox INBOX.Sent payslips saved-messages sent-mail-aug-2006 sent-mail-oct-2010 temp aidbot Drafts INBOX.Trash pinerc Sent sent-mail-jan-2009 sent-mail-sep-2008 Trash
Any ideas how I can get it to backup the rest of my mailboxes to mdbox?
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
On Thu, 2011-03-10 at 12:40 +0000, Chris Wilson wrote:
Added: http://hg.dovecot.org/dovecot-2.0/rev/0a1a0d33c1b2
Does
doveadm mailbox list -u chris '*'
list all the mailboxes? If it does, see if deleting the destination mdbox and running dsync again from scratch works.
Hi Timo,
On Sat, 12 Mar 2011, Timo Sirainen wrote:
Great, thanks :) I'm impressed by how responsive you are to requests for help and new features in Dovecot. Thank you :)
Yes, it appears to.
If it does, see if deleting the destination mdbox and running dsync again from scratch works.
Even worse this time:
chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox [sudo] password for chris: dsync(chris): Error: open() failed with mbox file /home/chris/mail/admin: Permission denied dsync(chris): Error: Failed to sync mailbox admin: Internal error occurred. Refer to server log for more information. [2011-03-16 11:00:45] dsync(chris): Info: spam: only in source dsync(chris): Info: temp: only in source dsync(chris): Info: Sent: only in source dsync(chris): Info: addressbook: only in source dsync(chris): Info: sent-mail-sep-2008: only in source dsync(chris): Info: sent-mail-oct-2010: only in source dsync(chris): Info: INBOX.Sent: only in source dsync(chris): Info: sent-mail: only in source dsync(chris): Info: INBOX.Trash: only in source You have mail in /var/spool/mail/chris
chris@one-mail(~)$ du -sk mdbox/ /var/spool/mail/chris mail 616368 mdbox/ 1020420 /var/spool/mail/chris 2082960 mail
So this time it only copied 600 MB of mail before silently giving up.
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
On Wed, 2011-03-16 at 15:23 +0000, Chris Wilson wrote:
Well, this is the problem. dsync aborts early because it can't open the mailbox.. Fixing it in the code didn't look as simple as I hoped, so since this is unlikely to be a common problem, maybe you could just remove the symlink for now or give enough permissions for the duration of dsync..
Hi Timo,
On Wed, 16 Mar 2011, Timo Sirainen wrote:
Thanks, I removed this symlink and one other, and fixed a similar problem where dsync would abort on my gzipped archive mailboxes, and this time it finished and seemed to be working well :)
Now I'm trying to migrate just my user account to mdbox, without affecting other users. I think I figured out that I have to use a SQL userdb with just an entry for me, falling back to the passwd userdb for everyone else. I think I got this configured, and Dovecot IMAP then appears to open my mdbox when I connect, but deliver still delivers to /var/spool/mail/chris.
doveconf -m lda doesn't pick up anything about the SQL userdb, so I don't see how deliver could even know that it should look here to pick up my mail location, but I think it should work out where the user is expecting their mail to be stored in order to deliver it to the correct place, shouldn't it? What am I missing?
chris@one-mail(~)$ doveconf -n
2.0.8: /etc/dovecot/dovecot.conf
doveconf: Warning: service anvil { client_limit=100 } is lower than required under max. load (203)
OS: Linux 2.6.18-164.10.1.el5xen x86_64 CentOS release 5.5 (Final)
auth_mechanisms = plain login auth_verbose = yes auth_worker_max_count = 10 default_client_limit = 100 mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail mbox_write_locks = fcntl mdbox_preallocate_space = yes mdbox_rotate_size = 16 M passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } passdb { driver = pam } service auth-worker { user = $default_internal_user } service imap-login { process_min_avail = 2 } service imap { process_limit = 80 } service pop3 { process_limit = 10 } ssl_cert = </etc/pki/dovecot/certs/mail.aptivate.org.chain.crt ssl_key = </etc/pki/dovecot/certs/mail.aptivate.org.key userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } userdb { driver = passwd } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 10 }
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
On Thu, 2011-03-17 at 11:26 +0000, Chris Wilson wrote:
SQL just for one user? That seems a bit wasteful. You could instead use http://wiki2.dovecot.org/AuthDatabase/PasswdFile
dovecot-lda doesn't know if you're using SQL or whatever, it just does a userdb lookup from auth process. You need to give -d <user> parameter to dovecot-lda for this to happen.
Hi Timo,
On Thu, 17 Mar 2011, Timo Sirainen wrote:
SQL just for one user? That seems a bit wasteful. You could instead use http://wiki2.dovecot.org/AuthDatabase/PasswdFile
Good idea, I hadn't thought of that, I've switched to passwd file now and it seems to be working.
OK, I've done that and now it seems to be working very well. It's extremely fast compared to mbox, as expected, which is excellent! Searching also feels faster for some reason.
Thanks for all your help!
Cheers, Chris.
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
participants (3)
-
Chris Wilson
-
Stan Hoeppner
-
Timo Sirainen