Hi Stan,

On Thu, 10 Mar 2011, Stan Hoeppner wrote:

Chris Wilson put forth on 3/9/2011 10:02 AM:

...

I think it would be a good idea for me to move from mbox to mdbox, as I have 57,000 messages in my inbox, and clients hang for a minute whenever they request a CHECK, or Dovecot feels the need to run one itself.

Why do you believe switching to mdbox will fix this problem?

Because CHECK takes forever, I can understand why it's necessary on mbox files, I don't see that it should be necessary on an mdbox store, and so I'm hoping that mdbox will be much faster.

If you have 57k messages in your INBOX as you say, then you have a much bigger problem to solve than your choice of mailbox format.

Well forgive me for wanting to be easily able to find any email I received in the last year from Pine (which makes switching folders a little painful).

Seriously, this is 2011, it shouldn't be impossible to index and randomly access 60,000 records containing a few GB of data in reasonable time.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.

Chris Wilson put forth on 3/10/2011 6:09 AM:

Apparently your server doesn't want me to reply:

A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed:

stan@hardwarefreak.com SMTP error from remote mail server after RCPT TO:stan@hardwarefreak.com: host greer.hardwarefreak.com [65.41.216.221]: 554 5.7.1 : Client host rejected: Access denied

Replying to the list isn't sufficient? Do you see a compelling reason why I would need two copies of your reply?

-- Stan

Hi Timo,

On Thu, 10 Mar 2011, Timo Sirainen wrote:

On 9.3.2011, at 18.02, Chris Wilson wrote:

...

I think it would be a good idea for me to move from mbox to mdbox, as I have 57,000 messages in my inbox, and clients hang for a minute whenever they request a CHECK, or Dovecot feels the need to run one itself.

mbox_very_dirty_syncs=yes should get rid of this problem even without switching to mdbox.

That option sounds scary :) I thought it would be safer to switch to a format where messages are read-only, especially as I currently have some strange symptoms (blank or nearly blank messages, messages cut in two, etc.) with mbox and I know it's not good in general, or good at handling large mailboxes in particular.

...

dsync(chris): Error: open() failed with mbox file /home/chris/mail/admin: Permission denied

So that is intentional?..

Not exactly, it's a symlink to /var/spool/mail/admin which for some reason I don't have permission to open at the moment.

...

chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox dsync(chris): Error: Next message unexpectedly corrupted in mbox file /home/chris/mail/spam at 404513741

mbox bug. Same would happen if you tried to read that same mail via imap. It fixed itself automatically though.

Okay good, that probably explains why when I tried to debug it (suspecting a bad Content-Length header in the previous messages) I couldn't see any problem.

...

dsync(chris): Info: INBOX.Sent: only in source dsync(chris): Info: sent-mail: only in source dsync(chris): Info: INBOX.Trash: only in source

Just saying that destination doesn't have these mailboxes yet.

OK, good.

...

Note: how is it possible for the same folder to be "only in source" and "only in dest"?

I think they have different GUIDs, or maybe Dovecot failed to get one of their GUID. Dsync works with mailbox GUIDs, not names. Anyway, I don't really know why it would suddenly lose one of them.

Perhaps it's worth printing the guids in that case? Is the guid stored in the first message in the mailbox?

Note that the mdbox for spam was created (maybe not completely) by a previous run of dsync, and I really can't see how the guid of the spam mbox could have changed (it definitely wasn't recreated yesterday).

...

chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox dsync(chris): Info: spam: highest_modseq changed: 2 != 3

Now the latter is all I get whenever I run dsync. Is it broken?

Nope. The modseq syncing isn't really perfect. So everything should be synced now.

I'm a bit surprised by how much less disk space it's taking:

chris@one-mail(~)$ du -sk /var/spool/mail/chris mail mdbox 1010516 /var/spool/mail/chris 2083872 mail 1016664 mdbox

It doesn't seem to have backed up most of my mailboxes:

chris@one-mail(~)$ ls mdbox/mailboxes/

addressbook INBOX.Sent INBOX.Trash Sent sent-mail sent-mail-oct-2010 sent-mail-sep-2008 spam temp

chris@one-mail(~)$ ls mail

addressbook archive INBOX.Drafts mail postponed-msgs sent-mail sent-mail-nov-2010 spam vcn admin badbox INBOX.Sent payslips saved-messages sent-mail-aug-2006 sent-mail-oct-2010 temp aidbot Drafts INBOX.Trash pinerc Sent sent-mail-jan-2009 sent-mail-sep-2008 Trash

Any ideas how I can get it to backup the rest of my mailboxes to mdbox?

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.

Hi Timo,

On Sat, 12 Mar 2011, Timo Sirainen wrote:

On Thu, 2011-03-10 at 12:40 +0000, Chris Wilson wrote:

...

...

I think they have different GUIDs, or maybe Dovecot failed to get one of their GUID. Dsync works with mailbox GUIDs, not names. Anyway, I don't really know why it would suddenly lose one of them.

Perhaps it's worth printing the guids in that case? Is the guid stored in the first message in the mailbox?

Added: http://hg.dovecot.org/dovecot-2.0/rev/0a1a0d33c1b2

Great, thanks :) I'm impressed by how responsive you are to requests for help and new features in Dovecot. Thank you :)

...

It doesn't seem to have backed up most of my mailboxes: .. Any ideas how I can get it to backup the rest of my mailboxes to mdbox?

Does

doveadm mailbox list -u chris '*'

list all the mailboxes?

Yes, it appears to.

If it does, see if deleting the destination mdbox and running dsync again from scratch works.

Even worse this time:

chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox [sudo] password for chris: dsync(chris): Error: open() failed with mbox file /home/chris/mail/admin: Permission denied dsync(chris): Error: Failed to sync mailbox admin: Internal error occurred. Refer to server log for more information. [2011-03-16 11:00:45] dsync(chris): Info: spam: only in source dsync(chris): Info: temp: only in source dsync(chris): Info: Sent: only in source dsync(chris): Info: addressbook: only in source dsync(chris): Info: sent-mail-sep-2008: only in source dsync(chris): Info: sent-mail-oct-2010: only in source dsync(chris): Info: INBOX.Sent: only in source dsync(chris): Info: sent-mail: only in source dsync(chris): Info: INBOX.Trash: only in source You have mail in /var/spool/mail/chris

chris@one-mail(~)$ du -sk mdbox/ /var/spool/mail/chris mail 616368 mdbox/ 1020420 /var/spool/mail/chris 2082960 mail

So this time it only copied 600 MB of mail before silently giving up.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.

Hi Timo,

On Wed, 16 Mar 2011, Timo Sirainen wrote:

On Wed, 2011-03-16 at 15:23 +0000, Chris Wilson wrote:

...

chris@one-mail(~)$ sudo dsync -v -u chris backup mdbox:/home/chris/mdbox [sudo] password for chris: dsync(chris): Error: open() failed with mbox file /home/chris/mail/admin: Permission denied

Well, this is the problem. dsync aborts early because it can't open the mailbox.. Fixing it in the code didn't look as simple as I hoped, so since this is unlikely to be a common problem, maybe you could just remove the symlink for now or give enough permissions for the duration of dsync..

Thanks, I removed this symlink and one other, and fixed a similar problem where dsync would abort on my gzipped archive mailboxes, and this time it finished and seemed to be working well :)

Now I'm trying to migrate just my user account to mdbox, without affecting other users. I think I figured out that I have to use a SQL userdb with just an entry for me, falling back to the passwd userdb for everyone else. I think I got this configured, and Dovecot IMAP then appears to open my mdbox when I connect, but deliver still delivers to /var/spool/mail/chris.

doveconf -m lda doesn't pick up anything about the SQL userdb, so I don't see how deliver could even know that it should look here to pick up my mail location, but I think it should work out where the user is expecting their mail to be stored in order to deliver it to the correct place, shouldn't it? What am I missing?

chris@one-mail(~)$ doveconf -n # 2.0.8: /etc/dovecot/dovecot.conf doveconf: Warning: service anvil { client_limit=100 } is lower than required under max. load (203) # OS: Linux 2.6.18-164.10.1.el5xen x86_64 CentOS release 5.5 (Final) auth_mechanisms = plain login auth_verbose = yes auth_worker_max_count = 10 default_client_limit = 100 mail_location = mbox:~/mail:INBOX=/var/mail/%u mail_privileged_group = mail mbox_write_locks = fcntl mdbox_preallocate_space = yes mdbox_rotate_size = 16 M passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } passdb { driver = pam } service auth-worker { user = $default_internal_user } service imap-login { process_min_avail = 2 } service imap { process_limit = 80 } service pop3 { process_limit = 10 } ssl_cert =

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.

Hi Timo,

On Thu, 17 Mar 2011, Timo Sirainen wrote:

SQL just for one user? That seems a bit wasteful. You could instead use http://wiki2.dovecot.org/AuthDatabase/PasswdFile

Good idea, I hadn't thought of that, I've switched to passwd file now and it seems to be working.

dovecot-lda doesn't know if you're using SQL or whatever, it just does a userdb lookup from auth process. You need to give -d <user> parameter to dovecot-lda for this to happen.

OK, I've done that and now it seems to be working very well. It's extremely fast compared to mbox, as expected, which is excellent! Searching also feels faster for some reason.

Thanks for all your help!

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.