[Dovecot] virtual mailbox / INTHREAD use case, issues, questions

[David Abrahams]

At Tue, 19 Jan 2010 16:51:04 +0200,
Timo Sirainen wrote:
> 
> On Wed, 2010-01-06 at 11:50 -0500, David Abrahams wrote:
> > At Wed, 6 Jan 2010 14:17:40 +0200,
> > Timo Sirainen wrote:
> > > 
> > > On 4.1.2010, at 20.47, David Abrahams wrote:
> > > 
> > > > 1. I had to manually create the virtual folder for all my users or
> > > >   they couldn't access their mail at all.  Is this fixed in 2.0?
> > > 
> > > No.
> > 
> > OK, is it considered a bug that should be addressed?  IMO there should
> > at least be a way to tell dovecot to ignore the non-existence of a
> > particular folder.
> 
> Actually it looks like v2.0 just creates the directory if it's missing.
> I'm not really sure if it's a good thing or not, but I guess I moved
> that part of the code to be common across all storages, so maybe it's
> good. :)

Well, my inclination would be to ignore missing directories until you
try to do something with them.  If I set up a fancy virtual folder
arrangement for advanced users I'd like ordinary users not to ever
have to encounter it.

> > > "lots of mailboxes" should cause error to be logged.
> > 
> > What does the error look like?  I could search my logs for it.
> 
> Easiest would be to make Dovecot log errors to a different file
> (log_path to different than info_log_path). Then you wouldn't have to
> search for errors, simply see if anything shows up in the error
> log. 

I'll try that, thanks.

> If
> you want to grep, I suppose anything with Error: or Fatal: or Panic:
> prefix would do.

Nothing in the current logs, but they may have been rotated out.

> > > > 7. I tried to create an IMAP search, rather than a virtual folder,
> > > >   that looked for x-mailbox INBOX header like the virtual folder
> > > >   does.  It too came up empty.  It doesn't exactly surprise me
> > > >   because I don't see an x-mailbox header in any of these messages.
> > > 
> > > x-mailbox doesn't use a header, it uses the actual mailbox name
> > > where the message exists.
> > 
> > Oh... what if the message exists in multiple mailboxes?  Typically
> > anything in INBOX can also be found in my "all mail" archive.
> ..
> > I guess that means it's crucial that, whatever else I do, INBOX should
> > be part of the "all mail" virtual folder in order for this to work.  I
> > didn't quite understand that before, and I think it's important to
> > have a description somewhere of how x-mailbox works that would help me
> > to get to that conclusion.  Certainly the existing description at
> > http://wiki.dovecot.org/Plugins/Virtual that it represents the
> > "original" mailbox isn't adequate for that purpose.
> 
> Hmm. If you're making a virtual mailbox out of another virtual mailbox,
> I'm not entirely sure what X-MAILBOX should do. Currently it returns the
> parent virtual mailbox's name. But should it instead return the mailbox
> name that physically contains the mails? I'm beginning to think that it
> probably should. 

Why?  Because fetching from there will be faster?

> > Having got the x-mailbox insight (I think---thanks), I have tried 
> > the following combinations:
> > 
> >   # ~/Maildir/virtual/INBOX/dovecot-virtual
> >   INBOX
> >   zz_archive*
> >     inthread refs x-mailbox inbox
> > 
> > shows only messages in INBOX. THe archives are actually in folders
> > like zz_archive.2010.01
> 
> Is this fast or slow then?

Oh, perfectly fast, but also perfectly ineffective :-)

> >   # ~/Maildir/virtual/all/dovecot-virtual
> >   *
> >     all
> > 
> >   # ~/Maildir/virtual/INBOX/dovecot-virtual
> >   virtual.all
> >     inthread refs x-mailbox inbox
> > 
> > Appears to hang
> 
> It's probably going to take a while to build the thread index. After
> that it should be pretty fast. It would be easier to see how far it gets
> by talking imap:
> 
> a login username password
> b select virtual.all
> c thread refs all
> d select virtual.INBOX

a login <username> <pw>
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH] Logged in
b select virtual.all
closed

(boom!)

I guess that explains my problem... after a fashion.

Ah, my error file contains lots of these:

Jan 21 18:06:01 IMAP(dave): Error: mmap() failed with index cache file /home/dave/Maildir/.zz_archive.2009.06/dovecot.index.cache: Cannot allocate memory
Jan 21 18:06:01 IMAP(dave): Fatal: pool_system_realloc(1048576): Out of memory

Okay, I bumped up mail_process_size from 256 to 1024 and it seems not
to be bailing out quite so soon.

b select virtual.all
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-3 unknown-5 unknown-6 $Forwarded $NotJunk unknown-0 unknown-2 NonJunk NotJunk gnus-forward $Junk Junk gnus-download gnus-save unknown-1 $Label1 JunkRecorded $MDNSent unknown-4 gnus-expire unknown-8 receipt-handled)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-3 unknown-5 unknown-6 $Forwarded $NotJunk unknown-0 unknown-2 NonJunk NotJunk gnus-forward $Junk Junk gnus-download gnus-save unknown-1 $Label1 JunkRecorded $MDNSent unknown-4 gnus-expire unknown-8 receipt-handled \*)] Flags permitted.
* 149262 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1264126364] UIDs valid
* OK [UIDNEXT 149263] Predicted next UID
* OK [HIGHESTMODSEQ 24] Highest
b OK [READ-WRITE] Select completed.
c thread refs all
c BAD Error in IMAP command THREAD: Missing search parameters

But, whatever is wrong with command c, it does seem to be working!

> After the c) command is finished, the thread thread index is built and
> d) command should be fast enough. In my tests with 40k messages the d)
> command takes two seconds. Most of that time goes to opening
> virtual.all, which in turn is slow because of the number of mailboxes
> and could be made faster by mailbox list indexes.

Thanks for your help; I'll be playing with this and seeing how well it
performs in practice.

-- 
Dave Abrahams           Meet me at BoostCon: http://www.boostcon.com
BoostPro Computing
http://www.boostpro.com