IMAP flags and dovecot-keywords not working as expected

[Mark Foley]

I think I've partially sorted out my issues.  First off, as the wiki:
http://wiki2.dovecot.org/MailboxFormat/Maildir points out: "The file [dovecot-keywords] must
not be directly modified ...", which is what I tried.  So the dovecot-keywords file I manually
created (entries 0-6):

0 Board_and_Committee
1 Completed
2 Health_Care_meetings
3 Notifications
4 OSHP-DAS
5 personal_or_To_Do
6 Retirement_exits
7 $label5
8 Junk
9 $Forwarded
10 $MDNSent
11 $label2

Was not used by Thunderbird. So, I removed that file, went into Thunderbird, and manually set a
random message to tag 1 which I had renamed as "Board and Committee". And behold! a new
dovecot-keyword file was created with the single entry:

0 $label1

As you (Peter) point out, Thunderbird retrieves the text for these from prefs.js.  The reason,
therefore, I have e.g.  "11 $label2" in my original hand-edited file is that Thunderbird caused
what I had designated as "Health Care Meeting" to be added to #11 since the previous slots were
already used by other things which it did not recognize.  Hence, they ended up with a IMAP flag
of 'l'. 

I repeated this for a total of 4 more tags, resulting in dovecot-keyword entries:

0 $label1
1 $label2
2 $label3
3 $label4
4 $label5

Now, setting the tag in Tbird results in the correct text and color being shown for the message
and the resulting IMAP tag being set correspondingly to 'a' thru 'e'. Flush with victory, I
then foolish violated the "don't modify directly principle" by adding:

5 $label6
6 $label7

Which I thought would correspond to my remaining TB tags. Of course, wrong. When I then set a
message to the 6th tag (keyword entry 5, $label6), it did not do it. Instead, for my last 2
tags it created 2 new entries:

7 personal_or_to_do
8 retirement_exits

So, IMAP flags 'f' and 'g' are skipped and "Personal or To Do" tags get an 'h' flag and
"Retirement Exits" get an 'i' flag. What was I thinking! Not worth starting over as this user
had no Outlook categories set to these values, so so-what if that user's files get flagged with
'h', 'i'.

Also interesting to note that the actual text of these last 2 tags is stored in the
dovecot-keywords file. I supposed this means that Thunderbird has a limit of 5 (or 7?) $label's
and any added beyond that get stored differently.

But, I'm not finished.  Of course, Thunderbird expects to set tags/IMAPflas on virgin messages
in the user's mail folders.  However, my need is to import Outlook categories and have
Thunderbird interpret the IMAP flags correctly having never set them directly. 

So, I repeated the same procedure in other folders with messages having Outlook categories. 
The problem here is that setting a message to "Board and Committee" (Tbird tag 1) did not
necessarily resulting in an IMAP flag of 'a'.  In some folders it was 'b', some 'c' and some
'd'.  I think my problem here was that I had manually set the IMAP flags on these message
beforehand.  So (I think) when Thunderbird, coordinating with Dovecot, went to set Tag 1, it
saw that flag 'a' was already used and it picked the next free flag.  Some folders had IMAP
files with 2 and 3 flags set, hence Tag 1 getting set to 'c' or 'd' in those folders.  I
believe if I had not initially pre-set the IMAP flags on these files I would have seen the
correct correspondance: Tag 1 = flag 'a', Tag 2 = flag 'b', etc.

What I then did in these cases was first, find out what IMAP flag a Tbird tagged file would be
set to (e.g.  'c'), then renamed all my *a files to e.g. *c. That worked.  Suddenly, folders
with 100 flags thusly set showed up with correct tags and colors in Thunderbird. 

To summarize, and this what I will do next time:

1. DO NOT pre-add IMAP flags to mail files.

2. For each folder in which there are Outlook categorized message, set an arbitrary message in
that folder to each of the Thunderbird flags corresponding to Outlook categories to determine
the correct flag letter -- which also causes Thunderbird and dovecot to save their respective
settings. 

3. Then, rename each mail file to have the correct IMAP flag(s). That can be quicky done using a
script and the list of message having Outlook categories.

I believe this procedure will work correctly and I will confirm that when I process my 2nd
Outlook user (who is, btw, the organization Director and who is a super user of Outlook
categories! Failure is not an option!). 

For anyone needing to do the same thing (migrate Outlook categories), I've included below a VB
script to run in Outlook which will output a list of all messages having categories.  The IMAP
mail file's Message ID can be used to locate the categorized message file in the IMAP folder
hierarchy and the category name (following the "~") indicates which Thunderbird tag to map it
to.  I'll not include the bash script to mass-append IMAP flags to these files as that script
will need some revising based on my recent experimentation, but should be a rather simple bash
exercise in any case. 

Note that the Outlook messages are also the same MAPI files, only the client used (Outlook
versus Thunderbird) are different. Outlook does not set IMAP flags to designate categories.
Categories are apparently stored in the user's .pst file.

------------CUT----------
Public Sub ListOutlookFolders()

    Dim olApp As Outlook.Application
    Dim olNamespace As Outlook.Namespace
    Dim olFolder As Outlook.MAPIFolder

    Set olApp = New Outlook.Application
    Set olNamespace = olApp.GetNamespace("MAPI")

    For Each olFolder In olNamespace.Folders
        Debug.Print olFolder.Name; ":", olFolder.Description
        ListFolders olFolder, 1
    Next

    Set olFolder = Nothing
    Set olNamespace = Nothing
    Set olApp = Nothing

End Sub

Sub ListFolders(myFolder As Outlook.MAPIFolder, Level As Integer)
    Dim olFolder As Outlook.MAPIFolder
'  go through each email
    scanFolder myFolder

'  Now we'll check for subfolders
    For Each olFolder In myFolder.Folders
'        Debug.Print ":"; String(Level * 2, "-"); olFolder.Name

'        go through each email
        scanFolder olFolder

        If olFolder.Folders.Count > 0 Then
            ListFolders olFolder, Level + 1
        End If
    Next
End Sub

Sub scanFolder(sFolder As Outlook.MAPIFolder)
Dim src As Folder
Dim oItem As Object
Dim propertyAccessor As Outlook.propertyAccessor
Set src = sFolder

Dim strHeader As String

For Each oItem In src.Items
    If TypeOf oItem Is Outlook.MailItem And oItem.Categories <> "" Then
'        Debug.Print "Cat: " + oItem.Categories
        Set propertyAccessor = oItem.propertyAccessor
        header =
propertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x007D001E")
        Dim headerLines() As String
        headerLines() = Split(header, vbCrLf)

        Dim thisHeader As Variant

        For Each thisHeader In headerLines
            If InStr(thisHeader, "Message-ID:") > 0 Then
               Debug.Print thisHeader + "~" + oItem.Categories
               Exit For
           End If
        Next
    End If
Next
End Sub
----------CUT---------

Example of resulting output:

Message-ID: <201109011105.p81B5666028910 at webserver.ohprs.org>~Red Category

Hopefully someone finds this useful.

THX --Mark

-----Original Message-----
> Subject: Re: IMAP flags and dovecot-keywords not working as expected
> To: dovecot at dovecot.org
> From: Peter Chiochetti <pch at myzel.net>
> Date: Sat, 30 Jul 2016 11:26:09 +0200
>
> Am 2016-07-30 um 08:00 schrieb Mark Foley:
> ?
> >
> > However, none of the tags show up correctly in Thunderbird.  If I manually set a message to
> > have a tag of 0, the corresponding IMAP file gets a flag of 'm', not 'a' and the following is
> > added to the dovecot-keywords files:
> >
> > 12 $label1
> >
> > How can I fix this? Where is "$label1" text defined? Why did Thunderbird not snag the text for
> > '0' from the dovecot-keywords file and give the IMAP file a tag of 'a'?
>
> Thunderbird flags are stored in the users prefs.js, eg:
> - user_pref("mailnews.tags.$label1.tag", "Important");
> - user_pref("mailnews.tags.$label1.color", "#FF0000");
>
> A kind of key->value assignment. The "$label[1-9]" keys are special, 
> where the number magically corresponds to the keyboard shorcut to tag 
> messages, 0 meaning clear all tags.
>
> There can be more than nine tags, but they wont have a shortcut then:
> - user_pref("mailnews.tags.ten.tag", "ten");
>
> 1) The server will only ever see the key. The user will only ever see 
> the value.
>
> 2) If you rename a label in TB, then only the value will change and the 
> server will still see the same key as before.
>
> 3) If you rename a key in dovecot, TB will not create a label for it and 
> the affected messages will no longer appear tagged, if TB does not know 
> about the key.
>
> 4) Dovecot adds to the keywords as it receives requests from clients: 
> Very likely there is a limit of 26 (letters of the alphabet) per 
> account; a-d=0-3 are already taken for internal use, so 22 remain.
>
>
> > My current theory is that the "Default" Thunderbird Tags corresponding to IMAP flags are not
> > changeable and if new tags are create in Tbird, they get new flag letters.  That would, of
> > course, mean that if a user changes Thunderbird tag name, they would lose all tag settings on
> > existing message.  That doesn't seem right and I hope my theory is wrong.
>
> I think you are mostly wrong: as long as you only use TB to work and as 
> long as you do not exceed the limit you should be save.
>
> Notice that tags are a scarce resource: any key you ever created counts 
> toward the limit - reusing old tags requires you to text-edit both 
> dovecot-keywords and TB prefs.js.
>
> -- 
> peter
>