[Dovecot] Dovecot deliver logging problem and procmail

[Gerhard Wiesinger]

Hello!

I think I found a bug in dovecots mailbox (mbox) handling:
I delivered the mailbox with the following commands and my procmail patch 
applied:
cat testmail3.txt | procmail -t -Y -a hostname -d user
It is delivered with:
/usr/libexec/dovecot/deliver -m testmail -d user -c /etc/dovecot.conf

Test case was as the following:
1.) Deliver one mail (no mailbox exists, no indeces exist)
2.) Open the mailbox with IMAP to update indices, etc
3.) Deliver them again (e.g. 12 times)
4.) Copy it for backup to diff
5.) Open the mailbox with IMAP to update indices, etc
6.) Make diff (see below)

Problem looks like that beginning with message 4 no X-UID header is 
updated. Some delivery I made in parallel, but the mailbox should be 
locked so this shouldn't be the problem.

It also looks like that dovecot.index.cache isn't updated (I looked inside 
the files for the headers). It is empty after step 2 (even no subject from 
message 1) and "full" after step 5. Is this per design or a bug?

Dovecot is 1.0rc2.

Thank you for the answer.

Ciao,
Gerhard

diff -ur xmailtest/testmail Mail/testmail
--- xmailtest/testmail	2006-07-08 08:05:22.000000000 +0200
+++ Mail/testmail	2006-07-08 08:08:03.000000000 +0200
@@ -8,7 +8,7 @@
  Status: O
  X-Keywords:
  Content-Length: 46820698
-X-IMAPbase: 1152338241 0000000001
+X-IMAPbase: 1152338241 0000000013
  X-UID: 1

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -608158,8 +608158,8 @@
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
  X-UID: 2
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -1216309,8 +1216309,8 @@
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
  X-UID: 3
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -1824459,9 +1824459,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 4

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -2432609,9 +2432610,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 5

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -3040759,9 +3040761,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 6

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -3648909,9 +3648912,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 7

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -4257059,9 +4257063,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 8

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -4865209,9 +4865214,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 9

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -5473359,9 +5473365,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 10

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -6081509,9 +6081516,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 11

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -6689659,9 +6689667,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 12

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
@@ -7297809,9 +7297818,10 @@
  Subject: Test from procmail
  Date: Sun, 02 Jul 2006 09:56:33 +0200
  X-mytest: on
-Status: 
-X-Keywords: 
+Status: O
+X-Keywords:
  Content-Length: 46820698
+X-UID: 13

  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest


On Wed, 5 Jul 2006, Gerhard Wiesinger wrote:

> Hello Timo!
>
> Any ideas why it may take so long time to open big mbox files with indexes 
> and delivered through dovecot deliver?
>
> So it looks like that something is wrong with the indexes or the algorithm.
>
> For details have a look below.
>
> BTW: procmail filtering through deliver works well now for me in production 
> environment. I'll make some code cleanups and configuration options. Maybe I 
> can release the patch today.
>
> Ciao,
> Gerhard
>
> On Mon, 3 Jul 2006, Timo Sirainen wrote:
>
>> On Jul 3, 2006, at 11:07 PM, Gerhard Wiesinger wrote:
>> 
>>> As far as I saw dovecot's deliver process updates some header fields and 
>>> the index files:
>>> Status:
>>> X-Keywords: (with a lot of spaces, why?)
>>> Content-Length: 6
>> 
>> The extra spaces are there so that if Dovecot needs to update some headers 
>> it can take the needed space from there so it doesn't have to move the 
>> whole mbox.
>> 
>
> Yes, I saw it after I wrote the mail ...
>
>>> After access with dovecot the following header fields change:
>>> Status: O
>>> X-IMAPbase: 115195111 0000000009
>>> X-UID: 2
>>> 
>>> So basically this isn't an advantage over the procmail or any other 
>>> delivering process top mbox. The problem is that after the first access 
>>> the X-UID and X-IMAPbase header must be added, so on large mbox files this 
>>> might take a long time.
>> 
>> Well, first of all Dovecot should take the space it needs from the 
>> X-Keywords spaces, so it doesn't need to write all that much data.
>> 
>> Second, the X-UID etc. headers are added by deliver if the mbox file is 
>> fully synced against the indexes. So after you had opened it once with 
>> IMAP, after that deliver should write the X-UID headers also as long as you 
>> don't use something else than Dovecot to modify the mbox.
>
> Ok, that worked well. I tried to deliver some large mails (~50MB) which 
> worked well. After sending around 500MB (10 Mails) I opened the mailbox with 
> pine over imap. It still took a lot of time (>20s) to open the mbox. Indexes 
> should be up2date, so why does it take so long time?
>
> After opening a second time everything is fine.
>
> Any ideas?
>
> Ciao,
> Gerhard
>