Re: [Dovecot] Dovecot deliver logging problem and procmail
Hello Timo!
Any ideas why it may take so long time to open big mbox files with indexes and delivered through dovecot deliver?
So it looks like that something is wrong with the indexes or the algorithm.
For details have a look below.
BTW: procmail filtering through deliver works well now for me in production environment. I'll make some code cleanups and configuration options. Maybe I can release the patch today.
Ciao, Gerhard
On Mon, 3 Jul 2006, Timo Sirainen wrote:
On Jul 3, 2006, at 11:07 PM, Gerhard Wiesinger wrote:
As far as I saw dovecot's deliver process updates some header fields and the index files: Status: X-Keywords: (with a lot of spaces, why?) Content-Length: 6
The extra spaces are there so that if Dovecot needs to update some headers it can take the needed space from there so it doesn't have to move the whole mbox.
Yes, I saw it after I wrote the mail ...
After access with dovecot the following header fields change: Status: O X-IMAPbase: 115195111 0000000009 X-UID: 2
So basically this isn't an advantage over the procmail or any other delivering process top mbox. The problem is that after the first access the X-UID and X-IMAPbase header must be added, so on large mbox files this might take a long time.
Well, first of all Dovecot should take the space it needs from the X-Keywords spaces, so it doesn't need to write all that much data.
Second, the X-UID etc. headers are added by deliver if the mbox file is fully synced against the indexes. So after you had opened it once with IMAP, after that deliver should write the X-UID headers also as long as you don't use something else than Dovecot to modify the mbox.
Ok, that worked well. I tried to deliver some large mails (~50MB) which worked well. After sending around 500MB (10 Mails) I opened the mailbox with pine over imap. It still took a lot of time (>20s) to open the mbox. Indexes should be up2date, so why does it take so long time?
After opening a second time everything is fine.
Any ideas?
Ciao, Gerhard
Hello! I think I found a bug in dovecots mailbox (mbox) handling: I delivered the mailbox with the following commands and my procmail patch applied: cat testmail3.txt | procmail -t -Y -a hostname -d user It is delivered with: /usr/libexec/dovecot/deliver -m testmail -d user -c /etc/dovecot.conf Test case was as the following: 1.) Deliver one mail (no mailbox exists, no indeces exist) 2.) Open the mailbox with IMAP to update indices, etc 3.) Deliver them again (e.g. 12 times) 4.) Copy it for backup to diff 5.) Open the mailbox with IMAP to update indices, etc 6.) Make diff (see below) Problem looks like that beginning with message 4 no X-UID header is updated. Some delivery I made in parallel, but the mailbox should be locked so this shouldn't be the problem. It also looks like that dovecot.index.cache isn't updated (I looked inside the files for the headers). It is empty after step 2 (even no subject from message 1) and "full" after step 5. Is this per design or a bug? Dovecot is 1.0rc2. Thank you for the answer. Ciao, Gerhard diff -ur xmailtest/testmail Mail/testmail --- xmailtest/testmail 2006-07-08 08:05:22.000000000 +0200 +++ Mail/testmail 2006-07-08 08:08:03.000000000 +0200 @@ -8,7 +8,7 @@ Status: O X-Keywords: Content-Length: 46820698 -X-IMAPbase: 1152338241 0000000001 +X-IMAPbase: 1152338241 0000000013 X-UID: 1 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -608158,8 +608158,8 @@ Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on X-UID: 2 -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -1216309,8 +1216309,8 @@ Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on X-UID: 3 -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -1824459,9 +1824459,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 4 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -2432609,9 +2432610,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 5 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -3040759,9 +3040761,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 6 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -3648909,9 +3648912,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 7 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -4257059,9 +4257063,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 8 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -4865209,9 +4865214,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 9 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -5473359,9 +5473365,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 10 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -6081509,9 +6081516,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 11 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -6689659,9 +6689667,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 12 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest @@ -7297809,9 +7297818,10 @@ Subject: Test from procmail Date: Sun, 02 Jul 2006 09:56:33 +0200 X-mytest: on -Status: -X-Keywords: +Status: O +X-Keywords: Content-Length: 46820698 +X-UID: 13 testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest On Wed, 5 Jul 2006, Gerhard Wiesinger wrote:
Hello Timo!
Any ideas why it may take so long time to open big mbox files with indexes and delivered through dovecot deliver?
So it looks like that something is wrong with the indexes or the algorithm.
For details have a look below.
BTW: procmail filtering through deliver works well now for me in production environment. I'll make some code cleanups and configuration options. Maybe I can release the patch today.
Ciao, Gerhard
On Mon, 3 Jul 2006, Timo Sirainen wrote:
On Jul 3, 2006, at 11:07 PM, Gerhard Wiesinger wrote:
As far as I saw dovecot's deliver process updates some header fields and the index files: Status: X-Keywords: (with a lot of spaces, why?) Content-Length: 6
The extra spaces are there so that if Dovecot needs to update some headers it can take the needed space from there so it doesn't have to move the whole mbox.
Yes, I saw it after I wrote the mail ...
After access with dovecot the following header fields change: Status: O X-IMAPbase: 115195111 0000000009 X-UID: 2
So basically this isn't an advantage over the procmail or any other delivering process top mbox. The problem is that after the first access the X-UID and X-IMAPbase header must be added, so on large mbox files this might take a long time.
Well, first of all Dovecot should take the space it needs from the X-Keywords spaces, so it doesn't need to write all that much data.
Second, the X-UID etc. headers are added by deliver if the mbox file is fully synced against the indexes. So after you had opened it once with IMAP, after that deliver should write the X-UID headers also as long as you don't use something else than Dovecot to modify the mbox.
Ok, that worked well. I tried to deliver some large mails (~50MB) which worked well. After sending around 500MB (10 Mails) I opened the mailbox with pine over imap. It still took a lot of time (>20s) to open the mbox. Indexes should be up2date, so why does it take so long time?
After opening a second time everything is fine.
Any ideas?
Ciao, Gerhard
On Sat, 2006-07-08 at 09:14 +0200, Gerhard Wiesinger wrote:
Test case was as the following: 1.) Deliver one mail (no mailbox exists, no indeces exist) 2.) Open the mailbox with IMAP to update indices, etc 3.) Deliver them again (e.g. 12 times) 4.) Copy it for backup to diff 5.) Open the mailbox with IMAP to update indices, etc 6.) Make diff (see below)
Problem looks like that beginning with message 4 no X-UID header is updated. Some delivery I made in parallel, but the mailbox should be locked so this shouldn't be the problem.
Wonder how you could get even that many added. The problem is that the sync stamp/size fields in index file aren't updated after the mbox has been changed, so it should have stopped adding X-UIDs after each added mail.
This should fix it: http://dovecot.org/list/dovecot-cvs/2006-August/006254.html
participants (2)
-
Gerhard Wiesinger
-
Timo Sirainen