[Dovecot] Huge IMAP folder takes forever

older
[Dovecot] Configuring for virtual...

Peter Lindgren

10 Dec 2006 10 Dec '06

8:07 p.m.

Hi! Just updated to 1.0 rc15, but the problem remains. I think it could be client-dependent to some extent.

I have only IMAPS access and Maildir. I have one mailbox which contains some 10,200 mails. That mailbox takes forever (have waited 30 minutes and then aborted) to open with Thunderbird 1.5.0.7. Other mailboxes (like dovecot :-), only 6984 mails) opens quickly.

When Thunderbird tries to open this folder, the imap process on the server uses roughly all cpu resources. There isn't much disk I/O, at least as far as I can tell by listening at the server.

The dovecot is upgraded from 0.99.something. Should I remove index files first, and in that case which? Is there something cached in Thunderbird?

Here's the directory listing for the troublesome folder:

-rw------- 1 peter peter 28 Jan 7 2006 .customflags -rw------- 1 peter peter 49248 Feb 7 2006 .imap.index -rw------- 1 peter peter 420998 Feb 7 2006 .imap.index.data -rw------- 1 peter peter 10256 Feb 7 2006 .imap.index.log -rw------- 1 peter peter 49200 Feb 7 2006 .imap.index.tree drwx------ 2 peter peter 686592 Dec 10 15:06 cur -rw------- 1 peter peter 64 Jun 6 2006 dovecot-keywords -rw------- 1 peter peter 619832 Dec 10 15:04 dovecot-uidlist -rw------- 1 peter peter 162992 Dec 10 15:21 dovecot.index -rw------- 1 peter peter 8826880 Dec 9 15:36 dovecot.index.cache -rw------- 1 peter peter 33776 Dec 10 15:22 dovecot.index.log -rw------- 1 peter peter 193660 Dec 3 12:50 dovecot.index.log.2 drwx------ 2 peter peter 512 Dec 10 15:06 new drwx------ 2 peter peter 512 Dec 10 15:04 tmp

Thanks in advance, Peter

Peter Lindgren, dataingenjör E-post: peter "at" norrskenkonsult punkt com http://www.norrskenkonsult.com

Show replies by date

Peter Lindgren

10 Dec 10 Dec

9:49 p.m.

Update: I did a kernel trace to see what the imap process was doing, and it was retrieving headers, but to me it seemed like there were an awful lot of null (\0) and control characters among the headers. Excerpts of the trace below, actual mail address hidden to protect the innocent.

It's difficult to trace a folder that works, since that imap process doesn't live long enough for me to trace it. Hints on that, other than being really quick on the keyboard? ;-) The traces I've managed has nice plaintext commands like these: 2994 imap 1165779379.972538 GIO fd 1 wrote 23 bytes "24 OK NOOP completed.\r " 2994 imap 1165779379.972620 RET write 23/0x17 ... 2994 imap 1165779379.977092 CALL read(0,0x6be31b,0xce5) 2994 imap 1165779379.977179 GIO fd 0 read 29 bytes "25 UID fetch 3657:* (FLAGS)\r " 2994 imap 1165779379.977245 RET read 29/0x1d

However, these reads/writes are to other file descriptors (0 and 1) than the erraneous reads below (fd 8).

My system: # uname -ap OpenBSD casiopea.norrskenkonsult.com 3.7 GENERIC#312 sparc SUNW,SPARCstation-5, MB86907 @ 170 MHz, on-chip FPU

# kdump -T -f imap.tracelog 9506 imap 1165775763.392950 EMUL "native" 9506 imap 1165775763.521181 CALL munmap(0xbef8000,0x100000) 9506 imap 1165775763.526077 RET munmap 0 9506 imap 1165775763.563349 CALL munmap(0xbeb7000,0x41000) 9506 imap 1165775763.564797 RET munmap 0 9506 imap 1165775763.564899 CALL munmap(0xbeb6000,0x1000) 9506 imap 1165775763.565074 RET munmap 0 9506 imap 1165775763.565192 CALL pread(0x8,0x8580000,0x1000,0,0,0x141000) 9506 imap 1165775763.566267 GIO fd 8 read 4088 bytes "\0\0\0\^\\0\0\0\0In-Reply-To: 1137747953.43d0a7f16017b@webbmail.thornqvist.se \0\0\0\0\^N\0\0\0Q\0\0\0\^W\0\0\0\0Message-ID: 6996e6f30601200332y1f7dcb9fi1397e97e41b9c04@mail.gmail.com \0\0\0\0\0\0\^Q\0\0\0t\0\0\0 \0\0\0\0References: 1137747953.43d0a7f16017b@webbmail.thornqvist.se \0\0\0\^R\0\0\0*\0\0\0\^[\0\0\0\0Subject: Re: [exakt] Pod-TV i SVT \0\0\0\0\0\^S\0\0\0.\0\0\0\^Z\0\0\0\0To: Exakt IT exakt@lists.frobbit.se \0\0\0\0\0 \0\0\0\0\0\0\0

\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0\f\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\^N\0\0\0\0\0\0\0\^O\0\0\0\0\0\0\0\^P\0\0\0\0\0\0\0\^Q\0\0\0\0\0\0\0\^R\0\0\0\0\0\0\0\

\^S\0\0\0\0\0\0\0\^T\0\0\0\0\0\0\0\0\0\0\0\b\0\0\0\b\0\0\0(\0\0\0H\0\0\0\0\0\0 \M-D\0\0\0\0\0\0

\M-w\0\0\0\0\0\0\^DG\0\0\0\0\0\0\^Dd\0\0\0\^]\0\^A3\M-X\0\0\^B\M-@\0\0\0 \0\0\0\\\0\0\0\^[\0\0\0\0Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response \0\0\0\v\0\0\0.\0\0\0\^Y\0\0\0\0Date: Fri, 20 Jan 2006 12:34:59 +0100 \0\0\0\0\0\f\0\0\0003\0\0\0\^T\0\0\0\0From: "Lars Ilshammar" xxxx@xxxxxxxxx.se \0\0\0\0\^N\0\0\0;\0\0\0\^S\0\0\0\0Message-ID: 004301c61db5$8b847e40$6400a8c0@LASSE \0\0\0\0\^Q\0\0\0o\0\0\0\^V\0\0\0\0References: 82ab3b60601192318q7c5d7cf1u6ee17edc0e11da92@mail.gmail.com 43D09D62.2080005@abm.uu.se \0\0\0\0\^R\0\0\0)\0\0\0\^X\0\0\0\0Subject: Re: [exakt] SMHI suger! \0\0\0\0\0\0\^S\0\0\0000\0\0\0\^U\0\0\0\0To: "Exakt IT" exakt@lists.frobbit.se \0\0\0\^T\0\0\0\^V\0\0\0\^^\0\0\0\0X-Priority: 3 \0\0\0\0\0 \0\0\0\0\0\0\0

\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0\f\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\^N\0\0\0\0\0\0\0\^O\0\0\0\0\0\0\0\^P\0\0\0\0\0\0\0\^Q\0\0\0\0\0\0\0\^R\0\0\0\0\0\0\0\

\^S\0\0\0\0\0\0\0\^T\0\0\0\0\0\0\0\0\0\0\0\b\0\0\0\b\0\0\0(\0\0\0H\0\0\0\0\0\0\b\M-<\0\0\0\0\0\0\b\M-l\0\0\0\0\0\0\^O\M-~\0\0\0\0\0\0\^PZ\0\0\0\\\0
\^A3\M-l\0\0\^B\M-T\0\0\0 \0\0\0F\0\0\0\^X\0\0\0\0Content-Type: text/plain; format=flowed; charset="iso-8859-1" \0\0\0\0\0\v\0\0\0.\0\0\0\^V\0\0\0\0Date: Fri, 20 Jan 2006 13:06:34 +0100

... continues like that until ...

9506 imap 1165775763.566267 GIO fd 8 read 8 bytes "\0\0\0\f\0\0\0H" 9506 imap 1165775763.566912 RET pread 4096/0x1000 9506 imap 1165775763.570811 CALL mmap(0,0x144000,0x3,0x1002,0xffffffff,0,0,0) 9506 imap 1165775763.570972 RET mmap 256024576/0xf42a000 9506 imap 1165775763.715347 CALL munmap(0x8481000,0x100000) 9506 imap 1165775763.719472 RET munmap 0 9506 imap 1165775763.757083 CALL munmap(0x843f000,0x42000) 9506 imap 1165775763.758449 RET munmap 0 9506 imap 1165775763.758569 CALL munmap(0x843e000,0x1000) 9506 imap 1165775763.758735 RET munmap 0 9506 imap 1165775763.758854 CALL pread(0x8,0xf56d000,0x1000,0,0,0x142000) 9506 imap 1165775763.759294 GIO fd 8 read 4088 bytes

...

/Peter

Peter Lindgren, dataingenjör E-post: peter "at" norrskenkonsult punkt com http://www.norrskenkonsult.com

Peter Lindgren

11:20 p.m.

New subject: [Dovecot] SOLVED: Huge IMAP folder takes forever

Problem solved, I removed all index files. Dovecot rebuilt them on access.

Peter Lindgren skrev:

...

Update: I did a kernel trace to see what the imap process was doing, and it was retrieving headers, but to me it seemed like there were an awful lot of null (\0) and control characters among the headers. Excerpts of the trace below, actual mail address hidden to protect the innocent.

It's difficult to trace a folder that works, since that imap process doesn't live long enough for me to trace it. Hints on that, other than being really quick on the keyboard? ;-) The traces I've managed has nice plaintext commands like these: 2994 imap 1165779379.972538 GIO fd 1 wrote 23 bytes "24 OK NOOP completed.\r " 2994 imap 1165779379.972620 RET write 23/0x17 ... 2994 imap 1165779379.977092 CALL read(0,0x6be31b,0xce5) 2994 imap 1165779379.977179 GIO fd 0 read 29 bytes "25 UID fetch 3657:* (FLAGS)\r " 2994 imap 1165779379.977245 RET read 29/0x1d

However, these reads/writes are to other file descriptors (0 and 1) than the erraneous reads below (fd 8).

My system: # uname -ap OpenBSD casiopea.norrskenkonsult.com 3.7 GENERIC#312 sparc SUNW,SPARCstation-5, MB86907 @ 170 MHz, on-chip FPU

# kdump -T -f imap.tracelog 9506 imap 1165775763.392950 EMUL "native" 9506 imap 1165775763.521181 CALL munmap(0xbef8000,0x100000) 9506 imap 1165775763.526077 RET munmap 0 9506 imap 1165775763.563349 CALL munmap(0xbeb7000,0x41000) 9506 imap 1165775763.564797 RET munmap 0 9506 imap 1165775763.564899 CALL munmap(0xbeb6000,0x1000) 9506 imap 1165775763.565074 RET munmap 0 9506 imap 1165775763.565192 CALL pread(0x8,0x8580000,0x1000,0,0,0x141000) 9506 imap 1165775763.566267 GIO fd 8 read 4088 bytes "\0\0\0\^\\0\0\0\0In-Reply-To: 1137747953.43d0a7f16017b@webbmail.thornqvist.se \0\0\0\0\^N\0\0\0Q\0\0\0\^W\0\0\0\0Message-ID: 6996e6f30601200332y1f7dcb9fi1397e97e41b9c04@mail.gmail.com \0\0\0\0\0\0\^Q\0\0\0t\0\0\0 \0\0\0\0References: 1137747953.43d0a7f16017b@webbmail.thornqvist.se \0\0\0\^R\0\0\0*\0\0\0\^[\0\0\0\0Subject: Re: [exakt] Pod-TV i SVT \0\0\0\0\0\^S\0\0\0.\0\0\0\^Z\0\0\0\0To: Exakt IT exakt@lists.frobbit.se \0\0\0\0\0 \0\0\0\0\0\0\0

\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0\f\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\^N\0\0\0\0\0\0\0\^O\0\0\0\0\0\0\0\^P\0\0\0\0\0\0\0\^Q\0\0\0\0\0\0\0\^R\0\0\0\0\0\0\0\

\^S\0\0\0\0\0\0\0\^T\0\0\0\0\0\0\0\0\0\0\0\b\0\0\0\b\0\0\0(\0\0\0H\0\0\0\0\0\0
    \M-D\0\0\0\0\0\0
\M-w\0\0\0\0\0\0\^DG\0\0\0\0\0\0\^Dd\0\0\0\^]\0\^A3\M-X\0\0\^B\M-@\0\0\0 \0\0\0\\\0\0\0\^[\0\0\0\0Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response \0\0\0\v\0\0\0.\0\0\0\^Y\0\0\0\0Date: Fri, 20 Jan 2006 12:34:59 +0100 \0\0\0\0\0\f\0\0\0003\0\0\0\^T\0\0\0\0From: "Lars Ilshammar" xxxx@xxxxxxxxx.se \0\0\0\0\^N\0\0\0;\0\0\0\^S\0\0\0\0Message-ID: 004301c61db5$8b847e40$6400a8c0@LASSE \0\0\0\0\^Q\0\0\0o\0\0\0\^V\0\0\0\0References: 82ab3b60601192318q7c5d7cf1u6ee17edc0e11da92@mail.gmail.com 43D09D62.2080005@abm.uu.se \0\0\0\0\^R\0\0\0)\0\0\0\^X\0\0\0\0Subject: Re: [exakt] SMHI suger! \0\0\0\0\0\0\^S\0\0\0000\0\0\0\^U\0\0\0\0To: "Exakt IT" exakt@lists.frobbit.se \0\0\0\^T\0\0\0\^V\0\0\0\^^\0\0\0\0X-Priority: 3 \0\0\0\0\0 \0\0\0\0\0\0\0

\0\0\0\0\0\0\0\v\0\0\0\0\0\0\0\f\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\^N\0\0\0\0\0\0\0\^O\0\0\0\0\0\0\0\^P\0\0\0\0\0\0\0\^Q\0\0\0\0\0\0\0\^R\0\0\0\0\0\0\0\

\^S\0\0\0\0\0\0\0\^T\0\0\0\0\0\0\0\0\0\0\0\b\0\0\0\b\0\0\0(\0\0\0H\0\0\0\0\0\0\b\M-<\0\0\0\0\0\0\b\M-l\0\0\0\0\0\0\^O\M-~\0\0\0\0\0\0\^PZ\0\0\0\\\0\
    \^A3\M-l\0\0\^B\M-T\0\0\0
    \0\0\0F\0\0\0\^X\0\0\0\0Content-Type: text/plain;
format=flowed; charset="iso-8859-1" \0\0\0\0\0\v\0\0\0.\0\0\0\^V\0\0\0\0Date: Fri, 20 Jan 2006 13:06:34 +0100

... continues like that until ...

9506 imap 1165775763.566267 GIO fd 8 read 8 bytes "\0\0\0\f\0\0\0H" 9506 imap 1165775763.566912 RET pread 4096/0x1000 9506 imap 1165775763.570811 CALL mmap(0,0x144000,0x3,0x1002,0xffffffff,0,0,0) 9506 imap 1165775763.570972 RET mmap 256024576/0xf42a000 9506 imap 1165775763.715347 CALL munmap(0x8481000,0x100000) 9506 imap 1165775763.719472 RET munmap 0 9506 imap 1165775763.757083 CALL munmap(0x843f000,0x42000) 9506 imap 1165775763.758449 RET munmap 0 9506 imap 1165775763.758569 CALL munmap(0x843e000,0x1000) 9506 imap 1165775763.758735 RET munmap 0 9506 imap 1165775763.758854 CALL pread(0x8,0xf56d000,0x1000,0,0,0x142000) 9506 imap 1165775763.759294 GIO fd 8 read 4088 bytes

...

/Peter

-- Peter Lindgren, dataingenjör E-post: peter "at" norrskenkonsult punkt com http://www.norrskenkonsult.com

6560

Age (days ago)

6560

Last active (days ago)

List overview

2 comments

1 participants

participants (1)

Peter Lindgren