Dovecot 2.3.8 - How to force index creation for user/mailbox? - dovecot

newer
ssl_min_protocol = TLSv1.3 does...

Dovecot 2.3.8 - How to force index creation for user/mailbox?

older
lmtp proxy 'Invalid FROM: Missing...

Tomek Lutelmowski

3 Dec 2019 3 Dec '19

6:30 p.m.

Hello,

I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

*doveadm -D -vvv index -u user 'Folder' *

But even if this folder have no dovecot.index* files - I always get:

doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.

So i tried:

*doveadm mailbox cache purge -u user 'Folder' *

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index:

*doveadm -vvv force-resync -u user 'Folder'*

Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?

Cheers, TomekL

Attachments:

attachment.html (text/html — 1.3 KB)

Show replies by date

Aki Tuomi

3 Dec 3 Dec

7:15 p.m.

...

On 03/12/2019 18:30 Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Hello,

I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

doveadm -D -vvv index -u user 'Folder'

But even if this folder have no dovecot.index* files - I always get:

doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.

So i tried: doveadm mailbox cache purge -u user 'Folder'

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'

Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?

Cheers, TomekL

Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.

Aki

Tomek Lutelmowski

9:20 p.m.

Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.

wt., 3 gru 2019 o 18:16 Aki Tuomi aki.tuomi@open-xchange.com napisał(a):

...

...
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Hello,

I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

doveadm -D -vvv index -u user 'Folder'

But even if this folder have no dovecot.index* files - I always get:

doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.

So i tried: doveadm mailbox cache purge -u user 'Folder'

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'

Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?

Cheers, TomekL

Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.

Aki

Aki Tuomi

9:23 p.m.

Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.

Aki

...

On 03/12/2019 21:20 Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.

wt., 3 gru 2019 o 18:16 Aki Tuomi aki.tuomi@open-xchange.com napisał(a):

...
...
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Hello,

I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

doveadm -D -vvv index -u user 'Folder'

But even if this folder have no dovecot.index* files - I always get:

doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.

So i tried: doveadm mailbox cache purge -u user 'Folder'

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'

Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?

Cheers, TomekL

Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.

Aki

Tomek Lutelmowski

9:31 p.m.

Neither of dovecot.index files are created (.cache, .log, .thread) - these files are created when I open the folder with IMAP client. When mailbox is not used for long time, the indexing takes very long time. So I'm looking for any solution (preferably with doveadm) to manually reindex folders in background.

wt., 3 gru 2019 o 20:23 Aki Tuomi aki.tuomi@open-xchange.com napisał(a):

...

Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.

Aki

...
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.

wt., 3 gru 2019 o 18:16 Aki Tuomi aki.tuomi@open-xchange.com napisał(a):

...
...
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <

...
...
Hello,

I'm using maildirs, and I have problems with big mailboxes (~ 20000

emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

...
doveadm -D -vvv index -u user 'Folder'

But even if this folder have no dovecot.index* files - I always get:

doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is

...
...
So i tried: doveadm mailbox cache purge -u user 'Folder'

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'

Is there any way to use doveadm tool to create/fix index? Shouldn't

...
...
Cheers, TomekL

Try first 'doveadm fetch -u victim body ALL'. If this does not

dovecot@dovecot.org> wrote: present. The index is created only when I log with IMAP client and check the folder. the index be updated on each email arrival? produce any useful results, try 'doveadm force-resync -u victim "*"'.

...
Aki

Sami Ketola

4 Dec 4 Dec

8:34 a.m.

Hi,

then please post doveconf -n

Sami

...

On 3 Dec 2019, at 21.31, Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Neither of dovecot.index files are created (.cache, .log, .thread) - these files are created when I open the folder with IMAP client. When mailbox is not used for long time, the indexing takes very long time. So I'm looking for any solution (preferably with doveadm) to manually reindex folders in background.

wt., 3 gru 2019 o 20:23 Aki Tuomi mailto:aki.tuomi@open-xchange.com> napisał(a): Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.

Aki

...
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot mailto:dovecot@dovecot.org> wrote:

Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.

wt., 3 gru 2019 o 18:16 Aki Tuomi mailto:aki.tuomi@open-xchange.com> napisał(a):

...
...
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot mailto:dovecot@dovecot.org> wrote:

Hello,

I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

doveadm -D -vvv index -u user 'Folder'

But even if this folder have no dovecot.index* files - I always get:

doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.

So i tried: doveadm mailbox cache purge -u user 'Folder'

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'

Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?

Cheers, TomekL

Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.

Aki

Tomek Lutelmowski

11:48 a.m.

Here is doveconf -n output:

# 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 5.4.1-gentoo x86_64 Gentoo Base System release 2.6 # Hostname: mail.xxxx.com auth_cache_negative_ttl = 0 auth_cache_size = 10 M auth_cache_ttl = 4 hours auth_default_realm = xxxx.com auth_master_user_separator = * auth_verbose = yes auth_verbose_passwords = plain default_vsz_limit = 1 G disable_plaintext_auth = no listen = * login_greeting = IMAP ready. mail_gid = mail mail_location = maildir:~ mail_plugins = zlib fts fts_lucene virtual acl mailbox_idle_check_interval = 3 secs managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { list = children location = maildir:/e/xxxx.com/mail-shared:INDEXPVT=~/Maildir/public/ mailbox poczta { auto = no } prefix = Wspólne/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes list = yes location = mailbox "Elementy usunięte" { auto = subscribe special_use = \Trash } mailbox "Elementy wysłane" { auto = subscribe special_use = \Sent } mailbox "Wersje robocze" { auto = subscribe special_use = \Drafts } mailbox Wiadomości-śmieci { auto = subscribe special_use = \Junk } prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-sql-master.conf.ext driver = sql master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars="@.-_()[]{}<>/\\+" mail_log_fields = uid box msgid from subject size sieve = file:~/sieve;active=~/sieve/sieve.active sieve_before = /etc/dovecot/sieve/global.sieve sieve_global = /etc/dovecot/sieve sieve_vacation_dont_check_recipient = yes sieve_vacation_use_original_recipient = yes zlib_save = xz zlib_save_level = 9 } pop3_uidl_format = %v.%u protocols = imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { process_min_avail = 10 service_count = 1 unix_listener imap { mode = 0666 } } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = $default_internal_user } service imap { client_limit = 1 executable = imap imap-postlogin } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } vsz_limit = 4 G } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } ssl_ca =

śr., 4 gru 2019 o 07:34 Sami Ketola sami.ketola@dovecot.fi napisał(a):

...

Hi,

then please post doveconf -n

Sami

On 3 Dec 2019, at 21.31, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:

Neither of dovecot.index files are created (.cache, .log, .thread) - these files are created when I open the folder with IMAP client. When mailbox is not used for long time, the indexing takes very long time. So I'm looking for any solution (preferably with doveadm) to manually reindex folders in background.

wt., 3 gru 2019 o 20:23 Aki Tuomi aki.tuomi@open-xchange.com napisał(a):

...
Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.

Aki

...
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.

wt., 3 gru 2019 o 18:16 Aki Tuomi aki.tuomi@open-xchange.com napisał(a):

...
...
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <

...
...
Hello,

I'm using maildirs, and I have problems with big mailboxes (~

20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:

...
doveadm -D -vvv index -u user 'Folder'

But even if this folder have no dovecot.index* files - I always

get:

...
doveadm(user): Info: Wersje robocze: Cache is already up to date

It seems that doveadm doesn't actually checks if the index is

...
...
So i tried: doveadm mailbox cache purge -u user 'Folder'

No joy either: doveadm(user): Error: Cache is unusable

And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'

Is there any way to use doveadm tool to create/fix index?

Shouldn't the index be updated on each email arrival?

...
Cheers, TomekL

Try first 'doveadm fetch -u victim body ALL'. If this does not

dovecot@dovecot.org> wrote: present. The index is created only when I log with IMAP client and check the folder. produce any useful results, try 'doveadm force-resync -u victim "*"'.

...
Aki

Sami Ketola

4:45 p.m.

...

On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

mail_location = maildir:~

can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.

Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/

Sami

Tomek Lutelmowski

5:15 p.m.

Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?

śr., 4 gru 2019 o 15:45 Sami Ketola sami.ketola@dovecot.fi napisał(a):

...

...
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:

mail_location = maildir:~

can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.

Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/

Sami

Sami Ketola

5:31 p.m.

...

On 4 Dec 2019, at 17.15, Tomek Lutelmowski via dovecot dovecot@dovecot.org wrote:

Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?

[root@example .INBOX]# pwd /vmail/sami@example.com/index/.INBOX [root@example .INBOX]# ls dovecot.index dovecot.index.cache dovecot.index.log [root@example .INBOX]# rm dovecot.index.log rm: remove regular file ‘dovecot.index.log’? y [root@example .INBOX]# doveadm force-resync -u sami INBOX [root@example .INBOX]# ls -al total 8228 drwx------ 2 vmail vmail 76 Dec 4 15:22 . drwx------ 13 vmail vmail 4096 Nov 25 08:58 .. -rw------- 1 vmail vmail 164080 Dec 4 15:22 dovecot.index -rw------- 1 vmail vmail 8247872 Dec 4 15:21 dovecot.index.cache -rw------- 1 vmail vmail 252 Dec 4 15:22 dovecot.index.log

[root@example .INBOX]# rm dovecot.index.* rm: remove regular file ‘dovecot.index.cache’? y rm: remove regular file ‘dovecot.index.log’? y

[root@example .INBOX]# doveadm index -u sami INBOX [root@example .INBOX]# doveadm force-resync -u sami INBOX

[root@example .INBOX]# ls -al total 180 drwx------ 2 vmail vmail 76 Dec 4 15:27 . drwx------ 13 vmail vmail 4096 Nov 25 08:58 .. -rw------- 1 vmail vmail 164080 Dec 4 15:27 dovecot.index -rw------- 1 vmail vmail 5748 Dec 4 15:27 dovecot.index.cache -rw------- 1 vmail vmail 308 Dec 4 15:27 dovecot.index.log

[root@example .INBOX]# doveconf -n | grep mail_location mail_location = Maildir:/vmail/%u/Maildir:INDEX=/vmail/%u/index

[root@ketola .INBOX]# doveadm user sami | grep home home /vmail/sami@example.com

Sami

Dmitry Donskih

5 Dec 5 Dec

3:53 p.m.

Yes.

mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir:INBOX=/var/vmail/%Ld/%Ln/Maildir/.INBOX:INDEX=/var-fast/dovecot/indexes/vmail/%Ld/%Ln

As Sami Ketola wrote earlier:

rm -f /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*

doveadm index -u user@domain.ru INBOX

doveadm force-resync-u user@domain.ru INBOX

ls /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*

/var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.cache /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.log

04.12.2019 18:15, Tomek Lutelmowski via dovecot wrote:

...

Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?

śr., 4 gru 2019 o 15:45 Sami Ketola mailto:sami.ketola@dovecot.fi> napisał(a):
> On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot
<dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
>
> mail_location = maildir:~
>

can't spot anything obvious wrong. Just spotted that you are
storing mails in users home root along with all other files such
as index files.

Maybe you should store the mailbox contents into Maildir:~/Maildir
under the users home and index to ~/index/

Sami

Tomek Lutelmowski

10:07 p.m.

Gents, thank you all - moving indexes to external folder worked like a charm! :-)

czw., 5 gru 2019 o 14:55 Dmitry Donskih via dovecot dovecot@dovecot.org napisał(a):

...

Yes.

mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir:INBOX=/var/vmail/%Ld/%Ln/Maildir/.INBOX:INDEX=/var-fast/dovecot/indexes/vmail/%Ld/%Ln

As Sami Ketola wrote earlier:

rm -f /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*

doveadm index -u user@domain.ru INBOX

doveadm force-resync -u user@domain.ru INBOX ls /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*

/var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.cache /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.log

04.12.2019 18:15, Tomek Lutelmowski via dovecot wrote:

Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?

śr., 4 gru 2019 o 15:45 Sami Ketola sami.ketola@dovecot.fi napisał(a):

...
...
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:

mail_location = maildir:~

can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.

Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/

Sami

Durga Prasad Malyala

8 Dec 8 Dec

12:17 p.m.

Just curious how does moving indexes to different folder help.?

Cheers/DP

On Fri, Dec 6, 2019, 01:38 Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:

...

Gents, thank you all - moving indexes to external folder worked like a charm! :-)

czw., 5 gru 2019 o 14:55 Dmitry Donskih via dovecot dovecot@dovecot.org napisał(a):

...
Yes.

mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir:INBOX=/var/vmail/%Ld/%Ln/Maildir/.INBOX:INDEX=/var-fast/dovecot/indexes/vmail/%Ld/%Ln

As Sami Ketola wrote earlier:

rm -f /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*

doveadm index -u user@domain.ru INBOX

doveadm force-resync -u user@domain.ru INBOX ls /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*

/var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.cache /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.log

04.12.2019 18:15, Tomek Lutelmowski via dovecot wrote:

Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?

śr., 4 gru 2019 o 15:45 Sami Ketola sami.ketola@dovecot.fi napisał(a):

...
...
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:

mail_location = maildir:~

can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.

Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/

Sami

1805

Age (days ago)

1810

Last active (days ago)

List overview

12 comments

5 participants

participants (5)

Aki Tuomi
Dmitry Donskih
Durga Prasad Malyala
Sami Ketola
Tomek Lutelmowski