Dovecot 2.3.8 - How to force index creation for user/mailbox?
Hello,
I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
*doveadm -D -vvv index -u user 'Folder' *
But even if this folder have no dovecot.index* files - I always get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.
So i tried:
*doveadm mailbox cache purge -u user 'Folder' *
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index:
*doveadm -vvv force-resync -u user 'Folder'*
Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?
Cheers, TomekL
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Hello,
I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
doveadm -D -vvv index -u user 'Folder'
But even if this folder have no dovecot.index* files - I always get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.
So i tried: doveadm mailbox cache purge -u user 'Folder'
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'
Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?
Cheers, TomekL
Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.
Aki
Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.
wt., 3 gru 2019 o 18:16 Aki Tuomi <aki.tuomi@open-xchange.com> napisał(a):
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Hello,
I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
doveadm -D -vvv index -u user 'Folder'
But even if this folder have no dovecot.index* files - I always get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.
So i tried: doveadm mailbox cache purge -u user 'Folder'
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'
Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?
Cheers, TomekL
Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.
Aki
Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.
Aki
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.
wt., 3 gru 2019 o 18:16 Aki Tuomi <aki.tuomi@open-xchange.com> napisał(a):
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Hello,
I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
doveadm -D -vvv index -u user 'Folder'
But even if this folder have no dovecot.index* files - I always get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.
So i tried: doveadm mailbox cache purge -u user 'Folder'
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'
Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?
Cheers, TomekL
Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.
Aki
Neither of dovecot.index files are created (.cache, .log, .thread) - these files are created when I open the folder with IMAP client. When mailbox is not used for long time, the indexing takes very long time. So I'm looking for any solution (preferably with doveadm) to manually reindex folders in background.
wt., 3 gru 2019 o 20:23 Aki Tuomi <aki.tuomi@open-xchange.com> napisał(a):
Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.
Aki
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.
wt., 3 gru 2019 o 18:16 Aki Tuomi <aki.tuomi@open-xchange.com> napisał(a):
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <
Hello,
I'm using maildirs, and I have problems with big mailboxes (~ 20000
emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
doveadm -D -vvv index -u user 'Folder'
But even if this folder have no dovecot.index* files - I always get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is
So i tried: doveadm mailbox cache purge -u user 'Folder'
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'
Is there any way to use doveadm tool to create/fix index? Shouldn't
Cheers, TomekL
Try first 'doveadm fetch -u victim body ALL'. If this does not
dovecot@dovecot.org> wrote: present. The index is created only when I log with IMAP client and check the folder. the index be updated on each email arrival? produce any useful results, try 'doveadm force-resync -u victim "*"'.
Aki
Hi,
then please post doveconf -n
Sami
On 3 Dec 2019, at 21.31, Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Neither of dovecot.index files are created (.cache, .log, .thread) - these files are created when I open the folder with IMAP client. When mailbox is not used for long time, the indexing takes very long time. So I'm looking for any solution (preferably with doveadm) to manually reindex folders in background.
wt., 3 gru 2019 o 20:23 Aki Tuomi <aki.tuomi@open-xchange.com <mailto:aki.tuomi@open-xchange.com>> napisał(a): Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.
Aki
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.
wt., 3 gru 2019 o 18:16 Aki Tuomi <aki.tuomi@open-xchange.com <mailto:aki.tuomi@open-xchange.com>> napisał(a):
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote:
Hello,
I'm using maildirs, and I have problems with big mailboxes (~ 20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
doveadm -D -vvv index -u user 'Folder'
But even if this folder have no dovecot.index* files - I always get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is present. The index is created only when I log with IMAP client and check the folder.
So i tried: doveadm mailbox cache purge -u user 'Folder'
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'
Is there any way to use doveadm tool to create/fix index? Shouldn't the index be updated on each email arrival?
Cheers, TomekL
Try first 'doveadm fetch -u victim body ALL'. If this does not produce any useful results, try 'doveadm force-resync -u victim "*"'.
Aki
Here is doveconf -n output:
# 2.3.8 (9df20d2db): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.8 (b7b03ba2) # OS: Linux 5.4.1-gentoo x86_64 Gentoo Base System release 2.6 # Hostname: mail.xxxx.com auth_cache_negative_ttl = 0 auth_cache_size = 10 M auth_cache_ttl = 4 hours auth_default_realm = xxxx.com auth_master_user_separator = * auth_verbose = yes auth_verbose_passwords = plain default_vsz_limit = 1 G disable_plaintext_auth = no listen = * login_greeting = IMAP ready. mail_gid = mail mail_location = maildir:~ mail_plugins = zlib fts fts_lucene virtual acl mailbox_idle_check_interval = 3 secs managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { list = children location = maildir:/e/xxxx.com/mail-shared:INDEXPVT=~/Maildir/public/ mailbox poczta { auto = no } prefix = Wspólne/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes list = yes location = mailbox "Elementy usunięte" { auto = subscribe special_use = \Trash } mailbox "Elementy wysłane" { auto = subscribe special_use = \Sent } mailbox "Wersje robocze" { auto = subscribe special_use = \Drafts } mailbox Wiadomości-śmieci { auto = subscribe special_use = \Junk } prefix = separator = / type = private } passdb { args = /etc/dovecot/dovecot-sql-master.conf.ext driver = sql master = yes pass = yes } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile fts = lucene fts_autoindex = yes fts_lucene = whitespace_chars="@.-_()[]{}<>/\\+" mail_log_fields = uid box msgid from subject size sieve = file:~/sieve;active=~/sieve/sieve.active sieve_before = /etc/dovecot/sieve/global.sieve sieve_global = /etc/dovecot/sieve sieve_vacation_dont_check_recipient = yes sieve_vacation_use_original_recipient = yes zlib_save = xz zlib_save_level = 9 } pop3_uidl_format = %v.%u protocols = imap lmtp sieve service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } } service imap-login { process_min_avail = 10 service_count = 1 unix_listener imap { mode = 0666 } } service imap-postlogin { executable = script-login /usr/local/bin/postlogin.sh user = $default_internal_user } service imap { client_limit = 1 executable = imap imap-postlogin } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } vsz_limit = 4 G } service managesieve-login { inet_listener sieve { address = 127.0.0.1 port = 4190 } } ssl_ca = </e/xxxx.com/cert/CA.pem ssl_cert = </e/ xxxx.com/cert/mail-chain.crt ssl_cipher_list = ALL:!LOW:!SSLv2:!EXP:!aNULL ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_require_crl = no userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lmtp { mail_plugins = zlib fts fts_lucene virtual acl sieve virtual } protocol lda { mail_plugins = zlib fts fts_lucene virtual acl sieve } protocol imap { mail_plugins = zlib fts fts_lucene virtual acl imap_acl }
śr., 4 gru 2019 o 07:34 Sami Ketola <sami.ketola@dovecot.fi> napisał(a):
Hi,
then please post doveconf -n
Sami
On 3 Dec 2019, at 21.31, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:
Neither of dovecot.index files are created (.cache, .log, .thread) - these files are created when I open the folder with IMAP client. When mailbox is not used for long time, the indexing takes very long time. So I'm looking for any solution (preferably with doveadm) to manually reindex folders in background.
wt., 3 gru 2019 o 20:23 Aki Tuomi <aki.tuomi@open-xchange.com> napisał(a):
Wait, did you mean 'dovecot.index' file? That file is only created occasionally, the actual index is kept in dovecot.index.log.
Aki
On 03/12/2019 21:20 Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Thanks, but both commands didn't create dovecot.index. Only when I open folder with IMAP client, the dovecot.index is created.
wt., 3 gru 2019 o 18:16 Aki Tuomi <aki.tuomi@open-xchange.com> napisał(a):
On 03/12/2019 18:30 Tomek Lutelmowski via dovecot <
Hello,
I'm using maildirs, and I have problems with big mailboxes (~
20000 emails, 30-50GB) - sometimes access to such mailbox takes very long (~10 minutes), I would like to be able to create index for mailbox in background, but I couldn't find any way to do it. So far I tried:
doveadm -D -vvv index -u user 'Folder'
But even if this folder have no dovecot.index* files - I always
get:
doveadm(user): Info: Wersje robocze: Cache is already up to date
It seems that doveadm doesn't actually checks if the index is
So i tried: doveadm mailbox cache purge -u user 'Folder'
No joy either: doveadm(user): Error: Cache is unusable
And command below doesn't actually do anything to index: doveadm -vvv force-resync -u user 'Folder'
Is there any way to use doveadm tool to create/fix index?
Shouldn't the index be updated on each email arrival?
Cheers, TomekL
Try first 'doveadm fetch -u victim body ALL'. If this does not
dovecot@dovecot.org> wrote: present. The index is created only when I log with IMAP client and check the folder. produce any useful results, try 'doveadm force-resync -u victim "*"'.
Aki
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
mail_location = maildir:~
can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.
Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/
Sami
Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?
śr., 4 gru 2019 o 15:45 Sami Ketola <sami.ketola@dovecot.fi> napisał(a):
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:
mail_location = maildir:~
can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.
Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/
Sami
On 4 Dec 2019, at 17.15, Tomek Lutelmowski via dovecot <dovecot@dovecot.org> wrote:
Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?
[root@example .INBOX]# pwd /vmail/sami@example.com/index/.INBOX [root@example .INBOX]# ls dovecot.index dovecot.index.cache dovecot.index.log [root@example .INBOX]# rm dovecot.index.log rm: remove regular file ‘dovecot.index.log’? y [root@example .INBOX]# doveadm force-resync -u sami INBOX [root@example .INBOX]# ls -al total 8228 drwx------ 2 vmail vmail 76 Dec 4 15:22 . drwx------ 13 vmail vmail 4096 Nov 25 08:58 .. -rw------- 1 vmail vmail 164080 Dec 4 15:22 dovecot.index -rw------- 1 vmail vmail 8247872 Dec 4 15:21 dovecot.index.cache -rw------- 1 vmail vmail 252 Dec 4 15:22 dovecot.index.log
[root@example .INBOX]# rm dovecot.index.* rm: remove regular file ‘dovecot.index.cache’? y rm: remove regular file ‘dovecot.index.log’? y
[root@example .INBOX]# doveadm index -u sami INBOX [root@example .INBOX]# doveadm force-resync -u sami INBOX
[root@example .INBOX]# ls -al total 180 drwx------ 2 vmail vmail 76 Dec 4 15:27 . drwx------ 13 vmail vmail 4096 Nov 25 08:58 .. -rw------- 1 vmail vmail 164080 Dec 4 15:27 dovecot.index -rw------- 1 vmail vmail 5748 Dec 4 15:27 dovecot.index.cache -rw------- 1 vmail vmail 308 Dec 4 15:27 dovecot.index.log
[root@example .INBOX]# doveconf -n | grep mail_location mail_location = Maildir:/vmail/%u/Maildir:INDEX=/vmail/%u/index
[root@ketola .INBOX]# doveadm user sami | grep home home /vmail/sami@example.com
Sami
Yes.
mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir:INBOX=/var/vmail/%Ld/%Ln/Maildir/.INBOX:INDEX=/var-fast/dovecot/indexes/vmail/%Ld/%Ln
As Sami Ketola wrote earlier:
rm -f /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*
doveadm index -u user@domain.ru INBOX
doveadm force-resync-u user@domain.ru INBOX
ls /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*
/var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.cache /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.log
04.12.2019 18:15, Tomek Lutelmowski via dovecot wrote:
Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?
śr., 4 gru 2019 o 15:45 Sami Ketola <sami.ketola@dovecot.fi <mailto:sami.ketola@dovecot.fi>> napisał(a):
> On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot <dovecot@dovecot.org <mailto:dovecot@dovecot.org>> wrote: > > mail_location = maildir:~ > can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files. Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/ Sami
Gents, thank you all - moving indexes to external folder worked like a charm! :-)
czw., 5 gru 2019 o 14:55 Dmitry Donskih via dovecot <dovecot@dovecot.org> napisał(a):
Yes.
mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir:INBOX=/var/vmail/%Ld/%Ln/Maildir/.INBOX:INDEX=/var-fast/dovecot/indexes/vmail/%Ld/%Ln
As Sami Ketola wrote earlier:
rm -f /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*
doveadm index -u user@domain.ru INBOX
doveadm force-resync -u user@domain.ru INBOX ls /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*
/var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.cache /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.log
04.12.2019 18:15, Tomek Lutelmowski via dovecot wrote:
Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?
śr., 4 gru 2019 o 15:45 Sami Ketola <sami.ketola@dovecot.fi> napisał(a):
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:
mail_location = maildir:~
can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.
Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/
Sami
Just curious how does moving indexes to different folder help.?
Cheers/DP
On Fri, Dec 6, 2019, 01:38 Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:
Gents, thank you all - moving indexes to external folder worked like a charm! :-)
czw., 5 gru 2019 o 14:55 Dmitry Donskih via dovecot <dovecot@dovecot.org> napisał(a):
Yes.
mail_location = maildir:/var/vmail/%Ld/%Ln/Maildir:INBOX=/var/vmail/%Ld/%Ln/Maildir/.INBOX:INDEX=/var-fast/dovecot/indexes/vmail/%Ld/%Ln
As Sami Ketola wrote earlier:
rm -f /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*
doveadm index -u user@domain.ru INBOX
doveadm force-resync -u user@domain.ru INBOX ls /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/*
/var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.cache /var-fast/dovecot/indexes/vmail/domain.ru/user/.INBOX/dovecot.index.log
04.12.2019 18:15, Tomek Lutelmowski via dovecot wrote:
Someone who have mailbox in different directory than indexes, can confirm if doveadm can recreate index (after deleting dovecot.index* files) or these files can be created only by connecting with IMAP client?
śr., 4 gru 2019 o 15:45 Sami Ketola <sami.ketola@dovecot.fi> napisał(a):
On 4 Dec 2019, at 11.48, Tomek Lutelmowski via dovecot < dovecot@dovecot.org> wrote:
mail_location = maildir:~
can't spot anything obvious wrong. Just spotted that you are storing mails in users home root along with all other files such as index files.
Maybe you should store the mailbox contents into Maildir:~/Maildir under the users home and index to ~/index/
Sami
participants (5)
-
Aki Tuomi
-
Dmitry Donskih
-
Durga Prasad Malyala
-
Sami Ketola
-
Tomek Lutelmowski