Hello
I have some performance problem with
doveadm quota get -u user@domain
after migrating from 2.3.21 to 2.4.1 and enabling fts-flatcurve plugin for mail account with 1330 folders and subfolders.
Quota on 2.3.21 was stored in maildirsize. After upgrading dovecot to 2.4.2 quota driver is set to count
When I run doveadm quota get -u user@domain command it takes about 15 minutes. Mail account quota is used about 6.7GB and about 10k messeges. Doveadm process use 100% CPU when it try to count quota values.
When I run command doveadm with "strace" to trace callbacks it looks like:
14:51:00.577061 newfstatat(AT_FDCWD, "/maildovecot/*******/Maildir/Foldery przechowywania/Zakupy/drogerie/rossmann/fts-flatcurve/index.714", {st_mode=S_IFDIR|0700, st_size=4096, ...}, 0) = 0 14:51:00.577295 openat(AT_FDCWD, "/maildovecot/*******/Maildir/Foldery przechowywania/Zakupy/drogerie/rossmann/fts-flatcurve/index.714", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 13 14:51:00.577680 rmdir("/maildovecot/*******/Maildir/Foldery przechowywania/Zakupy/drogerie/rossmann/fts-flatcurve/index.714") = -1 ENOTEMPTY (Directory not empty) 14:51:00.577765 rmdir("/maildovecot/*******/Maildir/Foldery przechowywania/Zakupy/drogerie/rossmann/fts-flatcurve/index.714") = -1 ENOTEMPTY (Directory not empty) 14:51:00.577847 rmdir("/maildovecot/*******/Maildir/Foldery przechowywania/Zakupy/drogerie/rossmann/fts-flatcurve/index.714") = -1 ENOTEMPTY (Directory not empty) 14:51:00.577929 rmdir("/maildovecot/*******Maildir/Foldery przechowywania/Zakupy/drogerie/rossmann/fts-flatcurve/index.714") = -1 ENOTEMPTY (Directory not empty)
or
14:33:44.529146 newfstatat(AT_FDCWD, "/maildovecot//******//Maildir/Foldery przechowywania/programy lojalnosciowe/vitay/fts-flatcurve/cur", 0x7fff0d9cf270, 0) = -1 ENOENT (No such file or directory)
and it loops 2374 times
grep fts-flatcurve strace_quota | grep rossmann | wc -l 2374
My configuration:
quota "User quota" { driver = count quota_storage_grace = 10M quota_mail_size = 50M quota_message_count = 1500000
quota_warning warn-80-storage {
quota_storage_percentage = 80
quota_warning_resource = storage
quota_warning_threshold = over
execute quota-warning {
args = 80 size %{user} %{userdb:quota_storage_size}
}
}
quota_warning warn-90-storage {
quota_storage_percentage = 90
quota_warning_resource = storage
quota_warning_threshold = over
execute quota-warning {
args = 90 size %{user} %{userdb:quota_storage_size}
}
}
quota_warning warn-90-message {
quota_message_percentage = 95
quota_warning_resource = message
quota_warning_threshold = over
execute quota-warning {
args = 90 inodes
}
}}
service quota-warning { executable = script /dovecot/bin/quota.sh user = mail unix_listener quota-warning { user = mail } }
cat conf.d/90-fts.conf
language_tokenizers = generic email-address language_tokenizer_generic_algorithm = simple language_filters = lowercase snowball stopwords fts_autoindex = yes fts_autoindex_max_recent_msgs = 100 fts_message_max_size = 20M fts_driver = flatcurve fts_decoder_driver = script fts_decoder_script_socket_path = decode2text
service decode2text { executable = script /dovecot/bin/attachment2text.sh unix_listener decode2text { mode = 0666 user = dovecot group = dovecot } }
fts_header_excludes { Authentication-Results = yes Received = yes DKIM-* = yes X-* = yes Comments = yes Date = yes Message-ID = yes Received = yes }
mailbox TRASH { fts_autoindex = no }
mailbox SPAM { fts_autoindex = no }
mailbox confirmed-spam { fts_autoindex = no }
mailbox confirmed-ham { fts_autoindex = no }
language en { default = yes filters = lowercase snowball english-possessive stopwords }
fts flatcurve { commit_limit = 500 min_term_size = 2 optimize_limit = 10 rotate_count = 5000 rotate_time = 5s substring_search = no }
My questions are:
Why "doveadm quota get" tries to open "fts-flatcurve" directories and tries to find "cur" or "ndex.714" directory? Is there any options to exclude all "fts-flatcurve" directories from quota ? Do you observe performance hit for mail accounts with many folders and subfolders after enabling fts-flatcurve?