[SOLVED] after replication with compression quotas are wrong
Arnaud Abélard
arnaud.abelard at univ-nantes.fr
Fri Nov 12 20:18:09 UTC 2021
Ah! At last, after comparing all default values using doveconf I found
the culprit: mailbox_list_index. Default is "no" on dovecot 2.2.7, but
"yes" on dovecot 2.3.4 (on debian, at least). Switching
mailbox_list_index to "no" fixes my quota miscalculation problem.
Arnaud
On 12/11/2021 20:09, Arnaud Abélard wrote:
>
>
> On 10/11/2021 15:52, Arnaud Abélard wrote:
>> I've just removed dummy-c-1's mailbox completely on the target server,
>> removed the quota plugin, forced a sync for dummy-c-1 only and once it
>> was done reactivated the quota plugins: double usage again.
>>
>> I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server
>> in order to have both server running the same version, without
>> improvement.
>
> Actually, I got that wrong, test server is running dovecot 2.3.4 and
> production server dovecot 2.2.7. I copied the dovecot.conf file from the
> production server to the test one, then rsync'ed the dummy-c-1 mailbox
> from the production server to the test server, restarted dovecot and the
> quota is still 115% after recalc. The only difference now is dovecot's
> version. Were there any changes related to quota on dovecot 2.3?
>
> Thanks,
>
> Arnaud
>
>>
>> I'm puzzled. Is dovecot storing anything outside of the user's
>> mailbox? Like a cache, a sqlite database of some kind somewhere?
>>
>> Arnaud
>>
>> On 08/11/2021 11:48, Arnaud Abélard wrote:
>>> On the target, I enabled the replication service without mail_replica
>>> and:
>>>
>>> doveadm replicator status 'dummy-c-1*'
>>> username priority fast sync full sync success sync failed
>>> dummy-c-1 none - - - -
>>>
>>> It only knows of dummy-c-1, no trace of his evil twin
>>> dummy-c-1 at univ-nantes.fr.
>>>
>>> On the target, I do have the same number of files
>>>
>>> find . -type f |wc -l
>>> 8705
>>>
>>> which is around half of what the quota is reporting (plus de index
>>> files, etc):
>>>
>>> ~# doveadm -D quota get -u dummy-c-1
>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules
>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
>>> Debug: Module loaded:
>>> /usr/lib/dovecot/modules/lib20_replication_plugin.so
>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
>>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so:
>>> undefined symbol: acl_user_module (this is usually intentional, so
>>> just ignore this message)
>>> Debug: Skipping module doveadm_expire_plugin, because dlopen()
>>> failed:
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
>>> undefined symbol: expire_set_deinit (this is usually intentional, so
>>> just ignore this message)
>>> Debug: Module loaded:
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
>>> Debug: Module loaded:
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
>>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen()
>>> failed:
>>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so:
>>> undefined symbol: lucene_index_iter_deinit (this is usually
>>> intentional, so just ignore this message)
>>> Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
>>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
>>> undefined symbol: fts_user_get_language_list (this is usually
>>> intentional, so just ignore this message)
>>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
>>> failed:
>>> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so:
>>> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
>>> intentional, so just ignore this message)
>>> doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1
>>> home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S
>>> doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting:
>>> plugin/quota_rule=*:backend=1000000000S
>>> doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000,
>>> home=/vmail/d/u/dummy-c-1/
>>> doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur
>>> backend=maildir args=
>>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur
>>> mailbox=* bytes=1000000000 messages=0
>>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur
>>> mailbox=INBOX.Trash bytes=+104857600 messages=0
>>> doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur
>>> bytes=100000000 (10%)
>>> doveadm(dummy-c-1): Debug: replication: No mail_replica setting -
>>> replication disabled
>>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX.,
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>>> location=maildir:/vmail/d/u/dummy-c-1/
>>> doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1,
>>> index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt=
>>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=,
>>> inbox=no, hidden=yes, list=no, subscriptions=no
>>> location=fail::LAYOUT=none
>>> doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=,
>>> inbox=, alt=
>>> doveadm(dummy-c-1): Debug: quota: quota_over_flag check:
>>> quota_over_script unset - skipping
>>> Quota name Type Value Limit
>>> %
>>> Quota Utilisateur STORAGE 1126751 976563
>>> 115
>>> Quota Utilisateur MESSAGE 16686 -
>>> 0
>>>
>>> The debug mode isn't much of any help here.
>>>
>>> Arnaud
>>>
>>> On 08/11/2021 11:30, Aki Tuomi wrote:
>>>> Recalculation won't fix replication mistakes.
>>>>
>>>> Did you ensure on the *target* server that it has only dummy-c-1 in
>>>> replication and that the file count on the target server matches
>>>> source server?
>>>>
>>>> You might also get something useful out of
>>>>
>>>> `doveadm -D quota recalc|get -u user`
>>>>
>>>> Aki
>>>>
>>>>> On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard at univ-nantes.fr>
>>>>> wrote:
>>>>>
>>>>> On 08/11/2021 10:39, Aki Tuomi wrote:
>>>>>> Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>>>>>
>>>>> I did so on the source server, checked indeed that
>>>>> dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator
>>>>> status list and it was indeed the case, so far so good. On the
>>>>> destination server, dummy-c-1's mailbox was still showing a 115% quota
>>>>> so I tried to make dovecot recalculate the quota but it did not change
>>>>> anything. I ended up deleting the user's mailbox and forced a
>>>>> replication and, to my surprise, the new mailbox still has a 115%
>>>>> quota.
>>>>>
>>>>> Arnaud
>>>>>
>>>>>>
>>>>>> Aki
>>>>>>
>>>>>>> On 08/11/2021 11:32 Arnaud Abélard
>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>
>>>>>>> Hello Aki,
>>>>>>>
>>>>>>> Thanks for the tip. I changed postfix configuration so it will
>>>>>>> not use
>>>>>>> the domain part anymore and added "auth_username_format = %Ln"
>>>>>>> just in
>>>>>>> case (and reloaded dovecot). So I shouldn't have any more new
>>>>>>> user at domain users but for all the existing users, doveadm replicator
>>>>>>> status still show user at domain clone and on the new server mails are
>>>>>>> still counted twice.
>>>>>>>
>>>>>>> Our userdb and passdb don't have the @domain defined anywhere so
>>>>>>> I don't
>>>>>>> think it's coming from there.
>>>>>>>
>>>>>>> Is there a local cache that still has a list of those user at domain
>>>>>>> usernames that I could reset or something?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Arnaud
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/11/2021 09:48, Aki Tuomi wrote:
>>>>>>>> This sounds like you are not normalizing usernames properly.
>>>>>>>>
>>>>>>>> Either use
>>>>>>>>
>>>>>>>> auth_username_format = %Ln
>>>>>>>>
>>>>>>>> as global setting, or return `user` attribute in both userdb and
>>>>>>>> passdb lookups.
>>>>>>>>
>>>>>>>> Aki
>>>>>>>>
>>>>>>>>> On 07/11/2021 20:31 Arnaud Abélard
>>>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>>>
>>>>>>>>> Hello again,
>>>>>>>>>
>>>>>>>>> I've found out that some mailboxes are actually duplicated.
>>>>>>>>> Doveadm
>>>>>>>>> replicator status on the production server gives this:
>>>>>>>>>
>>>>>>>>> ~# doveadm replicator status 'dummy-c-1*'
>>>>>>>>> username priority fast sync full sync
>>>>>>>>> success sync failed
>>>>>>>>> dummy-c-1 none 01:13:19 01:13:19
>>>>>>>>> 01:13:19 -
>>>>>>>>> dummy-c-1 at univ-nantes.fr none 00:15:28 00:15:28
>>>>>>>>> 00:15:28 -
>>>>>>>>>
>>>>>>>>> That'd explain why mails are counted twice when replicated on
>>>>>>>>> the new
>>>>>>>>> server but where does this come from since I don't have this quota
>>>>>>>>> problem on the production server?
>>>>>>>>>
>>>>>>>>> From the logs, it seems that postfix uses
>>>>>>>>> username at univ-nantes.fr when
>>>>>>>>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
>>>>>>>>> mailboxes are the erroneous ones.
>>>>>>>>>
>>>>>>>>> The users aren't duplicated in our ldap user backend and aren't
>>>>>>>>> using
>>>>>>>>> the @univ-nantes.fr part.
>>>>>>>>>
>>>>>>>>> ~# doveadm user 'dummy-c-1*'
>>>>>>>>> dummy-c-1
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Arnaud
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/11/2021 16:21, Arnaud Abélard wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> We are very long time happy dovecot users (since 2008 at
>>>>>>>>>> least). We have
>>>>>>>>>> around 90k mailboxes and since we had to move away from our
>>>>>>>>>> NAS storage
>>>>>>>>>> to a ceph storage I jumped on the opportunity to enable
>>>>>>>>>> compression with
>>>>>>>>>> the zlib plugin and dovecot's replication mecanism. We are using
>>>>>>>>>> debian's dovecot 2.2.27 packages on production and our new
>>>>>>>>>> server is
>>>>>>>>>> running dovecot's own ce-2.3.17 packages.
>>>>>>>>>>
>>>>>>>>>> On the production server everything works fine but on the new
>>>>>>>>>> server,
>>>>>>>>>> replicated mailboxes' quota is all wrong:
>>>>>>>>>>
>>>>>>>>>> on production:
>>>>>>>>>> # doveadm quota get -u dummy-c-1
>>>>>>>>>> Quota name Type Value
>>>>>>>>>> Limit %
>>>>>>>>>> Quota Utilisateur STORAGE 660026
>>>>>>>>>> 976563 67
>>>>>>>>>> Quota Utilisateur MESSAGE 8651 - 0
>>>>>>>>>>
>>>>>>>>>> on new server:
>>>>>>>>>> doveadm quota get -u dummy-c-1
>>>>>>>>>> Quota name Type Value
>>>>>>>>>> Limit %
>>>>>>>>>> Quota Utilisateur STORAGE 1125251 976563 115
>>>>>>>>>> Quota Utilisateur MESSAGE 16646 - 0
>>>>>>>>>>
>>>>>>>>>> If I add all the S flag from the filenames n both servers I
>>>>>>>>>> get exactly
>>>>>>>>>> the same usage, which is coherent with the quota on the
>>>>>>>>>> production server:
>>>>>>>>>>
>>>>>>>>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk
>>>>>>>>>> -F','
>>>>>>>>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
>>>>>>>>>> 675865938
>>>>>>>>>>
>>>>>>>>>> And I have exactly the same amountof mails on the two server, the
>>>>>>>>>> replication works as expected, no unwanted duplication of
>>>>>>>>>> mails occurs.
>>>>>>>>>>
>>>>>>>>>> Of course, I've tried to ask dovecot to recalculate quotas
>>>>>>>>>> with doveadm
>>>>>>>>>> quota recalc -u <username>, but it doesn't fix the problem.
>>>>>>>>>>
>>>>>>>>>> What am I missing?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Arnaud
>>>>>>>>>>
>>>>>>>>>> PS: Here is my doveconf -n output:
>>>>>>>>>>
>>>>>>>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
>>>>>>>>>> # Pigeonhole version 0.5.17 (054dddfa)
>>>>>>>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
>>>>>>>>>> # Hostname: gromel-test
>>>>>>>>>> auth_cache_size = 10 k
>>>>>>>>>> auth_verbose = yes
>>>>>>>>>> disable_plaintext_auth = no
>>>>>>>>>> doveadm_password = # hidden, use -P to show it
>>>>>>>>>> hostname = gromel1.univ-nantes.prive
>>>>>>>>>> lda_mailbox_autosubscribe = yes
>>>>>>>>>> listen = *
>>>>>>>>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>>>>>>>>>> login_trusted_networks = (...)
>>>>>>>>>> mail_gid = 5000
>>>>>>>>>> mail_location = maildir:%h
>>>>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>>>>> mail_privileged_group = vmail
>>>>>>>>>> mail_uid = 5000
>>>>>>>>>> maildir_stat_dirs = yes
>>>>>>>>>> maildir_very_dirty_syncs = yes
>>>>>>>>>> managesieve_notify_capability = mailto
>>>>>>>>>> managesieve_sieve_capability = fileinto reject envelope
>>>>>>>>>> encoded-character subaddress comparator-i;ascii-numeric
>>>>>>>>>> relational regex
>>>>>>>>>> imap4flags copy include variables body enotify environment
>>>>>>>>>> mailbox date
>>>>>>>>>> index ihave duplicate mime foreverypart extracttext
>>>>>>>>>> namespace {
>>>>>>>>>> inbox = yes
>>>>>>>>>> location =
>>>>>>>>>> prefix = INBOX.
>>>>>>>>>> separator = .
>>>>>>>>>> type = private
>>>>>>>>>> }
>>>>>>>>>> passdb {
>>>>>>>>>> args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>>> driver = ldap
>>>>>>>>>> }
>>>>>>>>>> plugin {
>>>>>>>>>> quota = maildir:Quota Utilisateur
>>>>>>>>>> quota_exceeded_message = Cet utilisateur a dépassé son
>>>>>>>>>> quota, votre
>>>>>>>>>> message n'a pu lui être livré.
>>>>>>>>>> quota_full_tempfail = yes
>>>>>>>>>> quota_rule = *:storage=1000M
>>>>>>>>>> quota_rule2 = INBOX.Trash:storage=+100M
>>>>>>>>>> sieve = ~/dovecot.sieve
>>>>>>>>>> sieve_dir = ~/sieve
>>>>>>>>>> sieve_extensions = -vacation
>>>>>>>>>> sieve_global_dir = /var/lib/dovecot/sieve/global/
>>>>>>>>>> sieve_max_redirects = 1
>>>>>>>>>> zlib_save = gz
>>>>>>>>>> zlib_save_level = 6
>>>>>>>>>> }
>>>>>>>>>> postmaster_address = postmaster@<snip>
>>>>>>>>>> protocols = imap pop3 sieve
>>>>>>>>>> replication_max_conns = 50
>>>>>>>>>> service auth {
>>>>>>>>>> client_limit = 49452
>>>>>>>>>> unix_listener auth-userdb {
>>>>>>>>>> group = vmail
>>>>>>>>>> mode = 0600
>>>>>>>>>> user = vmail
>>>>>>>>>> }
>>>>>>>>>> user = root
>>>>>>>>>> }
>>>>>>>>>> service dict {
>>>>>>>>>> unix_listener dict {
>>>>>>>>>> mode = 0600
>>>>>>>>>> user = vmail
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> service doveadm {
>>>>>>>>>> inet_listener {
>>>>>>>>>> port = 12345
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> service imap-login {
>>>>>>>>>> process_min_avail = 8
>>>>>>>>>> service_count = 0
>>>>>>>>>> user = mail
>>>>>>>>>> }
>>>>>>>>>> service imap {
>>>>>>>>>> executable = imap
>>>>>>>>>> process_limit = 16384
>>>>>>>>>> vsz_limit = 2 G
>>>>>>>>>> }
>>>>>>>>>> service managesieve-login {
>>>>>>>>>> inet_listener sieve {
>>>>>>>>>> port = 4190
>>>>>>>>>> }
>>>>>>>>>> process_min_avail = 8
>>>>>>>>>> service_count = 0
>>>>>>>>>> user = mail
>>>>>>>>>> vsz_limit = 2 G
>>>>>>>>>> }
>>>>>>>>>> service managesieve {
>>>>>>>>>> drop_priv_before_exec = yes
>>>>>>>>>> process_limit = 16384
>>>>>>>>>> }
>>>>>>>>>> service pop3-login {
>>>>>>>>>> process_min_avail = 8
>>>>>>>>>> service_count = 0
>>>>>>>>>> user = mail
>>>>>>>>>> }
>>>>>>>>>> service pop3 {
>>>>>>>>>> drop_priv_before_exec = yes
>>>>>>>>>> process_limit = 16384
>>>>>>>>>> }
>>>>>>>>>> ssl = no
>>>>>>>>>> userdb {
>>>>>>>>>> args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>>> driver = ldap
>>>>>>>>>> }
>>>>>>>>>> verbose_proctitle = yes
>>>>>>>>>> protocol imap {
>>>>>>>>>> mail_max_userip_connections = 50
>>>>>>>>>> mail_plugins = quota zlib notify replication imap_quota
>>>>>>>>>> zlib
>>>>>>>>>> }
>>>>>>>>>> protocol pop3 {
>>>>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>>>>> }
>>>>>>>>>> protocol sieve {
>>>>>>>>>> mail_max_userip_connections = 10
>>>>>>>>>> }
>>>>>>>>>> protocol lda {
>>>>>>>>>> mail_plugins = quota zlib notify replication sieve zlib
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arnaud Abélard
>>>>>>>>> Responsable pôle Système et Stockage
>>>>>>>>> Service Infrastructures
>>>>>>>>> DSIN Université de Nantes
>>>>>>>>> -
>>>>>>>
>>>>>>> --
>>>>>>> Arnaud Abélard
>>>>>>> Responsable pôle Système et Stockage
>>>>>>> Service Infrastructures
>>>>>>> DSIN Université de Nantes
>>>>>>> -
>>>>>
>>>>> --
>>>>> Arnaud Abélard
>>>>> Responsable pôle Système et Stockage
>>>>> Service Infrastructures
>>>>> DSIN Université de Nantes
>>>>> -
>>>
>>
>
--
Arnaud Abélard
Responsable pôle Système et Stockage
Service Infrastructures
DSIN Université de Nantes
-
More information about the dovecot
mailing list