[SOLVED] after replication with compression quotas are wrong

Arnaud Abélard arnaud.abelard at univ-nantes.fr
Fri Nov 12 20:18:09 UTC 2021


Ah! At last, after comparing all default values using doveconf I found 
the culprit: mailbox_list_index. Default is "no" on dovecot 2.2.7, but 
"yes" on dovecot 2.3.4 (on debian, at least). Switching 
mailbox_list_index to "no" fixes my quota miscalculation problem.

Arnaud

On 12/11/2021 20:09, Arnaud Abélard wrote:
> 
> 
> On 10/11/2021 15:52, Arnaud Abélard wrote:
>> I've just removed dummy-c-1's mailbox completely on the target server, 
>> removed the quota plugin, forced a sync for dummy-c-1 only and once it 
>> was done reactivated the quota plugins: double usage again.
>>
>> I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server 
>> in order to have both server running the same version, without 
>> improvement.
> 
> Actually, I got that wrong, test server is running dovecot 2.3.4 and 
> production server dovecot 2.2.7. I copied the dovecot.conf file from the 
> production server to the test one, then rsync'ed the dummy-c-1 mailbox 
> from the production server to the test server, restarted dovecot and the 
> quota is still 115% after recalc. The only difference now is dovecot's 
> version. Were there any changes related to quota on dovecot 2.3?
> 
> Thanks,
> 
> Arnaud
> 
>>
>> I'm puzzled. Is dovecot storing anything outside of the user's 
>> mailbox? Like a cache, a sqlite database of some kind somewhere?
>>
>> Arnaud
>>
>> On 08/11/2021 11:48, Arnaud Abélard wrote:
>>> On the target, I enabled the replication service without mail_replica 
>>> and:
>>>
>>> doveadm replicator status 'dummy-c-1*'
>>> username     priority fast sync full sync success sync failed
>>> dummy-c-1     none     -         -         -            -
>>>
>>> It only knows of dummy-c-1, no trace of his evil twin 
>>> dummy-c-1 at univ-nantes.fr.
>>>
>>> On the target, I do have the same number of files
>>>
>>> find . -type f |wc -l
>>> 8705
>>>
>>> which is around half of what the quota is reporting (plus de index 
>>> files, etc):
>>>
>>> ~# doveadm -D quota get -u dummy-c-1
>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules
>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
>>> Debug: Module loaded: 
>>> /usr/lib/dovecot/modules/lib20_replication_plugin.so
>>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
>>> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
>>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: 
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: 
>>> undefined symbol: acl_user_module (this is usually intentional, so 
>>> just ignore this message)
>>> Debug: Skipping module doveadm_expire_plugin, because dlopen() 
>>> failed: 
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: 
>>> undefined symbol: expire_set_deinit (this is usually intentional, so 
>>> just ignore this message)
>>> Debug: Module loaded: 
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
>>> Debug: Module loaded: 
>>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
>>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() 
>>> failed: 
>>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: 
>>> undefined symbol: lucene_index_iter_deinit (this is usually 
>>> intentional, so just ignore this message)
>>> Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
>>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: 
>>> undefined symbol: fts_user_get_language_list (this is usually 
>>> intentional, so just ignore this message)
>>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() 
>>> failed: 
>>> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: 
>>> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually 
>>> intentional, so just ignore this message)
>>> doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 
>>> home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S
>>> doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: 
>>> plugin/quota_rule=*:backend=1000000000S
>>> doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, 
>>> home=/vmail/d/u/dummy-c-1/
>>> doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur 
>>> backend=maildir args=
>>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur 
>>> mailbox=* bytes=1000000000 messages=0
>>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur 
>>> mailbox=INBOX.Trash bytes=+104857600 messages=0
>>> doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur 
>>> bytes=100000000 (10%)
>>> doveadm(dummy-c-1): Debug: replication: No mail_replica setting - 
>>> replication disabled
>>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., 
>>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes 
>>> location=maildir:/vmail/d/u/dummy-c-1/
>>> doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, 
>>> index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt=
>>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, 
>>> inbox=no, hidden=yes, list=no, subscriptions=no 
>>> location=fail::LAYOUT=none
>>> doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, 
>>> inbox=, alt=
>>> doveadm(dummy-c-1): Debug: quota: quota_over_flag check: 
>>> quota_over_script unset - skipping
>>> Quota name        Type      Value  Limit 
>>>                                              %
>>> Quota Utilisateur STORAGE 1126751 976563 
>>>                                            115
>>> Quota Utilisateur MESSAGE   16686      - 
>>>                                              0
>>>
>>> The debug mode isn't much of any help here.
>>>
>>> Arnaud
>>>
>>> On 08/11/2021 11:30, Aki Tuomi wrote:
>>>> Recalculation won't fix replication mistakes.
>>>>
>>>> Did you ensure on the *target* server that it has only dummy-c-1 in 
>>>> replication and that the file count on the target server matches 
>>>> source server?
>>>>
>>>> You might also get something useful out of
>>>>
>>>> `doveadm -D quota recalc|get -u user`
>>>>
>>>> Aki
>>>>
>>>>> On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> 
>>>>> wrote:
>>>>>
>>>>> On 08/11/2021 10:39, Aki Tuomi wrote:
>>>>>> Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>>>>>
>>>>> I did so on the source server, checked indeed that
>>>>> dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator
>>>>> status list and it was indeed the case, so far so good. On the
>>>>> destination server, dummy-c-1's mailbox was still showing a 115% quota
>>>>> so I tried to make dovecot recalculate the quota but it did not change
>>>>> anything. I ended up deleting the user's mailbox and forced a
>>>>> replication and, to my surprise, the new mailbox still has a 115% 
>>>>> quota.
>>>>>
>>>>> Arnaud
>>>>>
>>>>>>
>>>>>> Aki
>>>>>>
>>>>>>> On 08/11/2021 11:32 Arnaud Abélard 
>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>
>>>>>>> Hello Aki,
>>>>>>>
>>>>>>> Thanks for the tip. I changed postfix configuration so it will 
>>>>>>> not use
>>>>>>> the domain part anymore and added "auth_username_format = %Ln" 
>>>>>>> just in
>>>>>>> case (and reloaded dovecot). So I shouldn't have any more new
>>>>>>> user at domain users but for all the existing users, doveadm replicator
>>>>>>> status still show user at domain clone and on the new server mails are
>>>>>>> still counted twice.
>>>>>>>
>>>>>>> Our userdb and passdb don't have the @domain defined anywhere so 
>>>>>>> I don't
>>>>>>> think it's coming from there.
>>>>>>>
>>>>>>> Is there a local cache that still has a list of those user at domain
>>>>>>> usernames that I could reset or something?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Arnaud
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 08/11/2021 09:48, Aki Tuomi wrote:
>>>>>>>> This sounds like you are not normalizing usernames properly.
>>>>>>>>
>>>>>>>> Either use
>>>>>>>>
>>>>>>>> auth_username_format = %Ln
>>>>>>>>
>>>>>>>> as global setting, or return `user` attribute in both userdb and 
>>>>>>>> passdb lookups.
>>>>>>>>
>>>>>>>> Aki
>>>>>>>>
>>>>>>>>> On 07/11/2021 20:31 Arnaud Abélard 
>>>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>>>
>>>>>>>>> Hello again,
>>>>>>>>>
>>>>>>>>> I've found out that some mailboxes  are actually duplicated. 
>>>>>>>>> Doveadm
>>>>>>>>> replicator status on the production server gives this:
>>>>>>>>>
>>>>>>>>> ~# doveadm replicator status 'dummy-c-1*'
>>>>>>>>> username                              priority fast sync full sync
>>>>>>>>> success sync failed
>>>>>>>>> dummy-c-1                             none     01:13:19  01:13:19
>>>>>>>>> 01:13:19     -
>>>>>>>>> dummy-c-1 at univ-nantes.fr              none     00:15:28  00:15:28
>>>>>>>>> 00:15:28     -
>>>>>>>>>
>>>>>>>>> That'd explain why mails are counted twice when replicated on 
>>>>>>>>> the new
>>>>>>>>> server but where does this come from since I don't have this quota
>>>>>>>>> problem on the production server?
>>>>>>>>>
>>>>>>>>>     From the logs, it seems that postfix uses 
>>>>>>>>> username at univ-nantes.fr when
>>>>>>>>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
>>>>>>>>> mailboxes are the erroneous ones.
>>>>>>>>>
>>>>>>>>> The users aren't duplicated in our ldap user backend and aren't 
>>>>>>>>> using
>>>>>>>>> the @univ-nantes.fr part.
>>>>>>>>>
>>>>>>>>> ~# doveadm user 'dummy-c-1*'
>>>>>>>>> dummy-c-1
>>>>>>>>>
>>>>>>>>> Any ideas?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Arnaud
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/11/2021 16:21, Arnaud Abélard wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> We are very long time happy dovecot users (since 2008 at 
>>>>>>>>>> least). We have
>>>>>>>>>> around 90k mailboxes and since we had to move away from our 
>>>>>>>>>> NAS storage
>>>>>>>>>> to a ceph storage I jumped on the opportunity to enable 
>>>>>>>>>> compression with
>>>>>>>>>> the zlib plugin and dovecot's replication mecanism. We are using
>>>>>>>>>> debian's dovecot 2.2.27 packages on production and our new 
>>>>>>>>>> server is
>>>>>>>>>> running dovecot's own ce-2.3.17 packages.
>>>>>>>>>>
>>>>>>>>>> On the production server everything works fine but on the new 
>>>>>>>>>> server,
>>>>>>>>>> replicated mailboxes' quota is all wrong:
>>>>>>>>>>
>>>>>>>>>> on production:
>>>>>>>>>> # doveadm quota get -u dummy-c-1
>>>>>>>>>> Quota name        Type     Value 
>>>>>>>>>> Limit                             %
>>>>>>>>>> Quota Utilisateur STORAGE 660026 
>>>>>>>>>> 976563                            67
>>>>>>>>>> Quota Utilisateur MESSAGE   8651 -                             0
>>>>>>>>>>
>>>>>>>>>> on new server:
>>>>>>>>>> doveadm quota get -u dummy-c-1
>>>>>>>>>> Quota name        Type      Value 
>>>>>>>>>> Limit                           %
>>>>>>>>>> Quota Utilisateur STORAGE 1125251 976563 115
>>>>>>>>>> Quota Utilisateur MESSAGE   16646 -                           0
>>>>>>>>>>
>>>>>>>>>> If I add all the S flag from the filenames n both servers I 
>>>>>>>>>> get exactly
>>>>>>>>>> the same usage, which is coherent with the quota on the 
>>>>>>>>>> production server:
>>>>>>>>>>
>>>>>>>>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk 
>>>>>>>>>> -F','
>>>>>>>>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
>>>>>>>>>> 675865938
>>>>>>>>>>
>>>>>>>>>> And I have exactly the same amountof mails on the two server, the
>>>>>>>>>> replication works as expected, no unwanted duplication of 
>>>>>>>>>> mails occurs.
>>>>>>>>>>
>>>>>>>>>> Of course, I've tried to ask dovecot to recalculate quotas 
>>>>>>>>>> with doveadm
>>>>>>>>>> quota recalc -u <username>, but it doesn't fix the problem.
>>>>>>>>>>
>>>>>>>>>> What am I missing?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Arnaud
>>>>>>>>>>
>>>>>>>>>> PS: Here is my doveconf -n output:
>>>>>>>>>>
>>>>>>>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
>>>>>>>>>> # Pigeonhole version 0.5.17 (054dddfa)
>>>>>>>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
>>>>>>>>>> # Hostname: gromel-test
>>>>>>>>>> auth_cache_size = 10 k
>>>>>>>>>> auth_verbose = yes
>>>>>>>>>> disable_plaintext_auth = no
>>>>>>>>>> doveadm_password = # hidden, use -P to show it
>>>>>>>>>> hostname = gromel1.univ-nantes.prive
>>>>>>>>>> lda_mailbox_autosubscribe = yes
>>>>>>>>>> listen = *
>>>>>>>>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>>>>>>>>>> login_trusted_networks = (...)
>>>>>>>>>> mail_gid = 5000
>>>>>>>>>> mail_location = maildir:%h
>>>>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>>>>> mail_privileged_group = vmail
>>>>>>>>>> mail_uid = 5000
>>>>>>>>>> maildir_stat_dirs = yes
>>>>>>>>>> maildir_very_dirty_syncs = yes
>>>>>>>>>> managesieve_notify_capability = mailto
>>>>>>>>>> managesieve_sieve_capability = fileinto reject envelope
>>>>>>>>>> encoded-character subaddress comparator-i;ascii-numeric 
>>>>>>>>>> relational regex
>>>>>>>>>> imap4flags copy include variables body enotify environment 
>>>>>>>>>> mailbox date
>>>>>>>>>> index ihave duplicate mime foreverypart extracttext
>>>>>>>>>> namespace {
>>>>>>>>>>       inbox = yes
>>>>>>>>>>       location =
>>>>>>>>>>       prefix = INBOX.
>>>>>>>>>>       separator = .
>>>>>>>>>>       type = private
>>>>>>>>>> }
>>>>>>>>>> passdb {
>>>>>>>>>>       args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>>>       driver = ldap
>>>>>>>>>> }
>>>>>>>>>> plugin {
>>>>>>>>>>       quota = maildir:Quota Utilisateur
>>>>>>>>>>       quota_exceeded_message = Cet utilisateur a dépassé son 
>>>>>>>>>> quota, votre
>>>>>>>>>> message n'a pu lui être livré.
>>>>>>>>>>       quota_full_tempfail = yes
>>>>>>>>>>       quota_rule = *:storage=1000M
>>>>>>>>>>       quota_rule2 = INBOX.Trash:storage=+100M
>>>>>>>>>>       sieve = ~/dovecot.sieve
>>>>>>>>>>       sieve_dir = ~/sieve
>>>>>>>>>>       sieve_extensions = -vacation
>>>>>>>>>>       sieve_global_dir = /var/lib/dovecot/sieve/global/
>>>>>>>>>>       sieve_max_redirects = 1
>>>>>>>>>>       zlib_save = gz
>>>>>>>>>>       zlib_save_level = 6
>>>>>>>>>> }
>>>>>>>>>> postmaster_address = postmaster@<snip>
>>>>>>>>>> protocols = imap pop3 sieve
>>>>>>>>>> replication_max_conns = 50
>>>>>>>>>> service auth {
>>>>>>>>>>       client_limit = 49452
>>>>>>>>>>       unix_listener auth-userdb {
>>>>>>>>>>         group = vmail
>>>>>>>>>>         mode = 0600
>>>>>>>>>>         user = vmail
>>>>>>>>>>       }
>>>>>>>>>>       user = root
>>>>>>>>>> }
>>>>>>>>>> service dict {
>>>>>>>>>>       unix_listener dict {
>>>>>>>>>>         mode = 0600
>>>>>>>>>>         user = vmail
>>>>>>>>>>       }
>>>>>>>>>> }
>>>>>>>>>> service doveadm {
>>>>>>>>>>       inet_listener {
>>>>>>>>>>         port = 12345
>>>>>>>>>>       }
>>>>>>>>>> }
>>>>>>>>>> service imap-login {
>>>>>>>>>>       process_min_avail = 8
>>>>>>>>>>       service_count = 0
>>>>>>>>>>       user = mail
>>>>>>>>>> }
>>>>>>>>>> service imap {
>>>>>>>>>>       executable = imap
>>>>>>>>>>       process_limit = 16384
>>>>>>>>>>       vsz_limit = 2 G
>>>>>>>>>> }
>>>>>>>>>> service managesieve-login {
>>>>>>>>>>       inet_listener sieve {
>>>>>>>>>>         port = 4190
>>>>>>>>>>       }
>>>>>>>>>>       process_min_avail = 8
>>>>>>>>>>       service_count = 0
>>>>>>>>>>       user = mail
>>>>>>>>>>       vsz_limit = 2 G
>>>>>>>>>> }
>>>>>>>>>> service managesieve {
>>>>>>>>>>       drop_priv_before_exec = yes
>>>>>>>>>>       process_limit = 16384
>>>>>>>>>> }
>>>>>>>>>> service pop3-login {
>>>>>>>>>>       process_min_avail = 8
>>>>>>>>>>       service_count = 0
>>>>>>>>>>       user = mail
>>>>>>>>>> }
>>>>>>>>>> service pop3 {
>>>>>>>>>>       drop_priv_before_exec = yes
>>>>>>>>>>       process_limit = 16384
>>>>>>>>>> }
>>>>>>>>>> ssl = no
>>>>>>>>>> userdb {
>>>>>>>>>>       args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>>>       driver = ldap
>>>>>>>>>> }
>>>>>>>>>> verbose_proctitle = yes
>>>>>>>>>> protocol imap {
>>>>>>>>>>       mail_max_userip_connections = 50
>>>>>>>>>>       mail_plugins = quota zlib notify replication imap_quota 
>>>>>>>>>> zlib
>>>>>>>>>> }
>>>>>>>>>> protocol pop3 {
>>>>>>>>>>       mail_plugins = quota zlib notify replication
>>>>>>>>>> }
>>>>>>>>>> protocol sieve {
>>>>>>>>>>       mail_max_userip_connections = 10
>>>>>>>>>> }
>>>>>>>>>> protocol lda {
>>>>>>>>>>       mail_plugins = quota zlib notify replication sieve zlib
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Arnaud Abélard
>>>>>>>>> Responsable pôle Système et Stockage
>>>>>>>>> Service Infrastructures
>>>>>>>>> DSIN Université de Nantes
>>>>>>>>> -
>>>>>>>
>>>>>>> -- 
>>>>>>> Arnaud Abélard
>>>>>>> Responsable pôle Système et Stockage
>>>>>>> Service Infrastructures
>>>>>>> DSIN Université de Nantes
>>>>>>> -
>>>>>
>>>>> -- 
>>>>> Arnaud Abélard
>>>>> Responsable pôle Système et Stockage
>>>>> Service Infrastructures
>>>>> DSIN Université de Nantes
>>>>> -
>>>
>>
> 

-- 
Arnaud Abélard
Responsable pôle Système et Stockage
Service Infrastructures
DSIN Université de Nantes
-



More information about the dovecot mailing list