after replication with compression quotas are wrong

Arnaud Abélard arnaud.abelard at univ-nantes.fr
Mon Nov 8 10:48:12 UTC 2021


On the target, I enabled the replication service without mail_replica and:

doveadm replicator status 'dummy-c-1*'
username 
     priority fast sync full sync success sync failed
dummy-c-1 
     none     -         -         -            -

It only knows of dummy-c-1, no trace of his evil twin 
dummy-c-1 at univ-nantes.fr.

On the target, I do have the same number of files

find . -type f |wc -l
8705

which is around half of what the quota is reporting (plus de index 
files, etc):

~# doveadm -D quota get -u dummy-c-1
Debug: Loading modules from directory: /usr/lib/dovecot/modules
Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined 
symbol: acl_user_module (this is usually intentional, so just ignore 
this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: 
undefined symbol: expire_set_deinit (this is usually intentional, so 
just ignore this message)
Debug: Module loaded: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
Debug: Module loaded: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() 
failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: 
undefined symbol: lucene_index_iter_deinit (this is usually intentional, 
so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined 
symbol: fts_user_get_language_list (this is usually intentional, so just 
ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() 
failed: 
/usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: 
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually 
intentional, so just ignore this message)
doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 
home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S
doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: 
plugin/quota_rule=*:backend=1000000000S
doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, 
home=/vmail/d/u/dummy-c-1/
doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur 
backend=maildir args=
doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* 
bytes=1000000000 messages=0
doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur 
mailbox=INBOX.Trash bytes=+104857600 messages=0
doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur 
bytes=100000000 (10%)
doveadm(dummy-c-1): Debug: replication: No mail_replica setting - 
replication disabled
doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., 
sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes 
location=maildir:/vmail/d/u/dummy-c-1/
doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, 
indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt=
doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, 
inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none
doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, 
inbox=, alt=
doveadm(dummy-c-1): Debug: quota: quota_over_flag check: 
quota_over_script unset - skipping
Quota name        Type      Value  Limit 
                                             %
Quota Utilisateur STORAGE 1126751 976563 
                                           115
Quota Utilisateur MESSAGE   16686      - 
                                             0

The debug mode isn't much of any help here.

Arnaud

On 08/11/2021 11:30, Aki Tuomi wrote:
> Recalculation won't fix replication mistakes.
> 
> Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server?
> 
> You might also get something useful out of
> 
> `doveadm -D quota recalc|get -u user`
> 
> Aki
> 
>> On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
>>
>>   
>> On 08/11/2021 10:39, Aki Tuomi wrote:
>>> Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>>
>> I did so on the source server, checked indeed that
>> dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator
>> status list and it was indeed the case, so far so good. On the
>> destination server, dummy-c-1's mailbox was still showing a 115% quota
>> so I tried to make dovecot recalculate the quota but it did not change
>> anything. I ended up deleting the user's mailbox and forced a
>> replication and, to my surprise, the new mailbox still has a 115% quota.
>>
>> Arnaud
>>
>>>
>>> Aki
>>>
>>>> On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
>>>>
>>>>    
>>>> Hello Aki,
>>>>
>>>> Thanks for the tip. I changed postfix configuration so it will not use
>>>> the domain part anymore and added "auth_username_format = %Ln" just in
>>>> case (and reloaded dovecot). So I shouldn't have any more new
>>>> user at domain users but for all the existing users, doveadm replicator
>>>> status still show user at domain clone and on the new server mails are
>>>> still counted twice.
>>>>
>>>> Our userdb and passdb don't have the @domain defined anywhere so I don't
>>>> think it's coming from there.
>>>>
>>>> Is there a local cache that still has a list of those user at domain
>>>> usernames that I could reset or something?
>>>>
>>>> Thanks,
>>>>
>>>> Arnaud
>>>>
>>>>
>>>>
>>>> On 08/11/2021 09:48, Aki Tuomi wrote:
>>>>> This sounds like you are not normalizing usernames properly.
>>>>>
>>>>> Either use
>>>>>
>>>>> auth_username_format = %Ln
>>>>>
>>>>> as global setting, or return `user` attribute in both userdb and passdb lookups.
>>>>>
>>>>> Aki
>>>>>
>>>>>> On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>
>>>>>>     
>>>>>> Hello again,
>>>>>>
>>>>>> I've found out that some mailboxes  are actually duplicated. Doveadm
>>>>>> replicator status on the production server gives this:
>>>>>>
>>>>>> ~# doveadm replicator status 'dummy-c-1*'
>>>>>> username                              priority fast sync full sync
>>>>>> success sync failed
>>>>>> dummy-c-1                             none     01:13:19  01:13:19
>>>>>> 01:13:19     -
>>>>>> dummy-c-1 at univ-nantes.fr              none     00:15:28  00:15:28
>>>>>> 00:15:28     -
>>>>>>
>>>>>> That'd explain why mails are counted twice when replicated on the new
>>>>>> server but where does this come from since I don't have this quota
>>>>>> problem on the production server?
>>>>>>
>>>>>>     From the logs, it seems that postfix uses username at univ-nantes.fr when
>>>>>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
>>>>>> mailboxes are the erroneous ones.
>>>>>>
>>>>>> The users aren't duplicated in our ldap user backend and aren't using
>>>>>> the @univ-nantes.fr part.
>>>>>>
>>>>>> ~# doveadm user 'dummy-c-1*'
>>>>>> dummy-c-1
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Arnaud
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 05/11/2021 16:21, Arnaud Abélard wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> We are very long time happy dovecot users (since 2008 at least). We have
>>>>>>> around 90k mailboxes and since we had to move away from our NAS storage
>>>>>>> to a ceph storage I jumped on the opportunity to enable compression with
>>>>>>> the zlib plugin and dovecot's replication mecanism. We are using
>>>>>>> debian's dovecot 2.2.27 packages on production and our new server is
>>>>>>> running dovecot's own ce-2.3.17 packages.
>>>>>>>
>>>>>>> On the production server everything works fine but on the new server,
>>>>>>> replicated mailboxes' quota is all wrong:
>>>>>>>
>>>>>>> on production:
>>>>>>> # doveadm quota get -u dummy-c-1
>>>>>>> Quota name        Type     Value  Limit                             %
>>>>>>> Quota Utilisateur STORAGE 660026 976563                            67
>>>>>>> Quota Utilisateur MESSAGE   8651      -                             0
>>>>>>>
>>>>>>> on new server:
>>>>>>> doveadm quota get -u dummy-c-1
>>>>>>> Quota name        Type      Value  Limit                           %
>>>>>>> Quota Utilisateur STORAGE 1125251 976563                         115
>>>>>>> Quota Utilisateur MESSAGE   16646      -                           0
>>>>>>>
>>>>>>> If I add all the S flag from the filenames n both servers I get exactly
>>>>>>> the same usage, which is coherent with the quota on the production server:
>>>>>>>
>>>>>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
>>>>>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
>>>>>>> 675865938
>>>>>>>
>>>>>>> And I have exactly the same amountof mails on the two server, the
>>>>>>> replication works as expected, no unwanted duplication of mails occurs.
>>>>>>>
>>>>>>> Of course, I've tried to ask dovecot to recalculate quotas with doveadm
>>>>>>> quota recalc -u <username>, but it doesn't fix the problem.
>>>>>>>
>>>>>>> What am I missing?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Arnaud
>>>>>>>
>>>>>>> PS: Here is my doveconf -n output:
>>>>>>>
>>>>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
>>>>>>> # Pigeonhole version 0.5.17 (054dddfa)
>>>>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
>>>>>>> # Hostname: gromel-test
>>>>>>> auth_cache_size = 10 k
>>>>>>> auth_verbose = yes
>>>>>>> disable_plaintext_auth = no
>>>>>>> doveadm_password = # hidden, use -P to show it
>>>>>>> hostname = gromel1.univ-nantes.prive
>>>>>>> lda_mailbox_autosubscribe = yes
>>>>>>> listen = *
>>>>>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>>>>>>> login_trusted_networks = (...)
>>>>>>> mail_gid = 5000
>>>>>>> mail_location = maildir:%h
>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>> mail_privileged_group = vmail
>>>>>>> mail_uid = 5000
>>>>>>> maildir_stat_dirs = yes
>>>>>>> maildir_very_dirty_syncs = yes
>>>>>>> managesieve_notify_capability = mailto
>>>>>>> managesieve_sieve_capability = fileinto reject envelope
>>>>>>> encoded-character subaddress comparator-i;ascii-numeric relational regex
>>>>>>> imap4flags copy include variables body enotify environment mailbox date
>>>>>>> index ihave duplicate mime foreverypart extracttext
>>>>>>> namespace {
>>>>>>>       inbox = yes
>>>>>>>       location =
>>>>>>>       prefix = INBOX.
>>>>>>>       separator = .
>>>>>>>       type = private
>>>>>>> }
>>>>>>> passdb {
>>>>>>>       args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>       driver = ldap
>>>>>>> }
>>>>>>> plugin {
>>>>>>>       quota = maildir:Quota Utilisateur
>>>>>>>       quota_exceeded_message = Cet utilisateur a dépassé son quota, votre
>>>>>>> message n'a pu lui être livré.
>>>>>>>       quota_full_tempfail = yes
>>>>>>>       quota_rule = *:storage=1000M
>>>>>>>       quota_rule2 = INBOX.Trash:storage=+100M
>>>>>>>       sieve = ~/dovecot.sieve
>>>>>>>       sieve_dir = ~/sieve
>>>>>>>       sieve_extensions = -vacation
>>>>>>>       sieve_global_dir = /var/lib/dovecot/sieve/global/
>>>>>>>       sieve_max_redirects = 1
>>>>>>>       zlib_save = gz
>>>>>>>       zlib_save_level = 6
>>>>>>> }
>>>>>>> postmaster_address = postmaster@<snip>
>>>>>>> protocols = imap pop3 sieve
>>>>>>> replication_max_conns = 50
>>>>>>> service auth {
>>>>>>>       client_limit = 49452
>>>>>>>       unix_listener auth-userdb {
>>>>>>>         group = vmail
>>>>>>>         mode = 0600
>>>>>>>         user = vmail
>>>>>>>       }
>>>>>>>       user = root
>>>>>>> }
>>>>>>> service dict {
>>>>>>>       unix_listener dict {
>>>>>>>         mode = 0600
>>>>>>>         user = vmail
>>>>>>>       }
>>>>>>> }
>>>>>>> service doveadm {
>>>>>>>       inet_listener {
>>>>>>>         port = 12345
>>>>>>>       }
>>>>>>> }
>>>>>>> service imap-login {
>>>>>>>       process_min_avail = 8
>>>>>>>       service_count = 0
>>>>>>>       user = mail
>>>>>>> }
>>>>>>> service imap {
>>>>>>>       executable = imap
>>>>>>>       process_limit = 16384
>>>>>>>       vsz_limit = 2 G
>>>>>>> }
>>>>>>> service managesieve-login {
>>>>>>>       inet_listener sieve {
>>>>>>>         port = 4190
>>>>>>>       }
>>>>>>>       process_min_avail = 8
>>>>>>>       service_count = 0
>>>>>>>       user = mail
>>>>>>>       vsz_limit = 2 G
>>>>>>> }
>>>>>>> service managesieve {
>>>>>>>       drop_priv_before_exec = yes
>>>>>>>       process_limit = 16384
>>>>>>> }
>>>>>>> service pop3-login {
>>>>>>>       process_min_avail = 8
>>>>>>>       service_count = 0
>>>>>>>       user = mail
>>>>>>> }
>>>>>>> service pop3 {
>>>>>>>       drop_priv_before_exec = yes
>>>>>>>       process_limit = 16384
>>>>>>> }
>>>>>>> ssl = no
>>>>>>> userdb {
>>>>>>>       args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>       driver = ldap
>>>>>>> }
>>>>>>> verbose_proctitle = yes
>>>>>>> protocol imap {
>>>>>>>       mail_max_userip_connections = 50
>>>>>>>       mail_plugins = quota zlib notify replication imap_quota zlib
>>>>>>> }
>>>>>>> protocol pop3 {
>>>>>>>       mail_plugins = quota zlib notify replication
>>>>>>> }
>>>>>>> protocol sieve {
>>>>>>>       mail_max_userip_connections = 10
>>>>>>> }
>>>>>>> protocol lda {
>>>>>>>       mail_plugins = quota zlib notify replication sieve zlib
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Arnaud Abélard
>>>>>> Responsable pôle Système et Stockage
>>>>>> Service Infrastructures
>>>>>> DSIN Université de Nantes
>>>>>> -
>>>>
>>>> -- 
>>>> Arnaud Abélard
>>>> Responsable pôle Système et Stockage
>>>> Service Infrastructures
>>>> DSIN Université de Nantes
>>>> -
>>
>> -- 
>> Arnaud Abélard
>> Responsable pôle Système et Stockage
>> Service Infrastructures
>> DSIN Université de Nantes
>> -

-- 
Arnaud Abélard
Responsable pôle Système et Stockage
Service Infrastructures
DSIN Université de Nantes
-



More information about the dovecot mailing list