after replication with compression quotas are wrong
Arnaud Abélard
arnaud.abelard at univ-nantes.fr
Fri Nov 12 19:09:17 UTC 2021
On 10/11/2021 15:52, Arnaud Abélard wrote:
> I've just removed dummy-c-1's mailbox completely on the target server,
> removed the quota plugin, forced a sync for dummy-c-1 only and once it
> was done reactivated the quota plugins: double usage again.
>
> I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in
> order to have both server running the same version, without improvement.
Actually, I got that wrong, test server is running dovecot 2.3.4 and
production server dovecot 2.2.7. I copied the dovecot.conf file from the
production server to the test one, then rsync'ed the dummy-c-1 mailbox
from the production server to the test server, restarted dovecot and the
quota is still 115% after recalc. The only difference now is dovecot's
version. Were there any changes related to quota on dovecot 2.3?
Thanks,
Arnaud
>
> I'm puzzled. Is dovecot storing anything outside of the user's mailbox?
> Like a cache, a sqlite database of some kind somewhere?
>
> Arnaud
>
> On 08/11/2021 11:48, Arnaud Abélard wrote:
>> On the target, I enabled the replication service without mail_replica
>> and:
>>
>> doveadm replicator status 'dummy-c-1*'
>> username priority fast sync full sync success sync failed
>> dummy-c-1 none - - - -
>>
>> It only knows of dummy-c-1, no trace of his evil twin
>> dummy-c-1 at univ-nantes.fr.
>>
>> On the target, I do have the same number of files
>>
>> find . -type f |wc -l
>> 8705
>>
>> which is around half of what the quota is reporting (plus de index
>> files, etc):
>>
>> ~# doveadm -D quota get -u dummy-c-1
>> Debug: Loading modules from directory: /usr/lib/dovecot/modules
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib20_replication_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
>> Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
>> Debug: Skipping module doveadm_acl_plugin, because dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so:
>> undefined symbol: acl_user_module (this is usually intentional, so
>> just ignore this message)
>> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so:
>> undefined symbol: expire_set_deinit (this is usually intentional, so
>> just ignore this message)
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
>> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen()
>> failed:
>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so:
>> undefined symbol: lucene_index_iter_deinit (this is usually
>> intentional, so just ignore this message)
>> Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so:
>> undefined symbol: fts_user_get_language_list (this is usually
>> intentional, so just ignore this message)
>> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen()
>> failed:
>> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so:
>> undefined symbol: mail_crypt_box_get_pvt_digests (this is usually
>> intentional, so just ignore this message)
>> doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1
>> home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S
>> doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting:
>> plugin/quota_rule=*:backend=1000000000S
>> doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000,
>> home=/vmail/d/u/dummy-c-1/
>> doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur
>> backend=maildir args=
>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur
>> mailbox=* bytes=1000000000 messages=0
>> doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur
>> mailbox=INBOX.Trash bytes=+104857600 messages=0
>> doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur
>> bytes=100000000 (10%)
>> doveadm(dummy-c-1): Debug: replication: No mail_replica setting -
>> replication disabled
>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX.,
>> sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes
>> location=maildir:/vmail/d/u/dummy-c-1/
>> doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1,
>> index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt=
>> doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=,
>> inbox=no, hidden=yes, list=no, subscriptions=no
>> location=fail::LAYOUT=none
>> doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=,
>> inbox=, alt=
>> doveadm(dummy-c-1): Debug: quota: quota_over_flag check:
>> quota_over_script unset - skipping
>> Quota name Type Value Limit
>> %
>> Quota Utilisateur STORAGE 1126751 976563
>> 115
>> Quota Utilisateur MESSAGE 16686 -
>> 0
>>
>> The debug mode isn't much of any help here.
>>
>> Arnaud
>>
>> On 08/11/2021 11:30, Aki Tuomi wrote:
>>> Recalculation won't fix replication mistakes.
>>>
>>> Did you ensure on the *target* server that it has only dummy-c-1 in
>>> replication and that the file count on the target server matches
>>> source server?
>>>
>>> You might also get something useful out of
>>>
>>> `doveadm -D quota recalc|get -u user`
>>>
>>> Aki
>>>
>>>> On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard at univ-nantes.fr>
>>>> wrote:
>>>>
>>>> On 08/11/2021 10:39, Aki Tuomi wrote:
>>>>> Try doveadm replicator remove dummy-c-1 at univ-nantes.fr
>>>>
>>>> I did so on the source server, checked indeed that
>>>> dummy-c-1 at univ-nantes.fr wasn't showing up in the doveadm replicator
>>>> status list and it was indeed the case, so far so good. On the
>>>> destination server, dummy-c-1's mailbox was still showing a 115% quota
>>>> so I tried to make dovecot recalculate the quota but it did not change
>>>> anything. I ended up deleting the user's mailbox and forced a
>>>> replication and, to my surprise, the new mailbox still has a 115%
>>>> quota.
>>>>
>>>> Arnaud
>>>>
>>>>>
>>>>> Aki
>>>>>
>>>>>> On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard at univ-nantes.fr>
>>>>>> wrote:
>>>>>>
>>>>>> Hello Aki,
>>>>>>
>>>>>> Thanks for the tip. I changed postfix configuration so it will not
>>>>>> use
>>>>>> the domain part anymore and added "auth_username_format = %Ln"
>>>>>> just in
>>>>>> case (and reloaded dovecot). So I shouldn't have any more new
>>>>>> user at domain users but for all the existing users, doveadm replicator
>>>>>> status still show user at domain clone and on the new server mails are
>>>>>> still counted twice.
>>>>>>
>>>>>> Our userdb and passdb don't have the @domain defined anywhere so I
>>>>>> don't
>>>>>> think it's coming from there.
>>>>>>
>>>>>> Is there a local cache that still has a list of those user at domain
>>>>>> usernames that I could reset or something?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Arnaud
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 08/11/2021 09:48, Aki Tuomi wrote:
>>>>>>> This sounds like you are not normalizing usernames properly.
>>>>>>>
>>>>>>> Either use
>>>>>>>
>>>>>>> auth_username_format = %Ln
>>>>>>>
>>>>>>> as global setting, or return `user` attribute in both userdb and
>>>>>>> passdb lookups.
>>>>>>>
>>>>>>> Aki
>>>>>>>
>>>>>>>> On 07/11/2021 20:31 Arnaud Abélard
>>>>>>>> <arnaud.abelard at univ-nantes.fr> wrote:
>>>>>>>>
>>>>>>>> Hello again,
>>>>>>>>
>>>>>>>> I've found out that some mailboxes are actually duplicated.
>>>>>>>> Doveadm
>>>>>>>> replicator status on the production server gives this:
>>>>>>>>
>>>>>>>> ~# doveadm replicator status 'dummy-c-1*'
>>>>>>>> username priority fast sync full sync
>>>>>>>> success sync failed
>>>>>>>> dummy-c-1 none 01:13:19 01:13:19
>>>>>>>> 01:13:19 -
>>>>>>>> dummy-c-1 at univ-nantes.fr none 00:15:28 00:15:28
>>>>>>>> 00:15:28 -
>>>>>>>>
>>>>>>>> That'd explain why mails are counted twice when replicated on
>>>>>>>> the new
>>>>>>>> server but where does this come from since I don't have this quota
>>>>>>>> problem on the production server?
>>>>>>>>
>>>>>>>> From the logs, it seems that postfix uses
>>>>>>>> username at univ-nantes.fr when
>>>>>>>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
>>>>>>>> mailboxes are the erroneous ones.
>>>>>>>>
>>>>>>>> The users aren't duplicated in our ldap user backend and aren't
>>>>>>>> using
>>>>>>>> the @univ-nantes.fr part.
>>>>>>>>
>>>>>>>> ~# doveadm user 'dummy-c-1*'
>>>>>>>> dummy-c-1
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Arnaud
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 05/11/2021 16:21, Arnaud Abélard wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> We are very long time happy dovecot users (since 2008 at
>>>>>>>>> least). We have
>>>>>>>>> around 90k mailboxes and since we had to move away from our NAS
>>>>>>>>> storage
>>>>>>>>> to a ceph storage I jumped on the opportunity to enable
>>>>>>>>> compression with
>>>>>>>>> the zlib plugin and dovecot's replication mecanism. We are using
>>>>>>>>> debian's dovecot 2.2.27 packages on production and our new
>>>>>>>>> server is
>>>>>>>>> running dovecot's own ce-2.3.17 packages.
>>>>>>>>>
>>>>>>>>> On the production server everything works fine but on the new
>>>>>>>>> server,
>>>>>>>>> replicated mailboxes' quota is all wrong:
>>>>>>>>>
>>>>>>>>> on production:
>>>>>>>>> # doveadm quota get -u dummy-c-1
>>>>>>>>> Quota name Type Value
>>>>>>>>> Limit %
>>>>>>>>> Quota Utilisateur STORAGE 660026
>>>>>>>>> 976563 67
>>>>>>>>> Quota Utilisateur MESSAGE 8651 - 0
>>>>>>>>>
>>>>>>>>> on new server:
>>>>>>>>> doveadm quota get -u dummy-c-1
>>>>>>>>> Quota name Type Value
>>>>>>>>> Limit %
>>>>>>>>> Quota Utilisateur STORAGE 1125251 976563 115
>>>>>>>>> Quota Utilisateur MESSAGE 16646 - 0
>>>>>>>>>
>>>>>>>>> If I add all the S flag from the filenames n both servers I get
>>>>>>>>> exactly
>>>>>>>>> the same usage, which is coherent with the quota on the
>>>>>>>>> production server:
>>>>>>>>>
>>>>>>>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
>>>>>>>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
>>>>>>>>> 675865938
>>>>>>>>>
>>>>>>>>> And I have exactly the same amountof mails on the two server, the
>>>>>>>>> replication works as expected, no unwanted duplication of mails
>>>>>>>>> occurs.
>>>>>>>>>
>>>>>>>>> Of course, I've tried to ask dovecot to recalculate quotas with
>>>>>>>>> doveadm
>>>>>>>>> quota recalc -u <username>, but it doesn't fix the problem.
>>>>>>>>>
>>>>>>>>> What am I missing?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Arnaud
>>>>>>>>>
>>>>>>>>> PS: Here is my doveconf -n output:
>>>>>>>>>
>>>>>>>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
>>>>>>>>> # Pigeonhole version 0.5.17 (054dddfa)
>>>>>>>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
>>>>>>>>> # Hostname: gromel-test
>>>>>>>>> auth_cache_size = 10 k
>>>>>>>>> auth_verbose = yes
>>>>>>>>> disable_plaintext_auth = no
>>>>>>>>> doveadm_password = # hidden, use -P to show it
>>>>>>>>> hostname = gromel1.univ-nantes.prive
>>>>>>>>> lda_mailbox_autosubscribe = yes
>>>>>>>>> listen = *
>>>>>>>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>>>>>>>>> login_trusted_networks = (...)
>>>>>>>>> mail_gid = 5000
>>>>>>>>> mail_location = maildir:%h
>>>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>>>> mail_privileged_group = vmail
>>>>>>>>> mail_uid = 5000
>>>>>>>>> maildir_stat_dirs = yes
>>>>>>>>> maildir_very_dirty_syncs = yes
>>>>>>>>> managesieve_notify_capability = mailto
>>>>>>>>> managesieve_sieve_capability = fileinto reject envelope
>>>>>>>>> encoded-character subaddress comparator-i;ascii-numeric
>>>>>>>>> relational regex
>>>>>>>>> imap4flags copy include variables body enotify environment
>>>>>>>>> mailbox date
>>>>>>>>> index ihave duplicate mime foreverypart extracttext
>>>>>>>>> namespace {
>>>>>>>>> inbox = yes
>>>>>>>>> location =
>>>>>>>>> prefix = INBOX.
>>>>>>>>> separator = .
>>>>>>>>> type = private
>>>>>>>>> }
>>>>>>>>> passdb {
>>>>>>>>> args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>> driver = ldap
>>>>>>>>> }
>>>>>>>>> plugin {
>>>>>>>>> quota = maildir:Quota Utilisateur
>>>>>>>>> quota_exceeded_message = Cet utilisateur a dépassé son
>>>>>>>>> quota, votre
>>>>>>>>> message n'a pu lui être livré.
>>>>>>>>> quota_full_tempfail = yes
>>>>>>>>> quota_rule = *:storage=1000M
>>>>>>>>> quota_rule2 = INBOX.Trash:storage=+100M
>>>>>>>>> sieve = ~/dovecot.sieve
>>>>>>>>> sieve_dir = ~/sieve
>>>>>>>>> sieve_extensions = -vacation
>>>>>>>>> sieve_global_dir = /var/lib/dovecot/sieve/global/
>>>>>>>>> sieve_max_redirects = 1
>>>>>>>>> zlib_save = gz
>>>>>>>>> zlib_save_level = 6
>>>>>>>>> }
>>>>>>>>> postmaster_address = postmaster@<snip>
>>>>>>>>> protocols = imap pop3 sieve
>>>>>>>>> replication_max_conns = 50
>>>>>>>>> service auth {
>>>>>>>>> client_limit = 49452
>>>>>>>>> unix_listener auth-userdb {
>>>>>>>>> group = vmail
>>>>>>>>> mode = 0600
>>>>>>>>> user = vmail
>>>>>>>>> }
>>>>>>>>> user = root
>>>>>>>>> }
>>>>>>>>> service dict {
>>>>>>>>> unix_listener dict {
>>>>>>>>> mode = 0600
>>>>>>>>> user = vmail
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> service doveadm {
>>>>>>>>> inet_listener {
>>>>>>>>> port = 12345
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> service imap-login {
>>>>>>>>> process_min_avail = 8
>>>>>>>>> service_count = 0
>>>>>>>>> user = mail
>>>>>>>>> }
>>>>>>>>> service imap {
>>>>>>>>> executable = imap
>>>>>>>>> process_limit = 16384
>>>>>>>>> vsz_limit = 2 G
>>>>>>>>> }
>>>>>>>>> service managesieve-login {
>>>>>>>>> inet_listener sieve {
>>>>>>>>> port = 4190
>>>>>>>>> }
>>>>>>>>> process_min_avail = 8
>>>>>>>>> service_count = 0
>>>>>>>>> user = mail
>>>>>>>>> vsz_limit = 2 G
>>>>>>>>> }
>>>>>>>>> service managesieve {
>>>>>>>>> drop_priv_before_exec = yes
>>>>>>>>> process_limit = 16384
>>>>>>>>> }
>>>>>>>>> service pop3-login {
>>>>>>>>> process_min_avail = 8
>>>>>>>>> service_count = 0
>>>>>>>>> user = mail
>>>>>>>>> }
>>>>>>>>> service pop3 {
>>>>>>>>> drop_priv_before_exec = yes
>>>>>>>>> process_limit = 16384
>>>>>>>>> }
>>>>>>>>> ssl = no
>>>>>>>>> userdb {
>>>>>>>>> args = /etc/dovecot/dovecot-ldap.conf
>>>>>>>>> driver = ldap
>>>>>>>>> }
>>>>>>>>> verbose_proctitle = yes
>>>>>>>>> protocol imap {
>>>>>>>>> mail_max_userip_connections = 50
>>>>>>>>> mail_plugins = quota zlib notify replication imap_quota zlib
>>>>>>>>> }
>>>>>>>>> protocol pop3 {
>>>>>>>>> mail_plugins = quota zlib notify replication
>>>>>>>>> }
>>>>>>>>> protocol sieve {
>>>>>>>>> mail_max_userip_connections = 10
>>>>>>>>> }
>>>>>>>>> protocol lda {
>>>>>>>>> mail_plugins = quota zlib notify replication sieve zlib
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Arnaud Abélard
>>>>>>>> Responsable pôle Système et Stockage
>>>>>>>> Service Infrastructures
>>>>>>>> DSIN Université de Nantes
>>>>>>>> -
>>>>>>
>>>>>> --
>>>>>> Arnaud Abélard
>>>>>> Responsable pôle Système et Stockage
>>>>>> Service Infrastructures
>>>>>> DSIN Université de Nantes
>>>>>> -
>>>>
>>>> --
>>>> Arnaud Abélard
>>>> Responsable pôle Système et Stockage
>>>> Service Infrastructures
>>>> DSIN Université de Nantes
>>>> -
>>
>
--
Arnaud Abélard
Responsable pôle Système et Stockage
Service Infrastructures
DSIN Université de Nantes
-
More information about the dovecot
mailing list