after replication with compression quotas are wrong

Aki Tuomi aki.tuomi at open-xchange.com
Mon Nov 8 09:39:44 UTC 2021


Try doveadm replicator remove dummy-c-1 at univ-nantes.fr

Aki

> On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
> 
>  
> Hello Aki,
> 
> Thanks for the tip. I changed postfix configuration so it will not use 
> the domain part anymore and added "auth_username_format = %Ln" just in 
> case (and reloaded dovecot). So I shouldn't have any more new 
> user at domain users but for all the existing users, doveadm replicator 
> status still show user at domain clone and on the new server mails are 
> still counted twice.
> 
> Our userdb and passdb don't have the @domain defined anywhere so I don't 
> think it's coming from there.
> 
> Is there a local cache that still has a list of those user at domain 
> usernames that I could reset or something?
> 
> Thanks,
> 
> Arnaud
> 
> 
> 
> On 08/11/2021 09:48, Aki Tuomi wrote:
> > This sounds like you are not normalizing usernames properly.
> > 
> > Either use
> > 
> > auth_username_format = %Ln
> > 
> > as global setting, or return `user` attribute in both userdb and passdb lookups.
> > 
> > Aki
> > 
> >> On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
> >>
> >>   
> >> Hello again,
> >>
> >> I've found out that some mailboxes  are actually duplicated. Doveadm
> >> replicator status on the production server gives this:
> >>
> >> ~# doveadm replicator status 'dummy-c-1*'
> >> username                              priority fast sync full sync
> >> success sync failed
> >> dummy-c-1                             none     01:13:19  01:13:19
> >> 01:13:19     -
> >> dummy-c-1 at univ-nantes.fr              none     00:15:28  00:15:28
> >> 00:15:28     -
> >>
> >> That'd explain why mails are counted twice when replicated on the new
> >> server but where does this come from since I don't have this quota
> >> problem on the production server?
> >>
> >>   From the logs, it seems that postfix uses username at univ-nantes.fr when
> >> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
> >> mailboxes are the erroneous ones.
> >>
> >> The users aren't duplicated in our ldap user backend and aren't using
> >> the @univ-nantes.fr part.
> >>
> >> ~# doveadm user 'dummy-c-1*'
> >> dummy-c-1
> >>
> >> Any ideas?
> >>
> >> Thanks,
> >>
> >> Arnaud
> >>
> >>
> >>
> >>
> >>
> >> On 05/11/2021 16:21, Arnaud Abélard wrote:
> >>> Hello,
> >>>
> >>> We are very long time happy dovecot users (since 2008 at least). We have
> >>> around 90k mailboxes and since we had to move away from our NAS storage
> >>> to a ceph storage I jumped on the opportunity to enable compression with
> >>> the zlib plugin and dovecot's replication mecanism. We are using
> >>> debian's dovecot 2.2.27 packages on production and our new server is
> >>> running dovecot's own ce-2.3.17 packages.
> >>>
> >>> On the production server everything works fine but on the new server,
> >>> replicated mailboxes' quota is all wrong:
> >>>
> >>> on production:
> >>> # doveadm quota get -u dummy-c-1
> >>> Quota name        Type     Value  Limit                             %
> >>> Quota Utilisateur STORAGE 660026 976563                            67
> >>> Quota Utilisateur MESSAGE   8651      -                             0
> >>>
> >>> on new server:
> >>> doveadm quota get -u dummy-c-1
> >>> Quota name        Type      Value  Limit                           %
> >>> Quota Utilisateur STORAGE 1125251 976563                         115
> >>> Quota Utilisateur MESSAGE   16646      -                           0
> >>>
> >>> If I add all the S flag from the filenames n both servers I get exactly
> >>> the same usage, which is coherent with the quota on the production server:
> >>>
> >>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
> >>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
> >>> 675865938
> >>>
> >>> And I have exactly the same amountof mails on the two server, the
> >>> replication works as expected, no unwanted duplication of mails occurs.
> >>>
> >>> Of course, I've tried to ask dovecot to recalculate quotas with doveadm
> >>> quota recalc -u <username>, but it doesn't fix the problem.
> >>>
> >>> What am I missing?
> >>>
> >>> Thanks,
> >>>
> >>> Arnaud
> >>>
> >>> PS: Here is my doveconf -n output:
> >>>
> >>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
> >>> # Pigeonhole version 0.5.17 (054dddfa)
> >>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
> >>> # Hostname: gromel-test
> >>> auth_cache_size = 10 k
> >>> auth_verbose = yes
> >>> disable_plaintext_auth = no
> >>> doveadm_password = # hidden, use -P to show it
> >>> hostname = gromel1.univ-nantes.prive
> >>> lda_mailbox_autosubscribe = yes
> >>> listen = *
> >>> log_timestamp = "%Y-%m-%d %H:%M:%S "
> >>> login_trusted_networks = (...)
> >>> mail_gid = 5000
> >>> mail_location = maildir:%h
> >>> mail_plugins = quota zlib notify replication
> >>> mail_privileged_group = vmail
> >>> mail_uid = 5000
> >>> maildir_stat_dirs = yes
> >>> maildir_very_dirty_syncs = yes
> >>> managesieve_notify_capability = mailto
> >>> managesieve_sieve_capability = fileinto reject envelope
> >>> encoded-character subaddress comparator-i;ascii-numeric relational regex
> >>> imap4flags copy include variables body enotify environment mailbox date
> >>> index ihave duplicate mime foreverypart extracttext
> >>> namespace {
> >>>     inbox = yes
> >>>     location =
> >>>     prefix = INBOX.
> >>>     separator = .
> >>>     type = private
> >>> }
> >>> passdb {
> >>>     args = /etc/dovecot/dovecot-ldap.conf
> >>>     driver = ldap
> >>> }
> >>> plugin {
> >>>     quota = maildir:Quota Utilisateur
> >>>     quota_exceeded_message = Cet utilisateur a dépassé son quota, votre
> >>> message n'a pu lui être livré.
> >>>     quota_full_tempfail = yes
> >>>     quota_rule = *:storage=1000M
> >>>     quota_rule2 = INBOX.Trash:storage=+100M
> >>>     sieve = ~/dovecot.sieve
> >>>     sieve_dir = ~/sieve
> >>>     sieve_extensions = -vacation
> >>>     sieve_global_dir = /var/lib/dovecot/sieve/global/
> >>>     sieve_max_redirects = 1
> >>>     zlib_save = gz
> >>>     zlib_save_level = 6
> >>> }
> >>> postmaster_address = postmaster@<snip>
> >>> protocols = imap pop3 sieve
> >>> replication_max_conns = 50
> >>> service auth {
> >>>     client_limit = 49452
> >>>     unix_listener auth-userdb {
> >>>       group = vmail
> >>>       mode = 0600
> >>>       user = vmail
> >>>     }
> >>>     user = root
> >>> }
> >>> service dict {
> >>>     unix_listener dict {
> >>>       mode = 0600
> >>>       user = vmail
> >>>     }
> >>> }
> >>> service doveadm {
> >>>     inet_listener {
> >>>       port = 12345
> >>>     }
> >>> }
> >>> service imap-login {
> >>>     process_min_avail = 8
> >>>     service_count = 0
> >>>     user = mail
> >>> }
> >>> service imap {
> >>>     executable = imap
> >>>     process_limit = 16384
> >>>     vsz_limit = 2 G
> >>> }
> >>> service managesieve-login {
> >>>     inet_listener sieve {
> >>>       port = 4190
> >>>     }
> >>>     process_min_avail = 8
> >>>     service_count = 0
> >>>     user = mail
> >>>     vsz_limit = 2 G
> >>> }
> >>> service managesieve {
> >>>     drop_priv_before_exec = yes
> >>>     process_limit = 16384
> >>> }
> >>> service pop3-login {
> >>>     process_min_avail = 8
> >>>     service_count = 0
> >>>     user = mail
> >>> }
> >>> service pop3 {
> >>>     drop_priv_before_exec = yes
> >>>     process_limit = 16384
> >>> }
> >>> ssl = no
> >>> userdb {
> >>>     args = /etc/dovecot/dovecot-ldap.conf
> >>>     driver = ldap
> >>> }
> >>> verbose_proctitle = yes
> >>> protocol imap {
> >>>     mail_max_userip_connections = 50
> >>>     mail_plugins = quota zlib notify replication imap_quota zlib
> >>> }
> >>> protocol pop3 {
> >>>     mail_plugins = quota zlib notify replication
> >>> }
> >>> protocol sieve {
> >>>     mail_max_userip_connections = 10
> >>> }
> >>> protocol lda {
> >>>     mail_plugins = quota zlib notify replication sieve zlib
> >>> }
> >>>
> >>>
> >>>
> >>>
> >>
> >> -- 
> >> Arnaud Abélard
> >> Responsable pôle Système et Stockage
> >> Service Infrastructures
> >> DSIN Université de Nantes
> >> -
> 
> -- 
> Arnaud Abélard
> Responsable pôle Système et Stockage
> Service Infrastructures
> DSIN Université de Nantes
> -


More information about the dovecot mailing list