after replication with compression quotas are wrong

[Aki Tuomi]

This sounds like you are not normalizing usernames properly.

Either use

auth_username_format = %Ln

as global setting, or return `user` attribute in both userdb and passdb lookups. 

Aki

> On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
> 
>  
> Hello again,
> 
> I've found out that some mailboxes  are actually duplicated. Doveadm 
> replicator status on the production server gives this:
> 
> ~# doveadm replicator status 'dummy-c-1*'
> username                              priority fast sync full sync 
> success sync failed
> dummy-c-1                             none     01:13:19  01:13:19 
> 01:13:19     -
> dummy-c-1 at univ-nantes.fr              none     00:15:28  00:15:28 
> 00:15:28     -
> 
> That'd explain why mails are counted twice when replicated on the new 
> server but where does this come from since I don't have this quota 
> problem on the production server?
> 
>  From the logs, it seems that postfix uses username at univ-nantes.fr when 
> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' 
> mailboxes are the erroneous ones.
> 
> The users aren't duplicated in our ldap user backend and aren't using 
> the @univ-nantes.fr part.
> 
> ~# doveadm user 'dummy-c-1*'
> dummy-c-1
> 
> Any ideas?
> 
> Thanks,
> 
> Arnaud
> 
> 
> 
> 
> 
> On 05/11/2021 16:21, Arnaud Abélard wrote:
> > Hello,
> > 
> > We are very long time happy dovecot users (since 2008 at least). We have 
> > around 90k mailboxes and since we had to move away from our NAS storage 
> > to a ceph storage I jumped on the opportunity to enable compression with 
> > the zlib plugin and dovecot's replication mecanism. We are using 
> > debian's dovecot 2.2.27 packages on production and our new server is 
> > running dovecot's own ce-2.3.17 packages.
> > 
> > On the production server everything works fine but on the new server, 
> > replicated mailboxes' quota is all wrong:
> > 
> > on production:
> > # doveadm quota get -u dummy-c-1
> > Quota name        Type     Value  Limit                             %
> > Quota Utilisateur STORAGE 660026 976563                            67
> > Quota Utilisateur MESSAGE   8651      -                             0
> > 
> > on new server:
> > doveadm quota get -u dummy-c-1
> > Quota name        Type      Value  Limit                           %
> > Quota Utilisateur STORAGE 1125251 976563                         115
> > Quota Utilisateur MESSAGE   16646      -                           0
> > 
> > If I add all the S flag from the filenames n both servers I get exactly 
> > the same usage, which is coherent with the quota on the production server:
> > 
> > # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' 
> > '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
> > 675865938
> > 
> > And I have exactly the same amountof mails on the two server, the 
> > replication works as expected, no unwanted duplication of mails occurs.
> > 
> > Of course, I've tried to ask dovecot to recalculate quotas with doveadm 
> > quota recalc -u <username>, but it doesn't fix the problem.
> > 
> > What am I missing?
> > 
> > Thanks,
> > 
> > Arnaud
> > 
> > PS: Here is my doveconf -n output:
> > 
> > # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
> > # Pigeonhole version 0.5.17 (054dddfa)
> > # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
> > # Hostname: gromel-test
> > auth_cache_size = 10 k
> > auth_verbose = yes
> > disable_plaintext_auth = no
> > doveadm_password = # hidden, use -P to show it
> > hostname = gromel1.univ-nantes.prive
> > lda_mailbox_autosubscribe = yes
> > listen = *
> > log_timestamp = "%Y-%m-%d %H:%M:%S "
> > login_trusted_networks = (...)
> > mail_gid = 5000
> > mail_location = maildir:%h
> > mail_plugins = quota zlib notify replication
> > mail_privileged_group = vmail
> > mail_uid = 5000
> > maildir_stat_dirs = yes
> > maildir_very_dirty_syncs = yes
> > managesieve_notify_capability = mailto
> > managesieve_sieve_capability = fileinto reject envelope 
> > encoded-character subaddress comparator-i;ascii-numeric relational regex 
> > imap4flags copy include variables body enotify environment mailbox date 
> > index ihave duplicate mime foreverypart extracttext
> > namespace {
> >    inbox = yes
> >    location =
> >    prefix = INBOX.
> >    separator = .
> >    type = private
> > }
> > passdb {
> >    args = /etc/dovecot/dovecot-ldap.conf
> >    driver = ldap
> > }
> > plugin {
> >    quota = maildir:Quota Utilisateur
> >    quota_exceeded_message = Cet utilisateur a dépassé son quota, votre 
> > message n'a pu lui être livré.
> >    quota_full_tempfail = yes
> >    quota_rule = *:storage=1000M
> >    quota_rule2 = INBOX.Trash:storage=+100M
> >    sieve = ~/dovecot.sieve
> >    sieve_dir = ~/sieve
> >    sieve_extensions = -vacation
> >    sieve_global_dir = /var/lib/dovecot/sieve/global/
> >    sieve_max_redirects = 1
> >    zlib_save = gz
> >    zlib_save_level = 6
> > }
> > postmaster_address = postmaster@<snip>
> > protocols = imap pop3 sieve
> > replication_max_conns = 50
> > service auth {
> >    client_limit = 49452
> >    unix_listener auth-userdb {
> >      group = vmail
> >      mode = 0600
> >      user = vmail
> >    }
> >    user = root
> > }
> > service dict {
> >    unix_listener dict {
> >      mode = 0600
> >      user = vmail
> >    }
> > }
> > service doveadm {
> >    inet_listener {
> >      port = 12345
> >    }
> > }
> > service imap-login {
> >    process_min_avail = 8
> >    service_count = 0
> >    user = mail
> > }
> > service imap {
> >    executable = imap
> >    process_limit = 16384
> >    vsz_limit = 2 G
> > }
> > service managesieve-login {
> >    inet_listener sieve {
> >      port = 4190
> >    }
> >    process_min_avail = 8
> >    service_count = 0
> >    user = mail
> >    vsz_limit = 2 G
> > }
> > service managesieve {
> >    drop_priv_before_exec = yes
> >    process_limit = 16384
> > }
> > service pop3-login {
> >    process_min_avail = 8
> >    service_count = 0
> >    user = mail
> > }
> > service pop3 {
> >    drop_priv_before_exec = yes
> >    process_limit = 16384
> > }
> > ssl = no
> > userdb {
> >    args = /etc/dovecot/dovecot-ldap.conf
> >    driver = ldap
> > }
> > verbose_proctitle = yes
> > protocol imap {
> >    mail_max_userip_connections = 50
> >    mail_plugins = quota zlib notify replication imap_quota zlib
> > }
> > protocol pop3 {
> >    mail_plugins = quota zlib notify replication
> > }
> > protocol sieve {
> >    mail_max_userip_connections = 10
> > }
> > protocol lda {
> >    mail_plugins = quota zlib notify replication sieve zlib
> > }
> > 
> > 
> > 
> > 
> 
> -- 
> Arnaud Abélard
> Responsable pôle Système et Stockage
> Service Infrastructures
> DSIN Université de Nantes
> -