after replication with compression quotas are wrong
Arnaud Abélard
arnaud.abelard at univ-nantes.fr
Mon Nov 8 09:32:40 UTC 2021
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use
the domain part anymore and added "auth_username_format = %Ln" just in
case (and reloaded dovecot). So I shouldn't have any more new
user at domain users but for all the existing users, doveadm replicator
status still show user at domain clone and on the new server mails are
still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't
think it's coming from there.
Is there a local cache that still has a list of those user at domain
usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
> This sounds like you are not normalizing usernames properly.
>
> Either use
>
> auth_username_format = %Ln
>
> as global setting, or return `user` attribute in both userdb and passdb lookups.
>
> Aki
>
>> On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard at univ-nantes.fr> wrote:
>>
>>
>> Hello again,
>>
>> I've found out that some mailboxes are actually duplicated. Doveadm
>> replicator status on the production server gives this:
>>
>> ~# doveadm replicator status 'dummy-c-1*'
>> username priority fast sync full sync
>> success sync failed
>> dummy-c-1 none 01:13:19 01:13:19
>> 01:13:19 -
>> dummy-c-1 at univ-nantes.fr none 00:15:28 00:15:28
>> 00:15:28 -
>>
>> That'd explain why mails are counted twice when replicated on the new
>> server but where does this come from since I don't have this quota
>> problem on the production server?
>>
>> From the logs, it seems that postfix uses username at univ-nantes.fr when
>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr'
>> mailboxes are the erroneous ones.
>>
>> The users aren't duplicated in our ldap user backend and aren't using
>> the @univ-nantes.fr part.
>>
>> ~# doveadm user 'dummy-c-1*'
>> dummy-c-1
>>
>> Any ideas?
>>
>> Thanks,
>>
>> Arnaud
>>
>>
>>
>>
>>
>> On 05/11/2021 16:21, Arnaud Abélard wrote:
>>> Hello,
>>>
>>> We are very long time happy dovecot users (since 2008 at least). We have
>>> around 90k mailboxes and since we had to move away from our NAS storage
>>> to a ceph storage I jumped on the opportunity to enable compression with
>>> the zlib plugin and dovecot's replication mecanism. We are using
>>> debian's dovecot 2.2.27 packages on production and our new server is
>>> running dovecot's own ce-2.3.17 packages.
>>>
>>> On the production server everything works fine but on the new server,
>>> replicated mailboxes' quota is all wrong:
>>>
>>> on production:
>>> # doveadm quota get -u dummy-c-1
>>> Quota name Type Value Limit %
>>> Quota Utilisateur STORAGE 660026 976563 67
>>> Quota Utilisateur MESSAGE 8651 - 0
>>>
>>> on new server:
>>> doveadm quota get -u dummy-c-1
>>> Quota name Type Value Limit %
>>> Quota Utilisateur STORAGE 1125251 976563 115
>>> Quota Utilisateur MESSAGE 16646 - 0
>>>
>>> If I add all the S flag from the filenames n both servers I get exactly
>>> the same usage, which is coherent with the quota on the production server:
>>>
>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F','
>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l
>>> 675865938
>>>
>>> And I have exactly the same amountof mails on the two server, the
>>> replication works as expected, no unwanted duplication of mails occurs.
>>>
>>> Of course, I've tried to ask dovecot to recalculate quotas with doveadm
>>> quota recalc -u <username>, but it doesn't fix the problem.
>>>
>>> What am I missing?
>>>
>>> Thanks,
>>>
>>> Arnaud
>>>
>>> PS: Here is my doveconf -n output:
>>>
>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf
>>> # Pigeonhole version 0.5.17 (054dddfa)
>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11
>>> # Hostname: gromel-test
>>> auth_cache_size = 10 k
>>> auth_verbose = yes
>>> disable_plaintext_auth = no
>>> doveadm_password = # hidden, use -P to show it
>>> hostname = gromel1.univ-nantes.prive
>>> lda_mailbox_autosubscribe = yes
>>> listen = *
>>> log_timestamp = "%Y-%m-%d %H:%M:%S "
>>> login_trusted_networks = (...)
>>> mail_gid = 5000
>>> mail_location = maildir:%h
>>> mail_plugins = quota zlib notify replication
>>> mail_privileged_group = vmail
>>> mail_uid = 5000
>>> maildir_stat_dirs = yes
>>> maildir_very_dirty_syncs = yes
>>> managesieve_notify_capability = mailto
>>> managesieve_sieve_capability = fileinto reject envelope
>>> encoded-character subaddress comparator-i;ascii-numeric relational regex
>>> imap4flags copy include variables body enotify environment mailbox date
>>> index ihave duplicate mime foreverypart extracttext
>>> namespace {
>>> inbox = yes
>>> location =
>>> prefix = INBOX.
>>> separator = .
>>> type = private
>>> }
>>> passdb {
>>> args = /etc/dovecot/dovecot-ldap.conf
>>> driver = ldap
>>> }
>>> plugin {
>>> quota = maildir:Quota Utilisateur
>>> quota_exceeded_message = Cet utilisateur a dépassé son quota, votre
>>> message n'a pu lui être livré.
>>> quota_full_tempfail = yes
>>> quota_rule = *:storage=1000M
>>> quota_rule2 = INBOX.Trash:storage=+100M
>>> sieve = ~/dovecot.sieve
>>> sieve_dir = ~/sieve
>>> sieve_extensions = -vacation
>>> sieve_global_dir = /var/lib/dovecot/sieve/global/
>>> sieve_max_redirects = 1
>>> zlib_save = gz
>>> zlib_save_level = 6
>>> }
>>> postmaster_address = postmaster@<snip>
>>> protocols = imap pop3 sieve
>>> replication_max_conns = 50
>>> service auth {
>>> client_limit = 49452
>>> unix_listener auth-userdb {
>>> group = vmail
>>> mode = 0600
>>> user = vmail
>>> }
>>> user = root
>>> }
>>> service dict {
>>> unix_listener dict {
>>> mode = 0600
>>> user = vmail
>>> }
>>> }
>>> service doveadm {
>>> inet_listener {
>>> port = 12345
>>> }
>>> }
>>> service imap-login {
>>> process_min_avail = 8
>>> service_count = 0
>>> user = mail
>>> }
>>> service imap {
>>> executable = imap
>>> process_limit = 16384
>>> vsz_limit = 2 G
>>> }
>>> service managesieve-login {
>>> inet_listener sieve {
>>> port = 4190
>>> }
>>> process_min_avail = 8
>>> service_count = 0
>>> user = mail
>>> vsz_limit = 2 G
>>> }
>>> service managesieve {
>>> drop_priv_before_exec = yes
>>> process_limit = 16384
>>> }
>>> service pop3-login {
>>> process_min_avail = 8
>>> service_count = 0
>>> user = mail
>>> }
>>> service pop3 {
>>> drop_priv_before_exec = yes
>>> process_limit = 16384
>>> }
>>> ssl = no
>>> userdb {
>>> args = /etc/dovecot/dovecot-ldap.conf
>>> driver = ldap
>>> }
>>> verbose_proctitle = yes
>>> protocol imap {
>>> mail_max_userip_connections = 50
>>> mail_plugins = quota zlib notify replication imap_quota zlib
>>> }
>>> protocol pop3 {
>>> mail_plugins = quota zlib notify replication
>>> }
>>> protocol sieve {
>>> mail_max_userip_connections = 10
>>> }
>>> protocol lda {
>>> mail_plugins = quota zlib notify replication sieve zlib
>>> }
>>>
>>>
>>>
>>>
>>
>> --
>> Arnaud Abélard
>> Responsable pôle Système et Stockage
>> Service Infrastructures
>> DSIN Université de Nantes
>> -
--
Arnaud Abélard
Responsable pôle Système et Stockage
Service Infrastructures
DSIN Université de Nantes
-
More information about the dovecot
mailing list