after replication with compression quotas are wrong
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr when calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return user attribute in both userdb and passdb lookups.
Aki
On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr when calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return
userattribute in both userdb and passdb lookups.Aki
On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr when calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
Aki
On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return
userattribute in both userdb and passdb lookups.Aki
On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr when calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
I did so on the source server, checked indeed that dummy-c-1@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return
userattribute in both userdb and passdb lookups.Aki
On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr when calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
Recalculation won't fix replication mistakes.
Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server?
You might also get something useful out of
doveadm -D quota recalc|get -u user
Aki
On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
I did so on the source server, checked indeed that dummy-c-1@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return
userattribute in both userdb and passdb lookups.Aki
On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr when calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote:
Hello,
We are very long time happy dovecot users (since 2008 at least). We have around 90k mailboxes and since we had to move away from our NAS storage to a ceph storage I jumped on the opportunity to enable compression with the zlib plugin and dovecot's replication mecanism. We are using debian's dovecot 2.2.27 packages on production and our new server is running dovecot's own ce-2.3.17 packages.
On the production server everything works fine but on the new server, replicated mailboxes' quota is all wrong:
on production: # doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 660026 976563 67 Quota Utilisateur MESSAGE 8651 - 0
on new server: doveadm quota get -u dummy-c-1 Quota name Type Value Limit % Quota Utilisateur STORAGE 1125251 976563 115 Quota Utilisateur MESSAGE 16646 - 0
If I add all the S flag from the filenames n both servers I get exactly the same usage, which is coherent with the quota on the production server:
# find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l 675865938
And I have exactly the same amountof mails on the two server, the replication works as expected, no unwanted duplication of mails occurs.
Of course, I've tried to ask dovecot to recalculate quotas with doveadm quota recalc -u <username>, but it doesn't fix the problem.
What am I missing?
Thanks,
Arnaud
PS: Here is my doveconf -n output:
# 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 # Hostname: gromel-test auth_cache_size = 10 k auth_verbose = yes disable_plaintext_auth = no doveadm_password = # hidden, use -P to show it hostname = gromel1.univ-nantes.prive lda_mailbox_autosubscribe = yes listen = * log_timestamp = "%Y-%m-%d %H:%M:%S " login_trusted_networks = (...) mail_gid = 5000 mail_location = maildir:%h mail_plugins = quota zlib notify replication mail_privileged_group = vmail mail_uid = 5000 maildir_stat_dirs = yes maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = prefix = INBOX. separator = . type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } plugin { quota = maildir:Quota Utilisateur quota_exceeded_message = Cet utilisateur a dépassé son quota, votre message n'a pu lui être livré. quota_full_tempfail = yes quota_rule = *:storage=1000M quota_rule2 = INBOX.Trash:storage=+100M sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_extensions = -vacation sieve_global_dir = /var/lib/dovecot/sieve/global/ sieve_max_redirects = 1 zlib_save = gz zlib_save_level = 6 } postmaster_address = postmaster@<snip> protocols = imap pop3 sieve replication_max_conns = 50 service auth { client_limit = 49452 unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service dict { unix_listener dict { mode = 0600 user = vmail } } service doveadm { inet_listener { port = 12345 } } service imap-login { process_min_avail = 8 service_count = 0 user = mail } service imap { executable = imap process_limit = 16384 vsz_limit = 2 G } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 8 service_count = 0 user = mail vsz_limit = 2 G } service managesieve { drop_priv_before_exec = yes process_limit = 16384 } service pop3-login { process_min_avail = 8 service_count = 0 user = mail } service pop3 { drop_priv_before_exec = yes process_limit = 16384 } ssl = no userdb { args = /etc/dovecot/dovecot-ldap.conf driver = ldap } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 50 mail_plugins = quota zlib notify replication imap_quota zlib } protocol pop3 { mail_plugins = quota zlib notify replication } protocol sieve { mail_max_userip_connections = 10 } protocol lda { mail_plugins = quota zlib notify replication sieve zlib }
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
On the target, I enabled the replication service without mail_replica and:
doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - -
It only knows of dummy-c-1, no trace of his evil twin dummy-c-1@univ-nantes.fr.
On the target, I do have the same number of files
find . -type f |wc -l 8705
which is around half of what the quota is reporting (plus de index files, etc):
~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=1000000000S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=1000000000 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=100000000 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota name Type Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Utilisateur MESSAGE 16686 - 0
The debug mode isn't much of any help here.
Arnaud
On 08/11/2021 11:30, Aki Tuomi wrote:
Recalculation won't fix replication mistakes.
Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server?
You might also get something useful out of
doveadm -D quota recalc|get -u userAki
On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
I did so on the source server, checked indeed that dummy-c-1@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return
userattribute in both userdb and passdb lookups.Aki
On 07/11/2021 20:31 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello again,
I've found out that some mailboxes are actually duplicated. Doveadm replicator status on the production server gives this:
~# doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none 01:13:19 01:13:19 01:13:19 - dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 00:15:28 -
That'd explain why mails are counted twice when replicated on the new server but where does this come from since I don't have this quota problem on the production server?
From the logs, it seems that postfix uses username@univ-nantes.fr whencalling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' mailboxes are the erroneous ones.
The users aren't duplicated in our ldap user backend and aren't using the @univ-nantes.fr part.
~# doveadm user 'dummy-c-1*' dummy-c-1
Any ideas?
Thanks,
Arnaud
On 05/11/2021 16:21, Arnaud Abélard wrote: > Hello, > > We are very long time happy dovecot users (since 2008 at least). We have > around 90k mailboxes and since we had to move away from our NAS storage > to a ceph storage I jumped on the opportunity to enable compression with > the zlib plugin and dovecot's replication mecanism. We are using > debian's dovecot 2.2.27 packages on production and our new server is > running dovecot's own ce-2.3.17 packages. > > On the production server everything works fine but on the new server, > replicated mailboxes' quota is all wrong: > > on production: > # doveadm quota get -u dummy-c-1 > Quota name Type Value Limit % > Quota Utilisateur STORAGE 660026 976563 67 > Quota Utilisateur MESSAGE 8651 - 0 > > on new server: > doveadm quota get -u dummy-c-1 > Quota name Type Value Limit % > Quota Utilisateur STORAGE 1125251 976563 115 > Quota Utilisateur MESSAGE 16646 - 0 > > If I add all the S flag from the filenames n both servers I get exactly > the same usage, which is coherent with the quota on the production server: > > # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' > '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l > 675865938 > > And I have exactly the same amountof mails on the two server, the > replication works as expected, no unwanted duplication of mails occurs. > > Of course, I've tried to ask dovecot to recalculate quotas with doveadm > quota recalc -u <username>, but it doesn't fix the problem. > > What am I missing? > > Thanks, > > Arnaud > > PS: Here is my doveconf -n output: > > # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf > # Pigeonhole version 0.5.17 (054dddfa) > # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 > # Hostname: gromel-test > auth_cache_size = 10 k > auth_verbose = yes > disable_plaintext_auth = no > doveadm_password = # hidden, use -P to show it > hostname = gromel1.univ-nantes.prive > lda_mailbox_autosubscribe = yes > listen = * > log_timestamp = "%Y-%m-%d %H:%M:%S " > login_trusted_networks = (...) > mail_gid = 5000 > mail_location = maildir:%h > mail_plugins = quota zlib notify replication > mail_privileged_group = vmail > mail_uid = 5000 > maildir_stat_dirs = yes > maildir_very_dirty_syncs = yes > managesieve_notify_capability = mailto > managesieve_sieve_capability = fileinto reject envelope > encoded-character subaddress comparator-i;ascii-numeric relational regex > imap4flags copy include variables body enotify environment mailbox date > index ihave duplicate mime foreverypart extracttext > namespace { > inbox = yes > location = > prefix = INBOX. > separator = . > type = private > } > passdb { > args = /etc/dovecot/dovecot-ldap.conf > driver = ldap > } > plugin { > quota = maildir:Quota Utilisateur > quota_exceeded_message = Cet utilisateur a dépassé son quota, votre > message n'a pu lui être livré. > quota_full_tempfail = yes > quota_rule = *:storage=1000M > quota_rule2 = INBOX.Trash:storage=+100M > sieve = ~/dovecot.sieve > sieve_dir = ~/sieve > sieve_extensions = -vacation > sieve_global_dir = /var/lib/dovecot/sieve/global/ > sieve_max_redirects = 1 > zlib_save = gz > zlib_save_level = 6 > } > postmaster_address = postmaster@<snip> > protocols = imap pop3 sieve > replication_max_conns = 50 > service auth { > client_limit = 49452 > unix_listener auth-userdb { > group = vmail > mode = 0600 > user = vmail > } > user = root > } > service dict { > unix_listener dict { > mode = 0600 > user = vmail > } > } > service doveadm { > inet_listener { > port = 12345 > } > } > service imap-login { > process_min_avail = 8 > service_count = 0 > user = mail > } > service imap { > executable = imap > process_limit = 16384 > vsz_limit = 2 G > } > service managesieve-login { > inet_listener sieve { > port = 4190 > } > process_min_avail = 8 > service_count = 0 > user = mail > vsz_limit = 2 G > } > service managesieve { > drop_priv_before_exec = yes > process_limit = 16384 > } > service pop3-login { > process_min_avail = 8 > service_count = 0 > user = mail > } > service pop3 { > drop_priv_before_exec = yes > process_limit = 16384 > } > ssl = no > userdb { > args = /etc/dovecot/dovecot-ldap.conf > driver = ldap > } > verbose_proctitle = yes > protocol imap { > mail_max_userip_connections = 50 > mail_plugins = quota zlib notify replication imap_quota zlib > } > protocol pop3 { > mail_plugins = quota zlib notify replication > } > protocol sieve { > mail_max_userip_connections = 10 > } > protocol lda { > mail_plugins = quota zlib notify replication sieve zlib > } > > > >
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
I've just removed dummy-c-1's mailbox completely on the target server, removed the quota plugin, forced a sync for dummy-c-1 only and once it was done reactivated the quota plugins: double usage again.
I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in order to have both server running the same version, without improvement.
I'm puzzled. Is dovecot storing anything outside of the user's mailbox? Like a cache, a sqlite database of some kind somewhere?
Arnaud
On 08/11/2021 11:48, Arnaud Abélard wrote:
On the target, I enabled the replication service without mail_replica and:
doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - -
It only knows of dummy-c-1, no trace of his evil twin dummy-c-1@univ-nantes.fr.
On the target, I do have the same number of files
find . -type f |wc -l 8705
which is around half of what the quota is reporting (plus de index files, etc):
~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=1000000000S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=1000000000 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=100000000 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota name Type Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Utilisateur MESSAGE 16686 - 0
The debug mode isn't much of any help here.
Arnaud
On 08/11/2021 11:30, Aki Tuomi wrote:
Recalculation won't fix replication mistakes.
Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server?
You might also get something useful out of
doveadm -D quota recalc|get -u userAki
On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
I did so on the source server, checked indeed that dummy-c-1@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote:
This sounds like you are not normalizing usernames properly.
Either use
auth_username_format = %Ln
as global setting, or return
userattribute in both userdb and passdb lookups.Aki
> On 07/11/2021 20:31 Arnaud Abélard > <arnaud.abelard@univ-nantes.fr> wrote: > > Hello again, > > I've found out that some mailboxes are actually duplicated. Doveadm > replicator status on the production server gives this: > > ~# doveadm replicator status 'dummy-c-1*' > username priority fast sync full sync > success sync failed > dummy-c-1 none 01:13:19 01:13:19 > 01:13:19 - > dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 > 00:15:28 - > > That'd explain why mails are counted twice when replicated on the > new > server but where does this come from since I don't have this quota > problem on the production server? > > From the logs, it seems that postfix uses > username@univ-nantes.fr when > calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' > mailboxes are the erroneous ones. > > The users aren't duplicated in our ldap user backend and aren't > using > the @univ-nantes.fr part. > > ~# doveadm user 'dummy-c-1*' > dummy-c-1 > > Any ideas? > > Thanks, > > Arnaud > > > > > > On 05/11/2021 16:21, Arnaud Abélard wrote: >> Hello, >> >> We are very long time happy dovecot users (since 2008 at least). >> We have >> around 90k mailboxes and since we had to move away from our NAS >> storage >> to a ceph storage I jumped on the opportunity to enable >> compression with >> the zlib plugin and dovecot's replication mecanism. We are using >> debian's dovecot 2.2.27 packages on production and our new >> server is >> running dovecot's own ce-2.3.17 packages. >> >> On the production server everything works fine but on the new >> server, >> replicated mailboxes' quota is all wrong: >> >> on production: >> # doveadm quota get -u dummy-c-1 >> Quota name Type Value
>> Limit % >> Quota Utilisateur STORAGE 660026 >> 976563 67 >> Quota Utilisateur MESSAGE 8651
>> - 0 >> >> on new server: >> doveadm quota get -u dummy-c-1 >> Quota name Type Value
>> Limit % >> Quota Utilisateur STORAGE 1125251 976563
>> 115 >> Quota Utilisateur MESSAGE 16646
>> - 0 >> >> If I add all the S flag from the filenames n both servers I get >> exactly >> the same usage, which is coherent with the quota on the >> production server: >> >> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' >> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l >> 675865938 >> >> And I have exactly the same amountof mails on the two server, the >> replication works as expected, no unwanted duplication of mails >> occurs. >> >> Of course, I've tried to ask dovecot to recalculate quotas with >> doveadm >> quota recalc -u <username>, but it doesn't fix the problem. >> >> What am I missing? >> >> Thanks, >> >> Arnaud >> >> PS: Here is my doveconf -n output: >> >> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf >> # Pigeonhole version 0.5.17 (054dddfa) >> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 >> # Hostname: gromel-test >> auth_cache_size = 10 k >> auth_verbose = yes >> disable_plaintext_auth = no >> doveadm_password = # hidden, use -P to show it >> hostname = gromel1.univ-nantes.prive >> lda_mailbox_autosubscribe = yes >> listen = * >> log_timestamp = "%Y-%m-%d %H:%M:%S " >> login_trusted_networks = (...) >> mail_gid = 5000 >> mail_location = maildir:%h >> mail_plugins = quota zlib notify replication >> mail_privileged_group = vmail >> mail_uid = 5000 >> maildir_stat_dirs = yes >> maildir_very_dirty_syncs = yes >> managesieve_notify_capability = mailto >> managesieve_sieve_capability = fileinto reject envelope >> encoded-character subaddress comparator-i;ascii-numeric >> relational regex >> imap4flags copy include variables body enotify environment >> mailbox date >> index ihave duplicate mime foreverypart extracttext >> namespace { >> inbox = yes >> location = >> prefix = INBOX. >> separator = . >> type = private >> } >> passdb { >> args = /etc/dovecot/dovecot-ldap.conf >> driver = ldap >> } >> plugin { >> quota = maildir:Quota Utilisateur >> quota_exceeded_message = Cet utilisateur a dépassé son >> quota, votre >> message n'a pu lui être livré. >> quota_full_tempfail = yes >> quota_rule = *:storage=1000M >> quota_rule2 = INBOX.Trash:storage=+100M >> sieve = ~/dovecot.sieve >> sieve_dir = ~/sieve >> sieve_extensions = -vacation >> sieve_global_dir = /var/lib/dovecot/sieve/global/ >> sieve_max_redirects = 1 >> zlib_save = gz >> zlib_save_level = 6 >> } >> postmaster_address = postmaster@<snip> >> protocols = imap pop3 sieve >> replication_max_conns = 50 >> service auth { >> client_limit = 49452 >> unix_listener auth-userdb { >> group = vmail >> mode = 0600 >> user = vmail >> } >> user = root >> } >> service dict { >> unix_listener dict { >> mode = 0600 >> user = vmail >> } >> } >> service doveadm { >> inet_listener { >> port = 12345 >> } >> } >> service imap-login { >> process_min_avail = 8 >> service_count = 0 >> user = mail >> } >> service imap { >> executable = imap >> process_limit = 16384 >> vsz_limit = 2 G >> } >> service managesieve-login { >> inet_listener sieve { >> port = 4190 >> } >> process_min_avail = 8 >> service_count = 0 >> user = mail >> vsz_limit = 2 G >> } >> service managesieve { >> drop_priv_before_exec = yes >> process_limit = 16384 >> } >> service pop3-login { >> process_min_avail = 8 >> service_count = 0 >> user = mail >> } >> service pop3 { >> drop_priv_before_exec = yes >> process_limit = 16384 >> } >> ssl = no >> userdb { >> args = /etc/dovecot/dovecot-ldap.conf >> driver = ldap >> } >> verbose_proctitle = yes >> protocol imap { >> mail_max_userip_connections = 50 >> mail_plugins = quota zlib notify replication imap_quota zlib >> } >> protocol pop3 { >> mail_plugins = quota zlib notify replication >> } >> protocol sieve { >> mail_max_userip_connections = 10 >> } >> protocol lda { >> mail_plugins = quota zlib notify replication sieve zlib >> } >> >> >> >> > > -- > Arnaud Abélard > Responsable pôle Système et Stockage > Service Infrastructures > DSIN Université de Nantes > --- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
On 10/11/2021 15:52, Arnaud Abélard wrote:
I've just removed dummy-c-1's mailbox completely on the target server, removed the quota plugin, forced a sync for dummy-c-1 only and once it was done reactivated the quota plugins: double usage again.
I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in order to have both server running the same version, without improvement.
Actually, I got that wrong, test server is running dovecot 2.3.4 and production server dovecot 2.2.7. I copied the dovecot.conf file from the production server to the test one, then rsync'ed the dummy-c-1 mailbox from the production server to the test server, restarted dovecot and the quota is still 115% after recalc. The only difference now is dovecot's version. Were there any changes related to quota on dovecot 2.3?
Thanks,
Arnaud
I'm puzzled. Is dovecot storing anything outside of the user's mailbox? Like a cache, a sqlite database of some kind somewhere?
Arnaud
On 08/11/2021 11:48, Arnaud Abélard wrote:
On the target, I enabled the replication service without mail_replica and:
doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - -
It only knows of dummy-c-1, no trace of his evil twin dummy-c-1@univ-nantes.fr.
On the target, I do have the same number of files
find . -type f |wc -l 8705
which is around half of what the quota is reporting (plus de index files, etc):
~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=1000000000S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=1000000000 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=100000000 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota name Type Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Utilisateur MESSAGE 16686 - 0
The debug mode isn't much of any help here.
Arnaud
On 08/11/2021 11:30, Aki Tuomi wrote:
Recalculation won't fix replication mistakes.
Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server?
You might also get something useful out of
doveadm -D quota recalc|get -u userAki
On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
I did so on the source server, checked indeed that dummy-c-1@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
On 08/11/2021 11:32 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
Hello Aki,
Thanks for the tip. I changed postfix configuration so it will not use the domain part anymore and added "auth_username_format = %Ln" just in case (and reloaded dovecot). So I shouldn't have any more new user@domain users but for all the existing users, doveadm replicator status still show user@domain clone and on the new server mails are still counted twice.
Our userdb and passdb don't have the @domain defined anywhere so I don't think it's coming from there.
Is there a local cache that still has a list of those user@domain usernames that I could reset or something?
Thanks,
Arnaud
On 08/11/2021 09:48, Aki Tuomi wrote: > This sounds like you are not normalizing usernames properly. > > Either use > > auth_username_format = %Ln > > as global setting, or return
userattribute in both userdb and > passdb lookups. > > Aki > >> On 07/11/2021 20:31 Arnaud Abélard >> <arnaud.abelard@univ-nantes.fr> wrote: >> >> Hello again, >> >> I've found out that some mailboxes are actually duplicated. >> Doveadm >> replicator status on the production server gives this: >> >> ~# doveadm replicator status 'dummy-c-1*' >> username priority fast sync full sync >> success sync failed >> dummy-c-1 none 01:13:19 01:13:19 >> 01:13:19 - >> dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 >> 00:15:28 - >> >> That'd explain why mails are counted twice when replicated on >> the new >> server but where does this come from since I don't have this quota >> problem on the production server? >> >> From the logs, it seems that postfix uses >> username@univ-nantes.fr when >> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' >> mailboxes are the erroneous ones. >> >> The users aren't duplicated in our ldap user backend and aren't >> using >> the @univ-nantes.fr part. >> >> ~# doveadm user 'dummy-c-1*' >> dummy-c-1 >> >> Any ideas? >> >> Thanks, >> >> Arnaud >> >> >> >> >> >> On 05/11/2021 16:21, Arnaud Abélard wrote: >>> Hello, >>> >>> We are very long time happy dovecot users (since 2008 at >>> least). We have >>> around 90k mailboxes and since we had to move away from our NAS >>> storage >>> to a ceph storage I jumped on the opportunity to enable >>> compression with >>> the zlib plugin and dovecot's replication mecanism. We are using >>> debian's dovecot 2.2.27 packages on production and our new >>> server is >>> running dovecot's own ce-2.3.17 packages. >>> >>> On the production server everything works fine but on the new >>> server, >>> replicated mailboxes' quota is all wrong: >>> >>> on production: >>> # doveadm quota get -u dummy-c-1 >>> Quota name Type Value >>> Limit % >>> Quota Utilisateur STORAGE 660026 >>> 976563 67 >>> Quota Utilisateur MESSAGE 8651 - 0 >>> >>> on new server: >>> doveadm quota get -u dummy-c-1 >>> Quota name Type Value >>> Limit % >>> Quota Utilisateur STORAGE 1125251 976563 115 >>> Quota Utilisateur MESSAGE 16646 - 0 >>> >>> If I add all the S flag from the filenames n both servers I get >>> exactly >>> the same usage, which is coherent with the quota on the >>> production server: >>> >>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk -F',' >>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l >>> 675865938 >>> >>> And I have exactly the same amountof mails on the two server, the >>> replication works as expected, no unwanted duplication of mails >>> occurs. >>> >>> Of course, I've tried to ask dovecot to recalculate quotas with >>> doveadm >>> quota recalc -u <username>, but it doesn't fix the problem. >>> >>> What am I missing? >>> >>> Thanks, >>> >>> Arnaud >>> >>> PS: Here is my doveconf -n output: >>> >>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf >>> # Pigeonhole version 0.5.17 (054dddfa) >>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 >>> # Hostname: gromel-test >>> auth_cache_size = 10 k >>> auth_verbose = yes >>> disable_plaintext_auth = no >>> doveadm_password = # hidden, use -P to show it >>> hostname = gromel1.univ-nantes.prive >>> lda_mailbox_autosubscribe = yes >>> listen = * >>> log_timestamp = "%Y-%m-%d %H:%M:%S " >>> login_trusted_networks = (...) >>> mail_gid = 5000 >>> mail_location = maildir:%h >>> mail_plugins = quota zlib notify replication >>> mail_privileged_group = vmail >>> mail_uid = 5000 >>> maildir_stat_dirs = yes >>> maildir_very_dirty_syncs = yes >>> managesieve_notify_capability = mailto >>> managesieve_sieve_capability = fileinto reject envelope >>> encoded-character subaddress comparator-i;ascii-numeric >>> relational regex >>> imap4flags copy include variables body enotify environment >>> mailbox date >>> index ihave duplicate mime foreverypart extracttext >>> namespace { >>> inbox = yes >>> location = >>> prefix = INBOX. >>> separator = . >>> type = private >>> } >>> passdb { >>> args = /etc/dovecot/dovecot-ldap.conf >>> driver = ldap >>> } >>> plugin { >>> quota = maildir:Quota Utilisateur >>> quota_exceeded_message = Cet utilisateur a dépassé son >>> quota, votre >>> message n'a pu lui être livré. >>> quota_full_tempfail = yes >>> quota_rule = *:storage=1000M >>> quota_rule2 = INBOX.Trash:storage=+100M >>> sieve = ~/dovecot.sieve >>> sieve_dir = ~/sieve >>> sieve_extensions = -vacation >>> sieve_global_dir = /var/lib/dovecot/sieve/global/ >>> sieve_max_redirects = 1 >>> zlib_save = gz >>> zlib_save_level = 6 >>> } >>> postmaster_address = postmaster@<snip> >>> protocols = imap pop3 sieve >>> replication_max_conns = 50 >>> service auth { >>> client_limit = 49452 >>> unix_listener auth-userdb { >>> group = vmail >>> mode = 0600 >>> user = vmail >>> } >>> user = root >>> } >>> service dict { >>> unix_listener dict { >>> mode = 0600 >>> user = vmail >>> } >>> } >>> service doveadm { >>> inet_listener { >>> port = 12345 >>> } >>> } >>> service imap-login { >>> process_min_avail = 8 >>> service_count = 0 >>> user = mail >>> } >>> service imap { >>> executable = imap >>> process_limit = 16384 >>> vsz_limit = 2 G >>> } >>> service managesieve-login { >>> inet_listener sieve { >>> port = 4190 >>> } >>> process_min_avail = 8 >>> service_count = 0 >>> user = mail >>> vsz_limit = 2 G >>> } >>> service managesieve { >>> drop_priv_before_exec = yes >>> process_limit = 16384 >>> } >>> service pop3-login { >>> process_min_avail = 8 >>> service_count = 0 >>> user = mail >>> } >>> service pop3 { >>> drop_priv_before_exec = yes >>> process_limit = 16384 >>> } >>> ssl = no >>> userdb { >>> args = /etc/dovecot/dovecot-ldap.conf >>> driver = ldap >>> } >>> verbose_proctitle = yes >>> protocol imap { >>> mail_max_userip_connections = 50 >>> mail_plugins = quota zlib notify replication imap_quota zlib >>> } >>> protocol pop3 { >>> mail_plugins = quota zlib notify replication >>> } >>> protocol sieve { >>> mail_max_userip_connections = 10 >>> } >>> protocol lda { >>> mail_plugins = quota zlib notify replication sieve zlib >>> } >>> >>> >>> >>> >> >> -- >> Arnaud Abélard >> Responsable pôle Système et Stockage >> Service Infrastructures >> DSIN Université de Nantes >> --- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
Ah! At last, after comparing all default values using doveconf I found the culprit: mailbox_list_index. Default is "no" on dovecot 2.2.7, but "yes" on dovecot 2.3.4 (on debian, at least). Switching mailbox_list_index to "no" fixes my quota miscalculation problem.
Arnaud
On 12/11/2021 20:09, Arnaud Abélard wrote:
On 10/11/2021 15:52, Arnaud Abélard wrote:
I've just removed dummy-c-1's mailbox completely on the target server, removed the quota plugin, forced a sync for dummy-c-1 only and once it was done reactivated the quota plugins: double usage again.
I also downgraded from dovecot 2.3.17 to 2.2.27 on the target server in order to have both server running the same version, without improvement.
Actually, I got that wrong, test server is running dovecot 2.3.4 and production server dovecot 2.2.7. I copied the dovecot.conf file from the production server to the test one, then rsync'ed the dummy-c-1 mailbox from the production server to the test server, restarted dovecot and the quota is still 115% after recalc. The only difference now is dovecot's version. Were there any changes related to quota on dovecot 2.3?
Thanks,
Arnaud
I'm puzzled. Is dovecot storing anything outside of the user's mailbox? Like a cache, a sqlite database of some kind somewhere?
Arnaud
On 08/11/2021 11:48, Arnaud Abélard wrote:
On the target, I enabled the replication service without mail_replica and:
doveadm replicator status 'dummy-c-1*' username priority fast sync full sync success sync failed dummy-c-1 none - - - -
It only knows of dummy-c-1, no trace of his evil twin dummy-c-1@univ-nantes.fr.
On the target, I do have the same number of files
find . -type f |wc -l 8705
which is around half of what the quota is reporting (plus de index files, etc):
~# doveadm -D quota get -u dummy-c-1 Debug: Loading modules from directory: /usr/lib/dovecot/modules Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_replication_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(dummy-c-1)<24051><>: Debug: auth USER input: dummy-c-1 home=/vmail/d/u/dummy-c-1/ quota_rule=*:backend=1000000000S doveadm(dummy-c-1)<24051><>: Debug: Added userdb setting: plugin/quota_rule=*:backend=1000000000S doveadm(dummy-c-1): Debug: Effective uid=5000, gid=5000, home=/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: Quota root: name=Quota Utilisateur backend=maildir args= doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=* bytes=1000000000 messages=0 doveadm(dummy-c-1): Debug: Quota rule: root=Quota Utilisateur mailbox=INBOX.Trash bytes=+104857600 messages=0 doveadm(dummy-c-1): Debug: Quota grace: root=Quota Utilisateur bytes=100000000 (10%) doveadm(dummy-c-1): Debug: replication: No mail_replica setting - replication disabled doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=INBOX., sep=., inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/vmail/d/u/dummy-c-1/ doveadm(dummy-c-1): Debug: maildir++: root=/vmail/d/u/dummy-c-1, index=, indexpvt=, control=, inbox=/vmail/d/u/dummy-c-1, alt= doveadm(dummy-c-1): Debug: Namespace : type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, subscriptions=no location=fail::LAYOUT=none doveadm(dummy-c-1): Debug: none: root=, index=, indexpvt=, control=, inbox=, alt= doveadm(dummy-c-1): Debug: quota: quota_over_flag check: quota_over_script unset - skipping Quota name Type Value Limit % Quota Utilisateur STORAGE 1126751 976563 115 Quota Utilisateur MESSAGE 16686 - 0
The debug mode isn't much of any help here.
Arnaud
On 08/11/2021 11:30, Aki Tuomi wrote:
Recalculation won't fix replication mistakes.
Did you ensure on the *target* server that it has only dummy-c-1 in replication and that the file count on the target server matches source server?
You might also get something useful out of
doveadm -D quota recalc|get -u userAki
On 08/11/2021 12:22 Arnaud Abélard <arnaud.abelard@univ-nantes.fr> wrote:
On 08/11/2021 10:39, Aki Tuomi wrote:
Try doveadm replicator remove dummy-c-1@univ-nantes.fr
I did so on the source server, checked indeed that dummy-c-1@univ-nantes.fr wasn't showing up in the doveadm replicator status list and it was indeed the case, so far so good. On the destination server, dummy-c-1's mailbox was still showing a 115% quota so I tried to make dovecot recalculate the quota but it did not change anything. I ended up deleting the user's mailbox and forced a replication and, to my surprise, the new mailbox still has a 115% quota.
Arnaud
Aki
> On 08/11/2021 11:32 Arnaud Abélard > <arnaud.abelard@univ-nantes.fr> wrote: > > Hello Aki, > > Thanks for the tip. I changed postfix configuration so it will > not use > the domain part anymore and added "auth_username_format = %Ln" > just in > case (and reloaded dovecot). So I shouldn't have any more new > user@domain users but for all the existing users, doveadm replicator > status still show user@domain clone and on the new server mails are > still counted twice. > > Our userdb and passdb don't have the @domain defined anywhere so > I don't > think it's coming from there. > > Is there a local cache that still has a list of those user@domain > usernames that I could reset or something? > > Thanks, > > Arnaud > > > > On 08/11/2021 09:48, Aki Tuomi wrote: >> This sounds like you are not normalizing usernames properly. >> >> Either use >> >> auth_username_format = %Ln >> >> as global setting, or return
userattribute in both userdb and >> passdb lookups. >> >> Aki >> >>> On 07/11/2021 20:31 Arnaud Abélard >>> <arnaud.abelard@univ-nantes.fr> wrote: >>> >>> Hello again, >>> >>> I've found out that some mailboxes are actually duplicated. >>> Doveadm >>> replicator status on the production server gives this: >>> >>> ~# doveadm replicator status 'dummy-c-1*' >>> username priority fast sync full sync >>> success sync failed >>> dummy-c-1 none 01:13:19 01:13:19 >>> 01:13:19 - >>> dummy-c-1@univ-nantes.fr none 00:15:28 00:15:28 >>> 00:15:28 - >>> >>> That'd explain why mails are counted twice when replicated on >>> the new >>> server but where does this come from since I don't have this quota >>> problem on the production server? >>> >>> From the logs, it seems that postfix uses >>> username@univ-nantes.fr when >>> calling dovecot's LDA so I suppose that the non-'@univ-nantes.fr' >>> mailboxes are the erroneous ones. >>> >>> The users aren't duplicated in our ldap user backend and aren't >>> using >>> the @univ-nantes.fr part. >>> >>> ~# doveadm user 'dummy-c-1*' >>> dummy-c-1 >>> >>> Any ideas? >>> >>> Thanks, >>> >>> Arnaud >>> >>> >>> >>> >>> >>> On 05/11/2021 16:21, Arnaud Abélard wrote: >>>> Hello, >>>> >>>> We are very long time happy dovecot users (since 2008 at >>>> least). We have >>>> around 90k mailboxes and since we had to move away from our >>>> NAS storage >>>> to a ceph storage I jumped on the opportunity to enable >>>> compression with >>>> the zlib plugin and dovecot's replication mecanism. We are using >>>> debian's dovecot 2.2.27 packages on production and our new >>>> server is >>>> running dovecot's own ce-2.3.17 packages. >>>> >>>> On the production server everything works fine but on the new >>>> server, >>>> replicated mailboxes' quota is all wrong: >>>> >>>> on production: >>>> # doveadm quota get -u dummy-c-1 >>>> Quota name Type Value >>>> Limit % >>>> Quota Utilisateur STORAGE 660026 >>>> 976563 67 >>>> Quota Utilisateur MESSAGE 8651 - 0 >>>> >>>> on new server: >>>> doveadm quota get -u dummy-c-1 >>>> Quota name Type Value >>>> Limit % >>>> Quota Utilisateur STORAGE 1125251 976563 115 >>>> Quota Utilisateur MESSAGE 16646 - 0 >>>> >>>> If I add all the S flag from the filenames n both servers I >>>> get exactly >>>> the same usage, which is coherent with the quota on the >>>> production server: >>>> >>>> # find . -type f | grep 'S=' | awk -F'S=' '{print $2}' | awk >>>> -F',' >>>> '{print $1}' | awk -F':' '{print $1}' | paste -sd+ | bc -l >>>> 675865938 >>>> >>>> And I have exactly the same amountof mails on the two server, the >>>> replication works as expected, no unwanted duplication of >>>> mails occurs. >>>> >>>> Of course, I've tried to ask dovecot to recalculate quotas >>>> with doveadm >>>> quota recalc -u <username>, but it doesn't fix the problem. >>>> >>>> What am I missing? >>>> >>>> Thanks, >>>> >>>> Arnaud >>>> >>>> PS: Here is my doveconf -n output: >>>> >>>> # 2.3.17 (e2aa53df5b): /etc/dovecot/dovecot.conf >>>> # Pigeonhole version 0.5.17 (054dddfa) >>>> # OS: Linux 4.19.0-18-amd64 x86_64 Debian 10.11 >>>> # Hostname: gromel-test >>>> auth_cache_size = 10 k >>>> auth_verbose = yes >>>> disable_plaintext_auth = no >>>> doveadm_password = # hidden, use -P to show it >>>> hostname = gromel1.univ-nantes.prive >>>> lda_mailbox_autosubscribe = yes >>>> listen = * >>>> log_timestamp = "%Y-%m-%d %H:%M:%S " >>>> login_trusted_networks = (...) >>>> mail_gid = 5000 >>>> mail_location = maildir:%h >>>> mail_plugins = quota zlib notify replication >>>> mail_privileged_group = vmail >>>> mail_uid = 5000 >>>> maildir_stat_dirs = yes >>>> maildir_very_dirty_syncs = yes >>>> managesieve_notify_capability = mailto >>>> managesieve_sieve_capability = fileinto reject envelope >>>> encoded-character subaddress comparator-i;ascii-numeric >>>> relational regex >>>> imap4flags copy include variables body enotify environment >>>> mailbox date >>>> index ihave duplicate mime foreverypart extracttext >>>> namespace { >>>> inbox = yes >>>> location = >>>> prefix = INBOX. >>>> separator = . >>>> type = private >>>> } >>>> passdb { >>>> args = /etc/dovecot/dovecot-ldap.conf >>>> driver = ldap >>>> } >>>> plugin { >>>> quota = maildir:Quota Utilisateur >>>> quota_exceeded_message = Cet utilisateur a dépassé son >>>> quota, votre >>>> message n'a pu lui être livré. >>>> quota_full_tempfail = yes >>>> quota_rule = *:storage=1000M >>>> quota_rule2 = INBOX.Trash:storage=+100M >>>> sieve = ~/dovecot.sieve >>>> sieve_dir = ~/sieve >>>> sieve_extensions = -vacation >>>> sieve_global_dir = /var/lib/dovecot/sieve/global/ >>>> sieve_max_redirects = 1 >>>> zlib_save = gz >>>> zlib_save_level = 6 >>>> } >>>> postmaster_address = postmaster@<snip> >>>> protocols = imap pop3 sieve >>>> replication_max_conns = 50 >>>> service auth { >>>> client_limit = 49452 >>>> unix_listener auth-userdb { >>>> group = vmail >>>> mode = 0600 >>>> user = vmail >>>> } >>>> user = root >>>> } >>>> service dict { >>>> unix_listener dict { >>>> mode = 0600 >>>> user = vmail >>>> } >>>> } >>>> service doveadm { >>>> inet_listener { >>>> port = 12345 >>>> } >>>> } >>>> service imap-login { >>>> process_min_avail = 8 >>>> service_count = 0 >>>> user = mail >>>> } >>>> service imap { >>>> executable = imap >>>> process_limit = 16384 >>>> vsz_limit = 2 G >>>> } >>>> service managesieve-login { >>>> inet_listener sieve { >>>> port = 4190 >>>> } >>>> process_min_avail = 8 >>>> service_count = 0 >>>> user = mail >>>> vsz_limit = 2 G >>>> } >>>> service managesieve { >>>> drop_priv_before_exec = yes >>>> process_limit = 16384 >>>> } >>>> service pop3-login { >>>> process_min_avail = 8 >>>> service_count = 0 >>>> user = mail >>>> } >>>> service pop3 { >>>> drop_priv_before_exec = yes >>>> process_limit = 16384 >>>> } >>>> ssl = no >>>> userdb { >>>> args = /etc/dovecot/dovecot-ldap.conf >>>> driver = ldap >>>> } >>>> verbose_proctitle = yes >>>> protocol imap { >>>> mail_max_userip_connections = 50 >>>> mail_plugins = quota zlib notify replication imap_quota >>>> zlib >>>> } >>>> protocol pop3 { >>>> mail_plugins = quota zlib notify replication >>>> } >>>> protocol sieve { >>>> mail_max_userip_connections = 10 >>>> } >>>> protocol lda { >>>> mail_plugins = quota zlib notify replication sieve zlib >>>> } >>>> >>>> >>>> >>>> >>> >>> -- >>> Arnaud Abélard >>> Responsable pôle Système et Stockage >>> Service Infrastructures >>> DSIN Université de Nantes >>> - > > -- > Arnaud Abélard > Responsable pôle Système et Stockage > Service Infrastructures > DSIN Université de Nantes > --- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
-- Arnaud Abélard Responsable pôle Système et Stockage Service Infrastructures DSIN Université de Nantes
participants (2)
-
Aki Tuomi
-
Arnaud Abélard