- dovecot - dovecot.org

Confused about dovecot ACL
by HTMLServices.it 12 Jul '19

12 Jul '19

Hello everyone sorry i'm not very experienced and also my english I installed a centos 7 server with ispconfig postfix dovecot 2.2.36 and roundcube, this server is only a mail archive, so my need is that ALL the mailboxes are read-only on roundcube/imap and any user must NOT delete the messages. ... so I configured dovecot's ALC following the guide "https://wiki2.dovecot.org/ACL" I think I did everything correctly and I don't get errors but entering the webmail roundcube I CAN DELETE MESSAGES ..... it seems that the acl have no effect .... added to the dovecot configuration file /etc/dovecot/dovecot.conf ______________________________________ .... plugin { acl = vfile: / etc / dovecot / dovecot-acl } ..... protocol imap { mail_plugins = $ mail_plugins imap_acl } mail_plugins = acl ..... ______________________________________ created the "global" file /etc/dovecot/dovecot-acl and inserted that the test user has only lookup and reading rights (lr): * user=test(a)test.com lr ______________________________________ my dovecot.conf ********************************* listen = *,[::] protocols = imap pop3 auth_mechanisms = plain login disable_plaintext_auth = no log_timestamp = "%Y-%m-%d %H:%M:%S " mail_privileged_group = vmail ssl_cert = </etc/postfix/smtpd.cert ssl_key = </etc/postfix/smtpd.key ssl_protocols = !SSLv3 passdb { args = /etc/dovecot-sql.conf driver = sql } userdb { driver = prefetch } userdb { args = /etc/dovecot-sql.conf driver = sql } plugin { acl = vfile:/etc/dovecot/dovecot-acl quota = dict:user::file:/var/vmail/%d/%n/.quotausage sieve=/var/vmail/%d/%n/.sieve } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service imap-login { client_limit = 1000 process_limit = 500 } protocol imap { mail_plugins = $mail_plugins imap_acl mail_plugins = quota imap_quota } protocol pop3 { pop3_uidl_format = %08Xu%08Xv mail_plugins = quota } protocol lda { mail_plugins = sieve quota postmaster_address = root@localhost } protocol lmtp { postmaster_address = admin(a)htmlservices.it mail_plugins = quota sieve } mail_plugins = $mail_plugins quota mail_plugins = acl ********************************* my dovecot-acl ********************************* * user=test(a)test.com lr ********************************* "debug" ********************************* [root@archivio ~]# doveadm -Dv acl debug -u test(a)test.com INBOX Debug: Loading modules from directory: /usr/lib64/dovecot Debug: Module loaded: /usr/lib64/dovecot/lib01_acl_plugin.so Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_sieve_plugin.so Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(test(a)test.com): Debug: Added userdb setting: mail=maildir:/var/vmail/test.com/test/Maildir doveadm(test(a)test.com): Debug: Added userdb setting: plugin/quota_rule=*:storage=0B doveadm(test(a)test.com): Debug: Added userdb setting: plugin/sieve=/var/vmail/test.com/test/.sieve doveadm(test(a)test.com): Debug: Effective uid=5000, gid=5000, home=/var/vmail/test.com/test doveadm(test(a)test.com): Debug: acl: No acl_shared_dict setting - shared mailbox listing is disabled doveadm(test(a)test.com): Debug: maildir++: root=/var/vmail/test.com/test/Maildir, index=, indexpvt=, control=, inbox=/var/vmail/test.com/test/Maildir, alt= doveadm(test(a)test.com): Debug: acl: initializing backend with data: vfile:/etc/dovecot/dovecot-acl doveadm(test(a)test.com): Debug: acl: acl username = test(a)test.com doveadm(test(a)test.com): Debug: acl: owner = 1 doveadm(test(a)test.com): Debug: acl vfile: Global ACL file: /etc/dovecot/dovecot-acl doveadm(test(a)test.com): Info: Mailbox 'INBOX' is in namespace '' doveadm(test(a)test.com): Info: Mailbox path: /var/vmail/test.com/test/Maildir doveadm(test(a)test.com): Info: All message flags are shared across users in mailbox doveadm(test(a)test.com): Debug: Mailbox 'INBOX' matches global ACL pattern '*' doveadm(test(a)test.com): Debug: Mailbox 'INBOX' matches global ACL pattern '*' doveadm(test(a)test.com): Debug: Mailbox 'INBOX' matches global ACL pattern '*' doveadm(test(a)test.com): Debug: acl vfile: file /var/vmail/test.com/test/Maildir/dovecot-acl not found doveadm(test(a)test.com): Info: User test(a)test.com has rights: lookup read doveadm(test(a)test.com): Info: Mailbox in user's private namespace doveadm(test(a)test.com): Info: Mailbox INBOX is visible in LIST [root@archivio ~]# ********************************* if I see the line "Info: User test(a)test.com has rights: lookup read" it seems that the ACL (lookup and read) are correctly applied, but as I was saying above, entering the user test(a)test.com on the webmail, I can do everything I want to also delete the e-mails ..... these are days I try to understand but I don't understand what I'm wrong and how to solve .... thank you all in advance

2 1

Re: Best mail encryption solution for per-user
by Aki Tuomi 12 Jul '19

12 Jul '19

On 27.05.2018 21:16, mail(a)sjemm.net wrote: > May 27, 2018 8:52 AM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 26 May 2018 at 10:36 mail(a)sjemm.net wrote: >>> >>> May 23, 2018 10:10 AM, mail(a)sjemm.net wrote: >>> May 23, 2018 9:46 AM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> >>> On 23.05.2018 10:15, mail(a)sjemm.net wrote: >>> >>> May 23, 2018 8:31 AM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> >>> On 23.05.2018 09:13, mail(a)sjemm.net wrote: >>> May 20, 2018 8:01 PM, mail(a)sjemm.net wrote: >>> May 20, 2018 2:47 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 19 May 2018 at 16:40 mail(a)sjemm.net wrote: >>> >>> May 18, 2018 10:01 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 18 May 2018 at 21:44 mail(a)sjemm.net wrote: >>> >>> May 18, 2018 4:43 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 18 May 2018 at 17:38 mail(a)sjemm.net wrote: >>> >>> May 18, 2018 4:05 PM, "Aki Tuomi" <aki.tuomi(a)dovecot.fi> wrote: >>> On 18 May 2018 at 16:43 mail(a)sjemm.net wrote: >>> >>> Hi Tai74 and Aki, >>> I followed your conversation with interest on how to setup per user encryption in dovecot. >>> I have setup my dovecot with the following in a conf file: >>> >>> ============== >>> >>> mail_attribute_dict = file:%h/Maildir/dovecot-attributes >>> mail_plugins = $mail_plugins mail_crypt >>> plugin { >>> >>> mail_crypt_curve = secp521r1 >>> >>> mail_crypt_save_version = 2 >>> >>> } >>> >>> ============== >>> >>> This works nice, all emails are being encrypted and every user/folder has keys. >>> But as I understood from your conversation these keys are not protected. And I want them to be >>> protected by the users password used by imap. >>> >>> Those passwords are stored in a mysql DB file. ( I used a guide from workaround [dot] org to set up >>> the DB and postfix/dovecot) >>> >>> but how would i set it so, that the users password from the DB is used to encrypt the keys? >>> >>> should i use mail_crypt_private_password = ? >>> how do i point it to the mysql db then? >>> im unsure about this >>> >>> Do you have any hints on this? >>> >>> Kind regards, >>> Zjemm >>> >>> The passwords in your MySQL database are, hopefully, not in plaintext. If you want to secure your >>> user's keys using user's login password, you must have a TOOL that manages this. >>> >>> You can use mail_crypt_private_password = %w in (mysql) passdb fields to provide the user's login >>> password as private password. You might want to run it thru some hash, so %{sha1:password} might be >>> a good option. >>> >>> You can change the key password using 'doveadm mailbox cryptokey', this needs to be done every time >>> user changes his password. >>> >>> Also note that if you go down this road, and the user forgets his password, you will not be able to >>> recover the emails without backup copy of the private key. >>> >>> Aki >>> >>> Hi Aki >>> >>> I used the following command: >>> dovecot pw -s SHA256-CRYPT >>> >>> the output on the chosen password looks like: {SHA256-CRYPT}$5$Rokc06a7In4SF3bO$OQpGQWqg........ >>> >>> This output is used to store in the password fields in the database. So no plain text passwords no >>> :) >>> >>> You can use mail_crypt_private_password = %w in (mysql) passdb fields to provide the user's login >>> password as private password. >>> >>> can you explain this a bit more for me? >>> >>> for now i have in the 10-auth.conf file the following: >>> ============== >>> passdb { >>> driver = sql >>> >>> # Path for SQL configuration file, see example-config/dovecot-sql.conf.ext >>> args = /etc/dovecot/dovecot-sql.conf.ext >>> } >>> >>> and: >>> >>> userdb { >>> driver = static >>> args = uid=vmail gid=vmail home=/var/vmail/%d/%n >>> } >>> ============== >>> >>> then i have in dovecot-sql.conf.ext >>> ============== >>> driver = mysql >>> connect = host=x.x.x.x dbname=mailserver user=mailuser password=mailpasswordexample >>> default_pass_scheme = SHA256-CRYPT >>> password_query = SELECT email as user, password FROM virtual_users WHERE email='%u'; >>> ============== >>> Where do i need to set : mail_crypt_private_password = %w ? >>> >>> password as private password. You might want to run it thru some hash, so %{sha1:password} might be >>> a good option. >>> >>> the passwords are allready hashed in the DB using: dovecot pw -s SHA256-CRYPT to genereate the has. >>> so this step isnt nesesary anymore am i right? >>> >>> Thank you for your quick response, very helpfull >>> >>> Zjemm >>> >>> You misunderstood a bit. The idea is to use the *plaintext* password as the password for the >>> private key. Otherwise anyone could just decrypt it by looking at your database where the hashed >>> password is.. >>> >>> So: >>> >>> password_query = SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM >>> virtual_users WHERE email='%u' >>> >>> Aki >>> >>> Hi Aki, >>> >>> Thank you very much for your help, i realy appreciate that. >>> >>> Ok so if i understand it correctly i'll have to use: >>> >>> password_query = SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM >>> virtual_users WHERE email='%u' >>> >>> in my dovecot-sql.conf.ext file >>> >>> This query selects the user, the password, and %w >>> >>> if i run a little query myself: >>> MariaDB [mailserver]> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password >>> FROM virtual_users; >>> +------------------+----------------------------------+------------------------------------+ >>> | user | password | userdb_mail_crypt_private_password | >>> +------------------+----------------------------------+------------------------------------+ >>> | john(a)example.org | {SHA256-CRYPT}$5$M/GWzmtjsLroRWI | %w | >>> +------------------+----------------------------------+------------------------------------+ >>> >>> %w is a dovecot variable, and stands for the plaintext password, but the password is not stored as >>> plaintext in the DB, %w get filled with the actual plaintext password by dovecot upon the user that >>> is typing in the password when authenticating. >>> >>> is this correct? >>> >>> yes. >>> >>> so then i have the username the hashed password en the plaintext password as a result of the query. >>> >>> yes >>> >>> now userdb_mail_crypt_private_password = the plaintext password >>> do i need to reference it somewhere? or is userdb_mail_crypt_private_password autmatically used by >>> the dovecot mail_crypt plugin to encrypt the keys? or should it be mail_crypt_private_password? >>> >>> It gets injected into the mail process as 'mail_crypt_private_password', as if it was set in plugin >>> {} section. >>> >>> if i have this setup working i'm going to write a blog post on this topic to share this knowledge >>> >>> Thanks again and have a great weekend. >>> >>> Zjemm >>> >>> Aki >>> >>> Hi Aki, >>> >>> Cool i'm testing it right now. >>> I have set up a new mailserver (life is great with lxc containers :) ) >>> >>> postfix and dovecot are working like normal >>> >>> next i enable mail_crypt >>> >>> i did create a file: /etc/dovecot/conf.d/10-mailcrypt.conf >>> ========================== >>> mail_attribute_dict = file:%h/Maildir/dovecot-attributes >>> >>> mail_plugins = $mail_plugins mail_crypt >>> >>> plugin { >>> mail_crypt_curve = secp521r1 >>> mail_crypt_save_version = 2 >>> } >>> ========================== >>> >>> and then i changed the file: /etc/dovecot/dovecot-sql.conf.ext >>> >>> so the query is now the new query: >>> password_query = SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM >>> virtual_users WHERE email='%u'; >>> >>> then i restarted dovecot and postfix and send a test email to the one and only testuser that is in >>> there. >>> >>> when i open the mailbox with the tool mutt, i can see the new email, and when openening the email >>> the mutt client drops the connection. >>> >>> in the log i can see: >>> >>> May 19 13:34:48 mailserver1.example.local dovecot[600]: imap-login: Login: user=<john(a)example.org>, >>> method=PLAIN, rip=::1, lip=::1, mpid=713, TLS, session=<E3PnIY9sNM4AAAAAAAAAAAAAAAAAAAAB> >>> May 19 13:34:49 mailserver1.example.local dovecot[600]: imap(john(a)example.org): Error: read() >>> failed: >>> read(/var/vmail/example.org/john/Maildir/cur/1526736378.M161472P641.mailser…, >>> =571:2,) failed: Private key not available: Cannot decrypt key >>> bfc5bb25b1bf64290eea6dc14b516c6a0a25b64551b6e4f0f8677ba7274887cb: error:03070068:bignum >>> routines:BN_mpi2bn:encoding error (FETCH BODY[] for mailbox INBOX UID 8) >>> >>> i think i missed a step, but witch one? >>> >>> the userpassword hasnt been changed (that would be the next step in the testing process) >>> >>> should i've use doveadm first to encrypt the key with that userpassword? i thought it would do that >>> on the fly, because the initial keys where only just created when enabling the mail_crypt plugin >>> >>> please let me know your thougts >>> Zjemm >>> I noticed you replied directly to me, and not to the list, too... fixed that for you. >>> >>> mail_crypt_private_password is used when key is created, but if you have created it before using >>> password, you'll need to encrypt it before turning the setting on. >>> >>> Aki >>> Hi, Ok i'm sorry about that, thank you for fixing that. i'm a bit new to mailing lists :) >>> >>> are you a dovecot devoloper? or a enthusiastic user? >>> >>> anyway, i managed to get it working now with all your help :) >>> >>> to start fresh i did a: >>> rm -rf /var/vmail/example.org >>> >>> so all email is gone, and i assume all dovecot keys are gone that way. >>> >>> Then i tried to set a password first: >>> >>> root@mailserver1:/var/vmail# doveadm mailbox cryptokey password -u john(a)example.org -n summersun >>> result: Changed password for 0 key(s) >>> >>> (summersun = the password for this user) >>> >>> but then it says there are 0 keys, so then i send the test user an email and try to open the >>> mailbox, i'll get: >>> >>> imap(john(a)example.org): Error: read() failed: >>> read(/var/vmail/example.org/john/Maildir/cur/1526827992.M149712P883.mailser…, >>> =561:2,) failed: Private key not available: Cannot decrypt key >>> 71849013a70b0c631c06112077e7c2fe39b0b2737b4933b219793841209d4e7f: error:03070068:bignum >>> routines:BN_mpi2bn:encoding error >>> >>> but hey, now the key's are created, so i tried to set the password again: >>> >>> root@mailserver1:/var/vmail# doveadm mailbox cryptokey password -u john(a)example.org -n summersun >>> result: dcrypt_key_load_private(787701bd677dd69a26842547a3926cbaa625b0b5a91751f06678c3e9708343d9) >>> failed: password missing >>> ========= >>> >>> Lets start over again, i did a new: >>> rm -rf /var/vmail/example.org >>> >>> i disabled the mail_crypt_private_password in the sql query and send an email to the test user. >>> now again new keys are generated. >>> >>> then i did set the password via doveadm: >>> >>> root@mailserver1:/var/vmail# doveadm mailbox cryptokey password -u john(a)example.org -n summersun >>> result: Changed password for 1 key(s) >>> >>> i enabled the mail_crypt_private_password again in the sql query part and now it works :) >>> >>> YESSS!!! >>> >>> if i now change the password for the user, i cannot read the email anymore. that is what should >>> happen. >>> so now i changed the password for this user also with doveadm: >>> - doveadm mailbox cryptokey password -u john(a)example.org -n qwerty -o summersun >>> >>> and now i'm able to read the emails again. so it all works now as expected. >>> >>> i have now only one thing remaining. >>> >>> when i want to add a new user, i have to somehow: >>> - create the new user in the mysql DB >>> - start with dovecot not using mail_crypt_private_password (so disabled) >>> - send an initial email to let the users mailbox be created and generate the keys >>> - set the users password with doveadm >>> - restart dovecot again with mail_crypt_private_password (so enabled) >>> >>> but this would be disruptive to existing users, and i would have to script the whole thing as a >>> "add-new-user" script to make it usable i guess. >>> >>> do you know of a more easy way of acomplishing this? >>> could i use doveadm for all steps, and get around the mail_crypt_private_password disable part? >>> >>> or could i add a column in the mysql db called userdb_mail_crypt_private_password and fill that >>> column with %w for users that allready exist, and empty for new users. >>> when being empty, it might not set the mail_crypt_private_password for new users >>> >>> or is this the wrong way of thinking? >>> >>> let me know your thoughts on this. >>> >>> And last but not least, i really want to thank you for all your help. It is a really nice >>> experience. >>> Bless you! >>> >>> Zjemm >>> Hi Aki, >>> >>> You did fix the posting on the mail archive, i can see our conversation on there, all but the last >>> email i send to you. >>> Maybe you did not received the last email? >>> >>> would you be willing to take a look at my last question in the email See below: >>> Thank you very much. >>> >>> ================================ >>> i have now only one thing remaining. >>> >>> when i want to add a new user, i have to somehow: >>> - create the new user in the mysql DB >>> - start with dovecot not using mail_crypt_private_password (so disabled) >>> - send an initial email to let the users mailbox be created and generate the keys >>> - set the users password with doveadm >>> - restart dovecot again with mail_crypt_private_password (so enabled) >>> >>> but this would be disruptive to existing users, and i would have to script the whole thing as a >>> "add-new-user" script to make it usable i guess. >>> >>> do you know of a more easy way of acomplishing this? >>> could i use doveadm for all steps, and get around the mail_crypt_private_password disable part? >>> >>> or could i add a column in the mysql db called userdb_mail_crypt_private_password and fill that >>> column with %w for users that allready exist, and empty for new users. >>> when being empty, it might not set the mail_crypt_private_password for new users >>> >>> or is this the wrong way of thinking? >>> >>> let me know your thoughts on this. >>> >>> And last but not least, i really want to thank you for all your help. It is a really nice >>> experience. >>> Bless you! >>> >>> Zjemm >>> >>> If you have mail_crypt_private_password set *when* keys are generated, >>> dovecot should use that password to encrypt the user key. >>> >>> Aki >>> Hi Aki, >>> >>> well, if there is a new email user setup in the mysql DB, and no email has been send to that new >>> user. at that point there are no keys. >>> >>> So trying to set a password on those key's wont work. see: >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey password -u john2(a)example.org -n >>> qwerty >>> result: Changed password for 0 key(s) >>> >>> the keys need to be generated first. And that only happens when sending a mail to that new user. >>> Also, at the point of creating the keys, the mail_crypt_private_password should be disabled >>> otherwise the keys are generated and also encrypted with no user password, and then im unable to >>> set a new pasword for that key >>> >>> How can i set the password before the keys are generated? >>> or can i manually generate key's and then set the password? >>> >>> so at this point i have to do the following: >>> >>> You can use doveadm mailbox cryptokey generate -u user -U >>> >>> to generate a user key. >>> >>> Aki >>> >>> - create the new user in the mysql DB >>> - start with dovecot not using mail_crypt_private_password (so disabled) >>> - send an initial email to let the users mailbox be created and generate the keys >>> - set the users password with doveadm >>> - restart dovecot again with mail_crypt_private_password (so enabled) >>> can i somehow enable the mail_crypt_private_password per user? >>> because this mysql query always returns: %w >>> and thus allways sets: userdb_mail_crypt_private_password >>> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM virtual_users; >>> >>> Hi Aki, >>> >>> You are correct >>> >>> doveadm mailbox cryptokey generate -u user -U generates the keys, but i have to first remove: >>> '%w' AS userdb_mail_crypt_private_password >>> from dovecot configuration and restart dovecot. >>> >>> if i don't do that, the keys are generated, but also somehow encrypted using: no >>> password/empty/something else (i don't know) >>> >>> here you can see that happening: >>> >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey generate -u john2(a)example.org -U >>> Folder Public ID >>> ✓ <userkey> 53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4 >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey password -u john2(a)example.org -n >>> qwerty >>> result: dcrypt_key_load_private(53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4) >>> failed: password missing >>> >>> it expects an old password wich is never set in my opinion. >>> >>> so the last part of this setup would be to figure out a way to disable '%w' AS >>> userdb_mail_crypt_private_password for new users without manually removing that part from the query >>> and restarting dovecot. As that would mean connection interruption while adding new users >>> >>> if we could add new users to the DB and somehow not give '%w' AS userdb_mail_crypt_private_password >>> for that new user in the query, we can add the password afterward, and then enable '%w' AS >>> userdb_mail_crypt_private_password in the query for this user >>> >>> could we add a column in the user db lets say called: encrypted (yes/no) >>> >>> and then change the existing query: SELECT email as user, password, '%w' AS >>> userdb_mail_crypt_private_password FROM virtual_users; >>> >>> to firrt check if encryption is set to 'yes', if so retrun: >>> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM virtual_users; >>> >>> if not return: >>> SELECT email as user, password FROM virtual_users; >>> >>> but then in one mysql query so dovecot can work with it >>> >>> ? >>> >>> Hi Aki, >>> >>> would you be willing to respond to my last question i had send to you last week? >>> its this one: >>> ================================== >>> doveadm mailbox cryptokey generate -u user -U generates the keys, but i have to first remove: >>> '%w' AS userdb_mail_crypt_private_password >>> from dovecot configuration and restart dovecot. >>> >>> if i don't do that, the keys are generated, but also somehow encrypted using: no >>> password/empty/something else (i don't know) >>> >>> here you can see that happening: >>> >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey generate -u john2(a)example.org -U >>> Folder Public ID >>> ✓ <userkey> 53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4 >>> root@mailserver1:/var/vmail/example.org# doveadm mailbox cryptokey password -u john2(a)example.org -n >>> qwerty >>> result: dcrypt_key_load_private(53004acf9886f887ca081c83d7392a5aa3fac4b5a20ce013db91a341fb4644c4) >>> failed: password missing >>> >>> it expects an old password wich is never set in my opinion. >>> >>> could this be a bug? or am i doing it wrong? if needed i could send in a bug report? >>> >>> or might we need to enable the userdb_mail_crypt_private_password "per user" ? >>> >>> so the last part of this setup would be to figure out a way to disable '%w' AS >>> userdb_mail_crypt_private_password for new users without manually removing that part from the query >>> and restarting dovecot. As that would mean connection interruption while adding new users >>> >>> if we could add new users to the DB and somehow not give '%w' AS userdb_mail_crypt_private_password >>> for that new user in the query, we can add the password afterward, and then enable '%w' AS >>> userdb_mail_crypt_private_password in the query for this user >>> >>> could we add a column in the user db lets say called: encrypted (yes/no) >>> >>> and then change the existing query: SELECT email as user, password, '%w' AS >>> userdb_mail_crypt_private_password FROM virtual_users; >>> >>> to firrt check if encryption is set to 'yes', if so retrun: >>> SELECT email as user, password, '%w' AS userdb_mail_crypt_private_password FROM virtual_users; >>> >>> if not return: >>> SELECT email as user, password FROM virtual_users; >>> >>> but then in one mysql query so dovecot can work with it >>> >>> Please let me know your thougts, after this last thing im done :) >>> also let me know if i can support this project in some way, im willing to contribute >>> >>> Zjemm >> You can select it as userdb_mail_crypt_save_version=0 to disable encryption (it won't disable >> decryption). You can use value '2' for enabling encryption. DO NOT USE 1. >> >> Also I tried the SQL based thing, and if you had EMPTY mail_crypt_private_password, try these to >> change the password. If the one that works is the second one, let me know =) >> >> doveadm mailbox cryptokey password -u testuser -nnewpass -o "" >> >> doveadm mailbox cryptokey password -u testuser -nnewpass -o "yes" >> >> Aki > Hi Aki, > > Thanks for replying > > So if i understand you correctly i could enable encryption per user by using: userdb_mail_crypt_save_version= in the database as an extra column. the column would be called "userdb_mail_crypt_save_version" and the value is "0" or "2" > > if a user has: userdb_mail_crypt_save_version=0, encryption is disabled for that user > if a user has: userdb_mail_crypt_save_version=2, encryption is enabled for that user > > would this also override the setting: mail_crypt_save_version = 2 > in the mailcryp.conf file where that is a global setting? or should i delete it there and only use userdb_mail_crypt_save_version= from the database? > > for the password set command i hve testest some things: > > new user with no keys yet: > doveadm -o plugin/mail_crypt_private_password=password mailbox cryptokey generate -u john2(a)example.org -U > > i can set a password with: > new password: > doveadm mailbox cryptokey password -u john2(a)example.org -n newpass -o password > > > if i do not use new user doveadm: (doveadm -o plugin/mail_crypt_private_password=password mailbox cryptokey generate -u john2(a)example.org -U) > then use this: > doveadm mailbox cryptokey password -u john2(a)example.org -n newpass -o "yes" > > > so yes it is the second one > > Kind regards, > > Zjemm It's unfortunate problem with 2.2 that it defaults empty userdb variables into yes. This has been remedied in 2.3. Unfortunately there is no clean fix to this, they all resort to various levels of trickery. Variables set in passdb/userdb overwrite the ones in plugin {} section. Aki

2 2

Connection statistics per user and IP
by Lefteris Tsintjelis 12 Jul '19

12 Jul '19

Is it possible to enable statistics and keep track of the maximum number of connections per user and also per IP? Lefteris

3 2

Frequent Out of Memory for service(config)
by Root Kev 12 Jul '19

12 Jul '19

Hello Group, We have dovecot deployed as solely a Pop3 service that is used by our applications to pass mail from one application to another internally. We have roughly 4 applications that connect to the Pop3 service every 2 seconds, to check for new messages and pop them for processing if they are present. Depending on the site, we have between 1024-2048MB of memory set for default_vsz_limit. In all systems we see the Out of memory alert several times a day. We previously did not see this at all when running on CentOS6, with less memory. We have tried increasing the memory to the vsz_limit up to 2gb without success. We are running on CentOS 7 servers, running dovecot 2.3.6 (7eab80676) (from the dovecot repo). Can anyone advise any other settings that could be modified in order to correct these out of memory issues? # dovecot -n # 2.3.6 (7eab80676): /etc/dovecot/dovecot.conf # OS: Linux 3.10.0-957.5.1.el7.x86_64 x86_64 CentOS Linux release 7.6.1810 (Core) # Hostname: ****** #<modified for security> auth_cache_size = 10 M auth_verbose = yes default_vsz_limit = 1 G instance_name = Pop3 Mail Service listen = 10.*.*.* #<modified for security> log_path = /var/log/dovecot.log login_greeting = Pop3 Mail Service login_trusted_networks = 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* 10.*.*.* #<modified for security> mail_location = maildir:~/Maildir namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = cache_key=#hidden_use-P_to_show# driver = pam } protocols = pop3 ssl_cert = </etc/pki/dovecot/certs/dovecot.pem ssl_key = # hidden, use -P to show it userdb { driver = passwd } verbose_ssl = yes May 10 06:44:05 config: Fatal: pool_system_malloc(8192): Out of memory May 10 06:44:05 config: Fatal: master: service(config): child 27887 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 10 14:56:31 config: Fatal: pool_system_malloc(8192): Out of memory May 10 14:56:31 config: Fatal: master: service(config): child 2495 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 10 23:09:02 config: Fatal: pool_system_malloc(8192): Out of memory May 10 23:09:02 config: Fatal: master: service(config): child 10581 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 11 10:18:43 config: Fatal: pool_system_malloc(8192): Out of memory May 11 10:18:43 config: Fatal: master: service(config): child 30511 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 11 18:30:59 config: Fatal: pool_system_malloc(1608): Out of memory May 11 18:30:59 config: Fatal: master: service(config): child 12134 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 12 02:57:28 config: Fatal: pool_system_malloc(8192): Out of memory May 12 02:57:28 config: Fatal: master: service(config): child 24940 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 12 11:07:34 config: Fatal: pool_system_malloc(8192): Out of memory May 12 11:07:35 config: Fatal: master: service(config): child 8203 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 12 19:17:52 config: Fatal: pool_system_malloc(8192): Out of memory May 12 19:17:52 config: Fatal: master: service(config): child 14623 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 13 03:26:34 config: Fatal: pool_system_malloc(8192): Out of memory May 13 03:26:35 config: Fatal: master: service(config): child 21479 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 13 11:35:42 config: Fatal: pool_system_malloc(8192): Out of memory May 13 11:35:43 config: Fatal: master: service(config): child 26900 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump) May 13 19:34:26 config: Fatal: pool_system_malloc(8192): Out of memory May 13 19:34:26 config: Fatal: master: service(config): child 32187 returned error 83 (Out of memory (service config { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump). Thanks, Kevin

4 6

Re: Segfault of doveconf
by Claudio Prono 11 Jul '19

11 Jul '19

On 9.7.2019 13.40, Aki Tuomi via dovecot wrote: On 9.7.2019 12.54, Claudio Prono wrote: On 4.7.2019 19.00, Claudio Prono via dovecot wrote: Sorry to bother again, but the problem persist. I have checked all the configurations of dovecot, and seems all ok, but randomly the segfault pops up again, without no apparently reason. My packages are: ii dovecot-core 1:2.2.22-1ubuntu2.10 amd64 secure POP3/IMAP server - core files ii dovecot-imapd 1:2.2.22-1ubuntu2.10 amd64 secure POP3/IMAP server - IMAP daemon ii dovecot-managesieved 1:2.2.22-1ubuntu2.10 amd64 secure POP3/IMAP server - ManageSieve server ii dovecot-mysql 1:2.2.22-1ubuntu2.10 amd64 secure POP3/IMAP server - MySQL support ii dovecot-pop3d 1:2.2.22-1ubuntu2.10 amd64 secure POP3/IMAP server - POP3 daemon ii dovecot-sieve 1:2.2.22-1ubuntu2.10 amd64 secure POP3/IMAP server - Sieve filters support Can be a temporarly solution to uninstall dovecot-managesieved ? Cordially, Claudio. On 5.7.2019 09:59 Aki via dovecot wrote: You should install dovecot-dbg, and follow https://www.dovecot.org/bugreport-mail, then we could figure out why it's crashing in the first place. Aki I have followed the instructions to debug the problem, and I have uploaded the crash log at this location: https://justpaste.it/5wlot Yesterday the system have got some I/O Errors on the hard disk (virtual system hosted from Seeweb). The problems on the disk now are solved by the provider, and the crash reported is before that. So, now my question is: the doveconf crash can be linked to the hardware problem ? Or something else? Thanks for your help. Cordially, Claudio Prono. I tried unpacking the crash file you pastebin, can you please send it as attachment to me? Aki It seems you are at least missing postmaster_address in your config. I was not able to actually make dovecot crash when I tested, so can you try adding this setting and see if it helps? Aki On 9.7.2019 09:59 Claudio Prono via dovecot wrote: I have added it to the configuration, and restarted dovecot. Now I wait and see if this error again. Thank you for your time. Cheers, Claudio. ---- Just a rapid update, the problem of the segmentation fault of doveconf was not occourred again in the past two days. I think the problem was somehow related to the hardware failure on the virtual disk where the system is hosted. Now, the hardware problem is resolved from the virtual system maintainer, and doveconf has stopped to crash. Anyway, thanks for the support. Claudio.

1 0

Dovecot 2.3.6 on Solaris10: build issues, segfaults
by Joseph Tam 11 Jul '19

11 Jul '19

Hopefully, there is some fix for issue 3 which is beyond my skill to fix. Issue 1) Need recent gcc version Building Dovecot versions <=2.2.x using gcc 3.4.4 worked, but this gcc version fails to build 2.3.x properly: symptoms include compile failures and executable crashes that depended on the amount of optimization used, which is usually a sign of compiler bugs. (It could also be issue 3 in disguise.) Either way, I updated to gcc 9.1.0. Issue 2) Cannot build with --enable-hardening Using gcc 9.1.0, "configure" step fails because fd passing was broken, but the real problem was a compilation failure when "--enable-hardening" is used. Demonstration: # echo 'int main(){char a[1]; strcpy(a,a);} ' | gcc -w -fstack-protector-strong -x c - Undefined first referenced symbol in file __stack_chk_guard /var/tmp//cc12L9zV.o (symbol scope specifies local binding) ld: fatal: symbol referencing errors. No output written to a.out collect2: error: ld returned 1 exit status I'm not sure if this is a Solaris10 fumble, but configuring "--disable-hardening" removes the "-fstack-protector-strong" compiler option, which resolves this issue. Issue 3) dovecot/doveconf segfaults on startup It crashes here while processing dovecot.conf, as does "doveconf" (settings-parser.c:1519 in setting_copy()) *dest_size = *src_size; It appears *src_size is not an 8-byte address aligned (0x5597c). It inherits this value from the calling routine as the sum of "set" (8-byte aligned) + "def->offset"=20 => misaligned address. (settings-parser.c:1597 in settings_dup_full()) src = CONST_PTR_OFFSET(set, def->offset); (gdb) p set $2 = (const void *) 0x55968 (gdb) p *def $3 = {type = SET_SIZE, key = 0x2d548 "submission_max_mail_size", offset = 20, list_info = 0x0} (gdb) bt full #0 0xff190690 in setting_copy (type=SET_SIZE, src=0x5798c, dest=0xf7ed4, pool=0xf29a0, keep_values=false) at settings-parser.c:1519 src_size = 0x5798c dest_size = 0xf7ed4 __func__ = "setting_copy" #1 0xff190bc4 in settings_dup_full (info=0x2d9ac <submission_setting_parser_info>, set=0x57978, pool=0xf29a0, keep_values=false) at settings-parser.c:1600 def = 0x2d7d4 <submission_setting_defines+80> src = 0x5798c dest_set = 0xf7ec0 dest = 0xf7ed4 children = 0x2c i = 1105688 count = 4279837964 #2 0xff192724 in settings_parser_dup (old_ctx=0x9ec08, new_pool=0xf29a0) at settings-parser.c:1946 new_ctx = 0x10def0 iter = 0x0 links = {_table = 0xa23c0, _key = 0xa23c0, _keyp = 0xa23c0, _const_key = 0xa23c0, _value = 0xa23c0, _valuep = 0xa23c0} new_link = 0x10b488 value = 0x0 key = 0x0 i = 0 parser_pool = 0x10ded8 keep_values = false __func__ = "settings_parser_dup" #3 0x0001fc0c in config_filter_parsers_get (ctx=0x657a8, pool=0xf29a0, modules=0x0, filter=0x583e8, parsers_r=0xffbff9c4, output_r=0xffbff9bc, error_r=0xffbffa44) at config-filter.c:372 src = 0xf29d8 dest = 0xf2a48 error = 0x0 error_p = 0x52cfc i = 26 count = 33 __func__ = "config_filter_parsers_get" #4 0x00021be4 in config_all_parsers_check (ctx=0xffbffac8, new_filter=0x657a8, error_r=0xffbffa44) at config-parser.c:441 parsers = 0x58180 tmp_parsers = 0x4 output = {specific_services = 0xf2a20, service_uses_local = false, service_uses_remote = false, used_local = false, used_remote = false, permission_denied = false} i = 0 count = 3 ssl_set = 0x4 <error: Cannot access memory at address 0x4> global_ssl_set = 0x32188 "" tmp_pool = 0xf29a0 ssl_warned = false ret = 0 __func__ = "config_all_parsers_check" #5 0x00022ec8 in config_parse_finish (ctx=0xffbffac8, error_r=0xffbffb98) at config-parser.c:747 new_filter = 0x657a8 error = 0x33c22 "/user=%s" ret = 0 #6 0x00024230 in config_parse_file (path=0x50468 "/local/dovecot/etc/dovecot/dovecot.conf", expand_values=false, modules=0x0, error_r=0xffbffb98) at config-parser.c:1064 root = {prev = 0x0, input = 0x0, path = 0x50468 "/local/dovecot/etc/dovecot/dovecot.conf", linenum = 400} ctx = {pool = 0x52b58, path = 0x50468 "/local/dovecot/etc/dovecot/dovecot.conf", , all_parsers = {arr = {buffer = 0x58160, element_size = 4}, v = 0x58160, v_modifiable = 0x58160}, root_parsers = 0x52b70, cur_section = 0x58388, cur_input = 0x0, str = 0x58458, pathlen = 0, section_counter = 6, error = 0x0, old = 0x58580, seen_settings = {_table = 0x0, _key = 0x0, _keyp = 0x0, _const_key = 0x0, _value = 0x0, _valuep = 0x0}, filter = 0x0, expand_values = false, hide_errors = false} i = 33 count = 33 key = 0x0 value = 0x0 full_line = 0x0 type = CONFIG_LINE_TYPE_SKIP line = 0x0 fd = -1 ret = 0 handled = false #7 0x000188dc in main (argc=1, argv=0xffbffc94) at doveconf.c:979 master_service_flags = (MASTER_SERVICE_FLAG_STANDALONE | MASTER_SERVICE_FLAG_NO_INIT_DATASTACK_FRAME | MASTER_SERVICE_FLAG_DONT_SEND_STATS) scope = CONFIG_DUMP_SCOPE_ALL orig_config_path = 0x46300 "/local/dovecot/etc/dovecot/dovecot.conf" config_path = 0x50468 "/local/dovecot/etc/dovecot/dovecot.conf" module = 0x1c00 <error: Cannot access memory at address 0x1c00> module_names = {arr = {buffer = 0x46338, element_size = 4}, v = 0x46338, v_modifiable = 0x46338} filter = {service = 0x0, local_name = 0x0, local_host = 0x0, remote_host = 0x0, local_net = {family = 0, u = {ip6 = {_S6_un = {_S6_u8 = '\000' <repeats 15 times>, _S6_u32 = {0, 0, 0, 0}, __S6_align = 0}}, ip4 = {S_un = {S_un_b = { s_b1 = 0 '\000', s_b2 = 0 '\000', s_b3 = 0 '\000', s_b4 = 0 '\000'}, S_un_w = { s_w1 = 0, s_w2 = 0}, S_addr = 0}}}}, remote_net = {family = 0, u = {ip6 = { _S6_un = {_S6_u8 = '\000' <repeats 15 times>, _S6_u32 = {0, 0, 0, 0}, __S6_align = 0}}, ip4 = {S_un = {S_un_b = {s_b1 = 0 '\000', s_b2 = 0 '\000', s_b3 = 0 '\000', s_b4 = 0 '\000'}, S_un_w = {s_w1 = 0, s_w2 = 0}, S_addr = 0}}}}, local_bits = 0, remote_bits = 0} wanted_modules = 0x0 error = 0xfef47940 <_uberdata> "" exec_args = 0x0 setting_name_filters = 0x0 i = 5 c = -1 ret = -4195172 ret2 = 4 config_path_specified = false expand_vars = false hide_key = false parse_full_config = false simple_output = false dump_defaults = false host_verify = false print_plugin_banner = true hide_passwords = true Joseph Tam <jtam.home(a)gmail.com>

4 6

Dovecot behind Load Balancer
by Paolo Daniele 10 Jul '19

10 Jul '19

Hi, i've a question for you. I've two dovecot imap/pop server behind a zen load balancer. Load balancing is made by lx4nat so the public ip address of my load balancer contact directly the dovecot servers. Since few months i've a message from thunderbird that i've reached the imap limit login for ip. I've triend to increase the max user ip parameter but sometimes i've the same problem. It's a strange things that actually i'm able to mitigate by reduce the number of cached connections in Thunderbird but it's not normal. What do you think about that? Maybe there's some tuning that you can suggest. Thank you, Paolo

5 10

solr
by Maciej Milaszewski IQ PL 10 Jul '19

10 Jul '19

Hi I have set up SOLR in accordance with documentation and it runs well. I use solr like: ....... fts = solr fts_solr = debug url=http://IP:8983/solr/ (solr in external machine) ...... Is replication of this system really essential ?? Due to my tests, rollback of the solr server on external machine lasts less than minute and it is nearly non-visible from client side. On the other hand solr replication is quite complicated process and rollback or master-slave switch in this case is non-trivial task, that may have result in whole dataset inconsistency. Do you have any experience in such cases ? Maby load-balance in HAProxy colud do the thing ? Something like: ......... server search1 192.168.1.1:8983 check port 8983 inter 20s fastinter 2 server search2 192.168.1.2:8983 backup ......... Best Regards

2 1

Some questions
by Jérôme Bardot 10 Jul '19

10 Jul '19

Hello, This is my first email here. I want to understand well how dovecot is integrate with ldap in a postfix/dovecot/ldap setup. I use a debian server. More specifically what dovecot need in ldap to work. I saw we can use several "mode" related to virtual domain, etc. For "start" i only need one domain with several address. I currently use fusiondirectory for manage my ldap users. i guess i can use that schema to auto create users email (name.firstname(a)domain.tld for ie) ? I also want to setup some aliases and share directory based on ldap group/role can i do it ? An other question is can we have two domain name for imap.domain.tld && smtp.domain.tld ? IF some questions are out of topic let me know. thx jerome

4 3

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
by Arnold Opio Oree 10 Jul '19

10 Jul '19

Hello Timo, I have scheduled some time to carry through the test requested by Aki. I will add responding to you as part of that process. Not sure if I have said or done anything to offend you, as your tone appears quite abrasive, even where you do not have an actual response to an observation. It is my view that it is a privilege to provide an open source platform, and therefore a responsibility wherever feasible not only to properly maintain and support it, but also to properly document it; both of which from my experience and findings to date, Dovecot have failed to do. Kind regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree(a)parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Timo Sirainen via dovecot <dovecot(a)dovecot.org> Reply-To: Timo Sirainen <timo(a)sirainen.com> To: Arnold Opio Oree <arnold.oree(a)parallaxict.com> Cc: Dovecot Mailing List <dovecot(a)dovecot.org> Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Wed, 10 Jul 2019 11:09:57 +0300 On 7 Jul 2019, at 18.12, Arnold Opio Oree via dovecot < dovecot(a)dovecot.org> wrote: > > Dovecot Team, > > > > I'd like to report a number of bugs, that are to my view all > > critical. It would help to get your doveconf -n, example command lines causing the problems and the error messages it outputs or what the wrong behavior looks like in filesystem. It's now rather difficult to guess what exactly you tried and what happened. Also what kind of output does readpst make? I'm not sure why you're using DIRNAMEs here. > > doveadm-sync -1/general > > > > 1) If DIRNAMEs are not different between command line and > > mail_location doveadm sync will fail, saying that the source and > > destination directories are the same This sounds very strange. I'm not sure what exactly you did, and I couldn't reproduce with a small test. > > 2) The -n / -N flags do not work, and a sync will fail strangely if > > location is specified in the namespace definition Again, sounds strange. > > 3) Adds mbox to path name under mailbox directory (where syncing > from > > an mbox source) Probably with different parameters you could avoid it. > > 4) Not having the mailboxes at source named the same as those at > > destination causes errors and partial sync > > > > 5) Not having the target mailboxes formatted to receive the sync > > (/<mailboxroot>/DIRNAME/) will cause sync errors. I don't understand these. Target mailboxes are supposed to be empty initially, and after the initial sync they should be in the expected format. Why would they be different? > > doveadm-sync > > > > 1) With large synchronizations UIDs are corrupted where multiple > > syncs are executed and the program can no longer synchronize What exactly is the error message? > > dovecot > > > > 1) Panics and fails to expand ~ to user home: observed cases are > > where multiple namespaces are being used Panic message and more details would also help. > > With regards to the last error that I requested help on i.e. > > \Noselect. This has been resolved more-or-less by the workarounds > > that I have implemented for the bugs reported above. > > > > I have seen a number of threads whilst researching the \Noselect > > issue where people have been very confused. My finding was that > > \Noselect is a function of the IMAP specification server-side > > implementation RFC3501 ( > > https://tools.ietf.org/html/rfc3501#section-6.3.6) And for me the > > server was returning directories with \Noselect because the > mailboxes > > were malformed on account of dovadm-sync errors. In order to fix > this > > I formed a bash command to transverse the mailbox hierarchy and > > create the missing folders critical to the sdbox format, namely > > DIRNAME. Nowadays we have also an option to disable creation of \Noselect folders, because they confuse people. Using mail_location = ...:NO- NOSELECT - It won't affect existing folders immediately though. -----Original Message----- From: Arnold Opio Oree <arnoldoree(a)parallaxict.com> Reply-To: arnoldoree(a)parallaxict.com To: Aki Tuomi <aki.tuomi(a)open-xchange.com>, Dovecot Mailing List < dovecot(a)dovecot.org> Cc: debian-release(a)lists.debian.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Mon, 08 Jul 2019 13:21:51 +0100 Hello Aki, Thanks for looking into these. I will as requested attempt the relevant procedures under Dovecot 2.3.6. To make the test fair, I will need to fork the relevant production groupware stack (which is now stable and in operation, with our enterprise (email) data successfully migrated from Microsoft Exchange) to a new staging server; given that the current staging server is now of a materially different configuration to the production server (where the most controlled observations of these bugs were made). Kindly give me some time, as I have an urgent internal openstack deployment project to kick-off, that has been delayed by an overrun of this internal groupware stack deployment project. Best regards, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree(a)parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message----- From: Aki Tuomi via dovecot < dovecot(a)dovecot.org > Reply-To: Aki Tuomi < aki.tuomi(a)open-xchange.com > To: arnoldoree(a)parallaxict.com , Arnold Opio Oree < arnold.oree(a)parallaxict.com >, Arnold Opio Oree via dovecot < dovecot(a)dovecot.org > Cc: debian-release(a)lists.debian.org Subject: Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS! Date: Mon, 8 Jul 2019 08:48:08 +0300 (EEST) Hi! Thank you for reporting these. We will look into them. In the mean time, can you see if any of these are fixed in 2.3.6? Aki > On 07/07/2019 18:12 Arnold Opio Oree via dovecot < > dovecot(a)dovecot.org > > > wrote: > > > Dovecot Team, > > I'd like to report a number of bugs, that are to my view all > critical. > > System: Replicated on multiple Debian 10 (Buster) systems > Dovecot Version(s): 2.3.4.1 > > doveadm-sync -1/general > > 1) If DIRNAMEs are not different between command line and > mail_location doveadm sync will fail, saying that the source and > destination directories are the same > > 2) The -n / -N flags do not work, and a sync will fail strangely if > location is specified in the namespace definition > > 3) Adds mbox to path name under mailbox directory (where syncing from > an mbox source) > > 4) Not having the mailboxes at source named the same as those at > destination causes errors and partial sync > > 5) Not having the target mailboxes formatted to receive the sync > (/<mailboxroot>/DIRNAME/) will cause sync errors. > > doveadm-sync > > 1) With large synchronizations UIDs are corrupted where multiple > syncs are executed and the program can no longer synchronize > > dovecot > > 1) Panics and fails to expand ~ to user home: observed cases are > where multiple namespaces are being used > > Please let me know if you need me to elaborate or to provide any > further information that you may need to replicate the bugs, or if I > can help in any other way. > > With regards to the last error that I requested help on i.e. > \Noselect. This has been resolved more-or-less by the workarounds > that I have implemented for the bugs reported above. > > I have seen a number of threads whilst researching the \Noselect > issue where people have been very confused. My finding was that > \Noselect is a function of the IMAP specification server-side > implementation RFC3501 ( > https://tools.ietf.org/html/rfc3501#section-6.3.6 > ). And for me the > server was returning directories with \Noselect because the mailboxes > were malformed on account of dovadm-sync errors. In order to fix this > I formed a bash command to transverse the mailbox hierarchy and > create the missing folders critical to the sdbox format, namely > DIRNAME. > > Kind regards, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree(a)parallaxdt.com > > > http://www.parallaxdt.com > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard its contents and disclose them to no other persons. > > -----Original Message----- > From: Arnold Opio Oree via dovecot < > dovecot(a)dovecot.org > > > Reply-To: > arnoldoree(a)parallaxict.com > , Arnold Opio Oree < > arnold.oree(a)parallaxict.com > > > To: > dovecot(a)dovecot.org > > Cc: > rs(a)sys4.de > , > aki.tuomi(a)open-xchange.com > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 04 Jul 2019 14:52:28 +0100 > > > Hi all, > > The guidance provided so far has been really helpful, and has helped > a great deal to bringing down wasted energy on finding and executing > a viable path. I am now at the final due action to complete our > Dovecot application to our use-case, but am stuck on an issue that I > cannot find any easily accessible documentation on. > > Generally this is what has been done: > > 1. Uploaded the enterprise data PST to the target groupware server. > 2. Prepared the server by changing the mailbox format to sdbox and > the the Dovecot mail location to > mail_location=/var/vmail/domain/user/mail/ > 3. Converted the pst (on-server) to a recursive mbox hierarchy using > readpst > 4. Executed doveadm-sync to convert mbox hierarchy data into sdbox > and to copy it into the enterprise archive user's mailboxes > 4.i. The biggest issue I faced at this point was doveadm-sync saying > that the source and destination pointed to the same location, whereas > they clearly did not. > 4.i.a. I resolved this by removing the location= setting from the > target namespace, and allowing it to default to mail_location = > setting, and then using a completely different DIRNAME for the import > doveadm-sync execution (which was the desired final DIRNAME); I then > once the sync had been successful, changed the mail_location DIRNAME > so that it pointed to the imported mail DIRNAME; and hence the > imported email data was in the live mailboxes > 4.i.b. doveadm-import failed several times, and was throwing quite > inexplicable errors, so I moved onto doveadm-sync > 4.i.c. I also had to make sure that the source and destination folder > names matched, otherwise doveadm-syc threw very many errors and only > partially imported the data > 4.i.d. An issue which I decided just to live with is that an mbox > DIRNAME was added to each mailbox as well as the DIRNAME specified so > the path to mail is mbox/dbox-Mails. My thought is that with the data > live on an IMAP server it will be possible to do a dysync through TCP > to correct this problem. > > The final issue that I am facing now, is that when readpst finds > empty folders in the source pst hierarchy, it does not create an mbox > file in the mbox hierarchy folder space. This causes doveadm-sync to > not create the target data required for its mailbox structure i.e. > DIRNAME sub-folder and index file (with our configuration). At this > point either doveadm-sync or the dovecot process makes these empty > folders not selectable. > > The question now is how would I go about making all of these folders > selectable, e.g. with an internal or external command line tool to > change flags / create necessary sdbox mailbox constituent data? > > Many thanks, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree(a)parallaxdt.com > > > http://www.parallaxdt.com > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard its contents and disclose them to no other persons. > > -----Original Message----- > From: Arnold Opio Oree via dovecot < > dovecot(a)dovecot.org > > > Reply-To: > arnoldoree(a)parallaxict.com > , Arnold Opio Oree < > arnold.oree(a)parallaxict.com > > > To: Robert Schetterer < > rs(a)sys4.de > >, > dovecot(a)dovecot.org > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 27 Jun 2019 12:05:35 +0100 > > > Also you may run into client limits.... i ve seen this with > > outlook, > > apple mail, thunderbird via imap in the past > > Thanks for this note Robert, it was not really an aspect that I was > considering. > > We are operating our groupware services user access through both > Evolution Groupware and KDE Kontact / KMail on Debian Linux > workstations. Hopefully if there is a client issue it should be local > to only one groupware client. > > I will be sure to study / investigate in this - client - area should > any issues that are not traceable to the server-side arise. > > Many thanks, > > Arnold Opio Oree > Chief Executive Officer > Parallax Digital Technologies > > arnoldoree(a)parallaxdt.com > > > > http://www.parallaxdt.com > > > > tel : +44 (0) 333 577 8587 > fax : +44 (0) 20 8711 2477 > > Parallax Digital Technologies is a trading name of Parallax Global > Limited. U.K. Co. No. 08836288 > > The contents of this e-mail are confidential. If you are not the > intended recipient you are to delete this e-mail immediately, > disregard > its contents and disclose them to no other persons. > > > -----Original Message----- > From: Robert Schetterer via dovecot < > dovecot(a)dovecot.org > > > Reply-To: Robert Schetterer < > rs(a)sys4.de > > > To: > dovecot(a)dovecot.org > > > Subject: Re: Applying Dovecot for a large / deep folder-hierarchy > archive. > Date: Thu, 27 Jun 2019 12:53:49 +0200 > > Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot: > > On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: > > > Hello to you all, > > > > > > I'd like to ask about my intended application of Dovecot to > > > create > > > a folder-hierarchy for storing our enterprise emails, which are > > > treated as live data rather than archives for compliance or > > > occasional / reactive retrieval. > > > > > > The data is presently not that large (a few gigabytes), but it is > > > expected to grow rapidly. Up to this stage the data has been > > > contained > > > in a Microsoft Exchange mailbox (2013), and then in an offline > > > PST. > > > The move to the offline PST was by necessity, as the large number > > > of > > > folders, and depth of hierarchy to my best understanding caused > > > the > > > exchange server / outlook / evolution mail clients to begin to > > > malfunction. To cope with this the archive was broken up and the > > > bulk stored in the offline PST and the most active components > > > stored in > > > online Exchange mailboxes. > > > > > > I have some understanding of the fs mbox format, and also the > > > mitigations to be made for certain characters / strings. My main > > > concern is whether Dovecot is likely to be able to cope well with > > > a > > > large number of folders / depth of hierarcy. > > > > > > I will really appreciate any help / advice you can give. > > > > > > Best regards, > > > > > > Arnold Opio Oree > > > > > > > Hi! > > > > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only > > limiting > > factor is that the total name may not exceed 4096 bytes. Also > > individual > > folder names may not exceed 255 bytes. > > > > Prior to that the limit is 255 per folder up to 16 levels. > > > > I can't recommend using 'mbox' storage format, please consider > > using > > maildir or sdbox instead. > > > > Aki > > > > > > > > > > --- Aki Tuomi

2 2