- dovecot - dovecot.org

Applying Dovecot for a large / deep folder-hierarchy archive.
by Arnold Opio Oree 10 Jul '19

10 Jul '19

Hello to you all, I'd like to ask about my intended application of Dovecot to create a folder-hierarchy for storing our enterprise emails, which are treated as live data rather than archives for compliance or occasional / reactive retrieval. The data is presently not that large (a few gigabytes), but it is expected to grow rapidly. Up to this stage the data has been containedin a Microsoft Exchange mailbox (2013), and then in an offline PST. The move to the offline PST was by necessity, as the large number offolders, and depth of hierarchy to my best understanding caused the exchange server / outlook / evolution mail clients to begin tomalfunction. To cope with this the archive was broken up and the bulk stored in the offline PST and the most active components stored inonline Exchange mailboxes. I have some understanding of the fs mbox format, and also the mitigations to be made for certain characters / strings. My main concern is whether Dovecot is likely to be able to cope well with a large number of folders / depth of hierarcy. I will really appreciate any help / advice you can give. Best regards, Arnold Opio OreeChief Executive OfficerParallax Digital Technologies arnoldoree(a)parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587fax : +44 (0) 20 8711 2477 Parallax Digital Technologies78A Hatfeild MeadLondonSM4 5PF Parallax Digital Technologies is a trading name of Parallax GlobalLimited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not theintended recipient you are to delete this e-mail immediately, disregardits contents and disclose them to no other persons.

5 11

10-ssl.conf
by Nicholas Chua 10 Jul '19

10 Jul '19

Hi, I am having an error, doveconf: Fatal: Error in configuration file /usr/local/dovecot/etc/dovecot/conf.d/10-ssl.conf line 6: Unknown setting: service { ssl I am on dovecot 2.3.6. My 10-ssl.conf as follows https://pastebin.com/sYXsmm5b Any idea what went wrong? Thanks nic

2 1

Segfault of doveconf
by Claudio Prono 09 Jul '19

09 Jul '19

Hi, I have some problems of segfault at doveconf: Jul 2 08:13:16 mail kernel: [1415540.297823] doveconf[7103]: segfault at 560cb801e000 ip 00007f861da742d0 sp 00007ffe2d671548 error 6 in libc-2.23.so[7f861d9e5000+1c0000] Jul 2 15:45:38 mail kernel: [1442682.919193] doveconf[11915]: segfault at 563b32420000 ip 00007fcb127ae2d0 sp 00007ffecfc7b598 error 6 in libc-2.23.so[7fcb1271f000+1c0000] Jul 2 20:27:37 mail kernel: [1459601.781407] doveconf[14680]: segfault at 55618f01b000 ip 00007f44137202d0 sp 00007ffcbb8d8628 error 6 in libc-2.23.so[7f4413691000+1c0000] Jul 2 21:35:36 mail kernel: [1463680.435538] doveconf[15090]: segfault at 55ea7521d000 ip 00007f0bfe79f2d0 sp 00007ffd272793b8 error 6 in libc-2.23.so[7f0bfe710000+1c0000] Jul 3 00:44:28 mail kernel: [1475013.132492] doveconf[16253]: segfault at 561feea20000 ip 00007f34be2be2d0 sp 00007ffe731ef4b8 error 6 in libc-2.23.so[7f34be22f000+1c0000] System configuration follows: lsb_release -d Description: Ubuntu 16.04.6 LTS doveconf -n # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.13 (7b14904) # OS: Linux 4.4.0-45-generic x86_64 Ubuntu 16.04.6 LTS disable_plaintext_auth = no first_valid_gid = 8 first_valid_uid = 150 last_valid_gid = 8 last_valid_uid = 150 mail_location = mbox:~/mail:INBOX=/var/mail/%u managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX. } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /var/vmail/%d/%u/.dovecot.sieve sieve_dir = /var/vmail/%d/%u/sieve sieve_global_dir = /var/lib/dovecot/sieve/ sieve_storage = /var/vmail/%d/%u/sieve } protocols = " imap sieve pop3" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = mail mode = 0600 user = vmail } } service quota-warning { executable = script /usr/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = dovecot } ssl_ca = </etc/ssl/certs/SecureTrust_CA_mauri2.pem ssl_cert = </etc/ssl/certs/dovecot6.pem ssl_cipher_list = TLSv1+HIGH:!SSLv2:!RC4:!aNULL:!eNULL:!3DES:!IDEA:!SEED ssl_key = </etc/ssl/private/dovecot3.pem ssl_prefer_server_ciphers = yes userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = " sieve" } protocol pop3 { pop3_uidl_format = %Mf } dovecot --version 2.2.22 (fe789d2) apt-get update Hit:1 http://ubuntu.seeweb.it/ubuntu xenial InRelease Hit:2 http://ubuntu.seeweb.it/ubuntu xenial-updates InRelease Hit:3 http://ubuntu.seeweb.it/ubuntu xenial-backports InRelease Get:4 http://ubuntu.seeweb.it/ubuntu-security xenial-security InRelease [109 kB] Fetched 109 kB in 1s (99.1 kB/s) Reading package lists... Done apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Any hint about it ? Sometimes, from my webmail I receive an error communicating with server, but no other errors. Cordially, Claudio Prono. -- ---------------------------------------------------------------------------- Claudio Prono OPST System Developer @Mediaservice.net Srl Tel: +39-011-07.49.338 Via Santorelli, 15 Fax: +39-011-32.46.497 10095 Grugliasco (TO) http://mediaservice.net/disclaimer ITALY ---------------------------------------------------------------------------- PGP Key - http://keys.mediaservice.net/c_prono.asc

2 7

Orphaned processes after doveadm log reopen
by Tom Sommer 09 Jul '19

09 Jul '19

I rotate logs every night on my Director, running "doveadm log reopen" I've noticed a problem with processes not closing correctly, they can be open for days and cause corrupt index files because they are connected to different backends (so it writes to the wrong server when the process disconnects) 23518 ? S 140:53 dovecot/anvil [4 connections] 23519 ? S 444:45 dovecot/log 23520 ? S 497:19 dovecot/imap-login [8 pre-login + 27 proxies] 23531 ? S 0:27 dovecot/config 26201 ? S 551:53 dovecot/imap-login [7 pre-login + 33 proxies] 22126 ? S 960:15 dovecot/pop3-login [28 pre-login + 396 proxies] 59515 ? S 218:32 dovecot/pop3-login [44 pre-login + 468 proxies] Is there any way to force these orphaned processes to terminate after a certain time? -- Tom

2 4

Re: Applying Dovecot for a large / deep folder-hierarchy archive - BUG REPORTS!
by Arnold Opio Oree 07 Jul '19

07 Jul '19

Dovecot Team, I'd like to report a number of bugs, that are to my view all critical. System: Replicated on multiple Debian 10 (Buster) systemsDovecot Version(s): 2.3.4.1 doveadm-sync -1/general 1) If DIRNAMEs are not different between command line and mail_location doveadm sync will fail, saying that the source and destination directories are the same 2) The -n / -N flags do not work, and a sync will fail strangely if location is specified in the namespace definition 3) Adds mbox to path name under mailbox directory (where syncing from an mbox source) 4) Not having the mailboxes at source named the same as those at destination causes errors and partial sync 5) Not having the target mailboxes formatted to receive the sync (/<mailboxroot>/DIRNAME/) will cause sync errors. doveadm-sync 1) With large synchronizations UIDs are corrupted where multiple syncs are executed and the program can no longer synchronize dovecot 1) Panics and fails to expand ~ to user home: observed cases are where multiple namespaces are being used Please let me know if you need me to elaborate or to provide any further information that you may need to replicate the bugs, or if I can help in any other way. With regards to the last error that I requested help on i.e. \Noselect. This has been resolved more-or-less by the workarounds that I have implemented for the bugs reported above. I have seen a number of threads whilst researching the \Noselect issue where people have been very confused. My finding was that \Noselect is a function of the IMAP specification server-side implementation RFC3501 (https://tools.ietf.org/html/rfc3501#section-6.3.6) And for me the server was returning directories with \Noselect because the mailboxes were malformed on account of dovadm-sync errors. In order to fix this I formed a bash command to transverse the mailbox hierarchy and create the missing folders critical to the sdbox format, namely DIRNAME. Kind regards, Arnold Opio Oree -----Original Message-----From: Arnold Opio Oree via dovecot < dovecot(a)dovecot.org>Reply-To: arnoldoree(a)parallaxict.com, Arnold Opio Oree <arnold.oree(a)parallaxict.com>To: dovecot(a)dovecot.orgCc: rs(a)sys4.de , aki.tuomi(a)open-xchange.comSubject: Re: Applying Dovecot for a large / deep folder-hierarchy archive.Date: Thu, 04 Jul 2019 14:52:28 +0100 Hi all, The guidance provided so far has been really helpful, and has helped a great deal to bringing down wasted energy on finding and executing a viable path. I am now at the final due action to complete our Dovecot application to our use-case, but am stuck on an issue that I cannot find any easily accessible documentation on. Generally this is what has been done: 1. Uploaded the enterprise data PST to the target groupware server.2. Prepared the server by changing the mailbox format to sdbox and the the Dovecot mail location to mail_location=/var/vmail/domain/user/mail/3. Converted the pst (on-server) to a recursive mbox hierarchy using readpst4. Executed doveadm-sync to convert mbox hierarchy data into sdbox and to copy it into the enterprise archive user's mailboxes4.i. The biggest issue I faced at this point was doveadm-sync saying that the source and destination pointed to the same location, whereas they clearly did not. 4.i.a. I resolved this by removing the location= setting from the target namespace, and allowing it to default to mail_location = setting, and then using a completely different DIRNAME for the import doveadm-sync execution (which was the desired final DIRNAME); I then once the sync had been successful, changed the mail_location DIRNAME so that it pointed to the imported mail DIRNAME; and hence the imported email data was in the live mailboxes4.i.b. doveadm-import failed several times, and was throwing quite inexplicable errors, so I moved onto doveadm-sync4.i.c. I also had to make sure that the source and destination folder names matched, otherwise doveadm-syc threw very many errors and only partially imported the data4.i.d. An issue which I decided just to live with is that an mbox DIRNAME was added to each mailbox as well as the DIRNAME specified so the path to mail is mbox/dbox-Mails. My thought is that with the data live on an IMAP server it will be possible to do a dysync through TCP to correct this problem. The final issue that I am facing now, is that when readpst finds empty folders in the source pst hierarchy, it does not create an mbox file in the mbox hierarchy folder space. This causes doveadm-sync to not create the target data required for its mailbox structure i.e. DIRNAME sub- folder and index file (with our configuration). At this point either doveadm-sync or the dovecot process makes these empty folders not selectable. The question now is how would I go about making all of these folders selectable, e.g. with an internal or external command line tool to change flags / create necessary sdbox mailbox constituent data? Many thanks, Arnold Opio Oree Chief Executive Officer Parallax Digital Technologies arnoldoree(a)parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons. -----Original Message-----From: Arnold Opio Oree via dovecot < dovecot(a)dovecot.org>Reply-To: arnoldoree(a)parallaxict.com, Arnold Opio Oree <arnold.oree(a)parallaxict.com>To: Robert Schetterer <rs(a)sys4.de>, dovecot(a)dovecot.orgSubject: Re: Applying Dovecot for a large / deep folder-hierarchy archive.Date: Thu, 27 Jun 2019 12:05:35 +0100 > Also you may run into client limits.... i ve seen this with outlook, > apple mail, thunderbird via imap in the past Thanks for this note Robert, it was not really an aspect that I wasconsidering. We are operating our groupware services user access through bothEvolution Groupware and KDE Kontact / KMail on Debian Linuxworkstations. Hopefully if there is a client issue it should be localto only one groupware client. I will be sure to study / investigate in this - client - area should any issues that are not traceable to the server-side arise. Many thanks, Arnold Opio OreeChief Executive OfficerParallax Digital Technologies arnoldoree(a)parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax GlobalLimited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not theintended recipient you are to delete this e-mail immediately, disregardits contents and disclose them to no other persons. -----Original Message-----From: Robert Schetterer via dovecot < dovecot(a)dovecot.org>Reply-To: Robert Schetterer <rs(a)sys4.de>To: dovecot(a)dovecot.org Subject: Re: Applying Dovecot for a large / deep folder- hierarchyarchive.Date: Thu, 27 Jun 2019 12:53:49 +0200 Am 27.06.2019 um 07:35 schrieb Aki Tuomi via dovecot: > On 26.6.2019 22.12, Arnold Opio Oree via dovecot wrote: > > Hello to you all, > > > > I'd like to ask about my intended application of Dovecot to create > > a folder-hierarchy for storing our enterprise emails, which are > > treated as live data rather than archives for compliance or > > occasional / reactive retrieval. > > > > The data is presently not that large (a few gigabytes), but it is > > expected to grow rapidly. Up to this stage the data has been > > contained > > in a Microsoft Exchange mailbox (2013), and then in an offline PST. > > The move to the offline PST was by necessity, as the large number > > of > > folders, and depth of hierarchy to my best understanding caused the > > exchange server / outlook / evolution mail clients to begin to > > malfunction. To cope with this the archive was broken up and the > > bulk stored in the offline PST and the most active components > > stored in > > online Exchange mailboxes. > > > > I have some understanding of the fs mbox format, and also the > > mitigations to be made for certain characters / strings. My main > > concern is whether Dovecot is likely to be able to cope well with a > > large number of folders / depth of hierarcy. > > > > I will really appreciate any help / advice you can give. > > > > Best regards, > > > > Arnold Opio Oree > > > > Hi! > > Dovecot 2.2.34/2.3 supports unlimited folder depth, the only limiting > factor is that the total name may not exceed 4096 bytes. Also > individual > folder names may not exceed 255 bytes. > > Prior to that the limit is 255 per folder up to 16 levels. > > I can't recommend using 'mbox' storage format, please consider using > maildir or sdbox instead. > > Aki > Chief Executive Officer Parallax Digital Technologies arnoldoree(a)parallaxdt.com http://www.parallaxdt.com tel : +44 (0) 333 577 8587 fax : +44 (0) 20 8711 2477 Parallax Digital Technologies is a trading name of Parallax Global Limited. U.K. Co. No. 08836288 The contents of this e-mail are confidential. If you are not the intended recipient you are to delete this e-mail immediately, disregard its contents and disclose them to no other persons.

1 0

Dovecot 2.3.0 TLS
by Hauke Fath 07 Jul '19

07 Jul '19

All, our dovecot installation provides a bundle of intermedia CA certificates using the ssl_ca option. 2.3.0 does not supply the bundle, resulting in various clients either complaining about an unverifiable server cert, or quietly not connecting. The log has Jan 5 17:01:46 Bounce dovecot: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=XXX, lip=YYY, TLS handshaking: SSL_accept() failed: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown: SSL alert number 46, session=<uKK/kAlia+GCUyU5> We fixed the issue by downgrading to 2.2.33.2. Cheerio, hauke -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344

11 22

Repository / builds for Debian "buster"?
by Kostya Vasilyev 07 Jul '19

07 Jul '19

Speaking of repositories... Debian "buster" (future version 10) has for some time been "stage 1 frozen" (if I have the terminology right) and quite usable. Perhaps it's starting to make sense to make a repo / start producing builds? -- Kostya Vasilyev kman(a)fastmail.com

2 2

Reprocess mail through sieve?
by ＠lbutlr 06 Jul '19

06 Jul '19

is there a way to process a mailbox through an existing sieve script as it the mail was being delivered anew? I’ve cleaned up a lot of my list-sorting scripts and I would like to process a bunch of mail again so it gets sorted properly. If I can do this could I also do it for a specific date range? For example, let’s say io want to reprocess all the messages that are in “Archive” from between 20190501 and 20190630? -- Think of how stupid the average person is, and realize half of them are stupider than that.

2 1

dovecot/imap [blocking on log write]
by alpha_one_x86 06 Jul '19

06 Jul '19

Hi, My dovecot process seam blocked on dovecot/imap [blocking on log write], only restart fix it. How solve that's? Cheers, -- alpha_one_x86/BRULE Herman <alpha_one_x86(a)first-world.info> Main developer of Supercopier/Ultracopier/CatchChallenger, Esourcing and server management IT, OS, technologies, research & development, security and business department

4 4

Dovecot and Solr on the same server or on different
by Götz Reinicke 06 Jul '19

06 Jul '19

Hi. what would you suggest? What are the pros and cons for having dovecot and Solr on the same or different hots? I have about 800 accounts, some millions of mails and about 2 TB of zipped mails. From the recent experiences regarding maintenance I’d prefer different hosts for each. Thanks for suggestions and feedback . Götz

5 12