- dovecot - dovecot.org

2 panics, 1 segmentation fault in dovecot 2.3.0 (fts)
by Infoomatic 30 Jan '18

30 Jan '18

Hi, While testing our migration to dovecot 2.3.0, we encountered 2 panics, 1 segmentation fault while we wanted to rebuild the solr indizes. Problem [1] was encountered while rebuilding the indizes, and now I don't know how many mails/documents are indexed or where it got stuck. [2] and [3] encountered while just playing around, since I did not know of "tokenize" and did not find a lot of information about it. This is a ubuntu 16.04 machine, dovecot 2.3.0 compiled from source ( + the patch for the recent zlib problem). "doveadm index" runs on a ~ 4GB mdbox-zipped-mailbox for about 3-4 minutes before aborting with the message above, beforehand I did a "/opt/dovecot/bin/doveadm fts rescan -A". When I repeat "doveadm index" it aborts after about 4 seconds, after a "/opt/dovecot/bin/doveadm fts rescan -A" it runs again for 3-4 minutes. Apache tika logs, but those are probably not related to the problem, however, the devs might want investigate the last line: WARN Parse exception Expected ';', got "," determining content disposition javax.mail.internet.ParseException: Expected ';', got "," and org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes I would be happy if you could have a look at this, please, and of course just tell me what further information you need. Thanks, infoomatic [1] /opt/dovecot/bin/doveadm index -u user1 '*' doveadm(user1): Panic: file unichar.c: line 160 (uni_ucs4_to_utf8_c): assertion failed: (uni_is_valid_ucs4(chr)) doveadm(user1): Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5eea) [0x7ff27755beea] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5f59) [0x7ff27755bf59] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ff2774ce5e1] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(uni_ucs4_to_utf8_c+0xa0) [0x7ff277593300] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xa74f9) [0x7ff27753d4f9] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(mail_html2text_more+0xc5) [0x7ff27753d705] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0xcf5c) [0x7ff276460f5c] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_parser_more+0x27) [0x7ff276460c37] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(fts_build_mail+0x511) [0x7ff27645eb51] -> /opt/dovecot/lib/dovecot/lib20_fts_plugin.so(+0x110a6) [0x7ff2764650a6] -> /opt/dovecot/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x2e) [0x7ff27784618e] -> /opt/dovecot/bin/doveadm(+0x2f697) [0x55dd2f912697] -> /opt/dovecot/bin/doveadm(+0x29827) [0x55dd2f90c827] -> /opt/dovecot/bin/doveadm(+0x2a4bb) [0x55dd2f90d4bb] -> /opt/dovecot/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55dd2f90e2cd] -> /opt/dovecot/bin/doveadm(doveadm_cmd_run_ver2+0x575) [0x55dd2f91deb5] -> /opt/dovecot/bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55dd2f91df07] -> /opt/dovecot/bin/doveadm(main+0x1d2) [0x55dd2f8fd032] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff2770ec830] -> /opt/dovecot/bin/doveadm(_start+0x29) [0x55dd2f8fd439] Aborted (core dumped) [2] This was just about my curiosity since I haven't used "tokenize" and did not find too much info about it: /opt/dovecot/bin/doveadm fts tokenize -A '*' Panic: file doveadm-print.c: line 142: unreached Error: Raw backtrace: /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5eea) [0x7f38ac5c2eea] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(+0xc5f59) [0x7f38ac5c2f59] -> /opt/dovecot/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f38ac5355e1] -> /opt/dovecot/bin/doveadm(+0x3b3d3) [0x55c728e8b3d3] -> /opt/dovecot/bin/doveadm(+0x2a63a) [0x55c728e7a63a] -> /opt/dovecot/bin/doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x21d) [0x55c728e7b2cd] -> /opt/dovecot/bin/doveadm(doveadm_cmd_run_ver2+0x575) [0x55c728e8aeb5] -> /opt/dovecot/bin/doveadm(doveadm_cmd_try_run_ver2+0x37) [0x55c728e8af07] -> /opt/dovecot/bin/doveadm(main+0x1d2) [0x55c728e6a032] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f38ac153830] -> /opt/dovecot/bin/doveadm(_start+0x29) [0x55c728e6a439] Aborted (core dumped) [3] /opt/dovecot/bin/doveadm fts tokenize -u user1 "@" Segmentation fault (core dumped)

2 2

Dovecot 2.2.33.2 doveadm log reopen don't work
by Айхэд 30 Jan '18

30 Jan '18

dovecot -n # 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.4.21 (92477967) # OS: FreeBSD 11.1-RELEASE-p1 amd64 zfs from logrotate.conf: /var/log/dovecot/dovecot.log { rotate 180 missingok nomail noolddir compress create 0600 nobody nobody postrotate # /usr/local/bin/doveadm log reopen /usr/sbin/service dovecot reload endscript } After upgrade to 2.2.33.2 (prev version was 2.2.29.1) there are only messages from sieve in /var/log/dovecot/dovecot.log like this: Jan 29 00:00:12 lda(***): Info: sieve: msgid=<20180128210012.3A9A11138B8E@***>: stored mail into mailbox 'INBOX.' (INBOX) Messages from pop3, pop3-login, imap, imap-login are absent. If i replace /usr/local/bin/doveadm log reopen with /usr/sbin/service dovecot reload all works fine. Before upgrade all worked fine.

1 0

Change destination of dsync
by Sergio Belkin 29 Jan '18

29 Jan '18

Hi, I have the following command for retrieving mails from another IMAP mail server dsync -D -v backup -R -u jon.doe imapc: I't ok and works, but I'd want to copy messages outside of /var directory, is there a way to do that? Thanks in advance! -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org

2 1

problem with lda
by Infoomatic 28 Jan '18

28 Jan '18

Hello, I am currently testing the upgrade from our dovecot v2.2.23 to 2.3.0. I have a strange problem with lda. We use bounce mails for internal users if they try to send virus/spammails. The exact same config works with 2.2.23, but not with 2.3.0. The relevant postfix/master.cf part: dovecot unix - n n - - pipe flags=DRhu user=vmail:vmail argv=/opt/dovecot/libexec/dovecot/deliver -d ${recipient} -f ${sender} On the same machine, with 2.2.23 I get a bounce mail when trying to send a virus, with the subject "VIRUS in message apparently from you (Eicar-Test-Signature)", to both the users inbox as well as to the mailbox of virusalert(a)ourdomain.internal With 2.3.0, in the logs I get an error: relay=dovecot, delay=0.01, delays=0/0/0/0, dsn=5.3.0, status=bounced (command line usage error. Command output: lda: Fatal: Invalid -f parameter: Missing domain and the alert message is only sent to virusalert(a)ourdomain.internal, but not to the user who tried to send the mail. The relevant part of our dovecot.conf: protocol lda { mail_plugins = " quota zlib sieve acl mail_log notify" postmaster_address = postmaster(a)ourdomain.internal } Does anyone know how to solve this problem? Or is there a workaround via postfix/amavis/spamassassin? Any hints are highly appreciated, thanks! Regards, infoomatic

3 6

Send full X.509 client certificate to custom authentication policy server
by Jaime Hablutzel Egoavil 27 Jan '18

27 Jan '18

I'm working with Dovecot 2.3 and I'm wondering if I could send the full X.509 client certificate to my custom authentication policy server. I'm actually aware that I can send the client certificate validity status with something like: auth_policy_request_attributes = ... cert=%{cert} But I want the full X.509 certificate to be able to decide over the basis of certificate extensions, e.g. Certificate Policies extension. Is it currently possible?, what about Lua based authentication?, does Lua currently receive the full client certificate?. -- Jaime Hablutzel - RPC 994690880

2 1

exited on signal 6 (core dumped) when searching folder
by Kristjan Eentsalu 26 Jan '18

26 Jan '18

Hey, I'm getting messages exited on signal 6 (core dumped) when doing imap command "a UID SORT (DATE) UTF-8 BODY someword" message log file Jan 26 13:57:20 mail dovecot: imap(kristjan.eentsalu(a)yyy.yy): Panic: file charset-iconv.c: line 87 (charset_to_utf8_try): assertion failed: (srcleft <= CHARSET_MAX_PENDING_BUF_SIZE) Jan 26 13:57:20 mail dovecot: imap(kristjan.eentsalu(a)yyy.yy): Fatal: master: service(imap): child 6410 killed with signal 6 (core dumped) Problem seems to to be with one old mail. OS: FreeBSD 11.1-RELEASE-p4 Dovecot 2.2.33.2 , pigenhole 0.4.21 trace, conf and email available https://pastebin.com/7X6fZPPd Kristjan

2 1

disable deflate compression in rawlog files
by Daniel Betz 26 Jan '18

26 Jan '18

Hello list, is there an way to disable the deflate compression ( imap_zlib ) in the rawlog files, e.g. via postlogin script when dovecot.rawlog folder exists ? Rawlog shown at the moment like: 1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE COMPRESS=DEFLATE QUOTA] Logged in * NAMESPACE (("INBOX." ".")) NIL NIL 2 OK Namespace completed (0.001 + 0.000 secs). 3 OK Begin compression (0.001 + 0.000 secs). <D2>R<F0>tQ<D0>P<CA>K<CC>MURPr<C9>/KM<CE>/........ Regards, Daniel Betz System Design Engineer / Senior Systemadministration ___________________________________ domainfactory GmbH Oskar-Messter-Str. 33 85737 Ismaning Germany Telefon: +49 (0)89 / 55266-364 Telefax: +49 (0)89 / 55266-222 E-Mail: dbetz(a)df.eu Internet: www.df.eu Registergericht: Amtsgericht München HRB-Nummer 150294, Geschäftsführer: Dr. Claus Boyens, Tobias Mohr

1 0

Quota plugin creates IMAP folder
by Thorsten Hater 26 Jan '18

26 Jan '18

Hi everyone, I have noticed recently that Dovecot creates lock directories under users' Maildirs which look like IMAP folders. This is bad since users will see IMAP folders that are not supposed to be there. In my logs I see dovecot: imap(...): Error: file_create_locked(/.../Maildir/.vsize.lock) failed: Is a directory The directory in question # ls /.../Maildir/.vsize.lock/ cur/ dovecot.index.log dovecot-uidlist maildirfolder new/ tmp/ so Dovecot seems to think this to be an IMAP folder. I am using Dovecot 2.2.27 under Debian, built from source, the lock_method is set to fnctl, and the quota plugin uses the maildir method. Can somebody advice me on how to fix this? Best regards, Thorsten

3 7

Single mailbox optimizations
by Alex Shishkin 25 Jan '18

25 Jan '18

Hello, I have unconventional load profile for my email server and I would like to ask if someone can suggest optimizations in my case. I have only one mailbox on the server for collecting emails from variety of external mail servers. I use Postfix for collecting emails and Dovecot to provide IMAP for downloading emails from this mailbox. Currently Postfix configured to use LMTP for transferring emails to Dovecot. Dovecot mail storage format is mdbox. Everything was in order (no incoming and active queues in Postfix) when daily incoming email count was up to 1 million emails. But now I have to process up to 2 million incoming emails and I faced bottleneck of 10 emails per second for saving emails from Postfix active queue to mdbox using LMTP. Currently Postfix receives about 20 emails per second, incoming and active queues in Postfix are growing and I have to find the way to speed up transferring process from Postfix to mdbox. This setup is running on the virtual machine with 8 CPU and 8 GB RAM. CPU loaded only by dovecot/lmtp process but not more than 20% per core. I/O wait is 2-3%. Disk load is about 200 wMB/sec and 300 rKB/sec (and this sounds quite a bit for our storage capacity) . Postfix has destination concurrency limit 20 for default and virtual transport (I’m using virtual transport due to virtual domains configuration). Strace showed up that dovecot/lmtp processes have some delays only on waiting for write lock on truncation log file dovecot.index.log. I have temporary moved all index files including truncation log to tmpfs, but this did not help. The only thing that helped me to increase speed up to 20 emails per second is removing almost all emails from the mailbox (with empty Inbox I got even 30 emails per second). But I have to keep all incoming emails in the mailbox at least for one week so this way is unacceptable. So I would like to ask for suggestions, because I can’t find the root cause of this bottleneck. Dovecot version 2.2.31 Postfix version 2.10.1 ------------------------------------------------Dovecot conf ------------------------------------------------ mail_location = mdbox:~/mdbox mdbox_rotate_interval = 1 days mdbox_rotate_size = 200MB mail_plugins = " zlib” plugin { zlib_save = gz zlib_save_level = 6 } userdb { args = /etc/dovecot/passwd default_fields = uid=vmail gid=vmail home=/opt/vmail/%u driver = passwd-file } passdb { args = /etc/dovecot/passwd driver = passwd-file } service lmtp { process_min_avail = 8 service_count = 0 unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } vsz_limit = 0 } ------------------------------------------------------------------------------------------------ --------------------------------------Postfix conf------------------------------------------- virtual_alias_maps = hash:/etc/postfix/virtual virtual_mailbox_domains = test.com, example.com virtual_mailbox_maps = hash:/etc/postfix/vmailbox virtual_transport = lmtp:unix:private/dovecot-lmtp default_destination_concurrency_limit = 20 virtual_destination_concurrency_limit = 20 ------------------------------------------------------------------------------------------------

1 0

Re: dovecot Digest, Vol 177, Issue 55
by Matthew Brown 25 Jan '18

25 Jan '18

No reason to tell me about security, I just need chmod for my tests. Where to change it ? On Thu, Jan 25, 2018 at 12:00 PM, <dovecot-request(a)dovecot.org> wrote: > Send dovecot mailing list submissions to > dovecot(a)dovecot.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://dovecot.org/mailman/listinfo/dovecot > or, via email, send a message with subject or body 'help' to > dovecot-request(a)dovecot.org > > You can reach the person managing the list at > dovecot-owner(a)dovecot.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of dovecot digest..." > > > Today's Topics: > > 1. Re: Panic: data stack: Out of memory when allocating bytes > (Josef 'Jeff' Sipek) > 2. Re: 777 permissions on mailbox, subfolders subfiles new > emails (Nikolai Lusan) > 3. Re: problem with lda (Stephan Bosch) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 24 Jan 2018 17:39:36 -0500 > From: Josef 'Jeff' Sipek <jeff.sipek(a)dovecot.fi> > To: Thomas Robers <robers(a)tutech.de> > Cc: dovecot(a)dovecot.org > Subject: Re: Panic: data stack: Out of memory when allocating bytes > Message-ID: <20180124223936.GE14966@meili> > Content-Type: text/plain; charset=us-ascii > > On Wed, Jan 24, 2018 at 18:55:47 +0100, Thomas Robers wrote: > > Am 23.01.2018 um 20:07 schrieb Josef 'Jeff' Sipek: > > > On Tue, Jan 23, 2018 at 14:03:27 -0500, Josef 'Jeff' Sipek wrote: > > > > On Tue, Jan 23, 2018 at 18:21:38 +0100, Thomas Robers wrote: > ... > > > > 1. Do you have any idea what the imap process was doing at the time > of the > > > > allocation failure? > > > > Yes perhaps. We use shared mailboxes and at the time of failure the > > imap process is reading acl files in a shared mailbox (and subfolder). > > This shared mailbox has about 2800 subfolder and the acl files are read > > in about 10sec and and during this reading the imap process dies with > > the already mentioned error. It seems it has something to do with the > > shared mailbox, because since yesterday morning 5 core dumps have been > > created, 4 of them by one user accessing the shared mailbox and 1 > > by the user who is the owner of the shared mailbox. No other mailboxes > > are affected until now. > > Based on the stack trace, the client was creating a new mailbox, which > caused an acl list rebuild, which ended up triggering this oddly sized > allocation. > > > > > 2. You snipped all the important parts of the back trace. :) It > *starts* on > > > > the line: > > > > #0 0x00007f73f1386495 in raise () from /lib64/libc.so.6 > > > > > > In case you haven't used gdb before... after starting up gdb, run "bt > full" > > > at the gdb prompt. That'll print out a very detailed backtrace. (You > might > > > want to sanity check it to make sure there aren't any user passwords > in it > > > before posting it here...) > > > > Yes, sorry I'm not very familiar in using gdb, but here is the full > > backtrace: > > It looks like the binaries are stripped. There should be a "debug" package > you can install with symbol information. Then, the backtrace should be > much > more helpful. > > > > > Having the backtrace should help answer question number 1. > > > > 3. How big is this process when it dies? > > > > I don't know which imap process dies beforehand so how do i get this > > information? > > The size of the core file will give you an general idea. gdb can also > print > out lots of info via "maintenance info sections", but I don't think that'll > help figuring out why things blow up. > > Jeff. > > -- > Keyboard not found! > Press F1 to enter Setup > > > ------------------------------ > > Message: 2 > Date: Thu, 25 Jan 2018 13:09:40 +1000 > From: Nikolai Lusan <nikolai(a)lusan.id.au> > To: dovecot(a)dovecot.org > Subject: Re: 777 permissions on mailbox, subfolders subfiles new > emails > Message-ID: <1516849780.4714.5.camel(a)lusan.id.au> > Content-Type: text/plain; charset="UTF-8" > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > On Thu, 2018-01-25 at 00:35 +0200, Matthew Brown wrote: > > What I have to change, so dovecot on new email create, sets chmod 777 on > > Mailbox and all subfolders subfiles all new emails/sent emails etc. > > This is a really bad idea. It is possible to setup in dovecot if you're > using dovecot as your LDA. The only reason I can think of for anything > close to this is virtual mailboxes ... and even then I set things up with a > specific user to own the files and set them as 770 for the directories and > 660 for the files. With that setup dovecot limits users to their own mail > directories with values returned via the passdb/userdb. > > As a general rule anything on a *nix system that is chmod 777 is to be > avoided - it's a massive security risk. > > - -- > Nikolai Lusan <nikolai(a)lusan.id.au> > -----BEGIN PGP SIGNATURE----- > > iQIzBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAlppSnQACgkQ4ZaDRV2V > L6To/BAAjKeTi6reaRrcSClKm6PEbgewtF1KvHgO1zr+hv9T2XOnVGFSajeFwSXR > rDSXGQ/H9s97KDGZB4VKxuSKZsDRFkF9u7ILttuXOgj8kG8XZl4sqg/w0FIrTC1K > GD8T0TZWSJGFIzrLXTpgzoO1LeQXIuXgnZIMbcaneCHKZc1do/GwQ4DEFVdUnxRJ > n+76nbs8RdQwZKGXYGGTU/BXPniTSW1pmTFxnlruxE7fuWBYeEyQGQRke/CT2rFF > ET+d7LlV4Nds6eGjalyf7m5tCNS87Zm78xegANuMNaMdzPI8NlbnFO82bAlC7kP/ > WkCFSfhJL3s9daEnMWT/dxY2lNKy3kTyOZssXM6ROkNhOFRipNwRr77tO6qzu8l9 > pvaVxr8a0pMVQwxGAyW84vS9GWmeMNMMiY0k7XdPSBS/6CVlQm8eUhnDo6GFZoH9 > Tbn4XNkEKysbZEjImjXb6eQpRwQLniIULjeyoX8aqmN2sh4kDndLJwjxpz7Jd6v2 > j2tuS7+79G9qU8dqpCSwzvWYTteOLJeqJLy6s5F+zf8FHGLcHdNasoo+/Sdzg4+E > tZm31wp7iQdzKMwbOnRk9FceYXbb9WjxI1QY+7U11Rx25y44UDPzUYYOL+ZoWOXJ > zAr/F9fhUP/u8AubOgOn6BLaguiWcFK0g6HhCqDkVAUibEMzC+0= > =YUek > -----END PGP SIGNATURE----- > > > > ------------------------------ > > Message: 3 > Date: Thu, 25 Jan 2018 09:35:57 +0100 > From: Stephan Bosch <stephan(a)rename-it.nl> > To: Infoomatic <infoomatic(a)gmx.at>, dovecot(a)dovecot.org > Subject: Re: problem with lda > Message-ID: <2544728b-5441-c163-34fd-bd155e28f591(a)rename-it.nl> > Content-Type: text/plain; charset=utf-8 > > Op 1/24/2018 om 5:30 PM schreef Infoomatic: > > Hello, > > > > I am currently testing the upgrade from our dovecot v2.2.23 to 2.3.0. I > have a strange problem with lda. > > We use bounce mails for internal users if they try to send > virus/spammails. The exact same config works with 2.2.23, but not with > 2.3.0. > > > > The relevant postfix/master.cf part: > > dovecot unix - n n - - pipe > > flags=DRhu user=vmail:vmail argv=/opt/dovecot/libexec/dovecot/deliver > -d ${recipient} -f ${sender} > > > > On the same machine, with 2.2.23 I get a bounce mail when trying to send > a virus, with the subject "VIRUS in message apparently from you > (Eicar-Test-Signature)", to both the users inbox as well as to the mailbox > of virusalert(a)ourdomain.internal > > > > With 2.3.0, in the logs I get an error: > > > > relay=dovecot, delay=0.01, delays=0/0/0/0, dsn=5.3.0, status=bounced > (command line usage error. Command output: lda: Fatal: Invalid -f > parameter: Missing domain > > > > and the alert message is only sent to virusalert(a)ourdomain.internal, > but not to the user who tried to send the mail. > > > > The relevant part of our dovecot.conf: > > protocol lda { > > mail_plugins = " quota zlib sieve acl mail_log notify" > > postmaster_address = postmaster(a)ourdomain.internal > > } > > > > Does anyone know how to solve this problem? Or is there a workaround via > postfix/amavis/spamassassin? Any hints are highly appreciated, thanks! > > Apparently, the ${sender} has no @domain part. Why? > > Regards, > > Stephan. > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > dovecot mailing list > dovecot(a)dovecot.org > https://dovecot.org/mailman/listinfo/dovecot > > ------------------------------ > > End of dovecot Digest, Vol 177, Issue 55 > **************************************** > -- zxzzzzzzzzzzzzz

1 1