April 2017 - dovecot - dovecot.org

[Bug] FTS double escaping
by azurit＠pobox.sk 09 Apr '17

09 Apr '17

Hi, i'm trying to resolve few problems with indexing 'From' headers using FTS/Solr. I was tcpdumping the communication between Dovecot and Jetty/Solr and noticed that 'From' headers, which includes also sender's name, are double escaped. This is what was Dovecot sending to Solr: </field><field name="from">Name Surname &lt;test(a)example.com&gt;</field></doc></add> As you can see, characters < and > were escaped to < and > which were, again, escaped to &lt; and &gt;. This is doing problems while trying to index whole e-mail address, as Solr sees it as '<test(a)example.com>'. I spend hours trying to figure out why i'm able to search in all parts of e-mail addresses but searching for full and exact e-mail address was successfull ONLY for messages which doesn't include sender's name in 'From' header. Finally, after i found this bug, this fixed all search problems: <filter class="solr.PatternReplaceFilterFactory" pattern="&lt;" replacement=""/> <filter class="solr.PatternReplaceFilterFactory" pattern="&gt;" replacement=""/> I hope that, at least, this bug, reported by me, will be fixed. Thank you. azur

3 5

sieve does not seem to be working
by Robert Moskowitz 09 Apr '17

09 Apr '17

my local.conf has: # 90-sieve.conf plugin { sieve_before = /home/sieve/globalfilter.sieve } and cat /home/sieve/globalfilter.sieve require ["fileinto","mailbox"]; if anyof ( header :contains "X-Spam-Flag" "YES", header :contains "subject" "***SPAM***" ) { fileinto :create "Spam"; } There IS a globalfilter.svbin when I tried: sendmail -i testit3(a)test.htt-consult.com < sample-spam-GTUBE-junk.txt amavis is flagging it as ***Spam*** but it stays in inbox. So I tried: sieve-test -e -l /home/vmail/test.htt-consult.com/testit3/ /home/sieve/globalfilter.sieve /home/vmail/test.htt-consult.com/testit3/cur/1491512409.M371278P6513.z9m9z.test.htt-consult.com\,S\=1823\,W\=1868\:2\, info: msgid=<GTUBE1.1010101(a)example.net>: stored mail into mailbox 'Spam'. sieve-test(root): Info: final result: success And it DID get copied to Spam, not moved. I am now seeing it in inbox and Spam. Of course ownership on the message in Spam were wrong (root:root instead of vmail:mail, but I fixed that). So two questions, probably linked: Why did sieve not work. Is the subject test case sensitive? If it is case sensitive, why did the sieve-test work? thanks

2 2

Pigeonhole Sieve Extprograms Plugin
by KT Walrus 09 Apr '17

09 Apr '17

Just discovered the Pigeonhole Sieve Extprograms plugin (https://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/Extprograms <https://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/Extprograms>)… I currently have Postfix configured to run a script to store all incoming messages into a Mysql DB and a batch job that retrieves the messages in the DB, processes them and delivers them to the recipients' Dovecot servers. Seems to me it might be better to have Postfix deliver all incoming messages directly to a single Dovecot mailbox and use this Sieve plugin to do the whole job of updating the DB, processing the message, and delivering to the recipients. Also, I have a need to implement updating the DB when the recipient reads a message for the first time (setting the \Seen flag). Would this work best to execute a script using this Sieve plugin from a IMAP Sieve script that watches for \Seen flag changes? This might be a whole lot better than the way I was thinking of doing it before. I’m basically implementing mailing lists where the senders receive a notification reply with a link to a web page where the message may be edited/deleted before it is sent to recipients (after a fixed amount of time has elapsed). This gives the sender the opportunity to change his mind before it is too late or to send to recipients immediately (by marking the message as Urgent). Also, even after the message has been sent out, I want to allow the sender to be able to delete it in the recipients mailboxes if they have not read it yet (downloaded it to their email client) or to see which recipients have read the message (and when they read/downloaded it for the first time). Both the sender and the recipients will have Dovecot mailboxes so I want to give more control over the messages for their entire lifetimes (both for senders and for recipients) than normal email does. Any thoughts? Kevin

1 0

Released Pigeonhole v0.4.18.rc1 for Dovecot v2.2.29.rc1.
by Stephan Bosch 09 Apr '17

09 Apr '17

Hello Dovecot users, This is a rather small release. It fixes one important bug in the new discard script feature. It also adds a couple of new features for IMAPSieve. Changelog v0.4.18: + imapsieve plugin: Implemented the copy_source_after rule action. When this is enabled for a mailbox rule, the specified Sieve script is executed for the message in the source mailbox during a "COPY" event. This happens only after the Sieve script that is executed for the corresponding message in the destination mailbox finishes running successfully. + imapsieve plugin: Added non-standard Sieve environment items for the source and destination mailbox. - multiscript: The execution of the discard script had an implicit "keep", rather than an implicit "discard". The release is available as follows: http://pigeonhole.dovecot.org/releases/2.2/rc/dovecot-2.2-pigeonhole-0.4.18… http://pigeonhole.dovecot.org/releases/2.2/rc/dovecot-2.2-pigeonhole-0.4.18… Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this release candidate and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch stephan(a)rename-it.nl

1 0

Updating mysql db when messages first read by IMAP user
by KT Walrus 09 Apr '17

09 Apr '17

I need to implement updating a mysql db when the \SEEN flag is set by IMAP. New message delivery is done by having Postfix store all new inbound messages in a mysql db and a batch job watching the db for new messages and delivering them to the correct Dovecot server for the recipients. The batch job leaves a log entry in the db (for each recipient) with the current timestamp and adds a header with the log id to the outgoing message that is sent to Dovecot. The log table has a column for the first read timestamp which is set to 0 initially. I plan on implementing a Sieve IMAP script that watches for changes to the \SEEN flag and sends a mailto: notify message that contains the log id (from the message header) and the current date to a mail address dedicated for keep track of \SEEN flag changes. The notify message will be delivered to this mail address where a batch job watches that mail address and updates the db for the log id in the notify message with the timestamp and the current date if the row in the log table has a first read timestamp of 0. I plan on providing a web page that the sender can use to see whether the messages they have sent have been seen by the recipients. The web page will show other info too, but the part that I need to implement next is updating the db when a sent message has been read by the recipients of the message. Is this a good way to do this? Or, is there a better way than using an IMAP Sieve script to send notify messages? Is there any other way to have a Sieve script update a mysql database? Or, is there some other mechanism in Dovecot to use instead of an IMAP Sieve script? Kevin

1 0

IMAPSieve Changed flags cause?
by MRob 09 Apr '17

09 Apr '17

RFC 6785 says I should be able to run scripts when a command changes flags on a message but I can't understand what to put in imapsieve_mailboxXXX_causes. Dovecot logs something like STORE as an invalid cause. How do I run an administrator Sieve script caused from change in flags?

2 5

LAYOUT=fs and subfolders listing
by Dovecot＠9livesdata 07 Apr '17

07 Apr '17

dovecot-2.2.28-1.el6_31.wing.x86_64 I've created this public shared namespace with namespace public2 { type = public separator = . prefix = Public2. location = maildir:/mail/public2:INDEX=/mail/%u/public:LAYOUT=fs:DIRNAME=.store subscriptions = no } and I'm using the "imap_client_workarounds = tb-extra-mailbox-sep" and filesystem structure listed at the end. The problem is that with this configuration thunderbird (and roundcubemail) shows the Public2 namespace but empty - without any folders. I've tried with another DIRNAME=StoR - same effect If I will not use DIRNAME at all(and no .stor directories) TB sees the f1 f2 folders but not the f2/s1 subfolder. Just like it would not list any subfolders for subscription. Within TB I can create subfolder say f1/mysub1 but after it's unsubscribed cannot be subscribed again(not shown in listing) Can You provide any hints? I can live without the DIRNAME= but it has to list subfolders for subscription. BTW: it works well without LAYOUT=fs - but it's kind of requirement here. /mail/public2/: drwxrwsr-x 4 nobody havemail 4096 Apr 7 15:07 f1 drwxrwsr-x 3 nobody havemail 4096 Apr 7 15:02 f2 /mail/public2/f1: -rw-r--r-- 1 root havemail 42 Apr 7 15:07 dovecot-acl -r--r--r-- 1 nobody havemail 0 Oct 14 2015 dovecot-shared -rw-rw-r-- 1 nobody havemail 0 Oct 15 2015 dovecot-uidlist drwxrwsr-x 3 nobody havemail 4096 Apr 7 15:02 s1 drwxrws--- 5 nobody havemail 4096 Apr 7 15:02 .store /mail/public2/f1/s1: -rw-r--r-- 1 root havemail 40 Oct 16 2015 dovecot-acl -r--r--r-- 1 nobody havemail 0 Oct 14 2015 dovecot-shared -rw-rw-r-- 1 nobody havemail 0 Oct 15 2015 dovecot-uidlist drwxrws--- 5 nobody havemail 4096 Apr 7 15:02 .store /mail/public2/f1/s1/.store: drwxrws--- 2 nobody havemail 4096 Oct 15 2015 cur drwxrws--- 2 nobody havemail 4096 Oct 15 2015 new drwxrws--- 2 nobody havemail 4096 Oct 15 2015 tmp /mail/public2/f1/.store: drwxrws--- 2 nobody havemail 4096 Oct 15 2015 cur drwxrws--- 2 nobody havemail 4096 Oct 15 2015 new drwxrws--- 2 nobody havemail 4096 Oct 15 2015 tmp /mail/public2/f2: -rw-r--r-- 1 root havemail 40 Oct 16 2015 dovecot-acl -r--r--r-- 1 nobody havemail 0 Oct 14 2015 dovecot-shared -rw-rw-r-- 1 nobody havemail 0 Oct 15 2015 dovecot-uidlist drwxrws--- 5 nobody havemail 4096 Apr 7 15:02 .store /mail/public2/f2/.store: drwxrws--- 2 nobody havemail 4096 Oct 15 2015 cur drwxrws--- 2 nobody havemail 4096 Oct 15 2015 new drwxrws--- 2 nobody havemail 4096 Oct 15 2015 tmp

1 0

Running sievec as user vmail
by Robert Moskowitz 07 Apr '17

07 Apr '17

My sieve problem ended up a permissions problem. I ran sievec as root and .svbin needs vmail:mail ownership. I could always just add the chown command to my process, but I wonder if there is some 'clean' way to run sievec as user vmail while logged in as root? thanks

2 2

Doubt with imap/imap-login processes in proxy server
by Angel L. Mateo 07 Apr '17

07 Apr '17

Hi, I have a question I have not cleared after reading https://wiki.dovecot.org/LoginProcess. I'm developing a proxy server (without director, because the backend server is specified by a ldap lookup). My question is: the limit of imap (or pop3) concurrently users that the proxy can handle is determined by the imap-login limit or the imap one? -- Angel L. Mateo Martínez Sección de Telemática Área de Tecnologías de la Información y las Comunicaciones Aplicadas (ATICA) http://www.um.es/atica Tfo: 868889150 Fax: 868888337

2 2

Re: abrt reported "imap killed by SIGBUS"
by Aki Tuomi 07 Apr '17

07 Apr '17

Please keep responses on list. Can you please run valgrind on this, as it appears to be some sort of memory corruption issue. Aki > On April 7, 2017 at 5:37 AM Hongying Liu <hongyliu(a)redhat.com> wrote: > > > Thanks for your response. > > (gdb) p cache > $1 = (struct mail_cache *) 0x7fc4a811a320 > (gdb) p cache->hdr > $2 = (const struct mail_cache_header *) 0x7fc4a6ddb000 > (gdb) p cache->hdr->file_seq > Cannot access memory at address 0x7fc4a6ddb008 <---- > (gdb) p reset_id > $3 = 1488892251 > > > 2017-04-06 19:34 GMT+08:00 Aki Tuomi <aki.tuomi(a)dovecot.fi>: > > > Hi! > > > > Responses in middle. > > > > Aki > > > > On 06.04.2017 06:05, Hongying Liu wrote: > > > Hi sbr-services, > > > > > > Could you give me some idea? > > > > > > ### abrt reported the error as below. > > > [root@cupop4 log]# abrt-cli list --since 1488267244 id > > > ad716dbfd3a68bbe0f055e32ebfe562f4f75df43 > > > reason: imap killed by SIGBUS > > > time: Sun Mar 19 2017 10:58:27 AM JST > > > cmdline: dovecot/imap > > > package: dovecot-2.2.10-7.el7 > > > uid: 80180 (acaa2325) > > > count: 2 > > > Directory: /var/spool/abrt/ccpp-2017-03-19-10:58:27-4904 > > > 'abrt-cli report を実行する/var/spool/abrt/ccpp-2017-03-19-10:58:27-4904' > > > > > > > > > ### /var/log/messages > > > Mar 19 10:58:27 cupop4 abrt-hook-ccpp: Process 4904 (imap) of user 80180 > > > killed by SIGBUS - dumping core > > > > > > And, there are lots of ldap error in /var/log/messages. Dovecot uses the > > > ldap as userdb and passwddb. > > > Is the ldap error related to imap crash? > > > > > > Mar 19 10:58:21 cupop4 nslcd[1534]: [a378de] <group/member="postfix"> > > > ldap_result() failed: Invalid DN syntax: Invalid DN > > > > This looks like an invalid DN syntax. > > > > > > > > ### /var/log/dovecot > > > > > > Mar 19 10:58:27 cupop4 dovecot: imap(acaa2325): Fatal: master: > > > service(imap): child 4904 killed with signal 7 (core dumped) > > > > > Can you do p cache, p cache->hdr, p cache->hdr->file_seq and p reset_id > > > > > ### gdb > > > > > > Core was generated by `dovecot/imap'. > > > Program terminated with signal 7, Bus error. > > > #0 0x00007fc4a69f293e in mail_cache_lookup_offset (cache=0x7fc4a811a320, > > > view=<optimized out>, seq=<optimized out>, offset_r=offset_r@entry > > > =0x7ffe23a98900) > > > at mail-cache-lookup.c:95 > > > 95 while (cache->hdr->file_seq != reset_id) { > > > (gdb) bt > > > #0 0x00007fc4a69f293e in mail_cache_lookup_offset (cache=0x7fc4a811a320, > > > view=<optimized out>, seq=<optimized out>, offset_r=offset_r@entry > > > =0x7ffe23a98900) > > > at mail-cache-lookup.c:95 > > > #1 0x00007fc4a69f2afa in mail_cache_lookup_iter_init > > > (view=view@entry=0x7fc4a81332b0, > > > seq=<optimized out>, ctx_r=ctx_r@entry=0x7ffe23a988e0) > > > at mail-cache-lookup.c:154 > > > #2 0x00007fc4a69f2f8f in mail_cache_seq (seq=<optimized out>, > > > view=0x7fc4a81332b0) at mail-cache-lookup.c:322 > > > #3 mail_cache_field_exists (view=view@entry=0x7fc4a81332b0, > > seq=seq@entry=760, > > > field=field@entry=4) at mail-cache-lookup.c:352 > > > #4 0x00007fc4a69f3151 in mail_cache_lookup_field (view=0x7fc4a81332b0, > > > dest_buf=dest_buf@entry=0x7ffe23a98a00, seq=760, > > field_idx=field_idx@entry > > > =4) > > > at mail-cache-lookup.c:413 > > > #5 0x00007fc4a69d9439 in index_mail_cache_lookup_field > > > (mail=mail@entry=0x7fc4a8128cd0, > > > buf=buf@entry=0x7ffe23a98a00, field_idx=field_idx@entry=4) > > > at index-mail.c:68 > > > #6 0x00007fc4a69d94a7 in index_mail_get_fixed_field > > (mail=0x7fc4a8128cd0, > > > field=<optimized out>, data=<optimized out>, data_size=8) at > > > index-mail.c:130 > > > #7 0x00007fc4a69d9fed in index_mail_get_cached_virtual_size > > > (mail=mail@entry=0x7fc4a8128cd0, size_r=size_r@entry=0x7ffe23a98b20) at > > > index-mail.c:403 > > > #8 0x00007fc4a6987c1b in maildir_mail_get_virtual_size > > > (_mail=0x7fc4a8128cd0, size_r=0x7ffe23a98b20) at maildir-mail.c:388 > > > #9 0x00007fc4a69d9719 in index_mail_update_access_parts > > > (mail=mail@entry=0x7fc4a8128cd0) > > > at index-mail.c:1490 > > > #10 0x00007fc4a69dbb93 in index_mail_set_seq (_mail=0x7fc4a8128cd0, > > > seq=760, saving=<optimized out>) at index-mail.c:1521 > > > #11 0x00007fc4a69e28cb in search_more_with_mail (mail=0x7fc4a8128cd0, > > > ctx=0x7fc4a8132730) at index-search.c:1507 > > > #12 search_more_with_prefetching (mail_r=<optimized out>, ctx=<optimized > > > out>) at index-search.c:1579 > > > #13 search_more (ctx=ctx@entry=0x7fc4a8132730, > > > mail_r=mail_r@entry=0x7ffe23a98c20) > > > at index-search.c:1650 > > > #14 0x00007fc4a69e2ff4 in index_storage_search_next_nonblock > > > (_ctx=0x7fc4a8132730, mail_r=0x7fc4a81120d8, tryagain_r=0x7ffe23a98c87) > > at > > > index-search.c:1674 > > > #15 0x00007fc4a69bc4ef in mailbox_search_next_nonblock > > > (ctx=ctx@entry=0x7fc4a8132730, > > > mail_r=mail_r@entry=0x7fc4a81120d8, > > > tryagain_r=tryagain_r@entry=0x7ffe23a98c87) at mail-storage.c:1787 > > > #16 0x00007fc4a69bc55d in mailbox_search_next (ctx=0x7fc4a8132730, > > > mail_r=mail_r@entry=0x7fc4a81120d8) at mail-storage.c:1773 > > > #17 0x00007fc4a6e97874 in imap_fetch_more_int (ctx=ctx@entry > > =0x7fc4a8112078, > > > cancel=false) at imap-fetch.c:479 > > > #18 0x00007fc4a6e98852 in imap_fetch_more (ctx=0x7fc4a8112078, > > > cmd=cmd@entry=0x7fc4a8111ed0) > > > at imap-fetch.c:556 > > > #19 0x00007fc4a6e8c17d in cmd_fetch (cmd=0x7fc4a8111ed0) at > > cmd-fetch.c:284 > > > #20 0x00007fc4a6e9601c in command_exec (cmd=cmd@entry=0x7fc4a8111ed0) at > > > imap-commands.c:158 > > > #21 0x00007fc4a6e94f1f in client_command_input (cmd=cmd@entry > > =0x7fc4a8111ed0) > > > at imap-client.c:780 > > > #22 0x00007fc4a6e95005 in client_command_input (cmd=0x7fc4a8111ed0) at > > > imap-client.c:841 > > > #23 0x00007fc4a6e952fd in client_handle_next_command > > > (remove_io_r=<synthetic pointer>, client=0x7fc4a8111530) at > > > imap-client.c:879 > > > #24 client_handle_input (client=client@entry=0x7fc4a8111530) at > > > imap-client.c:891 > > > #25 0x00007fc4a6e956c5 in client_input (client=0x7fc4a8111530) at > > > imap-client.c:933 > > > #26 0x00007fc4a66e1a87 in io_loop_call_io (io=0x7fc4a811edd0) at > > > ioloop.c:388 > > > #27 0x00007fc4a66e290f in io_loop_handler_run > > > (ioloop=ioloop@entry=0x7fc4a80fc750) > > > at ioloop-epoll.c:220 > > > #28 0x00007fc4a66e15d8 in io_loop_run (ioloop=0x7fc4a80fc750) at > > > ioloop.c:412 > > > #29 0x00007fc4a668e9e3 in master_service_run (service=0x7fc4a80fc5e0, > > > callback=callback@entry=0x7fc4a6e9ed70 <client_connected>) at > > > master-service.c:571 > > > #30 0x00007fc4a6e89324 in main (argc=1, argv=0x7fc4a80fc390) at > > main.c:400 > > > > > > Thank you. > > > > > > Best regards, > > > Hongying Liu > > > >

1 0