To field was not correct indexed by FTS

newer
Multiple authentication instances...

TACHIBANA Masashi

20 Jul 2020 20 Jul '20

3:25 a.m.

Hi,

This To field was not correct indexed by FTS.

To: Yamada Taro yamada@example.com,=?UTF-8?B?dXNlcjJAZXhhbXBsZS5jb20=?= user2@example.com, =?UTF-8?B?dXNlcjNAZXhhbXBsZS5jb20=?= user3@example.com, user4 desu user4@example.com

--> Yamada Taro yamada@example.com , user2@example.com

And follow was correct indexed by FTS

To: Yamada Taro yamada@example.com,"user2@example.com" user2@example.com, "user3@example.com" user3@example.com, user4 desu user4@example.com

--> Yamada Taro yamada@example.com , user2@example.com user2@example.com , user3@example.com user3@example.com , user4 desu user4@example.com

Any hint?

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

Attachments:

invalid.eml (message/rfc822 — 307 bytes)
valid.eml (message/rfc822 — 272 bytes)

Show replies by date

TACHIBANA Masashi

20 Jul 20 Jul

2:24 p.m.

Hi, I'm Tachibana. Additionally, I found below: dovecot/src/plugins/fts/fts-build-mail.c: 187 i_debug("@@@@@ befor address parse:%s",hdr->full_value); 188 189 addr = message_address_parse(pool_datastack_create(), 190 hdr->full_value, 191 hdr->full_value_len, 192 UINT_MAX, 0); 193 str = t_str_new(hdr->full_value_len); 194 message_address_write(str, addr); 195 196 i_debug("@@@@@ after address parse:%s",str_data(str)); Mail addresses were deleted when passed message_address_parse(). So this debug logs came like following: Debug: @@@@@ befor address parse:uesr1 Debug: @@@@@ after address parse:uesr1 Debug: @@@@@ data:uesr1 Debug: @@@@@ befor address parse:Yamada Taro ,user2@example.com , user3@example.com , user4 desu Debug: @@@@@ after address parse:"Yamada Taro" , Debug: @@@@@ data:"Yamada Taro" , Debug: @@@@@ tmp_value:"Yamada Taro" , Regards, ----- Original Message -----

...

Hi,

This To field was not correct indexed by FTS.

To: Yamada Taro ,=?UTF-8?B?dXNlcjJAZXhhbXBsZS5jb20=?= , =?UTF-8?B?dXNlcjNAZXhhbXBsZS5jb20=?= , user4 desu

--> Yamada Taro , user2@example.com

And follow was correct indexed by FTS

To: Yamada Taro ,"user2@example.com" , "user3@example.com" , user4 desu

--> Yamada Taro , user2@example.com , user3@example.com , user4 desu

Any hint?

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp 株式会社クオリティア https://www.qualitia.co.jp/

Josef 'Jeff' Sipek

4:43 p.m.

On Mon, Jul 20, 2020 at 20:24:13 +0900, TACHIBANA Masashi wrote:

...

Hi,

I'm Tachibana. Additionally, I found below:

dovecot/src/plugins/fts/fts-build-mail.c:

187 i_debug("@@@@@ befor address parse:%s",hdr->full_value); 188 189 addr = message_address_parse(pool_datastack_create(), 190 hdr->full_value, 191 hdr->full_value_len, 192 UINT_MAX, 0); 193 str = t_str_new(hdr->full_value_len); 194 message_address_write(str, addr); 195 196 i_debug("@@@@@ after address parse:%s",str_data(str));

Mail addresses were deleted when passed message_address_parse(). So this debug logs came like following:

Debug: @@@@@ befor address parse:uesr1 Debug: @@@@@ after address parse:uesr1 Debug: @@@@@ data:uesr1 Debug: @@@@@ befor address parse:Yamada Taro ,user2@example.com , user3@example.com , user4 desu Debug: @@@@@ after address parse:"Yamada Taro" ,

Thanks for the report. I reproduced it locally, but I'm not sure what is causing it yet. Jeff.

...

Debug: @@@@@ data:"Yamada Taro" , Debug: @@@@@ tmp_value:"Yamada Taro" ,

Regards,

----- Original Message -----

...
Hi,

This To field was not correct indexed by FTS.

To: Yamada Taro ,=?UTF-8?B?dXNlcjJAZXhhbXBsZS5jb20=?= , =?UTF-8?B?dXNlcjNAZXhhbXBsZS5jb20=?= , user4 desu

--> Yamada Taro , user2@example.com

And follow was correct indexed by FTS

To: Yamada Taro ,"user2@example.com" , "user3@example.com" , user4 desu

--> Yamada Taro , user2@example.com , user3@example.com , user4 desu

Any hint?

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

-- Intellectuals solve problems; geniuses prevent them - Albert Einstein

Josef 'Jeff' Sipek

5:27 p.m.

On Mon, Jul 20, 2020 at 09:43:12 -0400, Josef 'Jeff' Sipek wrote:

...

On Mon, Jul 20, 2020 at 20:24:13 +0900, TACHIBANA Masashi wrote: ... Thanks for the report. I reproduced it locally, but I'm not sure what is causing it yet.

Alright, it is actually a parsing bug. The problem seems to be when the code encounters a "name <address>" format where the name is a valid address. This accidentally makes the parser stop parsing - making any additional addresses ignored.

E.g.,

test@example.com <test@example.com>, Name <other@example.com>

The parser stops after the first address's name thinking it is just a bare address (without a name). That is, after the first loop iterating while parsing, the "cursor" is at the first space and not at the comma.

                                   /- parsing should stop here
                                   v
test@example.com <test@example.com>, Name <other@example.com>
                ^
                `- parsing stops here

This is obviously wrong. I'll have to dig in more to figure out what it will take to fix this.

Thanks again,

Jeff.

-- FreeBSD 12.1

TACHIBANA Masashi

21 Jul 21 Jul

2:18 a.m.

Hi,

Thank you for your reply.

I hope this bug to be fixed soon. :) Thank you.

Tachibana

----- Original Message -----

...

On Mon, Jul 20, 2020 at 09:43:12 -0400, Josef 'Jeff' Sipek wrote:

...
On Mon, Jul 20, 2020 at 20:24:13 +0900, TACHIBANA Masashi wrote: ... Thanks for the report. I reproduced it locally, but I'm not sure what is causing it yet.

Alright, it is actually a parsing bug. The problem seems to be when the code encounters a "name <address>" format where the name is a valid address. This accidentally makes the parser stop parsing - making any additional addresses ignored.

E.g.,
test@example.com <test@example.com>, Name <other@example.com>
The parser stops after the first address's name thinking it is just a bare address (without a name). That is, after the first loop iterating while parsing, the "cursor" is at the first space and not at the comma.
                                   /- parsing should stop here
                                   v
test@example.com <test@example.com>, Name <other@example.com>
                ^
                `- parsing stops here
This is obviously wrong. I'll have to dig in more to figure out what it will take to fix this.

Thanks again,

Jeff.

-- FreeBSD 12.1

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

Josef 'Jeff' Sipek

5:51 p.m.

On Tue, Jul 21, 2020 at 08:18:21 +0900, TACHIBANA Masashi wrote:

...

Hi,

Thank you for your reply.

I hope this bug to be fixed soon. :)

It is a tricky one because as far as I can tell, the From header is invalid. So, we have to figure out the best way to handle such input.

The input is invalid because '@' is not allowed by the RFC to be in the name portion of an email address without quoting. In other words, these are ok:

test@example.com
<test@example.com>
Test <test@example.com>
"test@example.com" <test@example.com>

but this one is bad:

test@example.com <test@example.com>

In your case, the name is encoded but not quoted so even though the email contains:

=?UTF-8?B?dXNlcjJAZXhhbXBsZS5jb20=?= <user2@example.com>

the parser sees:

user2@example.com <user2@example.com>

Which is invalid.

Dovecot parsing needs to be improved, but as I already mentioned it is an improvement to handling of malformed input. So, as a workaround, if you control the software/system that generates these emails, try to change it so it doesn't generate headers that are invalid. As far as I can tell, the parser in Dovecot handles those just fine.

Jeff.

...

Thank you.

Tachibana

----- Original Message -----

...
On Mon, Jul 20, 2020 at 09:43:12 -0400, Josef 'Jeff' Sipek wrote:

...
On Mon, Jul 20, 2020 at 20:24:13 +0900, TACHIBANA Masashi wrote: ... Thanks for the report. I reproduced it locally, but I'm not sure what is causing it yet.

Alright, it is actually a parsing bug. The problem seems to be when the code encounters a "name <address>" format where the name is a valid address. This accidentally makes the parser stop parsing - making any additional addresses ignored.

E.g.,
test@example.com <test@example.com>, Name <other@example.com>
The parser stops after the first address's name thinking it is just a bare address (without a name). That is, after the first loop iterating while parsing, the "cursor" is at the first space and not at the comma.
                                   /- parsing should stop here
                                   v
test@example.com <test@example.com>, Name <other@example.com>
                ^
                `- parsing stops here
This is obviously wrong. I'll have to dig in more to figure out what it will take to fix this.

Thanks again,

Jeff.

-- FreeBSD 12.1
-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

-- Hegh QaQ law' quvHa'ghach QaQ puS

TACHIBANA Masashi

27 Jul 27 Jul

8:48 a.m.

Hi,

Thank you Jeff san.

It seems to be able to work fine in IMAP, without FTS. But I hope FTS to find addresses.

And I reported this to the developers of email solutions which adding such MIMEd display name. But for older mails or another email softwares, I hope this problems to be solved in Dovecot with FTS.

Thank you.

----- Original Message -----

...

On Tue, Jul 21, 2020 at 08:18:21 +0900, TACHIBANA Masashi wrote:

...
Hi,

Thank you for your reply.

I hope this bug to be fixed soon. :)

It is a tricky one because as far as I can tell, the From header is invalid. So, we have to figure out the best way to handle such input.

The input is invalid because '@' is not allowed by the RFC to be in the name portion of an email address without quoting. In other words, these are ok:

test@example.com test@example.com Test test@example.com "test@example.com" test@example.com

but this one is bad:

test@example.com test@example.com

In your case, the name is encoded but not quoted so even though the email contains:

=?UTF-8?B?dXNlcjJAZXhhbXBsZS5jb20=?= user2@example.com

the parser sees:

user2@example.com user2@example.com

Which is invalid.

Dovecot parsing needs to be improved, but as I already mentioned it is an improvement to handling of malformed input. So, as a workaround, if you control the software/system that generates these emails, try to change it so it doesn't generate headers that are invalid. As far as I can tell, the parser in Dovecot handles those just fine.

Jeff.

...
Thank you.

Tachibana

----- Original Message -----

...
On Mon, Jul 20, 2020 at 09:43:12 -0400, Josef 'Jeff' Sipek wrote:

...
On Mon, Jul 20, 2020 at 20:24:13 +0900, TACHIBANA Masashi wrote: ... Thanks for the report. I reproduced it locally, but I'm not sure what is causing it yet.

Alright, it is actually a parsing bug. The problem seems to be when the code encounters a "name <address>" format where the name is a valid address. This accidentally makes the parser stop parsing - making any additional addresses ignored.

E.g.,
test@example.com <test@example.com>, Name <other@example.com>
The parser stops after the first address's name thinking it is just a bare address (without a name). That is, after the first loop iterating while parsing, the "cursor" is at the first space and not at the comma.
                                   /- parsing should stop here
                                   v
test@example.com <test@example.com>, Name <other@example.com>
                ^
                `- parsing stops here
This is obviously wrong. I'll have to dig in more to figure out what it will take to fix this.

Thanks again,

Jeff.

-- FreeBSD 12.1
-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/
-- Hegh QaQ law' quvHa'ghach QaQ puS

-- TACHIBANA Masashi QUALITIA CO., LTD. mailto:tachibana@qualitia.co.jp

株式会社クオリティア https://www.qualitia.co.jp/

1572

Age (days ago)

1579

Last active (days ago)

List overview

6 comments

2 participants

participants (2)

Josef 'Jeff' Sipek
TACHIBANA Masashi