Solr, Dovecot & macOS / iOS

older
Dovecot Sieve doesn't move spam to...

James Brown

13 Aug 2019 13 Aug '19

2:58 p.m.

I’m thinking of getting Solr working with my Dovecot server. Server is new 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands of messages.

But I’m not sure if:

a) it will make enough of a difference and

b) does Mail.app and other mail clients on Macs or iOS devices perform searches on their local copy of mail or does it just send a search request to the server?

I’m guessing the searches are done locally so no point in Solr?

Does anyone know?

Thanks, James.

Attachments:

smime.p7s (application/x-pkcs7-signature — 4.6 KB)

Show replies by date

Sami Ketola

13 Aug 13 Aug

3:16 p.m.

...

On 13 Aug 2019, at 14.58, James Brown via dovecot dovecot@dovecot.org wrote:

I’m thinking of getting Solr working with my Dovecot server. Server is new 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands of messages.

But I’m not sure if:

a) it will make enough of a difference and

Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 100s of thousands of small files in same directory. If so, then use mdbox with periodic purges.

...

b) does Mail.app and other mail clients on Macs or iOS devices perform searches on their local copy of mail or does it just send a search request to the server?

None of the apple devices use IMAP SEARCH. They ALL maintain and use their own local search database on the device. Also they seem to refresh the database every now and then redownloading all emails.

...

I’m guessing the searches are done locally so no point in Solr?

No point if only apple client devices.

Sami

Jean-Daniel

3:37 p.m.

...

Le 13 août 2019 à 14:16, Sami Ketola via dovecot dovecot@dovecot.org a écrit :

...
On 13 Aug 2019, at 14.58, James Brown via dovecot dovecot@dovecot.org wrote:

I’m thinking of getting Solr working with my Dovecot server. Server is new 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands of messages.

But I’m not sure if:

a) it will make enough of a difference and

Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 100s of thousands of small files in same directory. If so, then use mdbox with periodic purges.

...
b) does Mail.app and other mail clients on Macs or iOS devices perform searches on their local copy of mail or does it just send a search request to the server?

None of the apple devices use IMAP SEARCH. They ALL maintain and use their own local search database on the device. Also they seem to refresh the database every now and then redownloading all emails.

Do you have a source for that. My experience is that without server search support, iOS is very slow at returning result. Moreover, it keep only latest messages and never download message until you read them.

Sami Ketola

3:53 p.m.

...

On 13 Aug 2019, at 15.37, Jean-Daniel via dovecot dovecot@dovecot.org wrote:

...
Le 13 août 2019 à 14:16, Sami Ketola via dovecot dovecot@dovecot.org a écrit :

...
On 13 Aug 2019, at 14.58, James Brown via dovecot dovecot@dovecot.org wrote:

I’m thinking of getting Solr working with my Dovecot server. Server is new 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands of messages.

But I’m not sure if:

a) it will make enough of a difference and

Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 100s of thousands of small files in same directory. If so, then use mdbox with periodic purges.

...
b) does Mail.app and other mail clients on Macs or iOS devices perform searches on their local copy of mail or does it just send a search request to the server?

None of the apple devices use IMAP SEARCH. They ALL maintain and use their own local search database on the device. Also they seem to refresh the database every now and then redownloading all emails.

Do you have a source for that. My experience is that without server search support, iOS is very slow at returning result. Moreover, it keep only latest messages and never download message until you read them.

I'm a apple device user myself. I have couple of iPhones, couple of iPads, couple if MacBooks and Mail.app on any of them is not using IMAP SEARCH. And I cannot find any configuration option to enable it. Only spotlight index is used. On Mac OS Mail.App seems to store the indexed data to:

samik@samikworkmac:~>ls -1 Library/Mail/V6/MailData/Envelope\ Index* Library/Mail/V6/MailData/Envelope Index Library/Mail/V6/MailData/Envelope Index-shm Library/Mail/V6/MailData/Envelope Index-wal

if those files are removed or spotlight search for mails is disabled Mail.App can't find anything anymore. It does not fall back to IMAP SEARCH.

Sami

Jean-Daniel

3:58 p.m.

...

Le 13 août 2019 à 14:53, Sami Ketola Sami.Ketola@Open-Xchange.com a écrit :

...
On 13 Aug 2019, at 15.37, Jean-Daniel via dovecot mailto:dovecot@dovecot.org> wrote:

...
Le 13 août 2019 à 14:16, Sami Ketola via dovecot mailto:dovecot@dovecot.org> a écrit :

...
On 13 Aug 2019, at 14.58, James Brown via dovecot mailto:dovecot@dovecot.org> wrote:

I’m thinking of getting Solr working with my Dovecot server. Server is new 6-core Mac Mini, mail store of over 1/2 TB. Mailboxes with 100s of thousands of messages.

But I’m not sure if:

a) it will make enough of a difference and

Choose mailbox format wisely. sdbox preferred unless HFS+ has problems with 100s of thousands of small files in same directory. If so, then use mdbox with periodic purges.

...
b) does Mail.app and other mail clients on Macs or iOS devices perform searches on their local copy of mail or does it just send a search request to the server?

None of the apple devices use IMAP SEARCH. They ALL maintain and use their own local search database on the device. Also they seem to refresh the database every now and then redownloading all emails.

Do you have a source for that. My experience is that without server search support, iOS is very slow at returning result. Moreover, it keep only latest messages and never download message until you read them.

I'm a apple device user myself. I have couple of iPhones, couple of iPads, couple if MacBooks and Mail.app on any of them is not using IMAP SEARCH. And I cannot find any configuration option to enable it. Only spotlight index is used. On Mac OS Mail.App seems to store the indexed data to:

samik@samikworkmac:~>ls -1 Library/Mail/V6/MailData/Envelope\ Index* Library/Mail/V6/MailData/Envelope Index Library/Mail/V6/MailData/Envelope Index-shm Library/Mail/V6/MailData/Envelope Index-wal

if those files are removed or spotlight search for mails is disabled Mail.App can't find anything anymore. It does not fall back to IMAP SEARCH.

My question was more about iOS. I know that macOS Mail does not rely on any way on remote indexing and has it’s own local index, but as it also store all messages locally, it’s an easy requirement. For iOS that only download messages meta-data by default, I was not so sure.

I’m accessing my mail server using Apple devices only, and see some imap SEARCH requests in dovecot stats, but can’t figure out where they came from though. So you may be right.

＠lbutlr

14 Aug 14 Aug

8:19 a.m.

On 13 Aug 19, at 05:58 , James Brown jlbrown@bordo.com.au wrote:

...

b) does Mail.app and other mail clients on Macs or iOS devices perform searches on their local copy of mail or does it just send a search request to the server?

Mail.app uses spotlight on the local data, so if your users are all Mac then solar is pointless.

1873

Age (days ago)

1874

Last active (days ago)

List overview

5 comments

4 participants

participants (4)

＠lbutlr
James Brown
Jean-Daniel
Sami Ketola