Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200, 000 emails in my Inbox
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
On 4/16/2022 11:32 PM, Turritopsis Dohrnii Teo En Ming wrote:
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
<snip>
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
<snip>
My IMAP server is Dovecot version 2.2.36.
I artificially created a situation like this in my mailbox. I created a new folder and copied all my other messages in my mailbox to it in the Linux commandline. Ended up with 154K messages in that folder. Then I had to force a complete reindex. Which I have a script for.
I'm using Solr for dovecot searching. If you do something similar and have dovecot connect to something external for search, that should solve the first problem.
The folder did eventually populate in roundcube, though it did take a while. My roundcube version is 1.5.2, and my dovecot version is 2:2.3.18-4+ubuntu20.04. Dovecot is installed from the official project APT repository on Ubuntu Server 20.04.
I seem to be having a problem with my Solr search engine. No matter what keyword I search for in that folder, it always gets 127168 hits. That is just not possible when searching for "dragon" and then "aardvark".
Thanks, Shawn
Dear Shawn Heisey,
My Dovecot, Postfix, Roundcube and Squirrelmail were automatically installed, configured and deployed by Virtualmin web hosting control panel install scripts.
How do I integrate Apache SOLR searching capability with Dovecot? Are there any excellent guides for this?
Thank you.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 17 Apr 2022 Sunday
On Sun, 17 Apr 2022 at 15:46, Shawn Heisey elyograg@elyograg.org wrote:
On 4/16/2022 11:32 PM, Turritopsis Dohrnii Teo En Ming wrote:
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
<snip>
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
<snip>
My IMAP server is Dovecot version 2.2.36.
I artificially created a situation like this in my mailbox. I created a new folder and copied all my other messages in my mailbox to it in the Linux commandline. Ended up with 154K messages in that folder. Then I had to force a complete reindex. Which I have a script for.
I'm using Solr for dovecot searching. If you do something similar and have dovecot connect to something external for search, that should solve the first problem.
The folder did eventually populate in roundcube, though it did take a while. My roundcube version is 1.5.2, and my dovecot version is 2:2.3.18-4+ubuntu20.04. Dovecot is installed from the official project APT repository on Ubuntu Server 20.04.
I seem to be having a problem with my Solr search engine. No matter what keyword I search for in that folder, it always gets 127168 hits. That is just not possible when searching for "dragon" and then "aardvark".
Thanks, Shawn
On 4/17/2022 7:44 AM, Turritopsis Dohrnii Teo En Ming wrote:
My Dovecot, Postfix, Roundcube and Squirrelmail were automatically installed, configured and deployed by Virtualmin web hosting control panel install scripts.
How do I integrate Apache SOLR searching capability with Dovecot? Are there any excellent guides for this?
I have this in /etc/dovecot/conf.d/10-mail.conf:
mail_plugins = $mail_plugins quota fts fts_solr
And in conf.d/90-plugin.conf, I have this:
plugin { fts = solr fts_solr = url=http://localhost:8983/solr/dovecot/ fts_autoindex = yes fts_enforced = yes }
I'm running a version of Solr that I built myself from the branch_8_11 source code -- 8.11.2-SNAPSHOT. I installed that with the service installer script that it includes. I have been customizing the index config, it started with the config and schema provided by dovecot, which did work. I'm going to have some questions for those who write the fts_solr plugin, and I still need to glance at the source code.
I am a committer on the Apache Solr project, so I do know a little bit about Solr. :) I am not as competent in the C code that makes dovecot as I am in the Java code that makes Solr. C was one of my first languages, but I have not spent any real time with it in decades.
My install of dovecot has a grand total of 180079 messages in it right now, 154K of which are in my mailbox. (I deleted the test folder I created to test your problem out). If you can upgrade roundcube and dovecot, plus incorporate Solr, things might work better for you.
I would suggest that you move most of the messages out of your inbox into some kind of archival folder. Keep between 3 months and one year of messages in the actual inbox so working with that folder is fast, and you can visit the archive if you need to look at something older.
I have never liked dealing with do-everything packages like webmin or virtualmin. I want to understand each of the pieces that make up the whole, and be able to upgrade and reconfigure each piece independently from the others. I find the configuration restraints of such a system very limiting. Very often if you customize something yourself outside the GUI, you lose the ability to administer it in the GUI, because it doesn't know how to deal with the change. Or if you do change something in the GUI afterwards, your customization might get deleted.
Thanks, Shawn
Dear Shawn Heisey,
Your instructions for integrating Apache SOLR with Dovecot is too much for me to understand. Do you have links/URLs to excellent guides for doing this?
Thank you.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Mon, 18 Apr 2022 at 07:27, Shawn Heisey elyograg@elyograg.org wrote:
On 4/17/2022 7:44 AM, Turritopsis Dohrnii Teo En Ming wrote:
My Dovecot, Postfix, Roundcube and Squirrelmail were automatically installed, configured and deployed by Virtualmin web hosting control panel install scripts.
How do I integrate Apache SOLR searching capability with Dovecot? Are there any excellent guides for this?
I have this in /etc/dovecot/conf.d/10-mail.conf:
mail_plugins = $mail_plugins quota fts fts_solr
And in conf.d/90-plugin.conf, I have this:
plugin { fts = solr fts_solr = url=http://localhost:8983/solr/dovecot/ fts_autoindex = yes fts_enforced = yes }
I'm running a version of Solr that I built myself from the branch_8_11 source code -- 8.11.2-SNAPSHOT. I installed that with the service installer script that it includes. I have been customizing the index config, it started with the config and schema provided by dovecot, which did work. I'm going to have some questions for those who write the fts_solr plugin, and I still need to glance at the source code.
I am a committer on the Apache Solr project, so I do know a little bit about Solr. :) I am not as competent in the C code that makes dovecot as I am in the Java code that makes Solr. C was one of my first languages, but I have not spent any real time with it in decades.
My install of dovecot has a grand total of 180079 messages in it right now, 154K of which are in my mailbox. (I deleted the test folder I created to test your problem out). If you can upgrade roundcube and dovecot, plus incorporate Solr, things might work better for you.
I would suggest that you move most of the messages out of your inbox into some kind of archival folder. Keep between 3 months and one year of messages in the actual inbox so working with that folder is fast, and you can visit the archive if you need to look at something older.
I have never liked dealing with do-everything packages like webmin or virtualmin. I want to understand each of the pieces that make up the whole, and be able to upgrade and reconfigure each piece independently from the others. I find the configuration restraints of such a system very limiting. Very often if you customize something yourself outside the GUI, you lose the ability to administer it in the GUI, because it doesn't know how to deal with the change. Or if you do change something in the GUI afterwards, your customization might get deleted.
Thanks, Shawn
On 4/20/22 08:03, Turritopsis Dohrnii Teo En Ming wrote:
Dear Shawn Heisey,
Your instructions for integrating Apache SOLR with Dovecot is too much for me to understand. Do you have links/URLs to excellent guides for doing this?
Here's the dovecot manual page on it:
https://doc.dovecot.org/configuration_manual/fts/solr/
This doesn't explain how to tell dovecot to index new incoming messages, or how to initiate a full reindex, but I can guide you on those.
Thanks, Shawn
Dear Shawn Heisey,
Noted with thanks. I shall probably try it one of these coming weekends.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 22:10, Shawn Heisey elyograg@elyograg.org wrote:
On 4/20/22 08:03, Turritopsis Dohrnii Teo En Ming wrote:
Dear Shawn Heisey,
Your instructions for integrating Apache SOLR with Dovecot is too much for me to understand. Do you have links/URLs to excellent guides for doing this?
Here's the dovecot manual page on it:
https://doc.dovecot.org/configuration_manual/fts/solr/
This doesn't explain how to tell dovecot to index new incoming messages, or how to initiate a full reindex, but I can guide you on those.
Thanks, Shawn
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
this is not the answer you are looking for but comes from years and years of experience with mail servers in general (cyrus first then dovecot)
First off imap clients can handle emails counts but vary on performance
- outlook for example is completely useless above 1000 count per
folder/sub folder, imap was kinda supported to outlook 2010 but after 2012 fell off largely due to microsoft pushing their sync & exchange servers.
- thunderbird will and can handle 10000~30000 emails but anything above
10000 really should be split into a subfolder, i usually take the inbox and split into the year of the messages to keep into a subfolder like 2020,2021 etc
- apple mail and iphone related seem to handle large emails boxes ok
but will slow down above the 1000+ per folder limit
- android you need to run bluemail or something like that but again
limits are like the iphone for around 1000 per folder, generic imap client does not work well with android.
As for the 200,000+ emails in the inbox no email system was ever designed for that - ever.
It needs to be understood that pending how your server hdd's are formatted (nfs, ufs, zfs etc) will directly effect the speed of email retreivals.
nfs being the slowest (dependant on network speeds, server speeds etc)
ufs is the old style unix partitions, ufs works ok but you need to tweek certain settings for how many inodes can be in memory at any one time to i believe a max of 10,000 (freebsd, other systems will vary)
zfs is the best as it will store dir inode etc into memory in the system so large file numbers in directories have a faster chance to respond, again system dependant suggest lots of memory
no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly.
forget about trying to index searches etc.
internet speed (even a 100M fibre connection) is also a factor.
this is a sample dir structure from my server
1 email = 1 file in a sub dir.
the operating system is where you are choking on its ability to handle 200,000 files in one directory is just unsustainable.
if you have to handle that many incoming emails i strongly suggest a sieve filter that automatically moves incoming emails into a folder based on the email to keep the counts under control.
I find dovecot can handle email counts better then the old cyrus but when i ran cyrus i was using ufs file system as well, when i upgraded to dovecot i had also upgraded to zfs.
# cd cur
[05:39:14] mail18.scom.ca [root:0] /data/dovecot/users/scom.ca/paul@scom.ca/cur # ll total 77333 drwx------ 2 vmail vmail uarch 361B Apr 18 05:35 . drwx------ 947 vmail vmail uarch 963B Apr 18 05:36 .. -rw------- 1 vmail vmail uarch 6.4K Feb 20 08:05 1645362302.M163606P37919.mail18.scom.ca,S=6536,W=6687:2,FS -rw------- 1 vmail vmail uarch 8.9K Feb 20 08:05 1645362302.M535115P37919.mail18.scom.ca,S=9074,W=9224:2,S -rw------- 1 vmail vmail uarch 5.5K Feb 20 08:05 1645362303.M133625P37919.mail18.scom.ca,S=5632,W=5729:2,S -rw------- 1 vmail vmail uarch 11K Feb 20 08:05 1645362305.M954336P38787.mail18.scom.ca,S=11438,W=11642:2,S -rw------- 1 vmail vmail uarch 1.9K Feb 20 08:05 1645362305.M954355P38787.mail18.scom.ca,S=1912,W=1965:2,S -rw------- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362306.M417932P37919.mail18.scom.ca,S=2333,W=2413:2,S -rw------- 1 vmail vmail uarch 2.2K Feb 20 08:05 1645362306.M569866P37919.mail18.scom.ca,S=2287,W=2366:2,S -rw------- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362307.M184339P37919.mail18.scom.ca,S=2371,W=2452:2,S -rw------- 1 vmail vmail uarch 22K Feb 20 08:05 1645362307.M706237P37919.mail18.scom.ca,S=22281,W=22692:2,S -rw------- 1 vmail vmail uarch 189K Feb 20 08:05 1645362313.M247319P37919.mail18.scom.ca,S=193354,W=196073:2,S -rw------- 1 vmail vmail uarch 182K Feb 20 08:05 1645362313.M68190P39861.mail18.scom.ca,S=186263,W=188880:2,S
fyi
Happy Monday !!! Thanks - paul
Paul Kudla
004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 On 4/17/2022 5:19 AM, dovecot-bounces@dovecot.org wrote:
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote:
As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. ... no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly.
I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for.
Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX.
Joseph Tam jtam.home@gmail.com
On Mon, Apr 18, 2022 at 03:23:10PM -0700, Joseph Tam wrote:
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote:
As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. ... no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly.
I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for.
Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX.
Gmail is *advertised* as working that way.
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
Maybe those issues have been fixed - I don't know. But unless they have been, Gmail is not really a panacea.
In any case, for Dovecot, Joseph & Paul's advice to divide mail into folders where possible seems sensible - especially for access over IMAP.
Sam
Gmail is *advertised* as working that way.
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared.
In any case, for Dovecot, Joseph & Paul's advice to divide mail into folders where possible seems sensible - especially for access over IMAP.
I also made some cronjobs, sieve scripts that archive/move message the most used folders/mailboxes to archive/2021, archive/2020 etc.
On 19. Apr 2022, at 10.36, Marc Marc@f1-outsourcing.eu wrote:
Gmail is *advertised* as working that way.
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared.
This is more about gmail “spam protection”. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted.
By using gmail there is no way of telling how many (important) emails you have not received because of this.
Sami
On Tue, 19 Apr 2022, Sami Ketola wrote:
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared.
This is more about gmail ?spam protection?. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted.
While Gmail's spam classifcation is frustratingly opaque (as with many other providers), there's one other weirdness that can make messages dissappear: when you send from Gmail through a forwarder back to the same Gmail account.
The mail will be accepted, but not appear in your INBOX. This is either some labelling weirdness (cannot be labelled as both INBOX and Sent) or maybe an anti-loop protection. This causes confusion when Gmail users test their mail forwarding I set up for them.
However, we're truly off-topic: my point was that Gmail users get a distorted sense of how most mail systems work (mailbox operations scale with message count), as they get brainwashed into the "Gmail" way of doing things. A relevant question is whether you can use Dovecot's virtual mailbox feature to define a catch-all virtual mailbox to placate these users which won't bring an imap process to its knees.
Joseph Tam jtam.home@gmail.com
The emails appeared in my Gmail inbox. Then sometime later the emails disappeared mysteriously. Not sure who deleted my emails from my Gmail email account.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 06:42, Joseph Tam jtam.home@gmail.com wrote:
On Tue, 19 Apr 2022, Sami Ketola wrote:
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared.
This is more about gmail ?spam protection?. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted.
While Gmail's spam classifcation is frustratingly opaque (as with many other providers), there's one other weirdness that can make messages dissappear: when you send from Gmail through a forwarder back to the same Gmail account.
The mail will be accepted, but not appear in your INBOX. This is either some labelling weirdness (cannot be labelled as both INBOX and Sent) or maybe an anti-loop protection. This causes confusion when Gmail users test their mail forwarding I set up for them.
However, we're truly off-topic: my point was that Gmail users get a distorted sense of how most mail systems work (mailbox operations scale with message count), as they get brainwashed into the "Gmail" way of doing things. A relevant question is whether you can use Dovecot's virtual mailbox feature to define a catch-all virtual mailbox to placate these users which won't bring an imap process to its knees.
Joseph Tam jtam.home@gmail.com
For me, it is not because of Gmail spam filters. I confirm I had the emails in my inbox previously, and then sometime later the emails were deleted mysteriously. Not sure who deleted my emails. Definitely not because of Gmail spam filters.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wed
On Tue, 19 Apr 2022 at 16:03, Sami Ketola sami@ketola.io wrote:
On 19. Apr 2022, at 10.36, Marc Marc@f1-outsourcing.eu wrote:
Gmail is *advertised* as working that way.
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared.
This is more about gmail “spam protection”. Gmail filters emails after accepting them with secret criteria and sometimes the email ends up in spam folder and sometime it is just silently deleted.
By using gmail there is no way of telling how many (important) emails you have not received because of this.
Sami
Recently, I discovered that some emails in one of my Gmail email accounts were missing. Not sure if Gmail/Google had deleted my email messages.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Tue, 19 Apr 2022 at 15:36, Marc Marc@f1-outsourcing.eu wrote:
Gmail is *advertised* as working that way.
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
They are also losing/deleting messages. I have had several occassions where I send a message and my delivering server registered the Sent status of the gmail server and it just disappeared.
In any case, for Dovecot, Joseph & Paul's advice to divide mail into folders where possible seems sensible - especially for access over IMAP.
I also made some cronjobs, sieve scripts that archive/move message the most used folders/mailboxes to archive/2021, archive/2020 etc.
I am planning to delete a lot of useless emails, which takes up a huge proportion of the 200,000 emails in my inbox.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Tue, 19 Apr 2022 at 14:49, Sam Kuper sampablokuper@posteo.net wrote:
On Mon, Apr 18, 2022 at 03:23:10PM -0700, Joseph Tam wrote:
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote:
As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. ... no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly.
I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for.
Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX.
Gmail is *advertised* as working that way.
In practice, though, Gmail used to exhibit search/browse bugs (e.g. failing to identify all relevant mails/threads) annoyingly often. This was sometimes true via the official Gmail web interface, and was especially true via the official mobile app, on at least some platforms.
Maybe those issues have been fixed - I don't know. But unless they have been, Gmail is not really a panacea.
In any case, for Dovecot, Joseph & Paul's advice to divide mail into folders where possible seems sensible - especially for access over IMAP.
Sam
I used to have 4-5 MILLION email messages in my Gmail email account, more than 10 years ago.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Tue, 19 Apr 2022 at 06:23, Joseph Tam jtam.home@gmail.com wrote:
On Mon, 18 Apr 2022, Paul Kudla (SCOM.CA Internet) wrote:
As for the 200,000+ emails in the inbox no email system was ever designed for that - ever. ... no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly.
I think Gmail does exaclty this -- their mail system really has one big mesage repository, and they simulate mailboxes by using labels. They seem to encourage piling the message high and using their search or auto-labelling features to find what you're looking for.
Users of mine who previously used Gmail expect our mail system to behave similarly, and I have to break them of their habit to packrat all their mail into their INBOX.
Joseph Tam jtam.home@gmail.com
It is not true that no email system was ever designed for 200,000 or more emails.
If I did not remember wrongly, more than 10 years ago, I used to have a Gmail email account, with 4-5 MILLION email messages in the Inbox. There was no problem searching even though I had 4-5 MILLION email messages in the Inbox.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Mon, 18 Apr 2022 at 17:44, Paul Kudla (SCOM.CA Internet) paul@scom.ca wrote:
this is not the answer you are looking for but comes from years and years of experience with mail servers in general (cyrus first then dovecot)
First off imap clients can handle emails counts but vary on performance
outlook for example is completely useless above 1000 count per folder/sub folder, imap was kinda supported to outlook 2010 but after 2012 fell off largely due to microsoft pushing their sync & exchange servers.
thunderbird will and can handle 10000~30000 emails but anything above 10000 really should be split into a subfolder, i usually take the inbox and split into the year of the messages to keep into a subfolder like 2020,2021 etc
apple mail and iphone related seem to handle large emails boxes ok but will slow down above the 1000+ per folder limit
android you need to run bluemail or something like that but again limits are like the iphone for around 1000 per folder, generic imap client does not work well with android.
As for the 200,000+ emails in the inbox no email system was ever designed for that - ever.
It needs to be understood that pending how your server hdd's are formatted (nfs, ufs, zfs etc) will directly effect the speed of email retreivals.
nfs being the slowest (dependant on network speeds, server speeds etc)
ufs is the old style unix partitions, ufs works ok but you need to tweek certain settings for how many inodes can be in memory at any one time to i believe a max of 10,000 (freebsd, other systems will vary)
zfs is the best as it will store dir inode etc into memory in the system so large file numbers in directories have a faster chance to respond, again system dependant suggest lots of memory
no system will support 200,000 + emails, even if the server can handle that and running imap where you only download the headers the email client would just spin trying to update the email box constantantly.
forget about trying to index searches etc.
internet speed (even a 100M fibre connection) is also a factor.
this is a sample dir structure from my server
1 email = 1 file in a sub dir.
the operating system is where you are choking on its ability to handle 200,000 files in one directory is just unsustainable.
if you have to handle that many incoming emails i strongly suggest a sieve filter that automatically moves incoming emails into a folder based on the email to keep the counts under control.
I find dovecot can handle email counts better then the old cyrus but when i ran cyrus i was using ufs file system as well, when i upgraded to dovecot i had also upgraded to zfs.
# cd cur
[05:39:14] mail18.scom.ca [root:0] /data/dovecot/users/scom.ca/paul@scom.ca/cur # ll total 77333 drwx------ 2 vmail vmail uarch 361B Apr 18 05:35 . drwx------ 947 vmail vmail uarch 963B Apr 18 05:36 .. -rw------- 1 vmail vmail uarch 6.4K Feb 20 08:05 1645362302.M163606P37919.mail18.scom.ca,S=6536,W=6687:2,FS -rw------- 1 vmail vmail uarch 8.9K Feb 20 08:05 1645362302.M535115P37919.mail18.scom.ca,S=9074,W=9224:2,S -rw------- 1 vmail vmail uarch 5.5K Feb 20 08:05 1645362303.M133625P37919.mail18.scom.ca,S=5632,W=5729:2,S -rw------- 1 vmail vmail uarch 11K Feb 20 08:05 1645362305.M954336P38787.mail18.scom.ca,S=11438,W=11642:2,S -rw------- 1 vmail vmail uarch 1.9K Feb 20 08:05 1645362305.M954355P38787.mail18.scom.ca,S=1912,W=1965:2,S -rw------- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362306.M417932P37919.mail18.scom.ca,S=2333,W=2413:2,S -rw------- 1 vmail vmail uarch 2.2K Feb 20 08:05 1645362306.M569866P37919.mail18.scom.ca,S=2287,W=2366:2,S -rw------- 1 vmail vmail uarch 2.3K Feb 20 08:05 1645362307.M184339P37919.mail18.scom.ca,S=2371,W=2452:2,S -rw------- 1 vmail vmail uarch 22K Feb 20 08:05 1645362307.M706237P37919.mail18.scom.ca,S=22281,W=22692:2,S -rw------- 1 vmail vmail uarch 189K Feb 20 08:05 1645362313.M247319P37919.mail18.scom.ca,S=193354,W=196073:2,S -rw------- 1 vmail vmail uarch 182K Feb 20 08:05 1645362313.M68190P39861.mail18.scom.ca,S=186263,W=188880:2,S
fyi
Happy Monday !!! Thanks - paul
Paul Kudla
004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3
Toronto 416.642.7266 Main 1.866.411.7266 Fax 1.888.892.7266 On 4/17/2022 5:19 AM, dovecot-bounces@dovecot.org wrote:
On 2022-04-17 05:32, Turritopsis Dohrnii Teo En Ming wrote:
Subject: Both Roundcube and Squirrelmail webmail cannot search for anything + cannot open many emails because there are more than 200,000 emails in my Inbox
Good day from Singapore,
I am using Virtualmin/Webmin web hosting control panel in my Virtual Private Servers (VPS) in Germany.
[1st issue]
Both Roundcube and Squirrelmail webmail cannot search for anything because there are more than 200,000 emails in my Inbox.
Roundcube webmail doesn't show any error. Search results simply don't show up after a long time.
However, I got the following error in Squirrelmail webmail (using Firefox web browser).
ERROR: Connection dropped by IMAP server.
Query: SEARCH CHARSET ISO-8859-1 ALL SUBJECT "keyword"
[2nd issue]
Both Roundcube and Squirrelmail webmail cannot open many emails because there are more than 200,000 emails in my inbox.
Roundcube webmail gives the following error:
SERVER ERROR! Could not load message from server.
Squirrelmail webmail gives the following error:
ERROR: Connection dropped by IMAP server. Query: FETCH 1 BODY[HEADER]
My IMAP server is Dovecot version 2.2.36.
Virtualmin version is 6.17. Webmin version is 1.990. Usermin version is 1.834. Authentic theme version is 19.85.1.
However, using Samsung Email app on my Samsung Galaxy A32 5G Android phone, I am able to open emails it seems. Only Roundcube and Squirrelmail webmail appears to have issues opening emails. Using Samsung Email app, I have deleted around 6,000 useless emails so far but still unable to search and open emails. Looks like I need to delete more and more useless emails.
How do I solve the above problems?
I am looking forward to your reply.
Thank you very much.
Mr. Turritopsis Dohrnii Teo En Ming, 44 years old as of 17 April 2022 Sunday, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a Systems Integrator (SI)/computer firm in Singapore. He is an IT enthusiast.
-----BEGIN EMAIL SIGNATURE-----
The Gospel for all Targeted Individuals (TIs):
[The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers
Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html
Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):
[1] https://tdtemcerts.wordpress.com/
[2] https://tdtemcerts.blogspot.sg/
[3] https://www.scribd.com/user/270125049/Teo-En-Ming
-----END EMAIL SIGNATURE-----
Hi there,
Try to increase ulimit for Dovecot and Solr service as well as the default OS one.
Also, you might want to consider opening a bug report in Roundcube git repository, if you weren't lucky to have response from here, Aleksander is active over there.
Good luck.
Zakaria
On 4/19/2022 5:12 PM, Turritopsis Dohrnii Teo En Ming wrote:
It is not true that no email system was ever designed for 200,000 or more emails.
If I did not remember wrongly, more than 10 years ago, I used to have a Gmail email account, with 4-5 MILLION email messages in the Inbox. There was no problem searching even though I had 4-5 MILLION email messages in the Inbox.
If you were using either a Google app or the gmail.com website to access that inbox, then it doesn't compare. I really doubt that either of those uses the IMAP protocol. They would be using something proprietary that is highly optimized for the way that Google stores data and leverages the enormous amounts of computing power that they maintain.
I would bet that if you accessed a gmail folder with 5 million messages in it using IMAP, you would have similar problems with it to those that have been described here in this thread. IMAP is a beautiful protocol, but I don't think it was designed for handling that many messages.
Thanks, Shawn
On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote:
I would bet that if you accessed a gmail folder with 5 million messages in it using IMAP, you would have similar problems with it to those that have been described here in this thread. IMAP is a beautiful protocol, but I don't think it was designed for handling that many messages.
This. Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with us to confirm.
Even at just ~100B for each message's headers, your IMAP2 client would likely need at least ~500MB free RAM to load 5 million mails.
By the time Mark stopped working on UW IMAP (the reference IMAP implementation, aka Panda IMAP), c.2010, even top-of-the-range smartphones typically had only ~512MB RAM total, and top-of-the-range ThinkPads had max ~4GiB (which was the upper limit of what 32-bit operating systems, still popular then, could handle).
When IMAP2 was invented, c.1988-1990, RAM like that was basically supercomputer territory.
Had Mark intended or expected IMAP2 users to have had supercomputers at their disposal, and to be searching such large volumes of mail over the protocol, I suspect he would have designed the protocol differently: for raw efficiency over human readability.
My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing.
Good luck in your quest!
Sam
My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing.
Wouldn't it be very tedious and time consuming to sync mailboxes from the server onto our local filesystems?
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 14:21, Sam Kuper sampablokuper@posteo.net wrote:
On Tue, Apr 19, 2022 at 07:26:10PM -0600, Shawn Heisey wrote:
I would bet that if you accessed a gmail folder with 5 million messages in it using IMAP, you would have similar problems with it to those that have been described here in this thread. IMAP is a beautiful protocol, but I don't think it was designed for handling that many messages.
This. Sadly, Mark Crispin (author of IMAP and IMAP2) is no longer with us to confirm.
Even at just ~100B for each message's headers, your IMAP2 client would likely need at least ~500MB free RAM to load 5 million mails.
By the time Mark stopped working on UW IMAP (the reference IMAP implementation, aka Panda IMAP), c.2010, even top-of-the-range smartphones typically had only ~512MB RAM total, and top-of-the-range ThinkPads had max ~4GiB (which was the upper limit of what 32-bit operating systems, still popular then, could handle).
When IMAP2 was invented, c.1988-1990, RAM like that was basically supercomputer territory.
Had Mark intended or expected IMAP2 users to have had supercomputers at their disposal, and to be searching such large volumes of mail over the protocol, I suspect he would have designed the protocol differently: for raw efficiency over human readability.
My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing.
Good luck in your quest!
Sam
On Wed, Apr 20, 2022 at 09:55:02PM +0800, Turritopsis Dohrnii Teo En Ming wrote:
My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing.
Wouldn't it be very tedious and time consuming to sync mailboxes from the server onto our local filesystems?
Setting it up can be tedious depending on your needs.
If your mailserver is self-hosted, giving you direct access to the filesystem, that gives you more options (rsync, Unison, Dovecot dsync, etc).
Even if not, it's still possible using isync, fetchmail, or whatever (see URL above).
Once the initial sync is performed, though, subsequent syncs should only need to transmit the difference between the local and remote mailstores, and therefore should usually be fast and, optionally, automatic.
Sam
My mail server is self-hosted. I think I would rather integrate Apache solr with Dovecot than transferring mailboxes from the server onto my laptop.
Thank you for your suggestion though.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 22:23, Sam Kuper sampablokuper@posteo.net wrote:
On Wed, Apr 20, 2022 at 09:55:02PM +0800, Turritopsis Dohrnii Teo En Ming wrote:
My advice for anyone who wants to be able to keep and search very large monolithic mailstores would be: synchronise (e.g. via https://isync.sourceforge.io/ or fetchmail or getmail or rsync or Unison or whatever) those mailstores from the server onto your local filesystem; use maildir on your local filesystem; and use either Mutt's "limiting" functions, or notmuch's index/search functions, for searching/browsing.
Wouldn't it be very tedious and time consuming to sync mailboxes from the server onto our local filesystems?
Setting it up can be tedious depending on your needs.
If your mailserver is self-hosted, giving you direct access to the filesystem, that gives you more options (rsync, Unison, Dovecot dsync, etc).
Even if not, it's still possible using isync, fetchmail, or whatever (see URL above).
Once the initial sync is performed, though, subsequent syncs should only need to transmit the difference between the local and remote mailstores, and therefore should usually be fast and, optionally, automatic.
Sam
I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname:
imap.gmail.com
TCP Port 993, SSL
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 09:26, Shawn Heisey elyograg@elyograg.org wrote:
On 4/19/2022 5:12 PM, Turritopsis Dohrnii Teo En Ming wrote:
It is not true that no email system was ever designed for 200,000 or more emails.
If I did not remember wrongly, more than 10 years ago, I used to have a Gmail email account, with 4-5 MILLION email messages in the Inbox. There was no problem searching even though I had 4-5 MILLION email messages in the Inbox.
If you were using either a Google app or the gmail.com website to access that inbox, then it doesn't compare. I really doubt that either of those uses the IMAP protocol. They would be using something proprietary that is highly optimized for the way that Google stores data and leverages the enormous amounts of computing power that they maintain.
I would bet that if you accessed a gmail folder with 5 million messages in it using IMAP, you would have similar problems with it to those that have been described here in this thread. IMAP is a beautiful protocol, but I don't think it was designed for handling that many messages.
Thanks, Shawn
On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote:
I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname:
imap.gmail.com
TCP Port 993, SSL
Indeed you CAN use IMAP with gmail. For work, I mostly use my email (which is gmail-based) via Thunderbird, with IMAP. I am familiar with it.
But if I use the Gmail app for Android or the gmail website, I feel absolutely certain that those applications are NOT using IMAP. Chances are that they use protocols developed by Google.
Regarding your next email on this thread, the initial sync of a large number of messages via IMAP does take a really long time. But once the initial sync is complete, syncing the differences is very fast. The big problem with accessing a large folder via a webmail app like Roundcube is that it never sets up a local copy of the mailbox. I actually don't know where the copy of all those message headers resides ... whether it is in the PHP interpreter or in the browser. But it's not persisted to disk as it would be with a more traditional IMAP client such as Thunderbird.
Thanks, Shawn
I would prefer to use Outlook as I have Microsoft Office desktop apps.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 22:12, Shawn Heisey elyograg@elyograg.org wrote:
On 4/20/22 07:51, Turritopsis Dohrnii Teo En Ming wrote:
I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname:
imap.gmail.com
TCP Port 993, SSL
Indeed you CAN use IMAP with gmail. For work, I mostly use my email (which is gmail-based) via Thunderbird, with IMAP. I am familiar with it.
But if I use the Gmail app for Android or the gmail website, I feel absolutely certain that those applications are NOT using IMAP. Chances are that they use protocols developed by Google.
Regarding your next email on this thread, the initial sync of a large number of messages via IMAP does take a really long time. But once the initial sync is complete, syncing the differences is very fast. The big problem with accessing a large folder via a webmail app like Roundcube is that it never sets up a local copy of the mailbox. I actually don't know where the copy of all those message headers resides ... whether it is in the PHP interpreter or in the browser. But it's not persisted to disk as it would be with a more traditional IMAP client such as Thunderbird.
Thanks, Shawn
On Wed, Apr 20, 2022 at 09:51:18PM +0800, Turritopsis Dohrnii Teo En Ming wrote:
I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname:
imap.gmail.com
TCP Port 993, SSL
Gmail offers an IMAP2 endpoint so that users can connect to Gmail with IMAP2 clients (Thunderbird, Claws, Apple's Mail.app, or whatever) if they wish to.
However, Gmail's web interface, and Gmail's proprietary mobile apps, almost certainly do *not* use IMAP2 to communicate with the Gmail servers. As has already been pointed out in this thread, they likely use a proprietary protocol optimised for that proprietary environment.
Sam
However, Gmail's web interface, and Gmail's proprietary mobile apps, almost certainly do *not* use IMAP2 to communicate with the Gmail servers. As has already been pointed out in this thread, they likely use a proprietary protocol optimised for that proprietary environment.
Noted and taken note of.
Thank you.
Regards,
Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore 20 Apr 2022 Wednesday
On Wed, 20 Apr 2022 at 22:16, Sam Kuper sampablokuper@posteo.net wrote:
On Wed, Apr 20, 2022 at 09:51:18PM +0800, Turritopsis Dohrnii Teo En Ming wrote:
I believe Gmail is using IMAP. The instructions for configuring Gmail email accounts in Outlook specifically mention IMAP server hostname:
imap.gmail.com
TCP Port 993, SSL
Gmail offers an IMAP2 endpoint so that users can connect to Gmail with IMAP2 clients (Thunderbird, Claws, Apple's Mail.app, or whatever) if they wish to.
However, Gmail's web interface, and Gmail's proprietary mobile apps, almost certainly do *not* use IMAP2 to communicate with the Gmail servers. As has already been pointed out in this thread, they likely use a proprietary protocol optimised for that proprietary environment.
Sam
participants (8)
-
hi＠zakaria.website
-
Joseph Tam
-
Marc
-
Paul Kudla (SCOM.CA Internet)
-
Sam Kuper
-
Sami Ketola
-
Shawn Heisey
-
Turritopsis Dohrnii Teo En Ming