On Tue, 10 Nov 2015 08:50:50 +0100 Christian Kivalo ml+dovecot@valo.at wrote:

Hi,

On 2015-11-10 01:44, mancyborg@gmail.com wrote:

...

Hello dear list, I've recently discovered 'doveadm stats' and I'm trying to use "doveadm stats dump user" and "doveadm stats dump session" to understand the pop/imap users that put more stress on the hard disks.

My problem is that some users refuse to delete their emails from the server, so they keep 20GB of maildir files on the server, the webmail (roundcube) takes forever to open the inbox, the imap searches takes forever and meanwhile all the users wait. (already tried roundcube + memcache(d) but didn't help)

What is forever in your context? I'm using roundcube and a folder with about 78k mails opens in < 1 sec unsorted. A folder with about 37k messages from a mailinglist and thread sort takes < 3 sec. My roundcube shows 200 messages per page by default. On a side note, are you using an imap proxy for roundcube? It doesn't help you with your dovecot problem but it speeds up roundcube.

To speed up imap searches i can recommend to implement fts-solr with dovecot (or maybe fts-elasticsearch, am wanting to try that but solr works...). That will speed up your searches after mailboxes are indexed.

...

So my problem is not the storage usage itself: I don't care if the user gets tons of emails with big attachments; my problem is when the user opens / searches an imap folder with more than 10K mails and iostat util goes 100% for minutes.

Dovecot should be very quick to open even folders with a huge amount of files due to its indexes.

I'm unable to reproduce any significant numbers in iostat when accessing large mailfolders with roundcube.

Whats your configuration, filesystem, ...

...

So I've enabled dovecot's stats and enjoying "doveadm stats top", "stats-top.pl" and "doveadm stats dump user/session", but talking about "doveadm stats dump user" and its output fields:

user reset_timestamp last_update num_logins num_cmds user_cpu sys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_path mail_lookup_attr mail_read_count mail_read_bytes mail_cache_hits

I'm not sure which of those fields can help me and I can't find any relevant documentation.

So here are my questions:

1. is there a documentation for those 21 fields and for 'doveadm stats' in general ?
2. what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ?
3. which field of those is, in your opinion, more representative for expressing the workload that gives me problems ?
4. which settings do I need to store 1 week worth of stats ?

I'm currenty using the 'standard' values:

stats_refresh = 30 secs stats_track_cmds = yes stats_memory_limit = 16 M stats_command_min_time = 1 mins stats_domain_min_time = 12 hours stats_ip_min_time = 12 hours stats_session_min_time = 15 mins stats_user_min_time = 1 hours

Can you please tell me the correct parameters to store 1 week of stats ?

For stats somebody else has to jump in, i have only enabled the plugin to see what to get out of it but not made any use of it.

Please share your doveconf -n output

...

Thank you, Mike

regards christian

By 'forever' I mean more than 1 minute.

So there is no documentation / manual for 'doveadm stats' ? Do I have to read the source to know which field does what ?

I mean the output fields of "doveadm stats dump user":

user reset_timestamp last_update num_logins num_cmds user_cpu sys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_path mail_lookup_attr mail_read_count mail_read_bytes mail_cache_hits

what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ?

(sorry to restate the same question, just making sure about it)

Thank you, Mike

On 2015-11-11 03:44, mancyborg@gmail.com wrote:

On Tue, 10 Nov 2015 08:50:50 +0100 Christian Kivalo ml+dovecot@valo.at wrote:

...

Hi,

On 2015-11-10 01:44, mancyborg@gmail.com wrote:

...

Hello dear list, I've recently discovered 'doveadm stats' and I'm trying to use "doveadm stats dump user" and "doveadm stats dump session" to understand the pop/imap users that put more stress on the hard disks.

My problem is that some users refuse to delete their emails from the server, so they keep 20GB of maildir files on the server, the webmail (roundcube) takes forever to open the inbox, the imap searches takes forever and meanwhile all the users wait. (already tried roundcube + memcache(d) but didn't help)

What is forever in your context? I'm using roundcube and a folder with about 78k mails opens in < 1 sec unsorted. A folder with about 37k messages from a mailinglist and thread sort takes < 3 sec. My roundcube shows 200 messages per page by default. On a side note, are you using an imap proxy for roundcube? It doesn't help you with your dovecot problem but it speeds up roundcube.

To speed up imap searches i can recommend to implement fts-solr with dovecot (or maybe fts-elasticsearch, am wanting to try that but solr works...). That will speed up your searches after mailboxes are indexed.

...

So my problem is not the storage usage itself: I don't care if the user gets tons of emails with big attachments; my problem is when the user opens / searches an imap folder with more than 10K mails and iostat util goes 100% for minutes.

Dovecot should be very quick to open even folders with a huge amount of files due to its indexes.

I'm unable to reproduce any significant numbers in iostat when accessing large mailfolders with roundcube.

Whats your configuration, filesystem, ...

...

So I've enabled dovecot's stats and enjoying "doveadm stats top", "stats-top.pl" and "doveadm stats dump user/session", but talking about "doveadm stats dump user" and its output fields:

user reset_timestamp last_update num_logins num_cmds user_cpu sys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_path mail_lookup_attr mail_read_count mail_read_bytes mail_cache_hits

I'm not sure which of those fields can help me and I can't find any relevant documentation.

So here are my questions:

1. is there a documentation for those 21 fields and for 'doveadm stats' in general ?
2. what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ?
3. which field of those is, in your opinion, more representative for expressing the workload that gives me problems ?
4. which settings do I need to store 1 week worth of stats ?

I'm currenty using the 'standard' values:

stats_refresh = 30 secs stats_track_cmds = yes stats_memory_limit = 16 M stats_command_min_time = 1 mins stats_domain_min_time = 12 hours stats_ip_min_time = 12 hours stats_session_min_time = 15 mins stats_user_min_time = 1 hours

Can you please tell me the correct parameters to store 1 week of stats ?

For stats somebody else has to jump in, i have only enabled the plugin to see what to get out of it but not made any use of it.

Please share your doveconf -n output

...

Thank you, Mike

regards christian

By 'forever' I mean more than 1 minute.

That is really long. This should not take that long.

So there is no documentation / manual for 'doveadm stats' ? Do I have to read the source to know which field does what ?

I don't know of more than whats on the dovecot wiki stats plugin page at http://wiki2.dovecot.org/Statistics

I mean the output fields of "doveadm stats dump user":

user reset_timestamp last_update num_logins num_cmds user_cpu sys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_path mail_lookup_attr mail_read_count mail_read_bytes mail_cache_hits

what's the difference between disk_output, read_bytes, read_count and mail_read_bytes ?

From the wiki page: disk_output: Number of bytes written to disk -> i'd go for disk_input if your interested in reads read_bytes: Number of bytes read using read() syscalls read_count: Number of read() syscalls mail_read_bytes: Number of message bytes read()

Not really much information but a base to start tests from.

Make yourself a testaccount and test. Thats the best way to figure stuff out by yourself.

(sorry to restate the same question, just making sure about it)

Thank you, Mike

Regards Christian

On Wed, 11 Nov 2015 09:50:29 -0600 Rick Romero rick@havokmon.com wrote:

LOL.

This is a horrible subject line.  I've been trying to resolve a DDOS issue, and ignoring a lot of email. 

Here I had been thinking this was a sex-spam, and I just got around to wondering why the spam system isn't working quite right and they kept coming in.  :P

Not sure if this was suggested, but use SSD's.  With a ton of mail combined with Maildir random reads/writes, SSD's are like night and day.

Rick

Very sorry about nasty subject ;)

Btw is it worth the offload on i/o by having some Dovecot index or cache on an SSD ?

On Wed, 11 Nov 2015 10:20:07 -0600 Rick Romero rick@havokmon.com wrote:

Quoting mancyborg@gmail.com:

...

On Wed, 11 Nov 2015 09:50:29 -0600 Rick Romero rick@havokmon.com wrote:

...

LOL.

This is a horrible subject line.  I've been trying to resolve a DDOS issue, and ignoring a lot of email. 

Here I had been thinking this was a sex-spam, and I just got around to wondering why the spam system isn't working quite right and they kept coming in.  :P

Not sure if this was suggested, but use SSD's.  With a ton of mail combined with Maildir random reads/writes, SSD's are like night and day.

Rick

Very sorry about nasty subject ;) Btw is it worth the offload on i/o by having some Dovecot index or cache on an SSD ?

Lately I've separated the indexes onto another SSD-based volume, just to save read/writes, but I don't think it actually improved anything.

Rick

I'm going to try dm-cache and flashcache asap ..