hunting the fatty
ml+dovecot at valo.at
Wed Nov 11 15:34:33 UTC 2015
On 2015-11-11 03:44, mancyborg at gmail.com wrote:
> On Tue, 10 Nov 2015 08:50:50 +0100
> Christian Kivalo <ml+dovecot at valo.at> wrote:
>> On 2015-11-10 01:44, mancyborg at gmail.com wrote:
>> > Hello dear list,
>> > I've recently discovered 'doveadm stats' and I'm trying to use
>> > "doveadm stats dump user" and "doveadm stats dump session"
>> > to understand the pop/imap users that put more stress on the hard
>> > disks.
>> > My problem is that some users refuse to delete their emails from the
>> > server,
>> > so they keep 20GB of maildir files on the server, the webmail
>> > (roundcube) takes forever to open the inbox,
>> > the imap searches takes forever
>> > and meanwhile all the users wait.
>> > (already tried roundcube + memcache(d) but didn't help)
>> What is forever in your context?
>> I'm using roundcube and a folder with about 78k mails opens in < 1 sec
>> unsorted. A folder with about 37k messages from a mailinglist and
>> sort takes < 3 sec. My roundcube shows 200 messages per page by
>> On a side note, are you using an imap proxy for roundcube? It doesn't
>> help you with your dovecot problem but it speeds up roundcube.
>> To speed up imap searches i can recommend to implement fts-solr with
>> dovecot (or maybe fts-elasticsearch, am wanting to try that but solr
>> works...). That will speed up your searches after mailboxes are
>> > So my problem is not the storage usage itself:
>> > I don't care if the user gets tons of emails with big attachments;
>> > my problem is when the user opens / searches an imap folder with more
>> > than 10K mails
>> > and iostat util goes 100% for minutes.
>> Dovecot should be very quick to open even folders with a huge amount
>> files due to its indexes.
>> I'm unable to reproduce any significant numbers in iostat when
>> large mailfolders with roundcube.
>> Whats your configuration, filesystem, ...
>> > So I've enabled dovecot's stats and enjoying "doveadm stats top",
>> > "stats-top.pl" and "doveadm stats dump user/session",
>> > but talking about "doveadm stats dump user" and its output fields:
>> > user reset_timestamp last_update num_logins num_cmds user_cpu sys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_path mail_lookup_attr mail_read_count mail_read_bytes mail_cache_hits
>> > I'm not sure which of those fields can help me
>> > and I can't find any relevant documentation.
>> > So here are my questions:
>> > 1. is there a documentation for those 21 fields and for 'doveadm
>> > stats' in general ?
>> > 2. what's the difference between disk_output, read_bytes, read_count
>> > and mail_read_bytes ?
>> > 3. which field of those is, in your opinion, more representative for
>> > expressing the workload that gives me problems ?
>> > 4. which settings do I need to store 1 week worth of stats ?
>> > I'm currenty using the 'standard' values:
>> > stats_refresh = 30 secs
>> > stats_track_cmds = yes
>> > stats_memory_limit = 16 M
>> > stats_command_min_time = 1 mins
>> > stats_domain_min_time = 12 hours
>> > stats_ip_min_time = 12 hours
>> > stats_session_min_time = 15 mins
>> > stats_user_min_time = 1 hours
>> > Can you please tell me the correct parameters to store 1 week of stats
>> > ?
>> For stats somebody else has to jump in, i have only enabled the plugin
>> to see what to get out of it but not made any use of it.
>> Please share your doveconf -n output
>> > Thank you,
>> > Mike
> By 'forever' I mean more than 1 minute.
That is really long. This should not take that long.
> So there is no documentation / manual for 'doveadm stats' ?
> Do I have to read the source to know which field does what ?
I don't know of more than whats on the dovecot wiki stats plugin page at
> I mean the output fields of "doveadm stats dump user":
> user reset_timestamp last_update num_logins num_cmds user_cpu sys_cpu min_faults maj_faults vol_cs invol_cs disk_input disk_output read_count read_bytes write_count write_bytes mail_lookup_path mail_lookup_attr mail_read_count mail_read_bytes mail_cache_hits
> what's the difference between disk_output, read_bytes, read_count and
> mail_read_bytes ?
From the wiki page:
disk_output: Number of bytes written to disk -> i'd go for disk_input if
your interested in reads
read_bytes: Number of bytes read using read() syscalls
read_count: Number of read() syscalls
mail_read_bytes: Number of message bytes read()
Not really much information but a base to start tests from.
Make yourself a testaccount and test. Thats the best way to figure stuff
out by yourself.
> (sorry to restate the same question, just making sure about it)
> Thank you,
More information about the dovecot