Scalability of Dovecot in the Cloud
Does anyone have any idea of how many IMAP connections a single cloud VM (4 vCores at 2.4GHz, 30GB RAM, local SSD storage - non-RAID) can be expected to handle in production. The mailboxes are fairly small (average 5MB total - 50MB max, as I don’t store attachments in Dovecot expect those saved through IMAP in the Sent/Drafts folders) and each user will probably have an average of 2 devices that have the mail clients configured to access each mailbox.
Can such a server handle 100,000 mailboxes (200,000 devices/clients)? Or is it more like 10,000? Or, even smaller?
I can scale the cloud VM up to 32 vCores and 240GB RAM (at 8 times the price) or split the mailboxes onto multiple VMs. The VM will also be running LMTP and other Dovecot services (I don’t plan on supporting POP3 at this time). The mailboxes will be sync’d to a backup VM running Dovecot for high availability so has some load from this background activity. LMTP will not be that high a load, I think, since most messages will be delivered by at night. But, clients will have IMAP connections 24/7.
Just trying to get an idea of the cost of running a potentially huge/growing mail service in the cloud… I’m going to have to support around a million mailboxes before the site will generate significant revenue to support operations.
Kevin
Zitat von KT Walrus <kevin@my.walr.us>:
Does anyone have any idea of how many IMAP connections a single
cloud VM (4 vCores at 2.4GHz, 30GB RAM, local SSD storage -
non-RAID) can be expected to handle in production. The mailboxes are
fairly small (average 5MB total - 50MB max, as I don’t store
attachments in Dovecot expect those saved through IMAP in the
Sent/Drafts folders) and each user will probably have an average of
2 devices that have the mail clients configured to access each
mailbox.Can such a server handle 100,000 mailboxes (200,000
devices/clients)? Or is it more like 10,000? Or, even smaller?
The bottleneck of IMAP-operations normally is the number of needed
IOPS for accessing the mail storage, which depends on the mean number
of concurrent users to be expected.
Can such a server handle this number of mailboxes? Most certainly yes.
The real question is: can such a server cope up with the expected load
of concurrent user sessions you are expecting?
On 04 Jun 2016, at 21:28, KT Walrus <kevin@my.walr.us> wrote:
Does anyone have any idea of how many IMAP connections a single cloud VM (4 vCores at 2.4GHz, 30GB RAM, local SSD storage - non-RAID) can be expected to handle in production. The mailboxes are fairly small (average 5MB total - 50MB max, as I don’t store attachments in Dovecot expect those saved through IMAP in the Sent/Drafts folders) and each user will probably have an average of 2 devices that have the mail clients configured to access each mailbox.
Can such a server handle 100,000 mailboxes (200,000 devices/clients)? Or is it more like 10,000? Or, even smaller?
I can scale the cloud VM up to 32 vCores and 240GB RAM (at 8 times the price) or split the mailboxes onto multiple VMs. The VM will also be running LMTP and other Dovecot services (I don’t plan on supporting POP3 at this time). The mailboxes will be sync’d to a backup VM running Dovecot for high availability so has some load from this background activity. LMTP will not be that high a load, I think, since most messages will be delivered by at night. But, clients will have IMAP connections 24/7.
Just trying to get an idea of the cost of running a potentially huge/growing mail service in the cloud… I’m going to have to support around a million mailboxes before the site will generate significant revenue to support operations.
Do you mean most of the IMAP clients will be IDLEing waiting for new mails, which mostly won't arrive until the next night? imap-hibernate feature will be very helpful there then.
Bottlenecks are commonly either the disk IO or the memory usage. With SSD you're probably less likely to run bottleneck in disk IO. Memory usage mainly depends on the number of active (non-hibernated) concurrent connections and also the mailbox sizes of the users.
I'd limit a single Dovecot VM to 64 GB of memory. Maybe more would work, but it might run into bottlenecks on the CPU usage side for services that are limited to a single process per instance.
Replication with dsync is going to increase the load and I'm not sure how big of an issue that is.
Anyway, if it's mostly IDLE connections, I'd expect 100k mailboxes/VM to be fine. Generally I'd expect about 10k active (non-hibernated) IMAP connections/VM for 32 GB of memory, but this depends a lot on the mailbox sizes.
Anyway, if it's mostly IDLE connections, I'd expect 100k mailboxes/VM to be fine. Generally I'd expect about 10k active (non-hibernated) IMAP connections/VM for 32 GB of memory, but this depends a lot on the mailbox sizes.
That is great news. 100k mailboxes/VM is a great number. I do expect most IMAP clients will be IDLEing. Do almost all email clients in use today do the IDLE command? Do most email clients open many connections per mailbox? Perhaps IDLEing on multiple namespaces/folders per mailbox? Would this affect your 100k mailboxes/VM estimate?
The cloud VM at 4 vCores, 30GB RAM, and local SSD storage is just $40/month (OVH Public Cloud). I had expected a cost of 10 cents per mailbox per month (with redundancies raising that cost to 25 cents per mailbox per month). But 100k mailboxes/VM would give me a total operating cost of less than 1 cent per month per mailbox, at scale. Maybe even 10 cents per year per mailbox for Public Cloud hosting fees?
Does anyone on this list run a large number of mailboxes per server in production? What is the largest number of Dovecot mailboxes/client connections you supported on a single server before you had to upgrade to multiple Dovecot servers?
Kevin
On Jun 11, 2016, at 6:37 PM, Timo Sirainen <tss@iki.fi> wrote:
On 04 Jun 2016, at 21:28, KT Walrus <kevin@my.walr.us <mailto:kevin@my.walr.us>> wrote:
Does anyone have any idea of how many IMAP connections a single cloud VM (4 vCores at 2.4GHz, 30GB RAM, local SSD storage - non-RAID) can be expected to handle in production. The mailboxes are fairly small (average 5MB total - 50MB max, as I don’t store attachments in Dovecot expect those saved through IMAP in the Sent/Drafts folders) and each user will probably have an average of 2 devices that have the mail clients configured to access each mailbox.
Can such a server handle 100,000 mailboxes (200,000 devices/clients)? Or is it more like 10,000? Or, even smaller?
I can scale the cloud VM up to 32 vCores and 240GB RAM (at 8 times the price) or split the mailboxes onto multiple VMs. The VM will also be running LMTP and other Dovecot services (I don’t plan on supporting POP3 at this time). The mailboxes will be sync’d to a backup VM running Dovecot for high availability so has some load from this background activity. LMTP will not be that high a load, I think, since most messages will be delivered by at night. But, clients will have IMAP connections 24/7.
Just trying to get an idea of the cost of running a potentially huge/growing mail service in the cloud… I’m going to have to support around a million mailboxes before the site will generate significant revenue to support operations.
Do you mean most of the IMAP clients will be IDLEing waiting for new mails, which mostly won't arrive until the next night? imap-hibernate feature will be very helpful there then.
Bottlenecks are commonly either the disk IO or the memory usage. With SSD you're probably less likely to run bottleneck in disk IO. Memory usage mainly depends on the number of active (non-hibernated) concurrent connections and also the mailbox sizes of the users.
I'd limit a single Dovecot VM to 64 GB of memory. Maybe more would work, but it might run into bottlenecks on the CPU usage side for services that are limited to a single process per instance.
Replication with dsync is going to increase the load and I'm not sure how big of an issue that is.
Anyway, if it's mostly IDLE connections, I'd expect 100k mailboxes/VM to be fine. Generally I'd expect about 10k active (non-hibernated) IMAP connections/VM for 32 GB of memory, but this depends a lot on the mailbox sizes.
This will depend on many more things also.
4 vcores, of unspecified dedication to your vm is not a good thing.
If you use something like gzip/bz2/lzma for compression of the emails,
will highly affect your cpu usage.
Searching and indexing will affect your cpu usage.
Quoting KT Walrus <kevin@my.walr.us>:
Anyway, if it's mostly IDLE connections, I'd expect 100k
mailboxes/VM to be fine. Generally I'd expect about 10k active
(non-hibernated) IMAP connections/VM for 32 GB of memory, but this
depends a lot on the mailbox sizes.That is great news. 100k mailboxes/VM is a great number. I do expect
most IMAP clients will be IDLEing. Do almost all email clients in
use today do the IDLE command? Do most email clients open many
connections per mailbox? Perhaps IDLEing on multiple
namespaces/folders per mailbox? Would this affect your 100k
mailboxes/VM estimate?The cloud VM at 4 vCores, 30GB RAM, and local SSD storage is just
$40/month (OVH Public Cloud). I had expected a cost of 10 cents per
mailbox per month (with redundancies raising that cost to 25 cents
per mailbox per month). But 100k mailboxes/VM would give me a total
operating cost of less than 1 cent per month per mailbox, at scale.
Maybe even 10 cents per year per mailbox for Public Cloud hosting
fees?Does anyone on this list run a large number of mailboxes per server
in production? What is the largest number of Dovecot
mailboxes/client connections you supported on a single server before
you had to upgrade to multiple Dovecot servers?Kevin
On Jun 11, 2016, at 6:37 PM, Timo Sirainen <tss@iki.fi> wrote:
On 04 Jun 2016, at 21:28, KT Walrus <kevin@my.walr.us
<mailto:kevin@my.walr.us>> wrote:Does anyone have any idea of how many IMAP connections a single
cloud VM (4 vCores at 2.4GHz, 30GB RAM, local SSD storage -
non-RAID) can be expected to handle in production. The mailboxes
are fairly small (average 5MB total - 50MB max, as I don’t store
attachments in Dovecot expect those saved through IMAP in the
Sent/Drafts folders) and each user will probably have an average
of 2 devices that have the mail clients configured to access each
mailbox.Can such a server handle 100,000 mailboxes (200,000
devices/clients)? Or is it more like 10,000? Or, even smaller?I can scale the cloud VM up to 32 vCores and 240GB RAM (at 8 times
the price) or split the mailboxes onto multiple VMs. The VM will
also be running LMTP and other Dovecot services (I don’t plan on
supporting POP3 at this time). The mailboxes will be sync’d to a
backup VM running Dovecot for high availability so has some load
from this background activity. LMTP will not be that high a load,
I think, since most messages will be delivered by at night. But,
clients will have IMAP connections 24/7.Just trying to get an idea of the cost of running a potentially
huge/growing mail service in the cloud… I’m going to have to
support around a million mailboxes before the site will generate
significant revenue to support operations.Do you mean most of the IMAP clients will be IDLEing waiting for
new mails, which mostly won't arrive until the next night?
imap-hibernate feature will be very helpful there then.Bottlenecks are commonly either the disk IO or the memory usage.
With SSD you're probably less likely to run bottleneck in disk IO.
Memory usage mainly depends on the number of active
(non-hibernated) concurrent connections and also the mailbox sizes
of the users.I'd limit a single Dovecot VM to 64 GB of memory. Maybe more would
work, but it might run into bottlenecks on the CPU usage side for
services that are limited to a single process per instance.Replication with dsync is going to increase the load and I'm not
sure how big of an issue that is.Anyway, if it's mostly IDLE connections, I'd expect 100k
mailboxes/VM to be fine. Generally I'd expect about 10k active
(non-hibernated) IMAP connections/VM for 32 GB of memory, but this
depends a lot on the mailbox sizes.
participants (4)
-
KT Walrus
-
Marc Stürmer
-
Patrick Domack
-
Timo Sirainen