On Tue, Jul 18, 2017 at 10:40 PM, Aki Tuomi <aki.tuomi@dovecot.fi> wrote:

On 19.07.2017 02:38, Mark Moseley wrote:

...

I've been playing with weakforced, so it fills in the 'fail2ban across a cluster' niche (not to mention RBLs). It seems to work well, once you've actually read the docs :)

I was curious if anyone had played with it and was *very* curious if anyone was using it in high traffic production. Getting things to 'work' versus getting them to work *and* handle a couple hundred dovecot servers is a very wide margin. I realize this is not a weakforced mailing list (there doesn't appear to be one anyway), but the users here are some of the likeliest candidates for having tried it out.

Mainly I'm curious if weakforced can handle serious concurrency and whether the cluster really works under load.

Hi!

Weakforced is used by some of our customers in quite large installations, and performs quite nicely.

Cool, good to know.

Do you have any hints/tips/guidelines for things like sizing, both in a per-server sense (memory, mostly) and in a cluster-sense (logins per sec :: node ratio)? I'm curious too how large is quite large. Not looking for details but just a ballpark figure. My largest install would have about 4 million mailboxes to handle, which I'm guessing falls well below 'quite large'. Looking at stats, our peak would be around 2000 logins/sec.

I'm also curious if -- assuming they're well north of 2000 logins/sec -- the replication protocol begins to overwhelm the daemon at very high concurrency.

Any rules of thumb on things like "For each additional 1000 logins/sec, add another # to setNumSiblingThreads and another # to setNumWorkerThreads" would be super appreciated too.

Thanks! And again, feel free to point me elsewhere if there's a better place to ask. For a young project, the docs are actually quite good.

Below is an answer by the current weakforced main developer. It overlaps partly with Samis answer.

---snip---

Do you have any hints/tips/guidelines for things like sizing, both in a per-server sense (memory, mostly) and in a cluster-sense (logins per sec :: node ratio)? I'm curious too how large is quite large. Not looking for details but just a ballpark figure. My largest install would have about 4 million mailboxes to handle, which I'm guessing falls well below 'quite large'. Looking at stats, our peak would be around 2000 logins/sec.

So in terms of overall requests per second, on a 4 CPU server, latencies start to rise pretty quickly once you get to around 18K requests per second. Now, bearing in mind that each login from Dovecot could generate 2 allow and 1 report requests, this leads to roughly 6K logins per second on a 4 CPU server.

In terms of memory usage, the more the better obviously, but it depends on your policy and how many time windows you have. Most of our customers have 24GB+.

I'm also curious if -- assuming they're well north of 2000 logins/sec -- the replication protocol begins to overwhelm the daemon at very high concurrency.

Eventually it will, but in tests it consumes a pretty tiny fraction of the overall CPU load compared to requests so it must be a pretty high limit. Also, if you don’t update the time windows DB in the allow function, then that doesn’t cause any replication. We’ve tested with three servers, each handling around 5-6000 logins/sec (i.e. 15-18K requests each) and the overall query rate was maintained.

Any rules of thumb on things like "For each additional 1000 logins/sec, add another # to setNumSiblingThreads and another # to setNumWorkerThreads" would be super appreciated too.

Actually the rule of thumb is more like:

• WorkerThreads - Set to number of CPUs. Set number of LuaContexts to WorkerThreads + 2
• SiblingThreads - Leave at 2 unless you see issues.

Thanks! And again, feel free to point me elsewhere if there's a better place to ask. Free free to ask questions using the weakforced issues on GitHub.

For a young project, the docs are actually quite good.

Thanks, that’s appreciated - we try to keep them up to date and comprehensive.

Neil