FYI: It was a kernel bug.
Do you mind sharing the actual technical background. Which kernel was affected and how?
We didn't track it down to a specific bug, but we finally decided that our setup using multiple IPs for source and destination was OK and should work as intended - but it didn't.
So we switched from Debians 3.2 kernel to a 3.14 from backports and - surprise - it worked as designed. Something in the 3.2 kernel limited the number of established connections to the frame given by the local port range, even incoming ones. I still don't know what, but I am quite sure it's not a new feature in 3.14 which makes our setup work, as it should be possible to have many connects from different source IPs on basically every linux kernel.
There are other limits (as some TCP hashtable sizes) which can be tuned, but that was not the limit we were hitting...
Old (not working): linux-image-3.2.0-4-amd64 3.2.63-2
New (working OK): linux-image-3.14-0.bpo.2-rt-amd64 3.14.15-2~bpo70+1
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein