I'm configuring a high traffic Linux server (RedHat) and running into a limit I haven't been able to track down. I need to be able to handle sustained 300 requests per second throughput using Nginx and JBoss.

The point of this server is to run checks on a user's account when that user signs in. Each request goes through Nginx to JBoss (specifically Torquebox with JBoss A7 with a Sinatra app) and then makes an LDAP request to bind that user and retrieve several attributes. It is during the bind that these errors occur. I'm able to reproduce this going directly to JBoss, so that rules out Nginx at least.

I get a variety of error messages, though oddly JBoss stopped writing to the log file recently. It used to report errors about creating native threads. Now I just see "java.net.SocketException: Connection reset" and "org.apache.http.conn.HttpHostConnectException: Connection to http://my.awesome.server:8080 refused" as responses in jmeter. To the best of my knowledge, I have plenty of available file handles, processes, sockets, and ports, yet the issue persists.

Unfortunately, I have very little experience tuning servers. I've found a couple useful documents - Ipsysctl tutorial 1.0.4 and Linux Tuning - but those documents are a bit over my head (and just entering the the configuration described in Linux Tuning doesn't fix my issue.

Here are the configuration changes I've tried (webproxy is the user that runs Nginx and JBoss):

/etc/security/limits.conf

webproxy soft nofile 65536
webproxy hard nofile 65536
webproxy soft nproc 65536
webproxy hard nproc 65536
root soft nofile 65536
root hard nofile 65536
root soft nproc 65536
root hard nofile 65536

First attempt /etc/sysctl.conf

sysctl net.core.somaxconn = 8192
sysctl net.ipv4.ip_local_port_range = 32768 65535
sysctl net.ipv4.tcp_fin_timeout = 15
sysctl net.ipv4.tcp_keepalive_time = 1800
sysctl net.ipv4.tcp_keepalive_intvl = 35
sysctl net.ipv4.tcp_tw_recycle = 1
sysctl net.ipv4.tcp_tw_reuse = 1

Second attempt /etc/sysctl.conf

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_congestion_control=htcp
net.ipv4.tcp_mtu_probing=1

Any ideas what might be happening here? Or better yet, are there some good documentation resources designed for beginners?