We have an apache webserver in front of Tomcat hosted on EC2, instance type is extra large with 34GB memory.

Our application deals with lot of external webservices and we have a very lousy external webservice which takes almost 300 seconds to respond to requests during peak hours.

During peak hours the server chokes at just about 300 httpd processes. ps -ef | grep httpd | wc -l =300

I have googled and found numerous suggestions but nothing seems to work.. following are some configuration i have done which are directly taken from online resources.

I have increased the limits of max connection and max clients in both apache and tomcat. here are the configuration details:

//apache

   <IfModule prefork.c>
    StartServers 100
    MinSpareServers 10
    MaxSpareServers 10
    ServerLimit 50000
    MaxClients 50000
    MaxRequestsPerChild 2000
    </IfModule>

//tomcat

    <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
           connectionTimeout="600000"
           redirectPort="8443"
           enableLookups="false" maxThreads="1500"
           compressableMimeType="text/html,text/xml,text/plain,text/css,application/x-javascript,text/vnd.wap.wml,text/vnd.wap.wmlscript,application/xhtml+xml,application/xml-dtd,application/xslt+xml"
           compression="on"/>

//Sysctl.conf

 net.ipv4.tcp_tw_reuse=1
 net.ipv4.tcp_tw_recycle=1
 fs.file-max = 5049800
 vm.min_free_kbytes = 204800
 vm.page-cluster = 20
 vm.swappiness = 90
 net.ipv4.tcp_rfc1337=1
 net.ipv4.tcp_max_orphans = 65536
 net.ipv4.ip_local_port_range = 5000 65000
 net.core.somaxconn = 1024

I have been trying numerous suggestions but in vain.. how to fix this? I'm sure m2xlarge server should serve more requests than 300, probably i might be going wrong with my configuration..

The server chokes only during peak hours and when there are 300 concurrent requests waiting for the [300 second delayed] webservice to respond.

Please help..