I have an apache 2 frontend with two nodes, and a backend with two instances of tomcat 6 balance with mod_proxy_ajp. The bbdd is in a separate machine. All machines use RHEL, 6.2 on the frontend, 5.5 on the backend. The infraestructure is virtualized using VMware.

    # This is the apache config in one of the virtualHost.
    ProxyPreserveHost On
    ProxyPass / balancer://liferay/
    <Proxy balancer://liferay>
            BalancerMember ajp://lrab:8009 route=liferaya
            BalancerMember ajp://lrbb:8009 route=liferayb status=+H
            ProxySet lbmethod=byrequests nofailover=on
    </Proxy>

The conector in tomcat is now configured like this:

   <!-- Define an AJP 1.3 Connector on port 8009 -->
   <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" 
    enableLookups="false" allowTrace="true" />

Do you think it could be useful to set a maxThreads parameter, like in this post?? in that case, How can i determine a proper number of threads?

From time to time, we get errors like this

[Tue Sep 18 17:57:02 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Sep 18 17:57:02 2012] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.1.104:8009 (lrab)

And apache switches to the pasive node (if its active) or fails with 503.

Some things i have tried so far:

• I think that i have some performance issues with one of the applications, Here you can see a threadDump But i'm not quite sure about it.

• I also started to monitor the network connection. I have noticed that there are some pings lost when i have a "ping -f " so maybe it could be a network issue, but the success rate is 100% (so the lost packets are only a few among the flood, but maybe, i don't know, enough to break the link betwen apache and tomcat). I wrote a python script to check connectivity with timestamps on the pings, so i can know when the network fails.

• After sniffing the network , i can also see some RST packets, but i don't know if that is a normal behaviour (some applications do that to end a network communication).

• I have also noticed that the applications have problems communicating with the database, but im not even sure if this could be related or not. If you think so, i can post more info about it.

• I changed the connector on the tomcats to use the native one, but still the same.

I need not even a solution to this, but maybe some guidance on how can i troubleshoot this better ¿Analyze threads, monitor mysql performance, sniff the traffic between apaches and tomcats?

Ultimately, all i need is to balance the tomcat instances in Active/pasive mode, so if there is another way to do it, i could give it a try.