I have an apache 2 frontend with two nodes, and a backend with two instances of
tomcat 6 balance with mod_proxy_ajp. The bbdd is in a separate machine. All machines use RHEL, 6.2 on the frontend, 5.5 on the backend. The infraestructure is virtualized using VMware.
# This is the apache config in one of the virtualHost.
ProxyPreserveHost On
ProxyPass / balancer://liferay/
<Proxy balancer://liferay>
BalancerMember ajp://lrab:8009 route=liferaya
BalancerMember ajp://lrbb:8009 route=liferayb status=+H
ProxySet lbmethod=byrequests nofailover=on
</Proxy>
The conector in
tomcat is now configured like this:
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8"
enableLookups="false" allowTrace="true" />
Do you think it could be useful to set a maxThreads parameter, like in this post?? in that case, How can i determine a proper number of threads?
From time to time, we get errors like this
[Tue Sep 18 17:57:02 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Sep 18 17:57:02 2012] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.1.104:8009 (lrab)
And apache switches to the pasive node (if its active) or fails with 503.
Some things i have tried so far:
I think that i have some performance issues with one of the applications, Here you can see a threadDump But i'm not quite sure about it.
I also started to monitor the network connection. I have noticed that there are some pings lost when i have a "ping -f " so maybe it could be a network issue, but the success rate is 100% (so the lost packets are only a few among the flood, but maybe, i don't know, enough to break the link betwen apache and tomcat). I wrote a python script to check connectivity with timestamps on the pings, so i can know when the network fails.
After sniffing the network , i can also see some RST packets, but i don't know if that is a normal behaviour (some applications do that to end a network communication).
I have also noticed that the applications have problems communicating with the database, but im not even sure if this could be related or not. If you think so, i can post more info about it.
I changed the connector on the tomcats to use the native one, but still the same.
I need not even a solution to this, but maybe some guidance on how can i troubleshoot this better ¿Analyze threads, monitor mysql performance, sniff the traffic between apaches and tomcats?
Ultimately, all i need is to balance the
tomcat instances in Active/pasive mode, so if there is another way to do it, i could give it a try.
