I have an apache process that suddenly appears on a particular server. When it shows up it starts consuming memory at a very rapid rate, then moves on to all the swap. In all it consumes about 11GB (including swap) of memory and the server eventually becomes unresponsive. The load on the server is under 1 at all other times. The process runs as nobody and I am having a hard time tracking down the source. If i run an strace on the process and all it did was continuously dump out mprotect over and over again

If i run lsof -p <pid>,

I get this, but only sometimes:

httpd   19229 nobody  152u  IPv4             175050 crawl-66-249-67-216.googlebot.com:62336 (CLOSE_WAIT)
httpd   19229 nobody  153u  IPv4             179104 crawl-66-249-71-167.googlebot.com:58012 (ESTABLISHED)

As long as I catch it, I can kill the process and the server almost immediately stabilizes. I have on site on the server that is getting a few thousand hits a a day that I think might be the source, but I still can't find the exact reason.

Also, this is a cPanel server and I have upcp'd the server, rebuilt apache with easy apache, and rebuilt httpd.conf.

It is not spawing any related processes, meaning I can find any php, mysql, cgi, etc. processes that relate to this process. It's just a loner process that balloons fast and consumes ever last MB of memory.

This is on a XenServer 5.6 based VM. No other servers in the cluster are having this issue.