Yesterday around 1am, our server ground to a crawl. This doesn't happen often, but I'm trying to get to the bottom of it.

There is no unusual traffic volume, no unusual processes running, just all of the sudden the server started killing fcgid processes.

[Thu Aug 02 01:17:32 2012] [warn] mod_fcgid: process 26460 graceful kill fail, sending SIGKILL

... for as many fcgid processes as we have...

CPU idle fell to 0% and I/O seemed to take up most of the load. The issue lasted about 5 minutes.

I suspect there was some swap activity, although I'm not sure if it was due to killed processes being swapped in to die, or if it was because some process ramped up memory usage faster than my process watching scripts can see them.

The oom-killer wasn't triggered (at least it's not logged), so I think this was Apache for some reason restarting the processes. This is not regular, and nothing obvious appears in cron.

Is there a normal Apache process which might cause this? We run dozens of different sites, and it was late at night, so volume was very, very low. (maybe 200 requests in a 10 minute period).