I am currently getting high CPU on a server that is just running a couple of sites with very low traffic. One of the sites is in still development going live soon. However, this site is very very slow...When browsing through its pages I can see that the CPU goes from 30% to 100% for httpd (see top output below).

I have tuned httpd & MySQL, Apache Solr, Tomcat for high performance, and I am using APC.

Not sure what to do from here or how to find the culprit as I have a bunch of messages on the httpd log and have been chasing dead ends for some time...any help is greatly appreciated.

Server: AuthenticAMD, Quad-Core AMD Opteron(tm) Processor 2352, RAM 16GB

Linux 2.6.27 64-bit, Centos 5.5

Plesk 9.5.4, MySQL 5.1.48, PHP 5.2.17

Apache/2.2.3 (CentOS) DAV/2 mod_jk/1.2.15 mod_ssl/2.2.3 OpenSSL/0.9.8e-fips-rhel5 PHP/5.2.17 mod_perl/2.0.4 Perl/v5.8.8

Tomcat6-6.0.29-1.jpp5, Tomcat-native-1.1.20-1.el5, Apache Solr

top

17595 apache    20   0 1825m 507m  10m R 100.4  3.2   0:17.50 httpd
17596 apache    20   0 1565m 247m 9936 R 83.1  1.5   0:10.86 httpd
17598 apache    20   0 1430m 110m 6472 S 54.5  0.7   0:08.66 httpd
17599 apache    20   0 1438m 124m  12m S 37.2  0.8   0:11.20 httpd
16197 mysql     20   0 13.0g 2.0g 5440 S  9.6 12.6 297:12.79 mysqld
17617 root      20   0 12748 1172  812 R  0.7  0.0   0:00.88 top
8169 tomcat    20   0 4613m 268m 6056 S  0.3  1.7   6:40.56 java

httpd error_log

[debug] prefork.c(991): AcceptMutex: sysvsem (default: sysvsem)
[info] mod_fcgid: Process manager 17593 started
[debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 17594 for worker proxy:reverse
[debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 17594 for (*)
[debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 17595 for worker proxy:reverse
[debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized

[notice] child pid 22782 exit signal Segmentation fault (11)

[error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed
[debug] util_ldap.c(2021): LDAP merging Shared Cache conf: shm=0x7fd29a5478c0 rmm=0x7fd29a547918 for VHOST: example.com
[info] APR LDAP: Built with OpenLDAP LDAP SDK
[info] LDAP: SSL support available
[info] Init: Seeding PRNG with 256 bytes of entropy
[info] Init: Generating temporary RSA private keys (512/1024 bits)
[info] Init: Generating temporary DH parameters (512/1024 bits)
[debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4265 indexes
[debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[debug] ssl_scache_shmcb.c(623): division_offset = 96
[debug] ssl_scache_shmcb.c(625): division_size = 15997
[debug] ssl_scache_shmcb.c(627): queue_size = 2136
[debug] ssl_scache_shmcb.c(629): index_num = 133
[debug] ssl_scache_shmcb.c(631): index_offset = 8
[debug] ssl_scache_shmcb.c(633): index_size = 16
[debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[debug] ssl_scache_shmcb.c(637): cache_data_size = 13853
[debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()