I am currently getting high CPU on a server that is just running a couple of sites with very low traffic. One of the sites is in still development going live soon. However, this site is very very slow...When browsing through its pages I can see that the CPU goes from 30% to 100% for httpd (see top output below).
I have tuned httpd & MySQL, Apache Solr, Tomcat for high performance, and I am using APC.
Not sure what to do from here or how to find the culprit as I have a bunch of messages on the httpd log and have been chasing dead ends for some time...any help is greatly appreciated.
Server:
AuthenticAMD, Quad-Core AMD Opteron(tm) Processor 2352, RAM 16GB
Linux 2.6.27 64-bit, Centos 5.5
Plesk 9.5.4, MySQL 5.1.48, PHP 5.2.17
Apache/2.2.3 (CentOS) DAV/2 mod_jk/1.2.15 mod_ssl/2.2.3 OpenSSL/0.9.8e-fips-rhel5 PHP/5.2.17 mod_perl/2.0.4 Perl/v5.8.8
Tomcat6-6.0.29-1.jpp5, Tomcat-native-1.1.20-1.el5, Apache Solr
top
17595 apache 20 0 1825m 507m 10m R 100.4 3.2 0:17.50 httpd
17596 apache 20 0 1565m 247m 9936 R 83.1 1.5 0:10.86 httpd
17598 apache 20 0 1430m 110m 6472 S 54.5 0.7 0:08.66 httpd
17599 apache 20 0 1438m 124m 12m S 37.2 0.8 0:11.20 httpd
16197 mysql 20 0 13.0g 2.0g 5440 S 9.6 12.6 297:12.79 mysqld
17617 root 20 0 12748 1172 812 R 0.7 0.0 0:00.88 top
8169 tomcat 20 0 4613m 268m 6056 S 0.3 1.7 6:40.56 java
httpd error_log
[debug] prefork.c(991): AcceptMutex: sysvsem (default: sysvsem)
[info] mod_fcgid: Process manager 17593 started
[debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 17594 for worker proxy:reverse
[debug] proxy_util.c(1967): proxy: initialized single connection worker 0 in child 17594 for (*)
[debug] proxy_util.c(1854): proxy: grabbed scoreboard slot 0 in child 17595 for worker proxy:reverse
[debug] proxy_util.c(1873): proxy: worker proxy:reverse already initialized
[notice] child pid 22782 exit signal Segmentation fault (11)
[error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed
[debug] util_ldap.c(2021): LDAP merging Shared Cache conf: shm=0x7fd29a5478c0 rmm=0x7fd29a547918 for VHOST: example.com
[info] APR LDAP: Built with OpenLDAP LDAP SDK
[info] LDAP: SSL support available
[info] Init: Seeding PRNG with 256 bytes of entropy
[info] Init: Generating temporary RSA private keys (512/1024 bits)
[info] Init: Generating temporary DH parameters (512/1024 bits)
[debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4265 indexes
[debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[debug] ssl_scache_shmcb.c(623): division_offset = 96
[debug] ssl_scache_shmcb.c(625): division_size = 15997
[debug] ssl_scache_shmcb.c(627): queue_size = 2136
[debug] ssl_scache_shmcb.c(629): index_num = 133
[debug] ssl_scache_shmcb.c(631): index_offset = 8
[debug] ssl_scache_shmcb.c(633): index_size = 16
[debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[debug] ssl_scache_shmcb.c(637): cache_data_size = 13853
[debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()