task blocked for more than

Posted by Manuel Sopena Ballesteros on Server Fault See other posts from Server Fault or by Manuel Sopena Ballesteros
Published on 2014-08-22T09:04:36Z Indexed on 2014/08/22 10:21 UTC
Read the original article Hit count: 259

Filed under:

linux-kernel

|

centos6.5

|

kernel-panic

I have a webserver with the configuration below:

VMWare ESXi environemt CPanel installed CentOS release 6.5 (Final) 4 CPUs 2G RAM 2x VM disks 100G each LVM system

My issue is I am getting kernel panic quite frequently. These is a list of some processes blocked I could see from the console:

mysqld
queueprocd
httpd
suphp
vmtoolsd
loop0
auditd

this is my sar logs

Linux 2.6.32-431.3.1.el6.x86_64 (test01)        08/22/2014      _x86_64_        (4 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all     26.86      0.01      0.98      0.57      0.00     71.57
12:20:01 AM     all      1.78      0.02      1.03      0.08      0.00     97.09
12:30:01 AM     all     26.34      0.02      0.85      0.05      0.00     72.74
12:40:01 AM     all     27.12      0.01      1.11      1.22      0.00     70.54
12:50:01 AM     all      1.59      0.02      0.94      0.13      0.00     97.32
01:00:01 AM     all     26.10      0.01      0.77      0.04      0.00     73.07
01:10:01 AM     all     27.51      0.01      1.16      0.14      0.00     71.18
01:20:01 AM     all      1.80      0.07      1.06      0.08      0.00     96.99
01:30:01 AM     all     26.19      0.01      0.78      0.05      0.00     72.96
01:40:01 AM     all     26.62      0.02      0.87      0.05      0.00     72.45
01:50:02 AM     all      1.35      0.01      0.87      0.02      0.00     97.75
02:00:01 AM     all     26.11      0.02      0.69      0.02      0.00     73.17
02:10:01 AM     all     26.73      0.02      0.89      0.14      0.00     72.21
02:20:01 AM     all      1.45      0.01      0.92      0.04      0.00     97.58
02:30:01 AM     all     26.59      0.01      1.06      0.03      0.00     72.31
02:40:01 AM     all     26.27      0.01      0.72      0.05      0.00     72.95
02:50:01 AM     all      0.86      0.01      0.50      0.09      0.00     98.53
03:00:01 AM     all     25.61      0.02      0.39      0.03      0.00     73.96
03:10:01 AM     all     26.30      0.08      0.66      0.14      0.00     72.82
03:20:01 AM     all      0.81      0.01      0.51      0.04      0.00     98.63
03:30:02 AM     all     26.15      0.02      0.53      0.07      0.00     73.24
03:40:01 AM     all     26.06      0.01      0.47      0.04      0.00     73.42
03:50:01 AM     all      0.96      0.02      0.51      0.03      0.00     98.48
Average:        all     17.69      0.02      0.79      0.14      0.00     81.36

06:58:14 AM       LINUX RESTART

07:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:10:01 AM     all      1.04      0.02      0.57      0.95      0.00     97.42
07:20:02 AM     all      0.66      0.01      0.39      0.06      0.00     98.87
07:30:01 AM     all     25.71      0.01      0.45      0.16      0.00     73.67
07:40:01 AM     all     25.88      0.01      0.35      0.08      0.00     73.68

As you can see the server became unresponsive at 03.50 AM and I had to reset the VM at 06.58 AM to fix it.

dmesg does not show any relevant information.

I don't see any bottleneck in sar, any idea what can I check next?

thank you very much

Developer IT