High system cpu load (%sys), system locks

Posted by Mark on Server Fault See other posts from Server Fault or by Mark
Published on 2011-11-03T16:38:26Z Indexed on 2012/11/30 23:07 UTC
Read the original article Hit count: 177

Filed under:

linux

|

cpu

For the last two weeks we are having intermittent severe spikes in system cpu usage (shown as %sys), which last for maybe half a minute, locking most processes, including ssh.

I've been trying to figure this out, but atop doesn't show anything relevant (system usage for processes it shows is insignificant), spikes are intermittent and I could not reproduce the spike using any workload for the web application this webserver hosts.

If you have any ideas on how to debug high %sys and (sometimes) %si CPU usage, please share them.

System specs (don't know if any of this is relevant): Dedicated server, CentOS 6, core i7 950, consistent 4 to 8 GB RAM free at any time, hard drives are in RAID-1.

Additional info:

dmesg output doesn't change between spikes
/var/log/messages doesn't change between spikes
Here is cat /proc/vmstat
Here is output of mpstat 1 during a typical spike

Add 07.11.11: looks like simple reboot restored system state, and we might never know what caused the disturbance in first place.

© Server Fault or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about cpu

Speedstep and Intel x5570?

as seen on Server Fault - Search for 'Server Fault'
Hi, My new server has 2 x X5570 CPUs. Now here is the output of grep -i hz /proc/cpuinfo model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz cpu MHz : 1600.231 model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz cpu MHz : 1600.231 model name : Intel(R) Xeon(R) CPU … >>> More
VMware ESXi - varying CPU time (CPU reservation)

as seen on Server Fault - Search for 'Server Fault'
Hello! I'm running FreeBSD 7.2 under VMware ESXi 3.5. Host has 2 physical CPUs and the BSD box is currently the only running VM. Only one virtual CPU is assigned to the VM. When measuring CPU time of a specific program, I get very different results from time to time. Processor usage is reported… >>> More
How to calculate CPU % based on raw CPU ticks in SNMP

as seen on Server Fault - Search for 'Server Fault'
According to http://net-snmp.sourceforge.net/docs/mibs/ucdavis.html#scalar_notcurrent ssCpuUser, ssCpuSystem, ssCpuIdle, etc are deprecated in favor of the raw variants (ssCpuRawUser, etc). The former values (which don't cover things like nice, wait, kernel, interrupt, etc) returned a percentage… >>> More
disparity between `top`'s given CPU % and process CPU usage total

as seen on Super User - Search for 'Super User'
I've noticed that there are sometimes (large) differences between the reported total CPU usage and a summation of the per-process CPU utilization given by apps like top and wmtop. As an example: I recently ran a git filter-branch --index-filter on a fairly large repo, with the index-filter command… >>> More
How to read oom-killer syslog messages?

as seen on Server Fault - Search for 'Server Fault'
I have a Ubuntu 12.04 server which sometimes dies completely - no SSH, no ping, nothing until it is physically rebooted. After the reboot, I see in syslog that the oom-killer killed, well, pretty much everything. There's a lot of detailed memory usage information in them. How do I read these logs… >>> More