We have several web servers running on Amazon (ec2) c1.xlarge, over Amazon AMI.
The servers are duplicates of each other, running the exact same hardware and software.
Each server spec is:
7 GB of memory
20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: c1.xlarge
A couple of weeks ago we have run a yum upgrade on one of the servers. Starting on this upgrade the upgraded server started showing a high load average.
Needless to say, we did not update the other servers and we can not do so until we understand the reason for this behavior.
The strange thing is that when we compare the servers using top or iostat, we can not find the reason for the high load.
Note that we have moved traffic from the "problematic" server to the others, which have made the "problematic" server less crowded in terms of requests, and still his load is higher.
Do you have any idea what could it be, or where else can we check?
Many thanks for the help!
Oz.
#
# proper server
# w command
#
00:42:26 up 2 days, 19:54, 2 users, load average: 0.41, 0.48, 0.49
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
pts/1 82.80.137.29 00:28 14:05 0.01s 0.01s -bash
pts/2 82.80.137.29 00:38 0.00s 0.02s 0.00s w
#
# proper server
# iostat command
#
Linux 3.2.12-3.2.4.amzn1.x86_64 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
9.03 0.02 4.26 0.17 0.13 86.39
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvdap1 1.63 1.50 55.00 367236 13444008
xvdfp1 4.41 45.93 70.48 11227226 17228552
xvdfp2 2.61 2.01 59.81 491890 14620104
xvdfp3 8.16 14.47 94.23 3536522 23034376
xvdfp4 0.98 0.79 45.86 192818 11209784
#
# problematic server
# w command
#
00:43:26 up 2 days, 21:52, 2 users, load average: 1.35, 1.10, 1.17
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
pts/0 82.80.137.29 00:28 15:04 0.02s 0.02s -bash
pts/1 82.80.137.29 00:38 0.00s 0.05s 0.00s w
#
# problematic server
# iostat command
#
Linux 3.2.20-1.29.6.amzn1.x86_64 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
7.97 0.04 3.43 0.19 0.07 88.30
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvdap1 2.10 1.49 76.54 374660 19253592
xvdfp1 5.64 40.98 85.92 10308946 21612112
xvdfp2 3.97 4.32 93.18 1087090 23439488
xvdfp3 10.87 30.30 115.14 7622474 28961720
xvdfp4 1.12 0.28 65.54 71034 16487112