We have several web servers running on Amazon (ec2) c1.xlarge, over Amazon AMI.

The servers are duplicates of each other, running the exact same hardware and software. Each server spec is: 7 GB of memory 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each) 1690 GB of instance storage 64-bit platform I/O Performance: High API name: c1.xlarge

A couple of weeks ago we have run a yum upgrade on one of the servers. Starting on this upgrade the upgraded server started showing a high load average. Needless to say, we did not update the other servers and we can not do so until we understand the reason for this behavior.

The strange thing is that when we compare the servers using top or iostat, we can not find the reason for the high load. Note that we have moved traffic from the "problematic" server to the others, which have made the "problematic" server less crowded in terms of requests, and still his load is higher.

Do you have any idea what could it be, or where else can we check? Many thanks for the help! Oz.

#
# proper server
# w command
#
 00:42:26 up 2 days, 19:54,  2 users,  load average: 0.41, 0.48, 0.49
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
      pts/1    82.80.137.29     00:28   14:05   0.01s  0.01s -bash
      pts/2    82.80.137.29     00:38    0.00s  0.02s  0.00s w


#
# proper server
# iostat command
#
Linux 3.2.12-3.2.4.amzn1.x86_64   _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.03    0.02    4.26    0.17    0.13   86.39

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvdap1            1.63         1.50        55.00     367236   13444008
xvdfp1            4.41        45.93        70.48   11227226   17228552
xvdfp2            2.61         2.01        59.81     491890   14620104
xvdfp3            8.16        14.47        94.23    3536522   23034376
xvdfp4            0.98         0.79        45.86     192818   11209784


#
# problematic server
# w command
#
 00:43:26 up 2 days, 21:52,  2 users,  load average: 1.35, 1.10, 1.17
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
      pts/0    82.80.137.29     00:28   15:04   0.02s  0.02s -bash
      pts/1    82.80.137.29     00:38    0.00s  0.05s  0.00s w


#
# problematic server
# iostat command
#
Linux 3.2.20-1.29.6.amzn1.x86_64          _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.97    0.04    3.43    0.19    0.07   88.30

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
xvdap1            2.10         1.49        76.54     374660   19253592
xvdfp1            5.64        40.98        85.92   10308946   21612112
xvdfp2            3.97         4.32        93.18    1087090   23439488
xvdfp3           10.87        30.30       115.14    7622474   28961720
xvdfp4            1.12         0.28        65.54      71034   16487112