Mysql performance problem & Failed DIMM
Posted
by
murdoch
on Server Fault
See other posts from Server Fault
or by murdoch
Published on 2011-03-14T21:01:45Z
Indexed on
2011/03/15
0:12 UTC
Read the original article
Hit count: 541
Hi
I have a dedicated mysql database server which has been having some performance problems recently, under normal load the server will be running fine, then suddenly out of the blue the performance will fall off a cliff. The server isn't using the swap file and there is 12GB of RAM in the server, more than enough for its needs.
After contacting my hosting comapnies support they have discovered that there is a failed 2GB DIMM in the server and have scheduled to replace it tomorow morning.
My question is could a failed DIMM result in the performance problems I am seeing or is this just coincidence?
My worry is that they will replace the ram tomorrow but the problems will persist and I will still be lost of explanations so I am just trying to think ahead.
The reason I ask is that there is plenty of RAM in the server, more than required and simply missing 2GB should be a problem, so if this failed DIMM is causing these performance problems then the OS must be trying to access the failed DIMM and slowing down as a result. Does that sound like a credible explanation?
This is what DELLs omreport program says about the RAM, notice one dimm is "Critical"
Memory Information
Health : Critical
Memory Operating Mode
Fail Over State : Inactive
Memory Operating Mode Configuration : Optimizer
Attributes of Memory Array(s)
Attributes : Location
Memory Array 1 : System Board or Motherboard
Attributes : Use
Memory Array 1 : System Memory
Attributes : Installed Capacity
Memory Array 1 : 12288 MB
Attributes : Maximum Capacity
Memory Array 1 : 196608 MB
Attributes : Slots Available
Memory Array 1 : 18
Attributes : Slots Used
Memory Array 1 : 6
Attributes : ECC Type
Memory Array 1 : Multibit ECC
Total of Memory Array(s)
Attributes : Total Installed Capacity
Value : 12288 MB
Attributes : Total Installed Capacity Available to the OS
Value : 12004 MB
Attributes : Total Maximum Capacity
Value : 196608 MB
Details of Memory Array 1
Index : 0
Status : Ok
Connector Name : DIMM_A1
Type : DDR3-Registered
Size : 2048 MB
Index : 1
Status : Ok
Connector Name : DIMM_A2
Type : DDR3-Registered
Size : 2048 MB
Index : 2
Status : Ok
Connector Name : DIMM_A3
Type : DDR3-Registered
Size : 2048 MB
Index : 3
Status : Critical
Connector Name : DIMM_B1
Type : DDR3-Registered
Size : 2048 MB
Index : 4
Status : Ok
Connector Name : DIMM_B2
Type : DDR3-Registered
Size : 2048 MB
Index : 5
Status : Ok
Connector Name : DIMM_B3
Type : DDR3-Registered
Size : 2048 MB
the command free -m shows this, the server seems to be using more than 10GB of ram which would suggest it is trying to use the DIMM
total used free shared buffers cached
Mem: 12004 10766 1238 0 384 4809
-/+ buffers/cache: 5572 6432
Swap: 2047 0 2047
iostat output while problem is occuring
avg-cpu: %user %nice %system %iowait %steal %idle
52.82 0.00 11.01 0.00 0.00 36.17
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 47.00 0.00 576.00 0 576
sda1 0.00 0.00 0.00 0 0
sda2 1.00 0.00 32.00 0 32
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 46.00 0.00 544.00 0 544
avg-cpu: %user %nice %system %iowait %steal %idle
53.12 0.00 7.81 0.00 0.00 39.06
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 49.00 0.00 592.00 0 592
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 49.00 0.00 592.00 0 592
avg-cpu: %user %nice %system %iowait %steal %idle
56.09 0.00 7.43 0.37 0.00 36.10
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 232.00 0.00 64520.00 0 64520
sda1 0.00 0.00 0.00 0 0
sda2 159.00 0.00 63728.00 0 63728
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 73.00 0.00 792.00 0 792
avg-cpu: %user %nice %system %iowait %steal %idle
52.18 0.00 9.24 0.06 0.00 38.51
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 49.00 0.00 600.00 0 600
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 49.00 0.00 600.00 0 600
avg-cpu: %user %nice %system %iowait %steal %idle
54.82 0.00 8.64 0.00 0.00 36.55
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 100.00 0.00 2168.00 0 2168
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 100.00 0.00 2168.00 0 2168
avg-cpu: %user %nice %system %iowait %steal %idle
54.78 0.00 6.75 0.00 0.00 38.48
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 84.00 0.00 896.00 0 896
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 84.00 0.00 896.00 0 896
avg-cpu: %user %nice %system %iowait %steal %idle
54.34 0.00 7.31 0.00 0.00 38.35
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 81.00 0.00 840.00 0 840
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 81.00 0.00 840.00 0 840
avg-cpu: %user %nice %system %iowait %steal %idle
55.18 0.00 5.81 0.44 0.00 38.58
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 317.00 0.00 105632.00 0 105632
sda1 0.00 0.00 0.00 0 0
sda2 224.00 0.00 104672.00 0 104672
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 93.00 0.00 960.00 0 960
avg-cpu: %user %nice %system %iowait %steal %idle
55.38 0.00 7.63 0.00 0.00 36.98
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 74.00 0.00 800.00 0 800
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 74.00 0.00 800.00 0 800
avg-cpu: %user %nice %system %iowait %steal %idle
56.43 0.00 7.80 0.00 0.00 35.77
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 72.00 0.00 784.00 0 784
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 72.00 0.00 784.00 0 784
avg-cpu: %user %nice %system %iowait %steal %idle
54.87 0.00 6.49 0.00 0.00 38.64
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 80.20 0.00 855.45 0 864
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 80.20 0.00 855.45 0 864
avg-cpu: %user %nice %system %iowait %steal %idle
57.22 0.00 5.69 0.00 0.00 37.09
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 33.00 0.00 432.00 0 432
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 33.00 0.00 432.00 0 432
avg-cpu: %user %nice %system %iowait %steal %idle
56.03 0.00 7.93 0.00 0.00 36.04
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 41.00 0.00 560.00 0 560
sda1 0.00 0.00 0.00 0 0
sda2 2.00 0.00 88.00 0 88
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 39.00 0.00 472.00 0 472
avg-cpu: %user %nice %system %iowait %steal %idle
55.78 0.00 5.13 0.00 0.00 39.09
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 29.00 0.00 392.00 0 392
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 29.00 0.00 392.00 0 392
avg-cpu: %user %nice %system %iowait %steal %idle
53.68 0.00 8.30 0.06 0.00 37.95
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 78.00 0.00 4280.00 0 4280
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 78.00 0.00 4280.00 0 4280
© Server Fault or respective owner
