I have a HP DL360 server with 4x 146GB SAS disks and a Smart Array P400i RAID controller with 256MB cache. The disks are in RAID 5 (3 disks + 1 hot spare).

The server is running VMware ESX 3i.

The disk write performance is really bad. Here are some numbers:

ns1:~# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 3364 MB in 2.00 seconds = 1685.69 MB/sec
Timing buffered disk reads: 18 MB in 3.79 seconds = 4.75 MB/sec

ns1:~# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=125000 && sync"
125000+0 records in
125000+0 records out
1024000000 bytes (1.0 GB) copied, 282.307 s, 3.6 MB/s

real 4m52.003s
user 0m2.160s
sys 3m10.796s

Compared to another server those number are terrible: Dell R200, 2x 500GB SATA disks, PERC raid controller (disks are mirrored).

web4:~# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 6584 MB in 2.00 seconds = 3297.79 MB/sec
Timing buffered disk reads: 316 MB in 3.02 seconds = 104.79 MB/sec

web4:~# time sh -c "dd if=/dev/zero of=ddfile bs=8k count=125000 && sync"
125000+0 records in
125000+0 records out
1024000000 bytes (1.0 GB) copied, 35.2919 s, 29.0 MB/s

real 0m36.570s
user 0m0.476s
sys 0m32.298s

The server isn't very loaded and the VMware Infrastructure Client performance monitor is showing 550KBps average read and 1208KBps average write for the last 30 minutes (highest write rate: 6.6MBps).

This has been a problem from the start. Any ideas?