We were troubleshooting a performance problem on one of our storage servers and after investigating almost everything in sight we saw that every 30 seconds, Vmware would go from 10k IOPS (NFS) to 30k, 50k, 100k or whatever the server would handle. Most of it were reads.

What could cause this raise in NFS operations per second every 30 seconds? The virtual machines are managed by external customers and there isn't much in common between them.

While breaking utilization down by filename, we discovered 5-10 virtual machines that contributed more to those peaks but it still doesn't explain why every 30 seconds. There are no other peaks outside that 30 sec period (ie. it stays in an almost constant average).

Is there an NFS tweak in Vmware to change that 30 second period? If that's really necessary, we would like to introduce some variation so all that workload isn't dropped all at once. It's causing NFS timeout on the ESX 3.5/4.0 hosts when the storage gets overloaded.