Problem
Network speed between a VM and another machine which is not residing on the same host, is 11MB/s at most.

Topology

Facts

• ESXi5 version is 5.0.0.504890
• VM has the latest Vmware Tools installed
• VM is using E1000 network driver
• Physical box has Win Srv 2008 R2 as the OS
• CrystalDiskMark says the drive on physical box can read/write 100MB/s
• vCenter is another vm on esx
• both vm and physical box are showing 1Gbps link speed
• Configuration > Networking shows vmnic0 as 1000 Full
• NTttcp is a client/server tool from Microsoft for measuring pure network throughput

Here's what I've done so far:

Test1:

• VM is running Filezilla FTP Server (default settings, one user account made)
• Physical box is running Filezilla FTP Client (default settings)
• Physical box is uploading a big file to FTP server
• Transfer speed (as observed by Windows Task Manager on both machines): ~11MB/s (bad)
• Physical box is downloading that file from FTP server
• Transfer speed (as observed by Windows Task Manager on both machines): still ~11MB/s (bad)

Could it be disk performance issue?

Test2:

• Physical box is running ntttcpr.exe -a 6 -m 6,0,VM_IP_ADDRESS
• VM is running ntttcps.exe -a 6 -m 6,0,PHY_BOX_IP_ADDRESS
• Transfer speed (as observed by Windows Task Manager on both machines): ~11MB/s (bad)

Could it be switch performance issue?

Test3:

• physical box is running vSphere Client
• I open Summary > Storage > datastore > Browse Datastore... from physical box and upload a file to datastore
• Transfer speed (as observed by Windows Task Manager on physical box): ~26-36MB/s (good)

Could it be a vm specific issue?

Test4:

• Installed ntttcp to another vm on the same esx server
• Measured network performance between vms on the same esx server with NTttcp
• Transfer speed (as observed by Windows Task Manager on physical box): ~90-120MB/s (excellent :)

Test5:
I have another esx server on the same site, connecting to the same datastore and same switch. Those two ESX servers have both 2 NICs. One NIC goes to switch while the other goes directly to the other ESX server.

• vMotioned one of the testing vms off to the other ESX host
• Measured network performance between vms on different esx servers with NTttcp
• Transfer speed (as observed by Windows Task Manager on physical box): ~11MB/s (bad)

While I'm aware of these:

• ESXi 4.1 slow file transfer
• ESXi 5 network performance is slow
• Debian Etch and ESXi slow network speeds
• VMWare ESXi slow file copy to guest

they did not help (or I must have been missed something)