Improving TCP performance over a gigabit network lots of connections and high traffic for storage and streaming services
Posted
by
Linux Guy
on Server Fault
See other posts from Server Fault
or by Linux Guy
Published on 2014-06-08T18:42:10Z
Indexed on
2014/06/08
21:27 UTC
Read the original article
Hit count: 308
I have two servers, Both servers hardware Specification are
Processor : Dual Processor
RAM : over 128 G.B
Hard disk : SSD Hard disk
Outging Traffic bandwidth : 3 Gbps
network cards speed : 10 Gbps
Server A : for Encoding videos
Server B : for storage videos andstream videos over web interface like youtube
The inbound bandwidth between two servers is 10Gbps , the outbound bandwidth internet bandwidth is 500Mpbs
Both servers using public ip addresses in public and private network
Both servers transfer and connection on nginx port , and the server B used for streaming media , like youtube stream videos
Both servers in same network , when i do ping from Server A to Server B i got high time latency above 1.0ms , the time range time=52.7 ms to time=215.7 ms -
This is the output of iftop utility
353Mb 707Mb 1.04Gb 1.38Gb 1.73Gb
mqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqq
server.example.com => ip.address 6.36Mb 4.31Mb 1.66Mb
<= 158Kb 94.8Kb 35.1Kb
server.example.com => ip.address 1.23Mb 4.28Mb 1.12Mb
<= 17.1Kb 83.5Kb 21.9Kb
server.example.com => ip.address 395Kb 3.89Mb 1.07Mb
<= 6.09Kb 109Kb 28.6Kb
server.example.com => ip.address 4.55Mb 3.83Mb 1.04Mb
<= 55.6Kb 45.4Kb 13.0Kb
server.example.com => ip.address 649Kb 3.38Mb 1.47Mb
<= 9.00Kb 38.7Kb 16.7Kb
server.example.com => ip.address 5.00Mb 3.32Mb 1.80Mb
<= 65.7Kb 55.1Kb 29.4Kb
server.example.com => ip.address 387Kb 3.13Mb 1.06Mb
<= 18.4Kb 39.9Kb 15.0Kb
server.example.com => ip.address 3.27Mb 3.11Mb 1.01Mb
<= 81.2Kb 64.5Kb 20.9Kb
server.example.com => ip.address 1.75Mb 3.08Mb 2.72Mb
<= 16.6Kb 35.6Kb 32.5Kb
server.example.com => ip.address 1.75Mb 2.90Mb 2.79Mb
<= 22.4Kb 32.6Kb 35.6Kb
server.example.com => ip.address 3.03Mb 2.78Mb 1.82Mb
<= 26.6Kb 27.4Kb 20.2Kb
server.example.com => ip.address 2.26Mb 2.66Mb 1.36Mb
<= 51.7Kb 49.1Kb 24.4Kb
server.example.com => ip.address 586Kb 2.50Mb 1.03Mb
<= 4.17Kb 26.1Kb 10.7Kb
server.example.com => ip.address 2.42Mb 2.49Mb 2.44Mb
<= 31.6Kb 29.7Kb 29.9Kb
server.example.com => ip.address 2.41Mb 2.46Mb 2.41Mb
<= 26.4Kb 24.5Kb 23.8Kb
server.example.com => ip.address 2.37Mb 2.39Mb 2.40Mb
<= 28.9Kb 27.0Kb 28.5Kb
server.example.com => ip.address 525Kb 2.20Mb 1.05Mb
<= 7.03Kb 26.0Kb 12.8Kb
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
TX: cum: 102GB peak: 1.65Gb rates: 1.46Gb 1.44Gb 1.48Gb
RX: 1.31GB 24.3Mb 19.5Mb 18.9Mb 20.0Mb
TOTAL: 103GB 1.67Gb 1.48Gb 1.46Gb 1.50Gb
I check the transfer speed using iperf utility From Server A to Server B
# iperf -c 0.0.0.2 -p 8777
------------------------------------------------------------
Client connecting to 0.0.0.2, TCP port 8777
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 3] local 0.0.0.1 port 38895 connected with 0.0.0.2 port 8777
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.8 sec 528 KBytes 399 Kbits/sec
My Current Connections in Server B
# netstat -an|grep ":8777"|awk '/tcp/ {print $6}'|sort -nr| uniq -c
2072 TIME_WAIT
28 SYN_RECV
1 LISTEN
189 LAST_ACK
139 FIN_WAIT2
373 FIN_WAIT1
3381 ESTABLISHED
34 CLOSING
Server A Network Card Information
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: external
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Server B Network Card Information
Settings for eth2:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: external
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
ifconfig server A
eth0 Link encap:Ethernet HWaddr 00:25:90:ED:9E:AA
inet addr:0.0.0.1 Bcast:0.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1202795665 errors:0 dropped:64334 overruns:0 frame:0
TX packets:2313161968 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:893413096188 (832.0 GiB) TX bytes:3360949570454 (3.0 TiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2207544 errors:0 dropped:0 overruns:0 frame:0
TX packets:2207544 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:247769175 (236.2 MiB) TX bytes:247769175 (236.2 MiB)
ifconfig Server B
eth2 Link encap:Ethernet HWaddr 00:25:90:82:C4:FE
inet addr:0.0.0.2 Bcast:0.0.0.2 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39973046980 errors:0 dropped:1828387600 overruns:0 frame:0
TX packets:69618752480 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3013976063688 (2.7 TiB) TX bytes:102250230803933 (92.9 TiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1049495 errors:0 dropped:0 overruns:0 frame:0
TX packets:1049495 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:129012422 (123.0 MiB) TX bytes:129012422 (123.0 MiB)
Netstat -i on Server B
# netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth2 9000 0 42098629968 0 2131223717 0 73698797854 0 0 0 BMRU
lo 65536 0 1077908 0 0 0 1077908 0 0 0 LRU
I Turn up send/receive buffers on the network card to 2048 and problem still persist
I increase the MTU for server A and problem still persist and i increase the MTU for server B for better connectivity and transfer speed but it couldn't transfer at all
The problem is : as you can see from iperf utility, the transfer speed from server A to server B slow when i restart network service in server B the transfer in server A at full speed, after 2 minutes , it's getting slow
How could i troubleshoot slow speed issue and fix it in server B ?
Notice : if there any other commands i should execute in servers for more information, so it might help resolve the problem , let me know in comments
© Server Fault or respective owner
