For some reason, ipvsadm does not seem to be equally balancing the connections between my real servers when using the wlc or lc schedulers. One real server gets absolutely hammered with requests while the others receive relatively few connections.
My ldirectord.cf file looks like this:
quiescent = yes
autoreload = yes
checktimeout = 10
checkinterval = 10
# *.site.com http
virtual = 111.111.111.111:http
real = 10.10.10.1:http ipip 10
real = 10.10.10.2:http ipip 10
real = 10.10.10.3:http ipip 10
real = 10.10.10.4:http ipip 10
real = 10.10.10.5:http ipip 10
scheduler = lc
protocol = tcp
service = http
checktype = negotiate
request = "/lb"
receive = "Up and running"
virtualhost = "site.com"
fallback = 127.0.0.1:http
The weird thing that I think may be causing the problem (but I'm really not sure) is that ipvsadm doesn't seem to be tracking active connections properly, they all appear as inactive connections
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 111.111.111.111:http lc
-> 10.10.10.1:http Tunnel 10 0 10
-> 10.10.10.2:http Tunnel 10 0 18
-> 10.10.10.3:http Tunnel 10 0 3
-> 10.10.10.4:http Tunnel 10 0 10
-> 10.10.10.5:http Tunnel 10 0 5
If I do ipvsadm -Lnc then I see lots of connections but only ever in ESTABLISHED & FIN_WAIT states.
I was using ldirectord previously on a Gentoo based load balancer and the activeconn used to be accurate, since moving to Ubuntu 10.4 LTS something seems to be different.
# ipvsadm -v
ipvsadm v1.25 2008/5/15 (compiled with popt and IPVS v1.2.1)
So, is ipvsadm not tracking active connections properly and thus making load balancing work incorrectly and if so, how do I get it to work properly again?
Edit: It gets weirder, if I cat /proc/net/ip_vs then it looks like the correct activeconns are there
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP B86A9732:0050 rr
-> 0AB42453:0050 Tunnel 10 1 24
-> 0AB4321D:0050 Tunnel 10 0 23
-> 0AB426B2:0050 Tunnel 10 2 25
-> 0AB4244C:0050 Tunnel 10 2 22
-> 0AB42024:0050 Tunnel 10 2 23
