xen 4.1 host priodically dropping network packets of domU
- by Dyutiman Chakraborty
I have xen 4.1 Host running on a ubuntu 12.04 LTS Server with ip 153.x.x.54. I have setup 2 VMs on it, namely, "dev.mydomain.com" and "web.mydomain.com" with ips 195.X.X.2 and 195.x.x.3 respectively.
For network the VMs connect through xendbr0 (xen-bridge), and can accces the network properly.
I can also login to the VMs with ssh with no issue.
However when I ping any of the VMs, there is a high amount of periodic packet drop.
If I the ping the xen host (dom0) there is no packet drop.
Following is a output of "tcpdump | grep ICMP" on dOM0 while I was pinging one of the domU
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
05:19:55.682493 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 30, length 64
05:19:56.691144 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 31, length 64
05:19:57.698776 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 32, length 64
05:19:58.706784 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 33, length 64
05:19:59.714751 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 34, length 64
05:20:00.723144 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 35, length 64
05:20:01.730349 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 36, length 64
05:20:02.739017 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 37, length 64
05:20:03.746806 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 38, length 64
05:20:06.770326 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 41, length 64
05:20:07.778801 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 42, length 64
05:20:08.786481 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 43, length 64
05:20:09.794720 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 44, length 64
05:20:10.802395 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 45, length 64
05:20:11.810770 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 46, length 64
05:20:12.818511 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 47, length 64
05:20:13.826817 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 48, length 64
05:20:14.835125 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 49, length 64
05:20:15.842138 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3460, seq 50, length 64
05:20:18.274072 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 1, length 64
05:20:19.282347 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 2, length 64
05:20:20.290746 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 3, length 64
05:20:21.297910 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 4, length 64
05:20:22.305656 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 5, length 64
05:20:23.314369 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 6, length 64
05:20:24.322055 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 7, length 64
05:20:25.329782 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 8, length 64
05:20:26.338473 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 9, length 64
05:20:27.346411 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 10, length 64
05:20:28.354175 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 11, length 64
05:20:29.361640 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 12, length 64
05:20:30.370026 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 13, length 64
05:20:31.377696 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 14, length 64
05:20:32.386151 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 15, length 64
05:20:33.394118 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 16, length 64
05:20:34.402058 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 17, length 64
05:20:35.409002 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 18, length 64
05:20:36.417692 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > web.mydomain.com: ICMP echo request, id 3461, seq 19, length 64
05:20:36.496916 IP6 fe80::3285:a9ff:feec:fc69 > ip6-allnodes: HBH ICMP6, multicast listener querymax resp delay: 1000 addr: ::, length 24
05:20:36.499112 IP6 fe80::21c:c0ff:fe6c:c091 > ff02::1:ff6c:c091: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff6c:c091, length 24
05:20:36.507041 IP6 fe80::227:eff:fe11:fa3f > ff02::1:ff00:2: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff00:2, length 24
05:20:36.523919 IP6 fe80::21c:c0ff:fe77:6257 > ff02::1:ff77:6257: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff77:6257, length 24
05:20:36.544785 IP6 fe80::54:ff:fe12:ea9a > ff02::1:ff12:ea9a: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff12:ea9a, length 24
05:20:36.581740 IP6 fe80::5604:a6ff:fef1:6da7 > ff02::1:fff1:6da7: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:fff1:6da7, length 24
05:20:36.600103 IP6 fe80::8a8:8aa0:5e18:917a > ff02::1:ff18:917a: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff18:917a, length 24
05:20:36.601989 IP6 fe80::227:eff:fe11:fa3e > ff02::1:ff11:fa3e: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff11:fa3e, length 24
05:20:36.611090 IP6 fe80::dcad:56ff:fe57:3bbe > ff02::1:ff57:3bbe: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff57:3bbe, length 24
05:20:36.660521 IP6 fe80::54:ff:fe02:1d31 > ff02::1:ff00:6: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff00:6, length 24
05:20:36.698871 IP6 fe80::21e:8cff:feb4:9f89 > ff02::1:ffb4:9f89: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ffb4:9f89, length 24
05:20:36.776548 IP6 fe80::54:ff:fe12:ea9a > ff02::1:ff01:7: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff01:7, length 24
05:20:36.781910 IP6 fe80::54:ff:fe8f:6dd > ff02::1:ff00:3: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff00:3, length 24
05:20:36.865475 IP6 fe80::21c:c0ff:fe4a:ae9f > ff02::1:ff4a:ae9f: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff4a:ae9f, length 24
05:20:36.908333 IP6 fe80::dcad:45ff:fe90:84db > ff02::1:ff90:84db: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff90:84db, length 24
05:20:36.919653 IP6 fe80::54:ff:fe12:ea9a > ff02::1:ff00:7: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff00:7, length 24
05:20:36.924276 IP6 fe80::59a2:2a4a:2082:6dee > ff02::1:ff82:6dee: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff82:6dee, length 24
05:20:37.001905 IP6 fe80::54:ff:fe8f:6dd > ff02::1:ff8f:6dd: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff8f:6dd, length 24
05:20:37.042403 IP6 fe80::54:ff:fe95:54f2 > ff02::1:ff95:54f2: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff95:54f2, length 24
05:20:37.090992 IP6 fe80::21c:c0ff:fe77:62ac > ff02::1:ff77:62ac: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff77:62ac, length 24
05:20:37.098118 IP6 fe80::d63d:7eff:fe01:b67f > ff02::1:ff01:b67f: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff01:b67f, length 24
05:20:37.118784 IP6 fe80::54:ff:fe12:ea9a > ff02::202: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::202, length 24
05:20:37.168548 IP6 fe80::54:ff:fe02:1d31 > ff02::1:ff02:1d31: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff02:1d31, length 24
05:20:41.743286 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 1, length 64
05:20:41.743542 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 1, length 64
05:20:42.743859 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 2, length 64
05:20:42.743952 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 2, length 64
05:20:43.745689 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 3, length 64
05:20:43.745777 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 3, length 64
05:20:44.746706 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 4, length 64
05:20:44.746796 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 4, length 64
05:20:45.747986 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 5, length 64
05:20:45.748082 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 5, length 64
05:20:46.749834 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 6, length 64
05:20:46.749920 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 6, length 64
05:20:47.750838 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 7, length 64
05:20:47.751182 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 7, length 64
05:20:48.751909 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 8, length 64
05:20:48.751991 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 8, length 64
05:20:49.752542 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 9, length 64
05:20:49.752620 IP dev.mydomain.com > ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in: ICMP echo reply, id 3463, seq 9, length 64
05:20:50.754246 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 10, length 64
05:20:51.753856 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 11, length 64
05:20:52.752868 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 12, length 64
05:20:53.754174 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 13, length 64
05:20:54.753972 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 14, length 64
05:20:55.753814 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 15, length 64
05:20:56.753391 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 16, length 64
05:20:57.753683 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 17, length 64
05:20:58.753487 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 18, length 64
05:20:59.754013 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 19, length 64
05:21:00.753169 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 20, length 64
05:21:01.753757 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 21, length 64
05:21:02.753307 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 22, length 64
05:21:03.753021 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 23, length 64
05:21:04.753628 IP ABTS-North-Dynamic-226.X.X.122.airtelbroadband.in > dev.mydomain.com: ICMP echo request, id 3463, seq 24, length 64
^C479 packets captured
718 packets received by filter
238 packets dropped by kernel
3 packets dropped by interface
You see the ping request is not responed to initially, then for a moment it is replied back and then again no reply.
I have tried everything (to the best of my knowledge) to fix this, but can't find any answer
Any help will be greatly appreciated
Thanks.