I have been struggling with this not easily reproducible issue since a while. I am using linux kernel v3.1.0, and sometimes routing to a few IP addresses does not work. What seems to happen is that instead of sending the packet to the gateway, the kernel treats the destination address as local, and tries to gets its MAC address via ARP.

For example, now my current IP address is 172.16.1.104/24, the gateway is 172.16.1.254:

# ifconfig eth0 eth0      Link encap:Ethernet  HWaddr 00:1B:63:97:FC:DC
          inet addr:172.16.1.104  Bcast:172.16.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:230772 errors:0 dropped:0 overruns:0 frame:0
          TX packets:171013 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:191879370 (182.9 Mb)  TX bytes:47173253 (44.9 Mb)
          Interrupt:17

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.1.254    0.0.0.0         UG    0      0        0 eth0
172.16.1.0      0.0.0.0         255.255.255.0   U     1      0        0 eth0

I can ping a few addresses, but not 172.16.0.59:

# ping -c1 172.16.1.254
PING 172.16.1.254 (172.16.1.254) 56(84) bytes of data.
64 bytes from 172.16.1.254: icmp_seq=1 ttl=64 time=0.383 ms

--- 172.16.1.254 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.383/0.383/0.383/0.000 ms
root@pozsybook:~# ping -c1 172.16.0.1
PING 172.16.0.1 (172.16.0.1) 56(84) bytes of data.
64 bytes from 172.16.0.1: icmp_seq=1 ttl=63 time=5.54 ms

--- 172.16.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.545/5.545/5.545/0.000 ms
root@pozsybook:~# ping -c1 172.16.0.2
PING 172.16.0.2 (172.16.0.2) 56(84) bytes of data.
64 bytes from 172.16.0.2: icmp_seq=1 ttl=62 time=7.92 ms

--- 172.16.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 7.925/7.925/7.925/0.000 ms
root@pozsybook:~# ping -c1 172.16.0.59
PING 172.16.0.59 (172.16.0.59) 56(84) bytes of data.
From 172.16.1.104 icmp_seq=1 Destination Host Unreachable

--- 172.16.0.59 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

When trying to ping 172.16.0.59, I can see in tcpdump that an ARP req was sent:

# tcpdump -n -i eth0|grep ARP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
15:25:16.671217 ARP, Request who-has 172.16.0.59 tell 172.16.1.104, length 28

and /proc/net/arp has an incomplete entry for 172.16.0.59:

# grep 172.16.0.59 /proc/net/arp
172.16.0.59      0x1         0x0         00:00:00:00:00:00     *        eth0

Please note, that 172.16.0.59 is accessible from this LAN from other computers.

Does anyone have any idea of what's going on? Thanks.

update: replies to the comments below:

• there are no interfaces besides eth0 and lo
• the ARP req cannot be seen on the other end, but that's how it should work. the main problem is that an ARP req should not even be sent at the first place
• the problem persist even if I add an explicit route with the command "route add -host 172.16.0.59 gw 172.16.1.254 dev eth0"