Conflicting ip routes with local table on attaching a virtual network interface
- by user1071840
I have an EC2 instance with these ip rules:
$ sudo ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
I can attach an elastic network interface to it with a private IP. Say the IP of my machine is 10.1.3.12 and the IP of the interface is 10.1.1.190. As soon as I attach the interface to my machine a new entry is added to the routing policy and local routing table:
sudo ip rule show
0: from all lookup local
32765: from 10.1.1.190 lookup 10003
32766: from all lookup main
32767: from all lookup default
$ sudo ip route show table local
broadcast 10.1.1.0 dev eth3 proto kernel scope link src 10.1.1.190
local 10.1.1.190 dev eth3 proto kernel scope host src 10.1.1.190
broadcast 10.1.1.255 dev eth3 proto kernel scope link src 10.1.1.190
broadcast 10.1.3.0 dev eth0 proto kernel scope link src 10.1.3.12
local 10.1.3.12 dev eth0 proto kernel scope host src 10.1.3.12
broadcast 10.1.3.255 dev eth0 proto kernel scope link src 10.1.3.12
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
I can send traffic to this ENI directly from a host that can have the same IP as the host the ENI is attached to. This is where the problem starts. I ran tcpdump on the port in question and saw multiple SYNs going to the ENI with src '10.1.3.12' and destination '10.1.1.190' but didn't see even a single ACK.
In my understanding if ACKs were being sent from the ENI they'd have destination as 10.1.3.12 i.e. the same as the local machine's IP and such packets will now be routed as local packets matching local routing policy:
local 10.1.3.12 dev eth0 proto kernel scope host src 10.1.3.12
I'd like to send all the packets originating from 10.1.1.190 (my ENI) to go back on the same interface i.e. eth3 in this case.
Contents of the nee table 10003 are:
$ sudo ip route show table 10003
default via 10.1.1.1 dev eth3
I think I can do the following:
I don't know if its possible but probably decrease the priority of local table so the packets match the table 10003.
Use iptables to mangle these packets and update the local table route to include the mark information
But I'm not sure if these are the right approaches.
