I recently ran into a situation where I needed two IP addresses on the same subnet assigned to one Linux host so that we could run two SSL/TLS sites. My first approach was to use IP aliasing, e.g. using eth0:0, eth0:1, etc, but our network admins have some fairly strict settings in place for security that squashed this idea:

1. They use DHCP snooping and normally don't allow static IP addresses. Static addressing is accomplished by using static DHCP entries, so the same MAC address always gets the same IP assignment. This feature can be disabled per switchport if you ask and you have a reason for it (thankfully I have a good relationship with the network guys and this isn't hard to do).
2. With the DHCP snooping disabled on the switchport, they had to put in a rule on the switch that said MAC address X is allowed to have IP address Y. Unfortunately this had the side effect of also saying that MAC address X is ONLY allowed to have IP address Y. IP aliasing required that MAC address X was assigned two IP addresses, so this didn't work.

There may have been a way around these issues on the switch configuration, but in an attempt to preserve good relations with the network admins I tried to find another way. Having two network interfaces seemed like the next logical step. Thankfully this Linux system is a virtual machine, so I was able to easily add a second network interface (without rebooting, I might add - pretty cool). A few keystrokes later I had two network interfaces up and running and both pulled IP addresses from DHCP.

But then the problem came in: the network admins could see (on the switch) the ARP entry for both interfaces, but only the first network interface that I brought up would respond to pings or any sort of TCP or UDP traffic.

After lots of digging and poking, here's what I came up with. It seems to work, but it also seems to be a lot of work for something that seems like it should be simple. Any alternate ideas out there?

Step 1: Enable ARP filtering on all interfaces:

# sysctl -w net.ipv4.conf.all.arp_filter=1
# echo "net.ipv4.conf.all.arp_filter = 1" >> /etc/sysctl.conf

From the file networking/ip-sysctl.txt in the Linux kernel docs:

arp_filter - BOOLEAN
    1 - Allows you to have multiple network interfaces on the same
    subnet, and have the ARPs for each interface be answered
    based on whether or not the kernel would route a packet from
    the ARP'd IP out that interface (therefore you must use source
    based routing for this to work). In other words it allows control
    of which cards (usually 1) will respond to an arp request.

    0 - (default) The kernel can respond to arp requests with addresses
    from other interfaces. This may seem wrong but it usually makes
    sense, because it increases the chance of successful communication.
    IP addresses are owned by the complete host on Linux, not by
    particular interfaces. Only for more complex setups like load-
    balancing, does this behaviour cause problems.

    arp_filter for the interface will be enabled if at least one of
    conf/{all,interface}/arp_filter is set to TRUE,
    it will be disabled otherwise

Step 2: Implement source-based routing

I basically just followed directions from http://lartc.org/howto/lartc.rpdb.multiple-links.html, although that page was written with a different goal in mind (dealing with two ISPs).

Assume that the subnet is 10.0.0.0/24, the gateway is 10.0.0.1, the IP address for eth0 is 10.0.0.100, and the IP address for eth1 is 10.0.0.101.

Define two new routing tables named eth0 and eth1 in /etc/iproute2/rt_tables:

... top of file omitted ...
1    eth0
2    eth1

Define the routes for these two tables:

# ip route add default via 10.0.0.1 table eth0
# ip route add default via 10.0.0.1 table eth1
# ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100 table eth0
# ip route add 10.0.0.0/24 dev eth1 src 10.0.0.101 table eth1

Define the rules for when to use the new routing tables:

# ip rule add from 10.0.0.100 table eth0
# ip rule add from 10.0.0.101 table eth1

The main routing table was already taken care of by DHCP (and it's not even clear that its strictly necessary in this case), but it basically equates to this:

# ip route add default via 10.0.0.1 dev eth0
# ip route add 130.127.48.0/23 dev eth0 src 10.0.0.100
# ip route add 130.127.48.0/23 dev eth1 src 10.0.0.101

And voila! Everything seems to work just fine. Sending pings to both IP addresses works fine. Sending pings from this system to other systems and forcing the ping to use a specific interface works fine (ping -I eth0 10.0.0.1, ping -I eth1 10.0.0.1). And most importantly, all TCP and UDP traffic to/from either IP address works as expected.

So again, my question is: is there a better way to do this? This seems like a lot of work for a seemingly simple problem.