i am trying to set up policy routing on my home server. My network looks like this:

Host routed          VPN gateway          Internet link
through VPN

192.168.0.35/24 ---> 192.168.0.5/24   ---> 192.168.0.1 DSL router
                     10.200.2.235/22  ....             .... 10.200.0.1  VPN server

The traffic from 192.168.0.32/27 should be and is routed through VPN. I wanted to define some routing policies to route some traffic from 192.168.0.5 through VPN as well - for start - from user with uid 2000. Policy routing is done using iptables mark target and ip rule fwmark.

The problem:

When connecting using user 2000 from 192.168.0.5 tcpdump shows outgoing packets, but nothing comes back. Traffic from 192.168.0.35 works fine (here I am not using fwmark but src policy).

Here is my VPN gateway setup:

# uname -a
Linux placebo 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:49:02 UTC 2012 i686 i686 i386 GNU/Linux
# iptables -V
iptables v1.4.12
# ip -V
ip utility, iproute2-ss111117

IPtables rules (all policies in table filter are ACCEPT)

# iptables -t mangle -nvL
Chain PREROUTING (policy ACCEPT 770K packets, 314M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 767K packets, 312M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 5520 packets, 1920K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 782K packets, 901M bytes)
 pkts bytes target     prot opt in     out     source               destination         
   74  4707 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner UID match 2000 MARK set 0x3

Chain POSTROUTING (policy ACCEPT 788K packets, 903M bytes)
 pkts bytes target     prot opt in     out     source               destination         


# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 996 packets, 51172 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 7 packets, 432 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1364 packets, 112K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 2302 packets, 160K bytes)
 pkts bytes target     prot opt in     out     source               destination         
  119  7588 MASQUERADE  all  --  *      vpn  0.0.0.0/0            0.0.0.0/0           

Routing:

# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lan state UNKNOWN qlen 1000
    link/ether 00:40:63:f9:c3:8f brd ff:ff:ff:ff:ff:ff
       valid_lft forever preferred_lft forever
3: lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 00:40:63:f9:c3:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.5/24 brd 192.168.0.255 scope global lan
    inet6 fe80::240:63ff:fef9:c38f/64 scope link 
       valid_lft forever preferred_lft forever
4: vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.200.2.235/22 brd 10.200.3.255 scope global vpn

# ip rule show
0:  from all lookup local 
32764:  from all fwmark 0x3 lookup VPN 
32765:  from 192.168.0.32/27 lookup VPN 
32766:  from all lookup main 
32767:  from all lookup default 

# ip route show table VPN
default via 10.200.0.1 dev vpn 
10.200.0.0/22 dev vpn  proto kernel  scope link  src 10.200.2.235 
192.168.0.0/24 dev lan  proto kernel  scope link  src 192.168.0.5

# ip route show
default via 192.168.0.1 dev lan  metric 100 
10.200.0.0/22 dev vpn  proto kernel  scope link  src 10.200.2.235 
192.168.0.0/24 dev lan  proto kernel  scope link  src 192.168.0.5 

TCP dump showing no traffic coming back when connection is made from 192.168.0.5 user 2000

# tcpdump -i vpn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn, link-type RAW (Raw IP), capture size 65535 bytes
### Traffic from user 2000 on 192.168.0.5 ###
10:19:05.629985 IP 10.200.2.235.37291 > 10.100-78-194.akamai.com.http: Flags [S], seq 2868799562, win 14600, options [mss 1460,sackOK,TS val 6887764 ecr 0,nop,wscale 4], length 0
10:19:21.678001 IP 10.200.2.235.37291 > 10.100-78-194.akamai.com.http: Flags [S], seq 2868799562, win 14600, options [mss 1460,sackOK,TS val 6891776 ecr 0,nop,wscale 4], length 0
### Traffic from 192.168.0.35 ###
10:23:12.066174 IP 10.200.2.235.49247 > 10.100-78-194.akamai.com.http: Flags [S], seq 2294159276, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 557451322 ecr 0,sackOK,eol], length 0
10:23:12.265640 IP 10.100-78-194.akamai.com.http > 10.200.2.235.49247: Flags [S.], seq 2521908813, ack 2294159277, win 14480, options [mss 1367,sackOK,TS val 388565772 ecr 557451322,nop,wscale 1], length 0
10:23:12.276573 IP 10.200.2.235.49247 > 10.100-78-194.akamai.com.http: Flags [.], ack 1, win 8214, options [nop,nop,TS val 557451534 ecr 388565772], length 0
10:23:12.293030 IP 10.200.2.235.49247 > 10.100-78-194.akamai.com.http: Flags [P.], seq 1:480, ack 1, win 8214, options [nop,nop,TS val 557451552 ecr 388565772], length 479
10:23:12.574773 IP 10.100-78-194.akamai.com.http > 10.200.2.235.49247: Flags [.], ack 480, win 7776, options [nop,nop,TS val 388566081 ecr 557451552], length 0