Packets marked by iptables only sent to the correct routing table sometimes
Posted
by
cookiecaper
on Super User
See other posts from Super User
or by cookiecaper
Published on 2012-10-27T00:39:10Z
Indexed on
2012/10/27
11:07 UTC
Read the original article
Hit count: 304
I am trying to route packets generated by a specific user out over a VPN. I have this configuration:
$ sudo iptables -S -t nat
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o tun0 -j MASQUERADE
$ sudo iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
$ sudo ip rule show
0: from all lookup local
32765: from all fwmark 0xb lookup 11
32766: from all lookup main
32767: from all lookup default
$ sudo ip route show table 11
10.8.0.5 dev tun0 proto kernel scope link src 10.8.0.6
10.8.0.6 dev tun0 scope link
10.8.0.1 via 10.8.0.5 dev tun0
0.0.0.0/1 via 10.8.0.5 dev tun0
$ sudo iptables -S -t raw
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
-A OUTPUT -m owner --uid-owner guy -j TRACE
-A OUTPUT -p tcp -m tcp --dport 80 -j TRACE
It seems that some sites work fine and use the VPN, but others don't and fall back to the normal interface. This is bad. This is a packet trace that used VPN:
Oct 27 00:24:28 agent kernel: [612979.976052] TRACE: raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
Oct 27 00:24:28 agent kernel: [612979.976105] TRACE: raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
Oct 27 00:24:28 agent kernel: [612979.976164] TRACE: mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
Oct 27 00:24:28 agent kernel: [612979.976210] TRACE: mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:24:28 agent kernel: [612979.976269] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:24:28 agent kernel: [612979.976320] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:24:28 agent kernel: [612979.976367] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:24:28 agent kernel: [612979.976414] TRACE: nat:POSTROUTING:rule:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
and this is one that didn't:
Oct 27 00:22:41 agent kernel: [612873.662559] TRACE: raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
Oct 27 00:22:41 agent kernel: [612873.662609] TRACE: raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
Oct 27 00:22:41 agent kernel: [612873.662664] TRACE: mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
Oct 27 00:22:41 agent kernel: [612873.662709] TRACE: mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:22:41 agent kernel: [612873.662761] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:22:41 agent kernel: [612873.662808] TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
Oct 27 00:22:41 agent kernel: [612873.662855] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
I have already tried "ip route flush cache", to no avail. I do not know why the first packet goes through the correct routing table, and the second doesn't. Both are marked.
Once again, I do not want ALL packets system-wide to go through the VPN, I only want packets from a specific user (UID=999) to go through the VPN. I am testing ipchicken.com and walmart.com via links, from the same user, same shell. walmart.com appears to use the VPN; ipchicken.com does not.
Any help appreciated. Will send 0.5 bitcoins to answerer who makes this fixed.
© Super User or respective owner
