In Windows there appear to be two ways to set up IPsec:

1. The IP Security Policy Management MMC snap-in (part of secpol.msc, introduced in Windows 2000).
2. The Windows Firewall with Advanced Security MMC snap-in (wf.msc, introduced in Windows 2008/Vista).

My question concerns #2 – I already figured out what I need to know for #1. (But I want to use the ‘new’ snap-in for its improved encryption capabilities.)

I have two Windows Server 2008 R2 computers in the same domain (domain members), on the same subnet:

server2  172.16.11.20
server3  172.16.11.30

My goal is to encrypt all communication between these two machines using IPsec in tunnel mode, so that the protocol stack is:

• IP
• ESP
• IP
• …etc.

First, on each computer, I created a Connection Security Rule:

• Endpoint 1: (local IP address), eg 172.16.11.20 for server2
• Endpoint 2: (remote IP address), eg 172.16.11.30
• Protocol: Any
• Authentication: Require inbound and outbound, Computer (Kerberos V5)
• IPsec tunnel:
  • Exempt IPsec protected connections
  • Local tunnel endpoint: Any
  • Remote tunnel endpoint: (remote IP address), eg 172.16.11.30

At this point, I can ping each machine, and Wireshark shows me the protocol stack; however, nothing is encrypted (which is expected at this point). I know that it's unencrypted because Wireshark can decode it (using the setting Attempt to detect/decode NULL encrypted ESP payloads) and the Monitor > Security Associations > Quick Mode display shows ESP Encryption: None.

Then on each server, I created Inbound and Outbound Rules:

• Protocol: Any
• Local IP addresses: (local IP address), eg 172.16.11.20
• Remote IP addresses: (remote IP address), eg 172.16.11.30
• Action: Allow the connection if it is secure
  • Require the connections to be encrypted

The problem: Though I create the Inbound and Outbound Rules on each server to enable encryption, the data is still going over the wire (wrapped in ESP) with NULL encryption. (You can see this in Wireshark.)

When the arrives at the receiving end, it's rejected (presumably because it's unencrypted). [And, disabling the Inbound rule on the receiving end causes it to lock up and/or bluescreen – fun!] The Windows Firewall log says, eg:

2014-05-30 22:26:28 DROP ICMP 172.16.11.20 172.16.11.30 - - 60 - - - - 8 0 - RECEIVE

I've tried varying a few things:

• In the Rules, setting the local IP address to Any
• Toggling the Exempt IPsec protected connections setting
• Disabling rules (eg disabling one or both sets of Inbound or Outbound rules)
• Changing the protocol (eg to just TCP)

But realistically there aren't that many knobs to turn.

Does anyone have any ideas? Has anyone tried to set up tunnel mode between two hosts using Windows Firewall?

I've successfully got it set up in transport mode (ie no tunnel) using exactly the same set of rules, so I'm a bit surprised that it didn't Just Work™ with the tunnel added.