To avoid having to parse X-Forwarded-For in Varnish, I'm trying to just set a header on the SSL terminator (currently Apache) that stores the direct client IP in a header.

On our development machine, this works:

RequestHeader set X-Foo %{REMOTE_ADDR}e

However, in staging it doesn't. Specifically, the header is empty, as illustrated by both varnishlog:

13 TxHeader     b X-Foo: (null)

(On the development machine, this shows the IP address as expected.)

Similarly, logging REMOTE_ADDR shows that it only appears to be populated on the dev machine:

# Config
LogFormat "%{X-Forwarded-For}i %{REMOTE_ADDR}e" combined
CustomLog "/var/log/httpd/access_log" combined

# Log file, staging
<my ip> -

# Log file, development
<my ip> <my ip>

Since the dev machine is, well, a dev machine, it is different in a number of ways; however, I can't track down which difference is causing this. The versions of Apache are the same (2.2.22), and I don't see anything relevant in any of the standard config files or /etc/sysconfig/httpd. And the rest of the system is reasonably similar, since they're built off the same CentOS 5 base image.

I can't even tell from the Apache documentation whether REMOTE_ADDR is expected to exist or not as an environment variable, but it clearly works on one machine, whether by fluke or design, and the inconsistency is driving me mad.