I have a fleet of webservers that server a LAMP webapp broken into multiple customer apps by virtualhost/domain. The platform is Ubuntu 10.04 VM + PHP 5.3 + Apache 2.2.14, on top of VMware ESX (v4 I think). This stuff's not too important, though -- I'm just setting up the background.

I have one customer that connects to a RADIUS server for authentication. We've found that the app responds as if some number of web servers are configured correctly and some are not. i.e. Apparently random authentication failures or successes, with no rhyme or reason.

I did a lot of analysis of our fleet, and resolved it down to the differences between two specific web servers. I'll call them "A" and "B". "A" works. "B" does not. "Works" means "connects to and gets authentication data successfully from the RADIUS server".

Ultimately, I'm looking for one thing that is different, and I've exhausted everything that I can come up with, so, looking for something else.

Here are things I've looked at

• PHP package versions (all from Ubuntu repos). These are exactly the same across servers.
• PECL package. There are no PECL packages that aren't installed by apt.
• Other libraries or packages. Nothing that was network-related or RADIUS-related was different among servers. (There were some minor package differences, though.)
• Network or hosting environment. I found that some of the working servers were on the same physical environment as some not-working ones (i.e. same ESX containers). So, probably, the physical network layer is not the problem.
• Test case. I created a test case as follows. It works on the working servers, and fails on the not-working servers, very consistently.

<?php 
$radius = radius_auth_open(); 
$username = 'theusername';
$password = 'thepassword';
$hostname = '12.34.56.78';
$radius_secret = '39wmmvxghg';
if (! radius_add_server($radius,$hostname,0,$radius_secret,5,3)) 
{ 
    die('Radius Error 1: ' . radius_strerror($radius) .  "\n"); 
} 

if (! radius_create_request($radius,RADIUS_ACCESS_REQUEST)) 
{ 
    die('Radius Error 2: ' . radius_strerror($radius) .  "\n"); 
} 

radius_put_attr($radius,RADIUS_USER_NAME,$username); 
radius_put_attr($radius,RADIUS_USER_PASSWORD,$password); 

switch (radius_send_request($radius)) 
{ 
    case RADIUS_ACCESS_ACCEPT: 
        echo 'GOOD LOGIN'; 
        break; 
    case RADIUS_ACCESS_REJECT: 
        echo 'BAD LOGIN'; 
        break; 
    case RADIUS_ACCESS_CHALLENGE: 
        echo 'CHALLENGE REQUESTED'; 
        break; 
    default: 
        die('Radius Error 3: ' . radius_strerror($radius) .  "\n"); 
} 
?>