i'm currently trying to work out some 3d picking code that I started years ago, but then lost interested the assignment was completed (this part wasn't actually part of the assignment).

I am not using the mouse coords for picking, i'm just using the position in 3d space and a ray directly out from there. A small hitch though is that I want to use a cone and not a ray.

Here are the variables i'm using:

float iReticleSlope = 95/3000; //inverse reticle slope
float baseReticle = 1; //radius of the reticle at z = 0
float maxRange = 3000; //max range to target
Quaternion orientation; //the cameras orientation
Vector3d position; //the cameras position

Then I loop through each object in the world:

Vector3d transformed; //object position after transformations
float d, r; //holder variables

for(i = 0; i < objects.length; i++) {

    transformed = position - objects[i].position; //transform the position relative to camera
    orientation.multiply(transformed); //orient the object relative to the camera

    if(transformed.z < 0) {

        d = sqrt(transformed[0] * transformed[0] + transformed[1] * transformed[1]);
        r = -transformed[2] * iReticleSlope + objects[i].radius;

        if(d < r && -transformed[2] - objects[i].radius <= maxRange) {

            //the object is under the reticle

        } else {

            //the object is not under the reticle

        }

    } else {
    //the object is not under the reticle
    }
}

Now this all works fine and dandy until the window ratio doesn't match the resolution ratio. Is there any simple way to account for that