First of all, this is kind of a follow up question on @byte56 excellent anwser on this question concerning picking algorithms.

I'm trying to convert one of his code examples to directX 11 however I have run in to some problems ( I can pick but the picking is way off), and I wanted to make sure I had done it rigth before moving on and checking the rest of my code. I am not that familiar with openGl but I can imagine openGl has diffrent coordinations systems, and functions that alters how you must implement to code abit.

This is his code example:

public Ray GetPickRay() {
    int mouseX = Mouse.getX();
    int mouseY = WORLD.Byte56Game.getHeight() - Mouse.getY();

    float windowWidth = WORLD.Byte56Game.getWidth();
    float windowHeight = WORLD.Byte56Game.getHeight();

   //get the mouse position in screenSpace coords
    double screenSpaceX = ((float) mouseX / (windowWidth / 2) - 1.0f) * aspectRatio;
    double screenSpaceY = (1.0f - (float) mouseY / (windowHeight / 2));

    double viewRatio = Math.tan(((float) Math.PI / (180.f/ViewAngle) / 2.00f))* zoomFactor;

    screenSpaceX = screenSpaceX * viewRatio;
    screenSpaceY = screenSpaceY * viewRatio;

    //Find the far and near camera spaces
    Vector4f cameraSpaceNear = new Vector4f((float) (screenSpaceX * NearPlane), (float)    (screenSpaceY * NearPlane), (float) (-NearPlane), 1);
    Vector4f cameraSpaceFar = new Vector4f((float) (screenSpaceX * FarPlane), (float) (screenSpaceY * FarPlane), (float) (-FarPlane), 1);


    //Unproject the 2D window into 3D to see where in 3D we're actually clicking
    Matrix4f tmpView = Matrix4f(view);
    Matrix4f invView = (Matrix4f) tmpView.invert();
    Vector4f worldSpaceNear = new Vector4f();
    Matrix4f.transform(invView, cameraSpaceNear, worldSpaceNear);

    Vector4f worldSpaceFar = new Vector4f();

    Matrix4f.transform(invView, cameraSpaceFar, worldSpaceFar);

    //calculate the ray position and direction
    Vector3f rayPosition = new Vector3f(worldSpaceNear.x, worldSpaceNear.y, worldSpaceNear.z);
    Vector3f rayDirection = new Vector3f(worldSpaceFar.x - worldSpaceNear.x, worldSpaceFar.y - worldSpaceNear.y, worldSpaceFar.z - worldSpaceNear.z);

    rayDirection.normalise();

    return new Ray(rayPosition, rayDirection);
}

All rigths reserved to him of course

This is my DirectX 11 code :

void GraphicEngine::pickRayVector(float mouseX, float mouseY,XMVECTOR&
pickRayInWorldSpacePos, XMVECTOR& pickRayInWorldSpaceDir)
{


    float PRVecX, PRVecY;
    float nearPlane = 0.1f;
    float farPlane = 200.0f;
    floar viewAngle = 0.4 * 3.14;

    PRVecX =  ((( 2.0f * mouseX) / ClientWidth ) - 1 ) * tan((viewAngle)/2);
    PRVecY =  (1-(( 2.0f * mouseY) / ClientHeight)) * tan((viewAngle)/2);

    XMVECTOR cameraSpaceNear = XMVectorSet(PRVecX * nearPlane,PRVecY * nearPlane, -nearPlane, 1.0f);
    XMVECTOR cameraSpaceFar = XMVectorSet(PRVecX * farPlane,PRVecY * farPlane, -farPlane, 1.0f);


    // Transform 3D Ray from View space to 3D ray in World space
    XMMATRIX invMat;
    XMVECTOR matInvDeter;

    invMat = XMMatrixInverse(&matInvDeter, cam->getCameraView());   //Inverse of View Space matrix is World space matrix
    XMVECTOR worldSpaceNear = XMVector3TransformCoord(cameraSpaceNear, invMat);
    XMVECTOR worldSpaceFar = XMVector3TransformCoord(cameraSpaceFar, invMat);

    pickRayInWorldSpacePos = worldSpaceNear;
    pickRayInWorldSpaceDir = worldSpaceFar-worldSpaceNear;
    pickRayInWorldSpaceDir = XMVector3Normalize(pickRayInWorldSpaceDir);

}

A couple of notes:

• The mouse coordinates are already converted so that the top left corner of the client window would be (0,0) and the bottom rigth (800,600) ( or whatever resolution you would have)
• I hadn't used any far or near plane before, so i just made some arbitrary number up for them. To my understanding it shouldnt matter as long as the object you are trying to pick is in between the range of thoese numbers
• The viewAngle is the same angle that I used when setting the camera view with XMMatrixPerspectiveFovLH , I just hadn't made it a member variable of my Camera class yet.
• I removed the variable aspectRation and zoomFactor because I assumed that they where related to some specific function of his game.

Now I'm not sure, but I think the problems lies either withing the mouse to viewspace conversion, maby that we use diffrent coordinations systems. Either that or how i transform the matrixes in the the end, because i know order is important when it comes to matrixes.

Any help is appriciated! Thanks in advance.

Edit: One more note, my code is in c++