First-Time GLSL Shadow Mapping Problems
- by Locke
I'm working on building out a 2.5D engine and having massive problems getting my shadows working.  I'm at a point where I'm VERY close.
So, let's see a picture to see what I have:
As you can see above, the image has lighting -- but the shadow map is displaying incorrectly.  The shadow map is shown in the bottom left hand side of the screen as a normal 2D texture, so we can see what it looks like at any given time.
If you notice, it appears that the shadows are generating backwards in the wrong direction -- I think.  But the problem is a little more deep -- I'm just plotting the shadow onto the screen, which I know is wrong -- I'm ignoring the actual test to see if we NEED to show a shadow.
The incoming parameters all appear to be correct -- so there has to be something wrong with my shader code somewhere.
Here's what my code looks like:
VERTEX:
    uniform mat4 LightModelViewProjectionMatrix;
    varying vec3 Normal;            // The eye-space normal of the current vertex.
    varying vec4 LightCoordinate;   // The texture coordinate of the light of the current vertex.
    varying vec3 LightDirection;    // The eye-space direction of the light.
    void main()
    {   
        Normal = normalize(gl_NormalMatrix * gl_Normal);    
        LightDirection = normalize(gl_NormalMatrix * gl_LightSource[0].position.xyz);
        LightCoordinate = LightModelViewProjectionMatrix * gl_Vertex;
        LightCoordinate.xy = ( LightCoordinate.xy * 0.5 ) + 0.5;
        gl_Position = ftransform();
        gl_TexCoord[0] = gl_MultiTexCoord0;
    }
FRAGMENT:
    uniform sampler2D DiffuseMap; 
    uniform sampler2D ShadowMap;
    varying vec3 Normal;            // The eye-space normal of the current vertex.
    varying vec4 LightCoordinate;   // The texture coordinate of the light of the current vertex.
    varying vec3 LightDirection;    // The eye-space direction of the light.
    void main()
    {   
        vec4 Texel = texture2D(DiffuseMap, vec2(gl_TexCoord[0]));
        // Directional lighting
        //Build ambient lighting
        vec4 AmbientElement = gl_LightSource[0].ambient;
        //Build diffuse lighting
        float Lambert = max(dot(Normal, LightDirection), 0.0); //max(abs(dot(Normal, LightDirection)), 0.0);
        vec4 DiffuseElement = ( gl_LightSource[0].diffuse * Lambert );
        vec4 LightingColor = ( DiffuseElement + AmbientElement );
        LightingColor.r = min(LightingColor.r, 1.0);
        LightingColor.g = min(LightingColor.g, 1.0);
        LightingColor.b = min(LightingColor.b, 1.0);
        LightingColor.a = min(LightingColor.a, 1.0);
        LightingColor *= Texel;
        //Everything up to this point is PERFECT
        // Shadow mapping
        // ------------------------------
        vec4 ShadowCoordinate = LightCoordinate / LightCoordinate.w;
        float DistanceFromLight = texture2D( ShadowMap, ShadowCoordinate.st ).z;
        float DepthBias = 0.001;
        float ShadowFactor = 1.0;
        if( LightCoordinate.w > 0.0 )
        {
            ShadowFactor = DistanceFromLight < ( ShadowCoordinate.z + DepthBias ) ? 0.5 : 1.0;
        }
        LightingColor.rgb *= ShadowFactor;
        //gl_FragColor = LightingColor;
        //Yes, I know this is wrong, but the line above (gl_FragColor = LightingColor;) produces the wrong effect
        gl_FragColor = LightingColor * texture2D( ShadowMap, ShadowCoordinate.st );
    }
I wanted to make sure the coordinates were correct for the shadow map -- so that's why you see it applied to the image as it is below.  But the depth for each point seems to be wrong -- the shadows SHOULD be opposite (look at how the image is -- the shaded areas from normal lighting are facing the opposite direction of the shadows).
Maybe my matrices are bad or something going in?  They're isolated and appear to be correct -- nothing else is going in unusual.  When I view from the light's view and get the MVP matrices for it, they're correct.
EDIT: Added an image so you can see what happens when I do the correct command at the end of the GLSL:
That's the image when the last line is just glFragColor = LightingColor;
Maybe someone has some idea of what I screwed up?