What could cause a pixel shader to paint outside the lines of the vertex shader output?

Posted by Rei Miyasaka on Game Development See other posts from Game Development or by Rei Miyasaka
Published on 2012-06-19T04:05:28Z Indexed on 2012/06/19 9:24 UTC
Read the original article Hit count: 369

Filed under:
|

From what I understand, the pixels that a pixel shader operates on are specified implicitly by the SV_POSITION output (in DirectX) of the vertex shader. What then could cause a pixel shader to render in the middle of nowhere?

I used the new Visual Studio 2012 graphics debugger to visualize my vertex and pixel shader output. This is the output from a DrawIndexed() call that draws a cube:

enter image description here

The pink part is the rendered output of the pixel shader, which takes the cube on its left as its input.

The vertex shader code:

cbuffer Buf { float4x4 final; };

struct In
{
    float4 pos:POSITION;
    float3 norm:NORMAL;
    float2 texuv:TEXCOORD;
};

struct Out
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

Out main(In input)
{
    Out output;
    output.pos = mul(input.pos, final);
    output.col = float4(1.0f, 0.5f, 0.5f, 1.0f);
    output.tex = input.texuv;

    return output;
}

And the pixel shader:

struct In
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

float4 main(In input) : SV_TARGET
{
    return input.col;
}

The raster stage is the only thing between the vertex shader and the pixel shader, so my suspicion is that it's some raster stage settings. But the raster stage shouldn't change the shape of the vertex shader output so drastically, should it?

© Game Development or respective owner

Related posts about directx

Related posts about shaders