What could cause a pixel shader to paint outside the lines of the vertex shader output?
- by Rei Miyasaka
From what I understand, the pixels that a pixel shader operates on are specified implicitly by the SV_POSITION output (in DirectX) of the vertex shader. What then could cause a pixel shader to render in the middle of nowhere?
I used the new Visual Studio 2012 graphics debugger to visualize my vertex and pixel shader output. This is the output from a DrawIndexed() call that draws a cube:
The pink part is the rendered output of the pixel shader, which takes the cube on its left as its input.
The vertex shader code:
cbuffer Buf { float4x4 final; };
struct In
{
float4 pos:POSITION;
float3 norm:NORMAL;
float2 texuv:TEXCOORD;
};
struct Out
{
float4 col:COLOR;
float2 tex:TEXCOORD;
float4 pos:SV_POSITION;
};
Out main(In input)
{
Out output;
output.pos = mul(input.pos, final);
output.col = float4(1.0f, 0.5f, 0.5f, 1.0f);
output.tex = input.texuv;
return output;
}
And the pixel shader:
struct In
{
float4 col:COLOR;
float2 tex:TEXCOORD;
float4 pos:SV_POSITION;
};
float4 main(In input) : SV_TARGET
{
return input.col;
}
The raster stage is the only thing between the vertex shader and the pixel shader, so my suspicion is that it's some raster stage settings. But the raster stage shouldn't change the shape of the vertex shader output so drastically, should it?
