HLSL tex2d sampler seemingly using inconsistent rounding; why?

Posted by RJFalconer on Stack Overflow See other posts from Stack Overflow or by RJFalconer
Published on 2010-04-16T14:23:13Z Indexed on 2010/04/21 13:33 UTC
Read the original article Hit count: 273

Filed under:
|

Hello all,

I have code that needs to render regions of my object differently depending on their location. I am trying to use a colour map to define these regions.

The problem is when I sample from my colour map, I get collisions. Ie, two regions with different colours in the colourmap get the same value returned from the sampler.

I've tried various formats of my colour map. I set the colours for each region to be "5" apart in each case;

  • Indexed colour
  • RGB, RGBA: region 1 will have RGB 5%,5%,5%. region 2 will have RGB 10%,10%,10% and so on.
  • HSV Greyscale: region 1 will have HSV 0,0,5%. region 2 will have HSV 0,0,10% and so on.

(Values selected in The Gimp)

The tex2D sampler returns a value [0..1].

[ I then intend to derive an int array index from region. Code to do with that is unrelated, so has been removed from the question ]

float region = tex2D(gColourmapSampler,In.UV).x;

Sampling the "5%" colour gave a "region" of 0.05098 in hlsl. From this I assume the 5% represents 5/100*255, or 12.75, which is rounded to 13 when stored in the texture OR when sampled by the sampler; can't tell which. (Reasoning: 0.05098 * 255 ~= 13)

By this logic, the 50% should be stored as 127.5.
Sampled, I get 0.50196 which implies it was stored as 128.

the 70% should be stored as 178.5.
Sampled, I get 0.698039, which implies it was stored as 178.

What rounding is going on here?
(127.5 becomes 128, 178.5 becomes 178 ?!)

Edit: OK, http://en.wikipedia.org/wiki/Bankers_rounding#Round_half_to_even Apparently this is "banker's rounding". Is this really what HLSL samplers use?


I am using Shader Model 2 and FX Composer. This is my sampler declaration;

//Colour map
texture gColourmapTexture <
    string ResourceName = "Globe_Colourmap_Regions_Greyscale.png";
    string ResourceType = "2D";
>;
sampler2D gColourmapSampler : register(s1) = sampler_state {
Texture = <gColourmapTexture>;
#if DIRECT3D_VERSION >= 0xa00
    Filter = MIN_MAG_MIP_LINEAR;
#else /* DIRECT3D_VERSION < 0xa00 */
    MinFilter = Linear;
    MipFilter = Linear;
    MagFilter = Linear;
#endif /* DIRECT3D_VERSION */
    AddressU = Clamp; 
    AddressV = Clamp;
};   

© Stack Overflow or respective owner

Related posts about hlsl

Related posts about fx-composer