Ruby BigDecimal Round: Is this an error?

Posted by peterdp on Stack Overflow See other posts from Stack Overflow or by peterdp
Published on 2010-01-13T02:32:31Z Indexed on 2010/03/31 10:53 UTC
Read the original article Hit count: 713

Filed under:
|
|
|
|

While writing a test with a value that gets represented as a BigDecimal, I ran into something weird and decided to dig into it. In brief, '0.00009' when rounded to two decimal places is returned as 0.01 instead of 0.00. Really. Here's my script/console capture:

>> bp = BigDecimal('0.09')
=> #<BigDecimal:210fe08,'0.9E-1',4(8)>
>> bp.round(2,BigDecimal::ROUND_HALF_DOWN).to_f
=> 0.09
>> bp = BigDecimal('0.009')
=> #<BigDecimal:210bcf4,'0.9E-2',4(8)>
>> bp.round(2,BigDecimal::ROUND_HALF_DOWN).to_f
=> 0.01
>> bp = BigDecimal('0.0009')
=> #<BigDecimal:2107a8c,'0.9E-3',4(12)>
>> bp.round(2,BigDecimal::ROUND_HALF_DOWN).to_f
=> 0.0
>> bp = BigDecimal('0.00009')
=> #<BigDecimal:2103428,'0.9E-4',4(12)>
>> bp.round(2,BigDecimal::ROUND_HALF_DOWN).to_f
=> 0.01
>> bp = BigDecimal('0.000009')
=> #<BigDecimal:20ff0f8,'0.9E-5',4(12)>
>> bp.round(2,BigDecimal::ROUND_HALF_DOWN).to_f
=> 0.0

Oh, and I get the same results if I use the default mode, like so:

>> bd = BigDecimal('0.00009')
=> #<BigDecimal:2152ed8,'0.9E-4',4(12)>
>> bd.round(2).to_f
=> 0.01

Here are my versions:

ruby 1.8.6 (2008-03-03 patchlevel 114) [i686-darwin9.2.2]
Rails 2.3.4

Has anyone seen anything like this?

© Stack Overflow or respective owner

Related posts about ruby-on-rails

Related posts about ruby