Search Results

Search found 22839 results on 914 pages for 'decimal point'.

Page 7/914 | < Previous Page | 3 4 5 6 7 8 9 10 11 12 13 14  | Next Page >

  • Real life example fo Floating Point error

    - by Rob
    Is there any examples of a company that was burned by floating point data that caused a rounding issue? We're implementing a new system and all the monetary values are stored in floats. I think if i can show actual examples of why this has failed it'll have more weight than the theory of why the values can't be stored properly.

    Read the article

  • Floating point arithmetics restricted to integers

    - by user396672
    I use doubles for a uniform implementation of some arithmetic calculations. These calculations may be actually applied to integers too, but there are no C++-like templates in Java and I don't want to duplicate the implementation code, so I simply use "double" version for ints. Does JVM spec guarantees the correctness of integer operations such a <=,=, +, -, *, and / (in case of remainder==0) when the operations are emulated as corresponding floating point ops? (Any integer, of course, has reasonable size to be represented in double's mantissa)

    Read the article

  • Floating point computer - Trouble with getting back correct results

    - by Francisco P.
    Having trouble with a challenge. Let's say I have a theoretical, base 10, floating point calculator with the following characteristics Only 3 digits for mantissa 1 digit for exponent Sign for mantissa and exponent How would this machine compute the following? 300 + \sum_{i=1}^{100} 0.2 The correct result is 320. The machine's result is 300. But why? Can't get where the 20 goes goes missing... Thanks for your time.

    Read the article

  • OOP Design of items in a Point-of-Sale system

    - by Jonas
    I am implementing a Point-of-Sale system. In the system I represent an Item in three places, and I wounder how I should represent them in OOP. First I have the WarehouseItem, that contains price, purchase price, info about the supplier, suppliers price, info about the product and quantity in warehouse. Then I have CartItem, which contains the same fields as WarehouseItem, but adds NrOfItems and Discount. And finally I have ReceiptItem, thats contains an item where I have stripped of info about the supplier, and only contains the price that was payed. Are there any OOP-recommendations, best-practices or design patterns that I could apply for this? I don't really know if CartItem should contain (wrap) an WarehouseItem, or extend it, or if I just should copy the fields that I need. Maybe I should create an Item-class where I keep all common fields, and then extend it to WarehouseItem, CartItem and ReceiptItem. Sometimes I think that it is good to keep the field of the item and just display the information that is needed.

    Read the article

  • Another floating point question

    - by jeffmax329
    I have read most of the posts on here regarding floating point, and I understand the basic underlying issue that using IEEE 754 (and just by the nature of storing numbers in binary) certain fractions cannot be represented. I am trying to figure out the following: If both Python and JavaScript use the IEEE 754 standard, why is it that executing the following in Python .1 + .1 Results in 0.20000000000000001 (which is to be expected) Where as in Javascript (in at least Chrome and Firefox) the answer is .2 However performing .1 + .2 In both languages results in 0.30000000000000004 In addition, executing var a = 0.3; in JavaScript and printing a results in 0.3 Where as doing a = 0.3 in Python results in 0.29999999999999999 I would like to understand the reason for this difference in behavior. In addition, many of the posts on OS link to a JavaScript port of Java's BigDecimal, but the link is dead. Does anyone have a copy?

    Read the article

  • Using write to print floating point numbers.

    - by Tom
    Hi, As an exercise to achieve something larger, i'm trying to use write to print a floating point number. I haven't done this in a while. I must be doing something wrong because I cant get it to work. Here is my code #include <unistd.h> int main(){ float f = 4.5; write(1,&f,sizeof float); return 0; } However, when running it im getting ?@ Any thoughts? Thanks in advance.

    Read the article

  • Floating point innacuracies

    - by Greg
    While writing a function which will perform some operation with each number in a range I ran into some problems with floating point inaccuracies. The problem can be seen in the code below: #include <iostream> using namespace std; int main() { double start = .99999, end = 1.00001, inc = .000001; int steps = (end - start) / inc; for(int i = 0; i <= steps; ++i) { cout << (start + (inc * i)) << endl; } } The problem is that the numbers the above program outputs look like this: 0.99999 0.999991 0.999992 0.999993 0.999994 0.999995 0.999996 0.999997 0.999998 0.999999 1 1 1 1 1 1.00001 1.00001 1.00001 1.00001 1.00001 1.00001 They only appear to be correct up to the first 1. What is the proper way to solve this problem?

    Read the article

  • Find max integer size that a floating point type can handle without loss of precision

    - by Checkers
    Double has range more than a 64-bit integer, but its precision is less dues to its representation (since double is 64-bit as well, it can't fit more actual values). So, when representing larger integers, you start to lose precision in the integer part. #include <boost/cstdint.hpp> #include <limits> template<typename T, typename TFloat> void maxint_to_double() { T i = std::numeric_limits<T>::max(); TFloat d = i; std::cout << std::fixed << i << std::endl << d << std::endl; } int main() { maxint_to_double<int, double>(); maxint_to_double<boost::intmax_t, double>(); maxint_to_double<int, float>(); return 0; } This prints: 2147483647 2147483647.000000 9223372036854775807 9223372036854775800.000000 2147483647 2147483648.000000 Note how max int can fit into a double without loss of precision and boost::intmax_t (64-bit in this case) cannot. float can't even hold an int. Now, the question: is there a way in C++ to check if the entire range of a given integer type can fit into a loating point type without loss of precision? Preferably, it would be a compile-time check that can be used in a static assertion, and would not involve enumerating the constants the compiler should know or can compute.

    Read the article

  • floating point equality in Python and in general

    - by eric.frederich
    I have a piece of code that behaves differently depending on whether I go through a dictionary to get conversion factors or whether I use them directly. The following piece of code will print 1.0 == 1.0 -> False But if you replace factors[units_from] with 10.0 and factors[units_to ] with 1.0 / 2.54 it will print 1.0 == 1.0 -> True #!/usr/bin/env python base = 'cm' factors = { 'cm' : 1.0, 'mm' : 10.0, 'm' : 0.01, 'km' : 1.0e-5, 'in' : 1.0 / 2.54, 'ft' : 1.0 / 2.54 / 12.0, 'yd' : 1.0 / 2.54 / 12.0 / 3.0, 'mile' : 1.0 / 2.54 / 12.0 / 5280, 'lightyear' : 1.0 / 2.54 / 12.0 / 5280 / 5.87849981e12, } # convert 25.4 mm to inches val = 25.4 units_from = 'mm' units_to = 'in' base_value = val / factors[units_from] ret = base_value * factors[units_to ] print ret, '==', 1.0, '->', ret == 1.0 Let me first say that I am pretty sure what is going on here. I have seen it before in C, just never in Python but since Python in implemented in C we're seeing it. I know that floating point numbers will change values going from a CPU register to cache and back. I know that comparing what should be two equal variables will return false if one of them was paged out while the other stayed resident in a register. Questions What is the best way to avoid problems like this?... In Python or in general. Am I doing something completely wrong? Side Note This is obviously part of a stripped down example but what I'm trying to do is come with with classes of length, volume, etc that can compare against other objects of the same class but with different units. Rhetorical Questions If this is a potentially dangerous problem since it makes programs behave in an undetermanistic matter, should compilers warn or error when they detect that you're checking equality of floats Should compilers support an option to replace all float equality checks with a 'close enough' function? Do compilers already do this and I just can't find the information.

    Read the article

  • How to correctly pass a float from C# to C++ (dll)

    - by RavelT
    I'm getting huge differences when I pass a float from C# to C++. I'm passing a dynamic float wich changes over time. With a debuger I get this: c++ lonVel -0.036019072 float c# lonVel -0.029392920 float I did set my MSVC++2010 floating point model to /fp:fast wich should be the standard in .NET if I'm not mistaken, but this didnt help. Now I cant give out the code but I can show a fraction of it. From C# side it looks like this: namespace Example { public class Wheel { public bool loging = true; #region Members public IntPtr nativeWheelObject; #endregion Members public Wheel() { this.nativeWheelObject = Sim.Dll_Wheel_Add(); return; } #region Wrapper methods public void SetVelocity(float lonRoadVelocity,float latRoadVelocity){Sim.Dll_Wheel_SetVelocity(this.nativeWheelObject,lonRoadVelocity,latRoadVelocity);} #endregion Wrapper methods } internal class Sim { #region PInvokes [DllImport(pluginName, CallingConvention=CallingConvention.Cdecl)] public static extern void Dll_Wheel_SetVelocity(IntPtr wheel,float lonRoadVelocity,float latRoadVelocity); #endregion PInvokes } } And in C++ side @ exportFunctions.cpp: EXPORT_API void Dll_Wheel_SetVelocity(CarWheel* wheel,float lonRoadVelocity,float latRoadVelocity){ wheel->SetVelocity(lonRoadVelocity,latRoadVelocity);} So any sugestions on what I should do in order to get 1:1 results or atleast 99% correct results.

    Read the article

  • asp.net compare validators to allow comma and dot (both!) as decimal separator

    - by DanC
    I am using a compare validator, which validates that the entered number is a valid double and also validates it against a given value (greater than zero). I am validating money amounts. Because of the location where the app is used, the locale sets the comma as the decimal separator. The problem is that when a user enters the value using the numeric keyboard, the number gets written with the dot as decimal separator, and is rejected by the validation. I'd like to have this validation done before triggering a postback (like a customvalidator would) and accepting both separators. Any ideas? Thanks

    Read the article

  • Given a start and end point, how can I constrain the end point so the resulting line segment is horizontal, vertical, or 45 degrees?

    - by GloryFish
    I have a grid of letters. The player clicks on a letter and drags out a selection. Using Bresenham's Algorithm I can create a line of highlighted letters representing the player's selection. However, what I really want is to have the line segment be constrained to 45 degree angles (as is common for crossword-style games). So, given a start point and an end point, how can I find the line that passes through the start point and is closest to the end point? Bonus: To make things super sweet I'd like to get a list of points in the grid that the line passes through, and for super MEGA bonus points, I'd like to get them in order of selection (i.e. from start point to end point).

    Read the article

  • About floating point precision and why do we still use it

    - by system_is_b0rken
    Floating point has always been troublesome for precision on large worlds. This article explains behind-the-scenes and offers the obvious alternative - fixed point numbers. Some facts are really impressive, like: "Well 64 bits of precision gets you to the furthest distance of Pluto from the Sun (7.4 billion km) with sub-micrometer precision. " Well sub-micrometer precision is more than any fps needs (for positions and even velocities), and it would enable you to build really big worlds. My question is, why do we still use floating point if fixed point has such advantages? Most rendering APIs and physics libraries use floating point (and suffer it's disadvantages, so developers need to get around them). Are they so much slower? Additionally, how do you think scalable planetary engines like outerra or infinity handle the large scale? Do they use fixed point for positions or do they have some space dividing algorithm?

    Read the article

  • understanding floating point variables

    - by Syom
    There is some problem, i can't understand anyway. look at this code please <script type="text/javascript"> function math(x) { var y; y = x*10; alert(y); } </script> <input type="button" onclick="math(0.011)"> What must be alerted after i click on button? i think 0.11, but no, it alerts 0.10999999999999999 explain please this behavior. thanks in advance

    Read the article

  • C++ floating point precision

    - by Davinel
    double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = 0; for (char i = 1; i <= 50; i++) { b = b + a; }; std::cout.precision(20); std::cout << b << std::endl; result: 15.000000000000014211 So.. 'a' is smaller than it should be. But if we take 'a' 50 times - result will be bigger than it should be. Why is this? And how to get correct result in this case?

    Read the article

  • how IEEE-754 floating point numbers work

    - by hatorade
    Let's say I have this: float i = 1.5 in binary, this float is represented as: 0 01111111 10000000000000000000000 I broke up the binary to represent the 'signed', 'exponent' and 'fraction' chunks. What I don't understand is how this represents 1.5. The exponent is 0 once you subtract the bias (127 - 127), and the fraction part with the implicit leading one is 1.1. How does 1.1 scaled by nothing = 1.5???

    Read the article

  • floating point precision in ruby on rails model validations

    - by Chris Allison
    Hello I am trying to validate a dollar amount using a regex: ^[0-9]+\.[0-9]{2}$ This works fine, but whenever a user submits the form and the dollar amount ends in 0(zero), ruby(or rails?) chops the 0 off. So 500.00 turns into 500.0 thus failing the regex validation. Is there any way to make ruby/rails keep the format entered by the user, regardless of trailing zeros?

    Read the article

  • how floating point numbers work in C

    - by hatorade
    Let's say I have this: float i = 1.5 in binary, this float is represented as: 0 01111111 10000000000000000000000 I broke up the binary to represent the 'signed', 'exponent' and 'fraction' chunks. What I don't understand is how this represents 1.5. The exponent is 0 once you subtract the bias (127 - 127), and the fraction part with the implicit leading one is 1.1. How does 1.1 scaled by nothing = 1.5???

    Read the article

  • Floating point precision and physics calculations

    - by Vee
    The gravity Vector2 in my physics world is (0; 0.1). The number 0.1 is known to be problematic, since "it cannot be represented exactly, but is approximately 1.10011001100110011001101 × 2-4". Having this value for the gravity gives me problems with collisions and creates quite nasty bugs. Changing the value to 0.11 solves these problems. Is there a more elegant solution that doesn't require changing the value at all?

    Read the article

  • Rejigging a floating point equation ...

    - by Jamie
    I'd like to know if there is a way to improve the accuracy of calculating a slope. (This came up a few months back here). It seems by changing: float get_slope(float dXa, float dXb, float dYa, float dYb) { return (dXa - dXb)/(dYa - dYb); } to float get_slope(float dXa, float dXb, float dYa, float dYb) { return dXa/(dYa - dYb) - dXb/(dYa - dYb); } might be an improvement. Suggestions? Edit: It's precision I'm after, not efficiency.

    Read the article

  • Floating point comparison in STL, BOOST

    - by Paul
    Is there in the STL or in Boost a set of generic simple comparison functions? The one I found are always requiring template parameters, and/or instantiation of a struct template. I'm looking for something with a syntax like : if ( is_greater(x,y) ) { ... } Which could be implemented as : template <typename T> bool is_greater(const T& x, const T& y) { return x > y + Precision<T>::eps; }

    Read the article

  • Floating point precision nuances.

    - by user247077
    Hi, I found this code in NVIDIA's cuda SDK samples. void computeGold( float* reference, float* idata, const unsigned int len) { reference[0] = 0; double total_sum = 0; unsigned int i; for( i = 1; i < len; ++i) { total_sum += idata[i-1]; reference[i] = idata[i-1] + reference[i-1]; } // Here it should be okay to use != because we have integer values // in a range where float can be exactly represented if (total_sum != reference[i-1]) printf("Warning: exceeding single-precision accuracy. Scan will be inaccurate.\n"); } (C) Nvidia Corp Can somebody please tell me a case where the warning would be printed, and most importantly, why. Thank you very much.

    Read the article

< Previous Page | 3 4 5 6 7 8 9 10 11 12 13 14  | Next Page >