Search Results

Search found 6604 results on 265 pages for 'double precision'.

Page 2/265 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

• x86-64 long double precision

  - by aaa

  hello. What is the actual precision of long double on Intel 64-bit platforms? is it 80 bits padded to 128 or actual 128 bit? if former, besides going gmp, is there another option to achieve true 128 precision?

  Read the article

• Difference between float and double

  - by VaioIsBorn

  I know, i've read about the difference between double precision and single precision etc. But they should give the same results on most cases right ? I was solving a problem on a programming contest and there were calculations with floating point numbers that were not really big so i decided to use float instead of double, and i checked it - i was…

  Read the article

• How to calculate a operation's time in micro second precision

  - by Sanjeet Daga

  I want to calculate performance of a function in micro second precision on Windows platform. Now Windows itself has milisecond granuality, so how can I achieve this. I tried following sample, but not getting correct results. LARGE_INTEGER ticksPerSecond = {0}; LARGE_INTEGER tick_1 = {0}; LARGE_INTEGER tick_2 = {0}; double uSec = 1000000; // Get…

  Read the article

• Best Square Root Method (Precision VS Speed)

  Square Root Methods Fast Algorithm Speed Precision computational Quake3 Fast Square Root

  Read the article

• When do you use float and when do you use double

  - by Jakub Zaverka

  Frequently in my programming experience I need to make a decision whether I should use float or double for my real numbers. Sometimes I go for float, sometimes I go for double, but really this feels more subjective. If I would be confronted to defend my decision, I would probably not give sound reasons. When do you use float and when do you use…

  Read the article

• About floating point precision and why do we still use it

  - by system_is_b0rken

  Floating point has always been troublesome for precision on large worlds. This article explains behind-the-scenes and offers the obvious alternative - fixed point numbers. Some facts are really impressive, like: "Well 64 bits of precision gets you to the furthest distance of Pluto from the Sun (7.4 billion km) with sub-micrometer precision. " …

  Read the article

• C++ floating point precision

  - by Davinel

  double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = 0; for (char i = 1; i <= 50; i++) { b = b + a; }; std::cout.precision(20); std::cout << b << std::endl; result: 15.000000000000014211 So.. 'a' is smaller than it should be. But if we…

  Read the article

• Setting minimum number of decimal places for std::ostream precision

  - by Phil Boltt

  Hi, Is there a way to set the "minimum" number of decimal places that a std::ostream will output? For example, say I have two doubles that I want to print: double a = 0; double b = 0.123456789; I can set my maximum decimal precision so that I output b exactly std::cout << std::setprecision(9) << b << std::endl; Is…

  Read the article

• Doble precision floating numbers in CUDA

  - by cuda-dev

  Does CUDA support double precision floating point numbers. Also need reasons for the same.

  Read the article

• C precision of double: compiler dependent?

  - by yCalleecharan

  Hi,on my 32-bit machine (with an Intel T7700 duo core), I have 15 precision digits for both double and long double types for the C language. I compared the parameters LDBL_DIG for long doubles and DBL_DIG for doubles and they are both 15. I got these answers using MVS2008. I was wondering if these results can be compiler dependent or do…

  Read the article

• Numerical precision of double type in Visual C++ 2008 Express debugger

  - by damik

  I'm using Visual C++ 2008 Express Edition and when i debug code: double x = 0.2; I see in debugging tooltip on x 0.20000000000000001 but: typedef numeric_limits< double > double_limit; int a = double_limit::digits10 gives me: a = 15 Why results in debugger are longer than maybe ? What is this strange precision based on ? …

  Read the article

• Return Double from Boost thread

  - by Benedikt Wutzi

  Hi I have an Boost thread which should return a double. The function looks like this: void analyser::findup(const double startwl, const double max, double &myret){ this->data.begin(); for(int i = (int)data.size() ; i >= 0;i--){ if(this->data[i].lambda > startwl){ if(this->data[i].db >= (max-30)) { …

  Read the article

• c++ floating point precision loss: 3015/0.00025298219406977296

  - by SigTerm

  The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == 0x40a78e0000000000 //*((unsigned __int64*)(&b)) == 0x3f30945640000000 double f = a/b;//3015/0.00025298219406977296; the result of calculation…

  Read the article

• MS SQL datetime precision problem

  - by Nailuj

  I have a situation where two persons might work on the same order (stored in an MS SQL database) from two different computers. To prevent data loss in the case where one would save his copy of the order first, and then a little later the second would save his copy and overwrite the first, I've added a check against the lastSaved field…

  Read the article

• How do calculators work with precision?

  - by zoul

  Hello! I wonder how calculators work with precision. For example the value of sin(M_PI) is not exactly zero when computed in double precision: #include <math.h> #include <stdio.h> int main() { double x = sin(M_PI); printf("%.20f\n", x); // 0.00000000000000012246 return 0; } Now I would certainly want to…

  Read the article

• Opening password protected Excel 2007 documents by double clicking from My documents does not work u

  - by erik-van-gorp

  When all of the following conditions are true, excel will open (most of the time) but will not open the document itself. No error is displayed. This only occurs with Excel files, Word and powerpoint do open perfectly. Conditions : OS is "Windows 7 Professional 64-bit" office is "Office 2007 Ultimate". excel file is in .xls…

  Read the article

• Determine precision and scale of particular number in Python

  - by jrdioko

  I have a variable in Python containing a floating point number (e.g. num = 24654.123), and I'd like to determine the number's precision and scale values (in the Oracle sense), so 123.45678 should give me (8,5), 12.76 should give me (4,2), etc. I was first thinking about using the string representation (via str or repr), but…

  Read the article

• Varying possible loss of precision

  - by szupie

  In Java, when you do int b; b = b + 1.0; You get a possible loss of precision error. But why is it that if you do int b; b += 1.0; There isn't any error?

  Read the article

• Varying behavior for possible loss of precision

  - by szupie

  In Java, when you do int b; b = b + 1.0; You get a possible loss of precision error. But why is it that if you do int b; b += 1.0; There isn't any error?

  Read the article

• C++ double division by 0.0 versus DBL_MIN

  - by wonsungi

  When finding the inverse square root of a double, is it better to clamp invalid non-positive inputs at 0.0 or MIN_DBL? (In my example below double b may end up being negative due to floating point rounding errors and because the laws of physics are slightly slightly fudged in the game.) Both division by 0.0 and MIN_DBL…

  Read the article

• Swap bits in c++ for a double

  - by hidayat

  Im trying to change from big endian to little endian on a double. One way to go is to use double val, tmp = 5.55; ((unsigned int *)&val)[0] = ntohl(((unsigned int *)&tmp)[1]); ((unsigned int *)&val)[1] = ntohl(((unsigned int *)&tmp)[0]); But then I get a warning: "dereferencing type-punned pointer…

  Read the article

• EF4 Code First Control Unicode and Decimal Precision, Scale with Attributes

  - by Dane Morgridge

  There are several attributes available when using code first with the Entity Framework 4 CTP5 Code First option.  When working with strings you can use [MaxLength(length)] to control the length and [Required] will work on all properties.  But there are a few things missing. By default all string will be…

  Read the article

• Dividing a double with integer

  - by hardcoder

  I am facing an issue while dividing a double with an int. Code snippet is : double db = 10; int fac = 100; double res = db / fac; The value of res is 0.10000000000000001 instead of 0.10. Does anyone know what is the reason for this? I am using cc to compile the code.

  Read the article

• Java string to double conversion.

  - by wretrOvian

  Hi, I've been reading up on the net about the issues with handling float and double types in java. Unfortunately, the image is still not clear. Hence, i'm asking here direct. :( My MySQL table has various DECIMAL(m,d) columns. The m may range from 5 to 30. d stays a constant at 2. Question 1. What equivalent…

  Read the article

• [C#] Convert string to double with 2 digit after decimal separator

  - by st.stoqnov

  All began with these simple lines of code: string s = "16.9"; double d = Convert.ToDouble(s); d*=100; The result should be 1690.0, but it's not. d is equal to 1689.9999999999998. All I want to do is to round a double to value with 2 digit after decimal separator. Here is my function. private double…

  Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >