Search Results

Search found 1806 results on 73 pages for 'numeric precision'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 1/73 | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • SQL SERVER – Precision of SMALLDATETIME – A 1 Minute Precision

    - by pinaldave
    I am myself surprised that I am writing this post today. I am going to present one of the very known facts of SQL Server SMALLDATETIME datatype. Even though this is a very well-known datatype, many a time, I have seen developers getting confused with precision of the SMALLDATETIME datatype. The precision of the datatype SMALLDATETIME is 1 minute. It discards the seconds by rounding up or rounding down any seconds greater than zero. Let us see the following example DECLARE @varSDate AS SMALLDATETIME SET @varSDate = '1900-01-01 12:12:01' SELECT @varSDate C_SDT SET @varSDate = '1900-01-01 12:12:29' SELECT @varSDate C_SDT SET @varSDate = '1900-01-01 12:12:30' SELECT @varSDate C_SDT SET @varSDate = '1900-01-01 12:12:59' SELECT @varSDate C_SDT Following is the result of the above script and note that any value between 0 (zero) and 59 is converted up or down. The part that confuses the developers is the value of the seconds in the display. I think if it is not maintained or recorded, it should not be displayed as well. Reference: Pinal Dave (http://blog.SQLAuthority.com) Filed under: Pinal Dave, SQL, SQL Authority, SQL DateTime, SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, T SQL, Technology

    Read the article

  • double precision in Ada?

    - by yCalleecharan
    Hi, I'm very new to Ada and was trying to see if it offers double precision type. I see that we have float and Put( Integer'Image( Float'digits ) ); on my machine gives a value of 6, which is not enough for numerical computations. Does Ada has double and long double types as in C? Thanks a lot...

    Read the article

  • Sql Exception: Error converting data type numeric to numeric

    - by Lucifer
    Hello We have a very strange issue with a database that has been moved from staging to production. The first time the database was moved it was by detaching, copying and reattaching, the second time we tried restoring from a backup of the staging. Both SQL Servers are the same version of MS SQL 2008, running on 64 bit hardware. The code accessing the database is the same build, built using the .net 2.0 framework. Here is the error message and some of the stack trace: Exception Details: System.Data.SqlClient.SqlException: Error converting data type numeric to numeric. Stack Trace: [SqlException (0x80131904): Error converting data type numeric to numeric.] System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection) +1953274 System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection) +4849707 System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj) +194 System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) +2392 System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString) +204 System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async) +954 System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) +162 System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe) +175 System.Data.SqlClient.SqlCommand.ExecuteNonQuery() +137 Version Information: Microsoft .NET Framework Version:2.0.50727.4200; ASP.NET Version:2.0.50727.4016

    Read the article

  • R Question. Numeric variable vs. Non-numeric and "names" function

    - by Michael
    > scores=cbind(UNCA.score, A.score, B.score, U.m.A, U.m.B) > names(scores)=c('UNCA.scores', 'A.scores', 'B.scores','UNCA.minus.A', 'UNCA.minus.B') > names(scores) [1] "UNCA.scores" "A.scores" "B.scores" "UNCA.minus.A" "UNCA.minus.B" > summary(UNCA.scores) X6.69230769230769 Min. : 4.154 1st Qu.: 7.333 Median : 8.308 Mean : 8.451 3rd Qu.: 9.538 Max. :12.000 > is.numeric(UNCA.scores) [1] FALSE > is.numeric(scores[,1]) [1] TRUE My question is, what is the difference between UNCA.scores and scores[,1]? UNCA.scores is the first column in the data.frame 'scores', but they are not the same thing, since one is numeric and the other isn't. If UNCA.scores is just a label here how can I make it be equivalent to 'scores[,1]? Thanks!

    Read the article

  • Required Working Precision for the BBP Algorithm?

    - by brainfsck
    Hello, I'm looking to compute the nth digit of Pi in a low-memory environment. As I don't have decimals available to me, this integer-only BBP algorithm in Python has been a great starting point. I only need to calculate one digit of Pi at a time. How can I determine the lowest I can set D, the "number of digits of working precision"? D=4 gives me many correct digits, but a few digits will be off by one. For example, computing digit 393 with precision of 4 gives me 0xafda, from which I extract the digit 0xa. However, the correct digit is 0xb. No matter how high I set D, it seems that testing a sufficient number of digits finds an one where the formula returns an incorrect value. I've tried upping the precision when the digit is "close" to another, e.g. 0x3fff or 0x1000, but cannot find any good definition of "close"; for instance, calculating at digit 9798 gives me 0xcde6 , which is not very close to 0xd000, but the correct digit is 0xd. Can anyone help me figure out how much working precision is needed to calculate a given digit using this algorithm? Thank you,

    Read the article

  • Find max integer size that a floating point type can handle without loss of precision

    - by Checkers
    Double has range more than a 64-bit integer, but its precision is less dues to its representation (since double is 64-bit as well, it can't fit more actual values). So, when representing larger integers, you start to lose precision in the integer part. #include <boost/cstdint.hpp> #include <limits> template<typename T, typename TFloat> void maxint_to_double() { T i = std::numeric_limits<T>::max(); TFloat d = i; std::cout << std::fixed << i << std::endl << d << std::endl; } int main() { maxint_to_double<int, double>(); maxint_to_double<boost::intmax_t, double>(); maxint_to_double<int, float>(); return 0; } This prints: 2147483647 2147483647.000000 9223372036854775807 9223372036854775800.000000 2147483647 2147483648.000000 Note how max int can fit into a double without loss of precision and boost::intmax_t (64-bit in this case) cannot. float can't even hold an int. Now, the question: is there a way in C++ to check if the entire range of a given integer type can fit into a loating point type without loss of precision? Preferably, it would be a compile-time check that can be used in a static assertion, and would not involve enumerating the constants the compiler should know or can compute.

    Read the article

  • BigDecimal precision not persisted with javax.persistence annotations

    - by dkaczynski
    I am using the javax.persistence API and Hibernate to create annotations and persist entities and their attributes in an Oracle 11g Express database. I have the following attribute in an entity: @Column(precision = 12, scale = 9) private BigDecimal weightedScore; The goal is to persist a decimal value with a maximum of 12 digits and a maximum of 9 of those digits to the right of the decimal place. After calculating the weightedScore, the result is 0.1234, but once I commit the entity with the Oracle database, the value displays as 0.12. I can see this by either by using an EntityManager object to query the entry or by viewing it directly in the Oracle Application Express (Apex) interface in a web browser. How should I annotate my BigDecimal attribute so that the precision is persisted correctly? Note: We use an in-memory HSQL database to run our unit tests, and it does not experience the issue with the lack of precision, with or without the @Column annotation.

    Read the article

  • negative precision values in ostream

    - by daz-fuller
    This is more of a question of curiosity but does anyone know how negative precision values are handled in C++? For example: double pi = 3.14159265; cout.precision(-10); cout.setf(ios::fixed, ios::floatfield); cout << pi << endl; I've tried this out and using GCC and it seems that the precision value is ignored but I was curious if there is some official line on what happens in this situation.

    Read the article

  • Delphi - Frac function losing precision.

    - by PeteDaMeat
    I have a TDateTime variable which is assigned a value at runtime of 40510.416667. When I extract the time to a TTime type variable using the Frac function, it sets it to 0.41666666666. Why has it changed the precision of the value and is there a workround to retain the precision from the original value ie. to set it to 0.416667.

    Read the article

  • x86-64 long double precision

    - by aaa
    hello. What is the actual precision of long double on Intel 64-bit platforms? is it 80 bits padded to 128 or actual 128 bit? if former, besides going gmp, is there another option to achieve true 128 precision?

    Read the article

  • About floating point precision and why do we still use it

    - by system_is_b0rken
    Floating point has always been troublesome for precision on large worlds. This article explains behind-the-scenes and offers the obvious alternative - fixed point numbers. Some facts are really impressive, like: "Well 64 bits of precision gets you to the furthest distance of Pluto from the Sun (7.4 billion km) with sub-micrometer precision. " Well sub-micrometer precision is more than any fps needs (for positions and even velocities), and it would enable you to build really big worlds. My question is, why do we still use floating point if fixed point has such advantages? Most rendering APIs and physics libraries use floating point (and suffer it's disadvantages, so developers need to get around them). Are they so much slower? Additionally, how do you think scalable planetary engines like outerra or infinity handle the large scale? Do they use fixed point for positions or do they have some space dividing algorithm?

    Read the article

  • C++ floating point precision

    - by Davinel
    double a = 0.3; std::cout.precision(20); std::cout << a << std::endl; result: 0.2999999999999999889 double a, b; a = 0.3; b = 0; for (char i = 1; i <= 50; i++) { b = b + a; }; std::cout.precision(20); std::cout << b << std::endl; result: 15.000000000000014211 So.. 'a' is smaller than it should be. But if we take 'a' 50 times - result will be bigger than it should be. Why is this? And how to get correct result in this case?

    Read the article

  • Setting minimum number of decimal places for std::ostream precision

    - by Phil Boltt
    Hi, Is there a way to set the "minimum" number of decimal places that a std::ostream will output? For example, say I have two doubles that I want to print: double a = 0; double b = 0.123456789; I can set my maximum decimal precision so that I output b exactly std::cout << std::setprecision(9) << b << std::endl; Is there a way to set "minimum" precision so that std::cout << a << std::endl; yields "0.0", not just "0"? Thanks! Phil

    Read the article

  • Loss of precision - int -> float or double

    - by stan
    I have an exam question i am revising for and the question is for 4 marks "In java we can assign a int to a double or a float". Will this ever loose infromation and why? I have put that because ints are normally of fixed length or size - the precision for sotring data is finite, where storing information in floating point can be infinite, essentially we loose infromation because of this Now i am a little sketchy as to whetehr or not i am hitting the right areas here. I very sure it will loose precision but i cant exactly put my finger on why. Can i getsome help please Thanks

    Read the article

  • Numerical precision of double type in Visual C++ 2008 Express debugger

    - by damik
    I'm using Visual C++ 2008 Express Edition and when i debug code: double x = 0.2; I see in debugging tooltip on x 0.20000000000000001 but: typedef numeric_limits< double > double_limit; int a = double_limit::digits10 gives me: a = 15 Why results in debugger are longer than maybe ? What is this strange precision based on ? My CPU is Intel Core 2 Duo T7100

    Read the article

  • MS Word Macro - Numeric field insertion with automatic calculation at end of page

    - by Will
    Hi, I am trying to duplicate a feature that exists in Multimate (Ashton Tate) word processor. Yes, the one that hasnt been supported for 20 years! If I can duplicate this one feature I can get all the users off MM and onto Word. The documents they create are billing documents. they consist of a descriptive paragraph of any length on the left side of the page, and a billing amount at the end of the paragraph over on the right hand side, like this (excuse the imperfect formatting).... +-----------------whole page--------------------+ |                                                                    | |    pppp-para 1-pppppppppp                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    pppppppppppppppppppp      $$$$$  | |                                                                    |  |                                                                    |  |    pppp-para 2-pppppppppp                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    p                                         p                   | |    pppppppppppppppppppp      $$$$$  | |                                                                    |  |                                                                    |  |                             etc                                  | +-----------------------------------------------------+ some of these bills can be a few hundred pages and have a dozen or so paragraphs on each page, which is why none of the users will leave MM until this efficient little feature can be duplicated. The thing that MM does really easily is that there is a function key that they can press at any time that will - - jump the cursor from the paragraph they are writing over to the right hand side - create a numeric field - allow them to enter a number into the numeric field - return them to the left hand side to start a new paragraph What MM also does is automatically total the numeric fields on each page and create a subtotal in the page footer. it also creates a total for the entire document and puts this in the footer of the last page. I would like to duplicate this feature in word with a macro, but have no idea where to start. Any suggestions or code would be great, thanks, will.

    Read the article

  • MS SQL datetime precision problem

    - by Nailuj
    I have a situation where two persons might work on the same order (stored in an MS SQL database) from two different computers. To prevent data loss in the case where one would save his copy of the order first, and then a little later the second would save his copy and overwrite the first, I've added a check against the lastSaved field (datetime) before saving. The code looks roughly like this: private bool orderIsChangedByOtherUser(Order localOrderCopy) { // Look up fresh version of the order from the DB Order databaseOrder = orderService.GetByOrderId(localOrderCopy.Id); if (databaseOrder != null && databaseOrder.LastSaved > localOrderCopy.LastSaved) { return true; } else { return false; } } This works for most of the time, but I have found one small bug. If orderIsChangedByOtherUser returns false, the local copy will have its lastSaved updated to the current time and then be persisted to the database. The value of lastSaved in the local copy and the DB should now be the same. However, if orderIsChangedByOtherUser is run again, it sometimes returns true even though no other user has made changes to the DB. When debugging in Visual Studio, databaseOrder.LastSaved and localOrderCopy.LastSaved appear to have the same value, but when looking closer they some times differ by a few milliseconds. I found this article with a short notice on the millisecond precision for datetime in SQL: Another problem is that SQL Server stores DATETIME with a precision of 3.33 milliseconds (0. 00333 seconds). The solution I could think of for this problem, is to compare the two datetimes and consider them equal if they differ by less than say 10 milliseconds. My question to you is then: are there any better/safer ways to compare two datetime values in MS SQL to see if they are exactly the same?

    Read the article

  • Precision error on matrix multiplication

    - by Wam
    Hello all, Coding a matrix multiplication in my program, I get precision errors (inaccurate results for large matrices). Here's my code. The current object has data stored in a flattened array, row after row. Other matrix B has data stored in a flattened array, column after column (so I can use pointer arithmetic). protected double[,] multiply (IMatrix B) { int columns = B.columns; int rows = Rows; int size = Columns; double[,] result = new double[rows,columns]; for (int row = 0; row < rows; row++) { for (int col = 0; col < columns; col++) { unsafe { fixed (float* ptrThis = data) fixed (float* ptrB = B.Data) { float* mePtr = ptrThis + row*rows; float* bPtr = ptrB + col*columns; double value = 0.0; for (int i = 0; i < size; i++) { value += *(mePtr++) * *(bPtr++); } result[row, col] = value; } } } } } Actually, the code is a bit more complicated : I do the multiply thing for several chunks (so instead of having i from 0 to size, I go from localStart to localStop), then sum up the resulting matrices. My problem : for a big matrix I get precision error : NUnit.Framework.AssertionException: Error at (0,1) expected: <6.4209571409444209E+18> but was: <6.4207619776304906E+18> Any idea ?

    Read the article

  • Binary files printing and desired precision

    - by yCalleecharan
    Hi, I'm printing a variable say z1 which is a 1-D array containing floating point numbers to a text file so that I can import into Matlab or GNUPlot for plotting. I've heard that binary files (.dat) are smaller than .txt files. The definition that I currently use for printing to a .txt file is: void create_out_file(const char *file_name, const long double *z1, size_t z_size){ FILE *out; size_t i; if((out = _fsopen(file_name, "w+", _SH_DENYWR)) == NULL){ fprintf(stderr, "***> Open error on output file %s", file_name); exit(-1); } for(i = 0; i < z_size; i++) fprintf(out, "%.16Le\n", z1[i]); fclose(out); } I have three questions: Are binary files really more compact than text files?; If yes, I would like to know how to modify the above code so that I can print the values of the array z1 to a binary file. I've read that fprintf has to be replaced with fwrite. My output file say dodo.dat should contain the values of array z1 with one floating number per line. I have %.16Le up in my code but I think that %.15Le is right as I have 15 precision digits with long double. I have put a dot (.) in the width position as I believe that this allows expansion to an arbitrary field to hold the desired number. Am I right? As an example with %.16Le, I can have an output like 1.0047914240730432e-002 which gives me 16 precision digits and the width of the field has the right width to display the number correctly. Is placing a dot (.) in the width position instead of a width value a good practice? Thanks a lot...

    Read the article

  • How do calculators work with precision?

    - by zoul
    Hello! I wonder how calculators work with precision. For example the value of sin(M_PI) is not exactly zero when computed in double precision: #include <math.h> #include <stdio.h> int main() { double x = sin(M_PI); printf("%.20f\n", x); // 0.00000000000000012246 return 0; } Now I would certainly want to print zero when user enters sin(p). I can easily round somewhere on 1e–15 to make this particular case work, but that’s a hack, not a solution. When I start to round like this and the user enters something like 1e–20, they get a zero back (because of the rounding). The same thing happens when the user enters 1/10 and hits the = key repeatedly — when he reaches the rounding treshold, he gets zero. And yet some calculators return plain zero for sin(p) and at the same time they can work with expressions such as (1e–20)/10 comfortably. Where’s the trick?

    Read the article

  • C# (4): double minus double giving precision problems

    - by thermal7
    I have come across a precision issue with double in .NET I thought this only applied to floats but now I see that double is a float. double test = 278.97 - 90.46; Debug.WriteLine(test) //188.51000000000005 //correct answer is 188.51 What is the correct way to handle this? Round? Lop off the unneeded decimal places?

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >