Search Results

Search found 680 results on 28 pages for 'precision'.

Page 5/28 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • warning: '0' flag ignored with precision and ‘%i’ gnu_printf format

    - by morpheous
    I am getting the following warning when compiling some legacy C code on Ubuntu Karmic, using gcc 4.4.1 The warning is: src/filename.c:385: warning: '0' flag ignored with precision and ‘%i’ gnu_printf format The snippet which causes the warning to be emitted is: char buffer[256] ; long fnum ; /* some initialization code here ... */ sprintf(buffer, "F%03.3i.DTA", (int)fnum); /* <- warning emitted here */ I think I understand the warning, but I would like to check in here to see if I am right, and also the (definite) correct way of resolving this.

    Read the article

  • Storing high precision latitude/longitude numbers in iOS Core Data

    - by Bryan
    I'm trying to store Latitude/Longitudes in core data. These end up being anywhere from 6-20 digit precision. And for whatever reason, i had them as floats in Core Data, its rounding them and not giving me the exact values back. I tried "decimal" type, with no luck either. Are NSStrings my only other option? EDIT NSManagedObject: @interface Event : NSManagedObject { } @property (nonatomic, retain) NSDecimalNumber * dec; @property (nonatomic, retain) NSDate * timeStamp; @property (nonatomic, retain) NSNumber * flo; @property (nonatomic, retain) NSNumber * doub; Here's the code for a sample number that I store into core data: NSNumber *n = [NSDecimalNumber decimalNumberWithString:@"-97.12345678901234567890123456789"]; Code to access it again: NSNumber *n = [managedObject valueForKey:@"dec"]; NSNumber *f = [managedObject valueForKey:@"flo"]; NSNumber *d = [managedObject valueForKey:@"doub"]; Printed values: Printing description of n: -97.1234567890124 Printing description of f: <CFNumber 0x603f250 [0xfef3e0]>{value = -97.12345678901235146441, type = kCFNumberFloat64Type} Printing description of d: <CFNumber 0x6040310 [0xfef3e0]>{value = -97.12345678901235146441, type = kCFNumberFloat64Type}

    Read the article

  • Multi-Precision Arithmetic on MIPS

    - by Rob
    Hi, I am just trying to implement multi-precision arithmetic on native MIPS. Assume that one 64-bit integer is in register $12 and $13 and another is in registers $14 and $15. The sum is to be placed in registers $10 and $11. The most significant word of the 64-bit integer is found in the even-numbered registers, and the least significant word is found in the odd-numbered registers. On the internet, it said, this is the shortest possible implementation. addu $11, $13, $15 # add least significant word sltu $10, $11, $15 # set carry-in bit addu $10, $10, $12 # add in first most significant word addu $10, $10, $14 # add in second most significant word I just wanna double check that I understand correctly. The sltu checks if the sum of the two least significant words is smaller or equal than one of the operands. If this is the case, than did a carry occur, is this right? To check if there occured a carry when adding the two most significant words and store the result in $9 I have to do: sltu $9, $10, $12 # set carry-in bit Does this make any sense?

    Read the article

  • Retain numerical precision in an R data frame?

    - by David
    When I create a dataframe from numeric vectors, R seems to truncate the value below the precision that I require in my analysis: data.frame(x=0.99999996) returns 1 (see update 1) I am stuck when fitting spline(x,y) and two of the x values are set to 1 due to rounding while y changes. I could hack around this but I would prefer to use a standard solution if available. example Here is an example data set d <- data.frame(x = c(0.668732936336141, 0.95351462456867, 0.994620622127435, 0.999602102672081, 0.999987126195509, 0.999999955814133, 0.999999999999966), y = c(38.3026509783688, 11.5895099585560, 10.0443344234229, 9.86152339768516, 9.84461434575695, 9.81648333804257, 9.83306725758297)) The following solution works, but I would prefer something that is less subjective: plot(d$x, d$y, ylim=c(0,50)) lines(spline(d$x, d$y),col='grey') #bad fit lines(spline(d[-c(4:6),]$x, d[-c(4:6),]$y),col='red') #reasonable fit Update 1 Since posting this question, I realize that this will return 1 even though the data frame still contains the original value, e.g. > dput(data.frame(x=0.99999999996)) returns structure(list(x = 0.99999999996), .Names = "x", row.names = c(NA, -1L), class = "data.frame") Update 2 After using dput to post this example data set, and some pointers from Dirk, I can see that the problem is not in the truncation of the x values but the limits of the numerical errors in the model that I have used to calculate y. This justifies dropping a few of the equivalent data points (as in the example red line).

    Read the article

  • Random Loss of precision in Python ReadLine()

    - by jackyouldon
    Hi all, We have a process which takes a very large csv (1.6GB) and breaks it down into pieces (in this case 3). This runs nightly and normally doesn't give us any problems. When it ran last night, however, the first of the output files had lost precision on the numeric fields in the data. The active ingredient in the script are the lines: while lineCounter <= chunk: oOutFile.write(oInFile.readline()) lineCounter = lineCounter + 1 and the normal output might be something like StringField1; StringField2; StringField3; StringField4; 1000000; StringField5; 0.000054454 etc. On this one occasion and in this one output file the numeric fields were all output with 6 zeros at the end i.e. StringField1; StringField2; StringField3; StringField4; 1000000.000000; StringField5; 0.000000 We are using Python v2.6 (and don't want to upgrade unless we really have to) but we can't afford to lose this data. Does anyone have any idea why this might have happened? If the readline is doing some kind of implicit conversion is there a way to do a binary read, because we really just want this data to pass through untouched? It is very wierd to us that this only affected one of the output files generated by the same script, and when it was rerun the output was as expected. thanks Jack

    Read the article

  • When must arbitrary precision arithmetic functions be used in PHP?

    - by Tjorriemorrie
    My colleague uses the Binary Calculator functions in bandwidth calculations; as much as terrabytes, and with percentage splitting on allocation. His usage of these functions appears correct in order not to lose a byte; although he seems to be using them now for everything. The manual only says: For arbitrary precision mathematics PHP offers the Binary Calculator which supports numbers of any size and precision, represented as strings. How much is any size? Is it really necessary? How big is the default float in PHP? Are there any good advice regarding this or things to keep in mind?

    Read the article

  • Read half precision float (float16 IEEE 754r) binary data in matlab

    - by Michael
    you have been a great help last time, i hope you can give me some advise this time, too. I read a binary file into matlab with bit16 (format = bitn) and i get a string of ones and zeros. bin = '1 00011 1111111111' (16 bits: 1. sign, 2-6. exponent, 7-16. mantissa) According to ftp://www.fox-toolkit.org/pub/fasthalffloatconversion.pdf it can be 'converted' like out = (-1)^bin(1) * 2^(bin(2:6)-15) * 1.bin(7:16) [are exponent and mantissa still binary?] Can someone help me out and tell me how to deal with the 'eeeee' and '1.mmmmmmmmmm' as mentioned in the pdf, please. Thanks a lot! Michael

    Read the article

  • How to compare two times in milliseconds precision?

    - by Marcos Issler
    I have a subtitle text file that works with standart srt format 00:00:00,000 Hour, minutes, seconds, milliseconds. I want to create a timer to update the subtitle screen and check the current time to know what subtitle show on screen. Which is the best to use? NSTimeInterval, NSDate? I think the best is to convert all to times to milliseconds number and compare. But NSTimeInterval works with seconds, not milliseconds. Some clue? Marcos

    Read the article

  • Multiplication algorithm for abritrary precision (bignum) integers.

    - by nn
    Hi, I'm writing a small bignum library for a homework project. I am to implement Karatsuba multiplication, but before that I would like to write a naive multiplication routine. I'm following a guide written by Paul Zimmerman titled "Modern Computer Arithmetic" which is freely available online. On page 4, there is a description of an algorithm titled BasecaseMultiply which performs gradeschool multiplication. I understand step 2, 3, where B^j is a digit shift of 1, j times. But I don't understand step 1 and 3, where we have A*b_j. How is this multiplication meant to be carried out if the bignum multiplication hasn't been defined yet? Would the operation "*" in this algorithm just be the repeated addition method? Here is the parts I have written thus far. I have unit tested them so they appear to be correct for the most part: The structure I use for my bignum is as follows: #define BIGNUM_DIGITS 2048 typedef uint32_t u_hw; // halfword typedef uint64_t u_w; // word typedef struct { unsigned int sign; // 0 or 1 unsigned int n_digits; u_hw digits[BIGNUM_DIGITS]; } bn; Currently available routines: bn *bn_add(bn *a, bn *b); // returns a+b as a newly allocated bn void bn_lshift(bn *b, int d); // shifts d digits to the left, retains sign int bn_cmp(bn *a, bn *b); // returns 1 if a>b, 0 if a=b, -1 if a<b

    Read the article

  • matlab precision determint problem

    - by ldigas
    I have the following program format compact; format short g; clear; clc; L = 140; J = 77; Jm = 10540; G = 0.8*10^8; d = L/3; for i=1:500000 omegan=1.+0.0001*i; a(1,1) = ((omegan^2)*(Jm/(G*J))*d^2)-2; a(1,2) = 2; a(1,3) = 0; a(1,4) = 0; a(2,1) = 1; a(2,2) = ((omegan^2)*(Jm/(G*J))*d^2)-2; a(2,3) = 1; a(2,4) = 0; a(3,1) = 0; a(3,2) = 1; a(3,3) = ((omegan^2)*(Jm/(G*J))*d^2)-2; a(3,4) = 1; a(4,1) = 0; a(4,2) = 0; a(4,3) = 2; a(4,4) = ((omegan^2)*(Jm/(G*J))*d^2)-2; if(abs(det(a))<1E-10) sprintf('omegan= %8.3f det= %8.3f',omegan,det(a)) end end Analytical solution of the above system, and the same program written in fortran gives out values of omegan equal to 16.3818 and 32.7636 (fortran values; analytical differ a little, but they're there somewhere). So, now I'm wondering ... where am I going wrong with this ? Why is matlab not giving the expected results ? (this is probably something terribly simple, but it's giving me headaches)

    Read the article

  • matlab precision determinant problem

    - by ldigas
    I have the following program format compact; format short g; clear; clc; L = 140; J = 77; Jm = 10540; G = 0.8*10^8; d = L/3; for i=1:500000 omegan=1.+0.0001*i; a(1,1) = ((omegan^2)*(Jm/(G*J))*d^2)-2; a(1,2) = 2; a(1,3) = 0; a(1,4) = 0; a(2,1) = 1; a(2,2) = ((omegan^2)*(Jm/(G*J))*d^2)-2; a(2,3) = 1; a(2,4) = 0; a(3,1) = 0; a(3,2) = 1; a(3,3) = ((omegan^2)*(Jm/(G*J))*d^2)-2; a(3,4) = 1; a(4,1) = 0; a(4,2) = 0; a(4,3) = 2; a(4,4) = ((omegan^2)*(Jm/(G*J))*d^2)-2; if(abs(det(a))<1E-10) sprintf('omegan= %8.3f det= %8.3f',omegan,det(a)) end end Analytical solution of the above system, and the same program written in fortran gives out values of omegan equal to 16.3818 and 32.7636 (fortran values; analytical differ a little, but they're there somewhere). So, now I'm wondering ... where am I going wrong with this ? Why is matlab not giving the expected results ? (this is probably something terribly simple, but it's giving me headaches)

    Read the article

  • Compare double till precision 2

    - by Ram
    Hi, I want to compare two double values till the second digit from decimal point. I am using following statement but I think it is not working properly. dbl1.ToString("g2", CultureInfo.InvariantCulture).Equals( dbl2.ToString("g2", CultureInfo.InvariantCulture)) Am I missing anything here?

    Read the article

  • how to increase the precision in Locate Me app

    - by thndrkiss
    Hi, I just installed the locate me app and started to understand the CLLocation stuff. I need to know how to increase the number of digits after the lat and long numbers displayed. Can any one help me. currently it shows 17.5968,82.8486. I need more number of digits after decimal(6 digits after decimal.

    Read the article

  • floating point precision in ruby on rails model validations

    - by Chris Allison
    Hello I am trying to validate a dollar amount using a regex: ^[0-9]+\.[0-9]{2}$ This works fine, but whenever a user submits the form and the dollar amount ends in 0(zero), ruby(or rails?) chops the 0 off. So 500.00 turns into 500.0 thus failing the regex validation. Is there any way to make ruby/rails keep the format entered by the user, regardless of trailing zeros?

    Read the article

  • Qwt setAxisScale() seems to lock up application when given numbers with greater precision than 2e-07

    - by Dane Larsen
    I'm using Qwt for some scientific graphing, and I'm working with some fairly small numbers, the smallest being around 1.0e-22. I'm trying to call setAxisScale(xaxis, xmin, xmax) //xmin = 0, xmax = 2.0e-10 But when I do, the application locks up. I haven't found anything in the documentation that refers to a minimum value. Xmin and xmax are both doubles, so that shouldn't be a problem. Is this a bug in Qwt, or am I doing something wrong? Thanks in advance

    Read the article

  • Sorting a vector of (double precision) reals and obtain their order

    - by Philipp
    Hello everyone, in C++ would like to sort a lengthy (2^20) vector of reals, obviously sort() does the trick. Having used R before I was used to the nice order() function which yields the permutation that leads to the sorted vector. Probably someone has done this in C++, maybe it's just my weak google-Fu that prevents me from finding it. And yeah, obivously my C++ newbness could stop me from spotting something straightforward. Example: x = {24, 55, 22, 1} then the permutation perm = {3, 2, 0, 1} maps the original x to the sorted x in ascending order. I can probably implement some bubble sort which does not only sort x but performs the same transpositions on the vector {0,1,2,...} and outputs both, but I believe someone must have thought about it and especially have done it efficiently. Thank you very much, Philipp

    Read the article

  • Floating point precision and physics calculations

    - by Vee
    The gravity Vector2 in my physics world is (0; 0.1). The number 0.1 is known to be problematic, since "it cannot be represented exactly, but is approximately 1.10011001100110011001101 × 2-4". Having this value for the gravity gives me problems with collisions and creates quite nasty bugs. Changing the value to 0.11 solves these problems. Is there a more elegant solution that doesn't require changing the value at all?

    Read the article

  • DateTime Comparison Precision

    - by mnh
    I'm doing DateTime comparison but I don't want to do comparison at second, millisecond and ticks level. What's the most elegant way? If I simply compare the DateTime, then they are seldom equal due to ticks differences.

    Read the article

  • Extend precision of MySQL's double datatype?

    - by tim82
    I am trying to save the value "6.714285714285714" into a DOUBLE datatype field. Unfortunately it does not fit at all and is cutted by one char. Storing larger numbers becomes less precise. Already searched in the mysql manual and it seems to be that double is the most precise data type available. Anyone knows a practicable workaround? Sorry for my bad english and thx a lot!

    Read the article

  • How to maintain decimal precision in calculations

    - by Blankman
    I need to sum 2 decimal values together, then divide by 2 and convert to string. My calculation currently is trimming to 2 decimal places, but I want to keep as many decimals as I can. city.Latitude = ( (lat.North + lat.South) / 2 ).ToString(); the values for lat.North and lat.Souch are like: 55.32342322

    Read the article

  • Losing DateTimeOffset precision when using C#

    - by Darvis Lombardo
    I have a SQL Server table with a CreatedDate field of type DateTimeOffset(2). A sample value which is in the table is 2010-03-01 15:18:58.57 -05:00 As an example, from within C# I retrieve this value like so: var cmd = new SqlCommand("SELECT CreatedDate FROM Entities WHERE EntityID = 2", cn); var da = new SqlDataAdapter(cmd); DataTable dt =new DataTable(); da.Fill(dt); And I look at the value: MessageBox.Show(dt.Rows[0][0].ToString()); The result is 2010-03-01 15:18:58 -05:00, which is missing the .57 that is stored in the database. If I look at dt.Rows[0][0] in the Watch window, I also do not see the .57, so it appears it has been truncated. Can someone shed some light on this? I need to use the date to match up with other records in the database and the .57 is needed. Thanks! Darvis

    Read the article

  • High precision event timer

    - by rahul jv
    #include "target.h" #include "xcp.h" #include "LocatedVars.h" #include "osek.h" /** * This task is activated every 10ms. */ long OSTICKDURATION; TASK( Task10ms ) { void XCP_FN_TYPE Xcp_CmdProcessor( void ); uint32 startTime = GetQueryPerformanceCounter(); /* Trigger DAQ for the 10ms XCP raster. */ if( XCPEVENT_DAQ_OVERLOAD & Xcp_DoDaqForEvent_10msRstr() ) { ++numDaqOverload10ms; } /* Update those variables which are modified every 10ms. */ counter16 += slope16; /* Trigger STIM for the 10ms XCP raster. */ if( enableBypass10ms ) { if( XCPEVENT_MISSING_DTO & Xcp_DoStimForEvent_10msRstr() ) { ++numMissingDto10ms; } } duration10ms = (uint32)( ( GetQueryPerformanceCounter() - startTime ) / STOPWATCH_TICKS_PER_US ); } What would be the easiest (and/or best) way to synchronise to some accurate clock to call a function at a specific time interval, with little jitter during normal circumstances, from C++? I am working on WINDOWS operating system now. The above code is for RTAS OSEK but I want to call a function at a specific time interval for windows operating system. Could anyone assist me in c++ language ??

    Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >