Search Results

Search found 1806 results on 73 pages for 'numeric precision'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 6/73 | < Previous Page | 2 3 4 5 6 7 8 9 10 11 12 13  | Next Page >

  • c# control location precision

    - by AC
    I need more precision then integer based locations when puttng controls on a form. It seems control.location only supports Point. Is there a work around for this? Point p = new Point(100, 200); this.Location = p;// this works of course PointF pF = new PointF(100.04f, 200.08f); this.Location = pF;// this does not work of course because Location expects a Point not PointF Is there some setting on the base form, or base control I can set to have more location precision?

    Read the article

  • Compare Date objects with different levels of precision

    - by brainimus
    I have a JUnit test that fails because the milliseconds are different. In this case I don't care about the milliseconds. How can I change the precision of the assert to ignore milliseconds (or any precision I would like it set to)? Example of a failing assert that I would like to pass: Date dateOne = new Date(); dateOne.setTime(61202516585000L); Date dateTwo = new Date(); dateTwo.setTime(61202516585123L); assertEquals(dateOne, dateTwo);

    Read the article

  • Coldfusion Components:- The argument passed to the function is not of type numeric

    - by salim.vali
    Hi I have a simple form form.cfm:- <cfset Registr = createObject("component", "forms.Registr") /> <cfset setFoo = createObject('component','forms.Registr).putNUsr(username,password,rating) /> <form name="regFrm" action="#cgi.script_name#" method="post" onsubmit="submitform();" > <tr><td>Username:</td> <td><input type="text" name=" Username" value="#form. Username#" ></td></tr> <tr><td>Password:</td> <td><input class="" type="password" name="password" value="#form.password#" ></td></tr> <tr><td>Rate:</td> <select name="rating" > <option value="" ></option> <cfloop query="qGetReting"> <option value="#rating_id#" <cfif form. rating eq prof_id>selected</cfif> >#rating#</option> </cfloop> </select> </td> </tr> </form> Now there is this cfc called Registr.cfc in the "forms" folder which has an insert-function called 'putNUsr'the code for 'Registr.cfc' is as follows. <cfcomponent> <cffunction name="putNUsr" returntype="void" displayname="" output="no"> <cfargument name="password" type="string" required="true"> <cfargument name="rating" type="numeric" required="true"> <cfargument name="username" type="string" required="true"> <cfquery datasource="#application.xyz#" name="q_putNUsr"> insert into users (username , password , rating) values( <cfqueryparam value="#arguments. username#" cfsqltype="CF_SQL_VARCHAR" />, <cfqueryparam value="#arguments.password#" cfsqltype="CF_SQL_VARCHAR" />, <cfqueryparam value="#arguments.rating#" cfsqltype="CF_SQL_INTEGER" ) </cfquery> </cffunction> </cfcomponent> I am able to populate the DB with the data if I do not use the form field "rating" which is numeric. Else i am getting the error as follows:- The RATING argument passed to the putNUsr function is not of type numeric. If the component name is specified as a type of this argument, it is possible that either a definition file for the component cannot be found or is not accessible. KINDLY HELP -S Vali

    Read the article

  • Convert varchar to numeric in Informix

    - by user302705
    Hi All I have problem while converting varchar type to Int type in Informix. Actually I dont know if the value is really varchar or not which I want to convert to INT. Its a sandbox system. As Example: I am trying to run this kind of Select telnumber from numbers n where Cast(n.telnumber AS INT) between 1234 and 9999 I got this error: "Character to numeric conversion error" If I run this query like this: Select telnumber from numbers n where n.telnumber between '1234' and '9999' it brings a resultset but not in the range that I defined. 130987 130710 130723 How can I convert telnumber to a numeric value and use it in "between" 1234 and 9999 range Thanks in advance.

    Read the article

  • AbsoluteTime with numeric argument behaves strangely.

    - by dreeves
    This is strange: DateList@AbsoluteTime[596523] returns {2078, 7, 2, 2, 42, 9.7849} But DateList@AbsoluteTime[596524] returns {1942, 5, 26, 20, 28, 39.5596} The question: What's going on? Note that AbsoluteTime with a numeric argument is undocumented. (I think I now know what it's doing but figured this is useful to have as a StackOverflow question for future reference; and I'm curious if there's some reason for that magic 596523 number.) PS: I encountered this when writing these utility functions for converting to and from unix time in Mathematica: (* Using Unix time (an integer) instead of Mathematica's AbsoluteTime... *) tm[x___] := AbsoluteTime[x] (* tm is an alias for AbsoluteTime. *) uepoch = tm[{1970}, TimeZone->0]; (* unixtm works analogously to tm. *) unixtm[x___] := Round[tm[x]-uepoch] (* tm & unixtm convert between unix & *) unixtm[x_?NumericQ] := Round[x-uepoch] (* mma epoch time when given numeric *) tm[t_?NumericQ] := t+uepoch (* args. Ie, they're inverses. *)

    Read the article

  • Numeric literals in sql server 2008

    - by costa
    What is the type that sql server assigns to the numeric literal: 2. , i.e. 2 followed by a dot? I was curious because: select convert(varchar(50), 2.) union all select convert(varchar(50), 2.0) returns: 2 2.0 which made me ask what's the difference between 2. and 2.0 type wise? Sql server seems to assign types to numeric literals depending on the number itself by finding the minimal storage type that can hold the number. A value of 1222333 is stored as int while 1152921504606846975 is stored as big int. thanks

    Read the article

  • c++ floating point precision loss: 3015/0.00025298219406977296

    - by SigTerm
    The problem. Microsoft Visual C++ 2005 compiler, 32bit windows xp sp3, amd 64 x2 cpu. Code: double a = 3015.0; double b = 0.00025298219406977296; //*((unsigned __int64*)(&a)) == 0x40a78e0000000000 //*((unsigned __int64*)(&b)) == 0x3f30945640000000 double f = a/b;//3015/0.00025298219406977296; the result of calculation (i.e. "f") is 11917835.000000000 (*((unsigned __int64*)(&f)) == 0x4166bb4160000000) although it should be 11917834.814763514 (i.e. *((unsigned __int64*)(&f)) == 0x4166bb415a128aef). I.e. fractional part is lost. Unfortunately, I need fractional part to be correct. Questions: 1) Why does this happen? 2) How can I fix the problem? Additional info: 0) The result is taken directly from "watch" window (it wasn't printed, and I didn't forget to set printing precision). I also provided hex dump of floating point variable, so I'm absolutely sure about calculation result. 1) The disassembly of f = a/b is: fld qword ptr [a] fdiv qword ptr [b] fstp qword ptr [f] 2) f = 3015/0.00025298219406977296; yields correct result (f == 11917834.814763514 , *((unsigned __int64*)(&f)) == 0x4166bb415a128aef ), but it looks like in this case result is simply calculated during compile-time: fld qword ptr [__real@4166bb415a128aef (828EA0h)] fstp qword ptr [f] So, how can I fix this problem? P.S. I've found a temporary workaround (i need only fractional part of division, so I simply use f = fmod(a/b)/b at the moment), but I still would like to know how to fix this problem properly - double precision is supposed to be 16 decimal digits, so such calculation isn't supposed to cause problems.

    Read the article

  • Numeric Data Entry in WPF

    - by Matt Hamilton
    How are you handling the entry of numeric values in WPF applications? Without a NumericUpDown control, I've been using a TextBox and handling its PreviewKeyDown event with the code below, but it's pretty ugly. Has anyone found a more graceful way to get numeric data from the user without relying on a third-party control?private void NumericEditPreviewKeyDown(object sender, KeyEventArgs e) { bool isNumPadNumeric = (e.Key >= Key.NumPad0 && e.Key <= Key.NumPad9) || e.Key == Key.Decimal; bool isNumeric = (e.Key >= Key.D0 && e.Key <= Key.D9) || e.Key == Key.OemPeriod; if ((isNumeric || isNumPadNumeric) && Keyboard.Modifiers != ModifierKeys.None) { e.Handled = true; return; } bool isControl = ((Keyboard.Modifiers != ModifierKeys.None && Keyboard.Modifiers != ModifierKeys.Shift) || e.Key == Key.Back || e.Key == Key.Delete || e.Key == Key.Insert || e.Key == Key.Down || e.Key == Key.Left || e.Key == Key.Right || e.Key == Key.Up || e.Key == Key.Tab || e.Key == Key.PageDown || e.Key == Key.PageUp || e.Key == Key.Enter || e.Key == Key.Return || e.Key == Key.Escape || e.Key == Key.Home || e.Key == Key.End); e.Handled = !isControl && !isNumeric && !isNumPadNumeric; }

    Read the article

  • How to parse kanji numeric characters using ICU?

    - by Aki
    I'm writing a function using ICU to parse an Unicode string which consists of kanji numeric character(s) and want to return the integer value of the string. "?" = 5 "???" = 31 "???????" = 5972 I'm setting the locale to Locale::getJapan() and using the NumberFormat::parse() to parse the character string. However, whenever I pass it any Kanji characters, the parse() method is returning U_INVALID_FORMAT_ERROR. Does anyone know if ICU supports Kanji character strings in the NumberFormat::parse() method? I was hoping that since I'm setting the Locale to Japanese that it would be able to parse Kanji numeric values. Thanks! #include <iostream> #include <unicode/numfmt.h> using namespace std; int main(int argc, char **argv) { const Locale &jaLocale = Locale::getJapan(); UErrorCode status = U_ZERO_ERROR; NumberFormat *nf = NumberFormat::createInstance(jaLocale, status); UChar number[] = {0x4E94}; // Character for '5' in Japanese '?' UnicodeString numStr(number); Formattable formattable; nf->parse(numStr, formattable, status); if (U_FAILURE(status)) { cout << "error parsing as number: " << u_errorName(status) << endl; return(1); } cout << "long value: " << formattable.getLong() << endl; }

    Read the article

  • Random Loss of precision in Python ReadLine()

    - by jackyouldon
    Hi all, We have a process which takes a very large csv (1.6GB) and breaks it down into pieces (in this case 3). This runs nightly and normally doesn't give us any problems. When it ran last night, however, the first of the output files had lost precision on the numeric fields in the data. The active ingredient in the script are the lines: while lineCounter <= chunk: oOutFile.write(oInFile.readline()) lineCounter = lineCounter + 1 and the normal output might be something like StringField1; StringField2; StringField3; StringField4; 1000000; StringField5; 0.000054454 etc. On this one occasion and in this one output file the numeric fields were all output with 6 zeros at the end i.e. StringField1; StringField2; StringField3; StringField4; 1000000.000000; StringField5; 0.000000 We are using Python v2.6 (and don't want to upgrade unless we really have to) but we can't afford to lose this data. Does anyone have any idea why this might have happened? If the readline is doing some kind of implicit conversion is there a way to do a binary read, because we really just want this data to pass through untouched? It is very wierd to us that this only affected one of the output files generated by the same script, and when it was rerun the output was as expected. thanks Jack

    Read the article

  • Looking for recommendations on OCR problem - tabular numeric data

    - by ldigas
    I have 20 pages of experiment measurement data which I need to digitalize. The results are in tabular form, scanned in 600 dpi resolution, and as far as scans go, they came up pretty clean and readable. For an example of how it looks see here (but beware: it is a rather big scan; about 5Mb; no problem for any broadband connection, but dialups should approach with caution!) ... and I need it finished by sunday afternoon (:-o) <-- smiley in a state of panic (then why did't you start sooner?)... yea, yeah ... I know ... but, it came up late, and I wasn't thinking I was gonna need this data also. So, I'm looking for recommendations. I haven't much experience with OCR programs, save scanning a page or two of pure text, but just to mention, I haven't the wish also to test out every OCR program out there. So this isn't a "name your OCR favourite". What I'm looking is advice from someone who's done something like that, and his/hers experience on what would be the best way to undertake. I need the data in txt form but since it will have to be checked (by drawing it, and just simply watching whether some points "jump out") I'll probably be entering it in Excel at first.

    Read the article

  • Problem installing iATKOS S3 Version 2 Snow Leopard 10.6.3 on DELL Precision T5500 Desktop

    - by Matias Dominoni
    Someone managed to install this right? I've used the following parameters: -v -x -f cpus=1 busratio=22 After installation, boot fails with a Kernel Panic. The exact error here: http://www.insanelymac.com/forum/index.php?showtopic=182609&mode=linearplus I'm aware that is very annoying installation. http://www.insanelymac.com/forum/index.php?showtopic=222386 Does anyone knows a guide to follow up or any other distrubution that works?

    Read the article

  • Excel CSV import treating quoted strings of numbers as numeric values, not strings

    - by MichaelOryl
    I've got a web application that is exporting its data to a CSV file. Here's one example row of the CSV file in question: 28,"65154",02/21/2013 00:00,"false","0316295","8316012,8315844","MALE" Since I can't post an image, I'll have to explain the results in Excel. The "0316295" field gets turned into a number and the leading 0 goes away. The "8316012,8315844" gets interpreted as one single number: 83,160,128,315,844. That is, most obviously, not the intended result. I've seen people recommend a leading single quote for such cases, but that doesn't really work either. 28,"65154",02/21/2013 00:00,"false","'0316295","'8316012,8315844","MALE" The single quote is visible at all times in the cell in Excel, though if I enter a number with a leading single quote myself, it shows just the intended string and not the single quote with the string. Importing is not the same as typing, it seems. Anybody have a solution here?

    Read the article

  • Dell Precision 670 no booting

    - by fristson
    Our dell running xp did not boot one day and was making a beep code I had a hard time finding on the internet. It was one of the RAM sticks so i replaced it. Before that i removed the motherboard battery to see if there was a problem with the battery. Now at start up it tells me that No Logical Drives Found and then BIOS not installed!. I have read over alot of post here and other places but everyone is talking about installation of the OS. I just want it to start with what it had. Please help.

    Read the article

  • Change the precision of all decimal columns in every table in the database

    - by Jon
    Hi all, I have a rather large database that has alot of decimal columns in alot of tables, the customer has now changed their mind and wants all the numbers (decimals) to have a precision of 3 d.p. instead of the original two. Is there any quick way of going through all the tables in a database and changing any decimal column in that table to have 3.d.p instead of 2 d.p? The db is on sql 2005. Any help would be great.

    Read the article

  • Possible loss of precision; extracting char from string

    - by Troy
    I am getting a string from the user and then doing some checking to make sure it is valid, here is the code I have been using; char digit= userInput.charAt(0) - '0'; This had been working fine until I did some work on another method, I went to compile and have been receiving a 'possible loss of precision' error since then. What am I doing wrong?

    Read the article

  • Arbitrary-precision arithmetic Explanation

    - by TT
    I'm trying to learn C and have come across the inability to work with REALLY big numbers (i.e., 100 digits, 1000 digits, etc.). I am aware that there exist libraries to do this, but I want to attempt to implement it myself. I just want to know if anyone has or can provide a very detailed, dumbed down explanation of arbitrary-precision arithmetic. Thanks!

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 2 3 4 5 6 7 8 9 10 11 12 13  | Next Page >