Search Results

Search found 680 results on 28 pages for 'precision'.

Page 3/28 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • Using a Higher Precision (than 8-bit unsigned integer) Buffered Image for Heightmaps in Java

    - by pl12
    I am generating a heightmap for every quad in my quadtree in openCL. The way I was creating the image is as follows: DataBufferInt dataBuffer = (DataBufferInt)img.getRaster().getDataBuffer(); int data[] = dataBuffer.getData(); //img is a bufferedimage inputImageMem = CL.clCreateImage2D( context, CL_MEM_READ_WRITE | CL_MEM_USE_HOST_PTR, new cl_image_format[]{imageFormat}, size, size, size * Sizeof.cl_uint, Pointer.to(data), null); This works ok but the major issue is that as the quads get smaller and smaller the 8-bit format of the buffered image starts to cause intolerable "stepping" issues as seen below: I was wondering if there was an alternate way I could go about doing this? Thanks for the time.

    Read the article

  • Precision of Interval for PL/SQL Function value

    - by Gary
    Generally, when you specify a function the scale/precision/size of the return datatype is undefined. For example, you say FUNCTION show_price RETURN NUMBER or FUNCTION show_name RETURN VARCHAR2. You are not allowed to have FUNCTION show_price RETURN NUMBER(10,2) or FUNCTION show_name RETURN VARCHAR2(20), and the function return value is unrestricted. This is documented functionality. Now, I get an precision error (ORA-01873) if I push 9999 hours (about 400 days) into the following. The limit is because the default days precision is 2 DECLARE v_int INTERVAL DAY (4) TO SECOND(0); FUNCTION hhmm_to_interval return INTERVAL DAY TO SECOND IS v_hhmm INTERVAL DAY (4) TO SECOND(0); BEGIN v_hhmm := to_dsinterval('PT9999H'); RETURN v_hhmm; -- END hhmm_to_interval; BEGIN v_int := hhmm_to_interval; end; / and it won't allow the precision to be specified directly as part of the datatype returned by the function. DECLARE v_int INTERVAL DAY (4) TO SECOND(0); FUNCTION hhmm_to_interval return INTERVAL DAY (4) TO SECOND IS v_hhmm INTERVAL DAY (4) TO SECOND(0); BEGIN v_hhmm := to_dsinterval('PT9999H'); RETURN v_hhmm; -- END hhmm_to_interval; BEGIN v_int := hhmm_to_interval; end; / I can use a SUBTYPE DECLARE subtype t_int is INTERVAL DAY (4) TO SECOND(0); v_int INTERVAL DAY (4) TO SECOND(0); FUNCTION hhmm_to_interval return t_int IS v_hhmm INTERVAL DAY (4) TO SECOND(0); BEGIN v_hhmm := to_dsinterval('PT9999H'); RETURN v_hhmm; -- END hhmm_to_interval; BEGIN v_int := hhmm_to_interval; end; / Any drawbacks to the subtype approach ? Any alternatives (eg some place to change a default precision) ? Working with 10gR2.

    Read the article

  • 3rd monitor with Dell Precision 490

    - by Animesh Kumar
    I am trying to go for a 3 monitor setup on my workstation, but need some information before I can invest in a new graphics card and 2 monitors. Would be glad to get some help here: I have a Dell Precision 490 workstation with Nvidia Quadro FX 4500 graphics card (PCI-E 16) on it. It has 2 DVI-D ports that can support 2 monitors upto 2560x1600 resolution. For the third monitor, I was looking at Nvidia NVS 295 card, however I am not sure if it is physically possible to hook another card to the motherboard. Here is the spec of Precision 490: http://www.dell.com/downloads/global/products/precn/en/spec_precn_490_en.pdf Can I attach NVS 295 to it? If not, what options do I have?

    Read the article

  • Floating point precision in Visual C++

    - by Luigi Giaccari
    HI, I am trying to use the robust predicates for computational geometry from Jonathan Richard Shewchuk. I am not a programmer, so I am not even sure of what I am saying, I may be doing some basic mistake. The point is the predicates should allow for precise aritmthetic with adaptive floating point precision. On my computer: Asus pro31/S (Core Due Centrino Processor) they do not work. The problem may stay in the fact the my computer may use some improvements in the floating point precision taht conflicts with the one used by Shewchuk. The author says: /* On some machines, the exact arithmetic routines might be defeated by the / / use of internal extended precision floating-point registers. Sometimes / / this problem can be fixed by defining certain values to be volatile, / / thus forcing them to be stored to memory and rounded off. This isn't / / a great solution, though, as it slows the arithmetic down. */ Now what I would like to know is that there is a way, maybe some compiler option, to turn off the internal extended precision floating-point registers. I really appriaciate your help

    Read the article

  • Convert Double to String without precision loss in javascript

    - by holger
    I would like to convert a floating point variable to a string without losing any precision. I.e. I would like the string to have the same information as my floating point variable contains, since I use the output for further processing (even if it means that the string will be very long and readable). To put this more clearly, I would like to have functions for cyclic conversion var dA = 323423.23423423e4; var sA = toString(dA); var dnA = toDouble(sA); and I would like dnA and dA to be equal Thanks PS: Sources on the internet usually talk about how to round strings but I have not found information on exact representation. Also I am not interested in Arbitrary Precision calculations, I just need double precision floating point arithmetic.

    Read the article

  • Is long long in C++ known to be very nasty in terms of precision?

    - by Kevin
    The Given Problem: Given a theater with n rows, m seats, and a list of seats that are reserved. Given these values, determine how many ways two friends can sit together in the same row. So, if the theater was a size of 2x3 and the very first seat in the first row was reserved, there would be 3 different seatings that these two guys can take. The Problem That I'm Dealing With The function itself is supposed to return the number of seatings that there are based on these constraints. The return value is a long long. I've gone through my code many many times...and I'm pretty sure that it's right. All I'm doing is incrementing this one value. However, ALL of the values that my function return differ from the actual solution by 1 or 2. Any ideas? And if you think that it's just something wrong with my code, please tell me. I don't mind being called an idiot just as long as I learn something.

    Read the article

  • double returning only one precision in java

    - by srinath
    in my code i am performing division on numbers and storing it in a double variable. but the variable is returning with only one precision. i want to increase the precision how could i do that. double x; for ( int i =0 ; i<10000 ; i++) { x= w[i] / l[i]; System.out.println(x); } in the above code w and l are integer arrays; i am getting output like 3.0 0.0 1.0 0.0 i want to increase the precision upto 4 atleast.

    Read the article

  • What is the registry key for 'Enhanced Pointer Precision' option in the Mouse Properties

    - by detj
    I use a Razer mouse at work, and though the mouse is supercool, it's driver always removes the 'Enhanced Pointer Precision' option from the Pointer Options tab found in the Mouse Properties of Control Panel on startup. If I could find the correct registry setting to set this option 'on' whenever Windows starts, it would solve my problem. I've tried finding the correct key using Process Monitor, I didn't get success. Anyone know the key??

    Read the article

  • ORA-01438: value larger than specified precision allows for this column

    - by bobir
    We get sometimes the following error from our partner's database: ORA-01438: value larger than specified precision allows for this column The full response looks like the following: ORA-01438: value larger than specified precision allows for this column ORA-06512: at "UMAIN.PAY_NET_V1_PKG", line 176 ORA-06512: at line 1 5592988 What can be the cause for this error? Thank you in advance.

    Read the article

  • How to bind a double precision using psycopg2

    - by user337636
    I'm trying to bind a float to a postgresql double precision using psycopg2. ele = 1.0/3.0 dic = {'name': 'test', 'ele': ele} sql = '''insert into waypoints (name, elevation) values (%(name)s, %(ele)s)''' cur = db.cursor() cur.execute(sql, dic) db.commit() sql = """select elevation from waypoints where name = 'test'""" cur.execute(sql_out) ele_out = cur.fetchone()[0] ele_out 0.33333333333300003 ele 0.33333333333333331 Obviously I don't need the precision, but I would like to be able to simply compare the values. I could use the struct module and save it as a string, but thought there should be a better way. Thanks

    Read the article

  • How to maintain precision using DateTime.Now.Ticks in C#

    - by nmr
    I know that when I use DateTime.Now.Ticks in C# it returns a long value but I need to store it in an int variable and I am confused as to whether or not I can maintain that precision. As of right now I just have a cast int timeStampValue = (int)DateTime.Now.Ticks; Any suggestions or advice on how to maintain the precision, if possible, would be much appreciated.

    Read the article

  • Sub-millisecond precision timing in C or C++

    - by andand
    What techniques / methods exist for getting sub-millisecond precision timing data in C or C++, and what precision and accuracy do they provide? I'm looking for methods that don't require additional hardware. The application involves waiting for approximately 50 microseconds +/- 1 microsecond while some external hardware collects data. EDIT: OS is Wndows, probably with VS2010. If I can get drivers and SDK's for the hardware on Linux, I can go there. Thanks.

    Read the article

  • Configuring NVIDIA Quadro with Dell Precision M4600

    - by vsecades
    After a frustrating couple of weeks when I recently bought my Dell Precision laptop, I managed to fix an issue where Ubuntu (yes, was NOT using Windows, get serious) would not recognize the video card and would cause all sorts of problems all over the place. I ended up one Saturday morning nearly throwing this thing away, when I managed to find a post about NVIDIA Optimus technology... ( http://www.pcmag.com/article2/0,2817,2358963,00.asp ). Now I am a huge advocate of disruptive new stuff, as long as we keep the broader audience in mind. Anyhow, disabling this (which as the BIOS settings state only work on Windows 7 or later), effectively allow the NVIDIA based Ubuntu driver to kick in full force. No need for a trash can anymore thankfully. As I saw multiple posts all over the place about this, check your BIOS, disable and try the video again to see if this corrects your issues. Best of luck!

    Read the article

  • How to clean this Dell Precision M6400

    - by Daniel Pratt
    I have (well, ok, my employer has and I use) a Dell Precision M6400 notebook. It's a decent piece of hardware, but I have at least one major gripe: It's a dust and...uh...crumb (I repent! I repent!) magnet! And I cannot seem to exorcise the dust/crumbs from it! There is a strip of metal above the keyboard that is punched full of tiny holes. Well, maybe it's better to describe them as 'pits'. If a sufficiently small particle finds its way into one of those pits, there is only about a 50% that I will manage to get it out. Consequently, there is now a chorus of tiny little particles silently chiding me about eating cookies a cracker whilst I browse the intarwebs. Does anyone have any suggestions about how I could remove these particles from this machine...while still preserving the function of the machine?

    Read the article

  • Precision of cos(atan2(y,x)) versus using complex <double>, C++

    - by Ivan
    Hi all, I'm writing some coordinate transformations (more specifically the Joukoswky Transform, Wikipedia Joukowsky Transform), and I'm interested in performance, but of course precision. I'm trying to do the coordinate transformations in two ways: 1) Calculating the real and complex parts in separate, using double precision, as below: double r2 = chi.x*chi.x + chi.y*chi.y; //double sq = pow(r2,-0.5*n) + pow(r2,0.5*n); //slow!!! double sq = sqrt(r2); //way faster! double co = cos(atan2(chi.y,chi.x)); double si = sin(atan2(chi.y,chi.x)); Z.x = 0.5*(co*sq + co/sq); Z.y = 0.5*si*sq; where chi and Z are simple structures with double x and y as members. 2) Using complex : Z = 0.5 * (chi + (1.0 / chi)); Where Z and chi are complex . There interesting part is that indeed the case 1) is faster (about 20%), but the precision is bad, giving error in the third decimal number after the comma after the inverse transform, while the complex gives back the exact number. So, the problem is on the cos(atan2), sin(atan2)? But if it is, how the complex handles that? Thanks!

    Read the article

  • How to flash Dell Precision 390 from linux (debian)

    - by malat
    I am trying to update my BIOS: $ sudo dmidecode -s bios-version 2.1.2 With a newer one: 2.6.0. I went to this page Dell Precision System BIOS, 2.6.0 After downloading the file WS390-020600.BIN, here is what it states: $ ./WS390-020600.BIN --help Usage: WS390-020600.BIN [options] Options: --help Print this text. --version Print package versions. If no options, update the BIOS. and $ ./WS390-020600.BIN --version Dell BIOS Update Installer 1.2 Copyright 2006 Dell Inc. All Rights Reserved. ./WS390-020600.BIN: 60: ./WS390-020600.BIN: ./flash: not found Does anyone knows where this flash command can be found ? Update: it looks like this is a self-extracting archive (need bash as per comment in header). $ head -30 WS390-020600.BIN [...] Extract() { tail -n +`awk '/^__ARC__/ { print NR + 1; exit 0; }' $0` $0 | gzip -cd >$_PRG So the flash command should have been auto-generated, however the above command does not appear to be running as original author intended. I do not see anything wrong with the command though.

    Read the article

  • The best cross platform (portable) arbitrary precision math library

    - by Siu Ching Pong - Asuka Kenji
    Dear ninjas / hackers / wizards, I'm looking for a good arbitrary precision math library in C or C++. Could you please give me some advices / suggestions? The primary requirements: It MUST handle arbitrarily big integers (my primary interest is on integers). In case that you don't know what the word arbitrarily big means, imagine something like 100000! (the factorial of 100000). The precision MUST NOT NEED to be specified during library initialization / object creation. The precision should ONLY be constrained by the available resources of the system. It SHOULD utilize the full power of the platform, and should handle "small" numbers natively. That means on a 64-bit platform, calculating 2^33 + 2^32 should use the available 64-bit CPU instructions. The library SHOULD NOT calculate this in the same way as it does with 2^66 + 2^65 on the same platform. It MUST handle addition (+), subtraction (-), multiplication (*), integer division (/), remainder (%), power (**), increment (++), decrement (--), gcd(), factorial(), and other common integer arithmetic calculations efficiently. Ability to handle functions like sqrt() (square root), log() (logarithm) that do not produce integer results is a plus. Ability to handle symbolic computations is even better. Here are what I found so far: Java's BigInteger and BigDecimal class: I have been using these so far. I have read the source code, but I don't understand the math underneath. It may be based on theories / algorithms that I have never learnt. The built-in integer type or in core libraries of bc / Python / Ruby / Haskell / Lisp / Erlang / OCaml / PHP / some other languages: I have ever used some of these, but I have no idea on which library they are using, or which kind of implementation they are using. What I have already known: Using a char as a decimal digit, and a char* as a decimal string and do calculations on the digits using a for-loop. Using an int (or a long int, or a long long) as a basic "unit" and an array of it as an arbitrary long integer, and do calculations on the elements using a for-loop. Booth's multiplication algorithm What I don't know: Printing the binary array mentioned above in decimal without using naive methods. Example of a naive method: (1) add the bits from the lowest to the highest: 1, 2, 4, 8, 16, 32, ... (2) use a char* string mentioned above to store the intermediate decimal results). What I appreciate: Good comparisons on GMP, MPFR, decNumber (or other libraries that are good in your opinion). Good suggestions on books / articles that I should read. For example, an illustration with figures on how a un-naive arbitrarily long binary to decimal conversion algorithm works is good. Any help. Please DO NOT answer this question if: you think using a double (or a long double, or a long long double) can solve this problem easily. If you do think so, it means that you don't understand the issue under discussion. you have no experience on arbitrary precision mathematics. Thank you in advance! Asuka Kenji

    Read the article

  • Hibernate: Found: float, expected: double precision

    - by Frederic Morin
    I have a problem with the mapping of Oracle Float double precision datatype to Java Double datatype. The hibernate schema validator seems to fail when the Java Double datatype is used. org.hibernate.HibernateException: Wrong column type in DB.TABLE for column amount. Found: float, expected: double precision The only way to avoid this is to disable schema validation and hope the schema is in sync with the app about to run. I must fix this before it goes out to production. App's evironment: - Grails 1.2.1 - Hibernate-core 3.3.1.GA - Oracle 10g

    Read the article

  • Arbitrary-precision random numbers in C: generation for Monte Carlo simulation without atmospheric n

    - by Yktula
    I know that there are other questions similar to this one, however the following question pertains to arbitrary-precision random number generation in C for use in Monte Carlo simulation. How can we generate good quality arbitrary-precision random numbers in C, when atmospheric noise isn't always available, without relying on disk i/o or network access that would create bottlenecks? libgmp is capable of generating random numbers, but, like other implementations of pseudo-random number generators, it requires a seed. As the manual mentions, "the system time is quite easy to guess, so if unpredictability is required then it should definitely not be the only source for the seed value." Is there a portable/ported library for generating random numbers, or seeds for random numbers? The libgmp also mentions that "On some systems there's a special device /dev/random which provides random data better suited for use as a seed." However, /dev/random and /dev/urandom can only be used on *nix systems.

    Read the article

  • Phantom updates due to decimal precision on calculated properties

    - by Jamie Ide
    This article describes my problem. I have several properties that are calculated. These are typed as decimal(9,2) in SQL Server and decimal in my C# classes. An example of the problem is: Object is loaded with a property value of 14.9 A calculation is performed and the property value is changed to 14.90393 When the session is flushed, NHibernate issues an update because the property is dirty Since the database field is decimal (9,2) the stored value doesn't change Basically a phantom update is issued every time this object is loaded. I don't want to truncate the calculations in my business objects because that tightly couples them to the database and I don't want to lose the precision in other calculations. I tried setting scale and precision or CustomType("Decimal(9,2)") in the mapping file but this appears to only affect schema generation. My only reasonable option appears to be creating an IUserType implementation to handle this. Is there a better solution?

    Read the article

  • c# control location precision

    - by AC
    I need more precision then integer based locations when puttng controls on a form. It seems control.location only supports Point. Is there a work around for this? Point p = new Point(100, 200); this.Location = p;// this works of course PointF pF = new PointF(100.04f, 200.08f); this.Location = pF;// this does not work of course because Location expects a Point not PointF Is there some setting on the base form, or base control I can set to have more location precision?

    Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >