Search Results

Search found 680 results on 28 pages for 'precision'.

Page 2/28 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

  • reading unformatted fortran file in matlab - which precision?

    - by Griff
    I have just written out a file: real*8 :: vol_cel real*8, dimension(256,256,256) :: dense [... some operations] open(unit=8,file=fname,form="unformatted") write(8)dense(:,:,:)/vol_cell close(8) dense and vol_cell are real*8 variables. My code to read this in in Matlab: fid = fopen(fname,'r'); mesh_raw = fread(fid,256*256*256,'double'); fclose(fid); The min and max values clearly show that it is not reading it in correctly (Min is 0 and max is a largish positive real*8). min = 3.3622e+38 max = -3.3661e+38 What precision do I need to set in Matlab to make it read in the unformatted Fortran file? A somewhat related question: This Matlab code I am using reads binary files OK but not unformatted files. Though I am generating this new data on my Mac OSX using gfortran. It doesn't recognize form="binary" so I can't do it that way. Do I need to add some library?

    Read the article

  • How to calculate a operation's time in micro second precision

    - by Sanjeet Daga
    I want to calculate performance of a function in micro second precision on Windows platform. Now Windows itself has milisecond granuality, so how can I achieve this. I tried following sample, but not getting correct results. LARGE_INTEGER ticksPerSecond = {0}; LARGE_INTEGER tick_1 = {0}; LARGE_INTEGER tick_2 = {0}; double uSec = 1000000; // Get the frequency QueryPerformanceFrequency(&ticksPerSecond); //Calculate per uSec freq double uFreq = ticksPerSecond.QuadPart/uSec; // Get counter b4 start of op QueryPerformanceCounter(&tick_1); // The ope itself Sleep(10); // Get counter after opfinished QueryPerformanceCounter(&tick_2); // And now the op time in uSec double diff = (tick_2.QuadPart/uFreq) - (tick_1.QuadPart/uFreq);

    Read the article

  • Rails precision error

    - by dontangg
    When I run this in my Rails application: my_envelope.transactions.sum(:amount) This SQL is shown in the log files: SQL (0.3ms) SELECT SUM("transactions"."amount") AS sum_id FROM "transactions" WHERE (envelope_id = 834498537) And this value is returned: <BigDecimal:1011be570,'0.2515999999 9999997E2',27(27)> As you can see, the value is 25.159999. It should be 25.16. When I run the same SQL on the database myself, the correct value is returned. I'm a little confused because I know that there are precision problems with Floats, but it is returning a BigDecimal. The SQL column type is decimal. Any ideas?

    Read the article

  • Precision problems of real numbers in Fortran

    - by saladoil
    I've been trying to use Fortran for my research project, with the GNU Fortran compiler (gfortran), latest version, but I've been encountering some problems in the way it processes real numbers. If you have for example the code: program test implicit none real :: y = 23.234, z z = y * 100000 write(*,*) y, z end program You'll get as output: 23.23999 2323400.0. I find this really strange. Can someone tell me what's exactly happening here? Looking at z I can see that y does retain its precision, so for calculations that shouldn't be a problem I suppose. But why is the output of y not exactly the same as the value that I've specified, and what can I do to make it exactly the same?

    Read the article

  • Using Taylor Series to Avoid Loss of Precision

    - by Zachary
    I'm trying to use Taylor series to develop a numerically sound algorithm for solving a function. I've been at it for quite a while, but haven't had any luck yet. I'm not sure what I'm doing wrong. The function is f(x)=1 + x - sin(x)/ln(1+x) x~0 Also: why does loss of precision even occur in this function? when x is close to zero, sin(x)/ln(1+x) isn't even close to being the same number as x. I don't see where significance is even being lost. In order to solve this, I believe that I will need to use the Taylor expansions for sin(x) and ln(1+x), which are x - x^3/3! + x^5/5! - x^7/7! + ... and x - x^2/2 + x^3/3 - x^4/4 + ... respectfully. I have attempted to use like denominators to combine the x and sin(x)/ln(1+x) components, and even to combine all three, but nothing seems to work out correctly in the end. Any help is appreciated.

    Read the article

  • Actual long double precision does not agree with std::numeric_limits

    - by dmb
    Working on Mac OS X 10.6.2, Intel, with i686-apple-darwin10-g++-4.2.1, and compiling with the -arch x86_64 flag, I just noticed that while... std::numeric_limits<long double>::max_exponent10 = 4932 ...as is expected, when a long double is actually set to a value with exponent greater than 308, it becomes inf--ie in reality it only has 64bit precision instead of 80bit. Also, sizeof() is showing long doubles to be 16 bytes, which they should be. Finally, using gives the same results as . Does anyone know where the discrepancy might be? long double x = 1e308, y = 1e309; cout << std::numeric_limits::max_exponent10 << endl; cout << x << '\t' << y << endl; cout << sizeof(x) << endl; gives 4932 1e+308 inf 16

    Read the article

  • Why differs floating-point precision in C# when separated by parantheses and when separated by state

    - by Andreas Larsen
    I am aware of how floating point precision works in the regular cases, but I stumbled on an odd situation in my C# code. Why aren't result1 and result2 the exact same floating point value here? const float A; // Arbitrary value const float B; // Arbitrary value float result1 = (A*B)*dt; float result2 = (A*B); result2 *= dt; From this page I figured float arithmetic was left-associative and that this means values are evaluated and calculated in a left-to-right manner. The full source code involves XNA's Quaternions. I don't think it's relevant what my constants are and what the VectorHelper.AddPitchRollYaw() does. The test passes just fine if I calculate the delta pitch/roll/yaw angles in the same manner, but as the code is below it does not pass: X Expected: 0.275153548f But was: 0.275153786f [TestFixture] internal class QuaternionPrecisionTest { [Test] public void Test() { JoystickInput input; input.Pitch = 0.312312432f; input.Roll = 0.512312432f; input.Yaw = 0.912312432f; const float dt = 0.017001f; float pitchRate = input.Pitch * PhysicsConstants.MaxPitchRate; float rollRate = input.Roll * PhysicsConstants.MaxRollRate; float yawRate = input.Yaw * PhysicsConstants.MaxYawRate; Quaternion orient1 = Quaternion.Identity; Quaternion orient2 = Quaternion.Identity; for (int i = 0; i < 10000; i++) { float deltaPitch = (input.Pitch * PhysicsConstants.MaxPitchRate) * dt; float deltaRoll = (input.Roll * PhysicsConstants.MaxRollRate) * dt; float deltaYaw = (input.Yaw * PhysicsConstants.MaxYawRate) * dt; // Add deltas of pitch, roll and yaw to the rotation matrix orient1 = VectorHelper.AddPitchRollYaw( orient1, deltaPitch, deltaRoll, deltaYaw); deltaPitch = pitchRate * dt; deltaRoll = rollRate * dt; deltaYaw = yawRate * dt; orient2 = VectorHelper.AddPitchRollYaw( orient2, deltaPitch, deltaRoll, deltaYaw); } Assert.AreEqual(orient1.X, orient2.X, "X"); Assert.AreEqual(orient1.Y, orient2.Y, "Y"); Assert.AreEqual(orient1.Z, orient2.Z, "Z"); Assert.AreEqual(orient1.W, orient2.W, "W"); } } Granted, the error is small and only presents itself after a large number of iterations, but it has caused me some great headackes.

    Read the article

  • Preoblem with Precision floating point operation in C

    - by Microkernel
    Hi Guys, For one of my course project I started implementing "Naive Bayesian classifier" in C. My project is to implement a document classifier application (especially Spam) using huge training data. Now I have problem implementing the algorithm because of the limitations in the C's datatype. ( Algorithm I am using is given here, http://en.wikipedia.org/wiki/Bayesian_spam_filtering ) PROBLEM STATEMENT: The algorithm involves taking each word in a document and calculating probability of it being spam word. If p1, p2 p3 .... pn are probabilities of word-1, 2, 3 ... n. The probability of doc being spam or not is calculated using Here, probability value can be very easily around 0.01. So even if I use datatype "double" my calculation will go for a toss. To confirm this I wrote a sample code given below. #define PROBABILITY_OF_UNLIKELY_SPAM_WORD (0.01) #define PROBABILITY_OF_MOSTLY_SPAM_WORD (0.99) int main() { int index; long double numerator = 1.0; long double denom1 = 1.0, denom2 = 1.0; long double doc_spam_prob; /* Simulating FEW unlikely spam words */ for(index = 0; index < 162; index++) { numerator = numerator*(long double)PROBABILITY_OF_UNLIKELY_SPAM_WORD; denom2 = denom2*(long double)PROBABILITY_OF_UNLIKELY_SPAM_WORD; denom1 = denom1*(long double)(1 - PROBABILITY_OF_UNLIKELY_SPAM_WORD); } /* Simulating lot of mostly definite spam words */ for (index = 0; index < 1000; index++) { numerator = numerator*(long double)PROBABILITY_OF_MOSTLY_SPAM_WORD; denom2 = denom2*(long double)PROBABILITY_OF_MOSTLY_SPAM_WORD; denom1 = denom1*(long double)(1- PROBABILITY_OF_MOSTLY_SPAM_WORD); } doc_spam_prob= (numerator/(denom1+denom2)); return 0; } I tried Float, double and even long double datatypes but still same problem. Hence, say in a 100K words document I am analyzing, if just 162 words are having 1% spam probability and remaining 99838 are conspicuously spam words, then still my app will say it as Not Spam doc because of Precision error (as numerator easily goes to ZERO)!!!. This is the first time I am hitting such issue. So how exactly should this problem be tackled?

    Read the article

  • Problem with Precision floating point operation in C

    - by Microkernel
    Hi Guys, For one of my course project I started implementing "Naive Bayesian classifier" in C. My project is to implement a document classifier application (especially Spam) using huge training data. Now I have problem implementing the algorithm because of the limitations in the C's datatype. ( Algorithm I am using is given here, http://en.wikipedia.org/wiki/Bayesian_spam_filtering ) PROBLEM STATEMENT: The algorithm involves taking each word in a document and calculating probability of it being spam word. If p1, p2 p3 .... pn are probabilities of word-1, 2, 3 ... n. The probability of doc being spam or not is calculated using Here, probability value can be very easily around 0.01. So even if I use datatype "double" my calculation will go for a toss. To confirm this I wrote a sample code given below. #define PROBABILITY_OF_UNLIKELY_SPAM_WORD (0.01) #define PROBABILITY_OF_MOSTLY_SPAM_WORD (0.99) int main() { int index; long double numerator = 1.0; long double denom1 = 1.0, denom2 = 1.0; long double doc_spam_prob; /* Simulating FEW unlikely spam words */ for(index = 0; index < 162; index++) { numerator = numerator*(long double)PROBABILITY_OF_UNLIKELY_SPAM_WORD; denom2 = denom2*(long double)PROBABILITY_OF_UNLIKELY_SPAM_WORD; denom1 = denom1*(long double)(1 - PROBABILITY_OF_UNLIKELY_SPAM_WORD); } /* Simulating lot of mostly definite spam words */ for (index = 0; index < 1000; index++) { numerator = numerator*(long double)PROBABILITY_OF_MOSTLY_SPAM_WORD; denom2 = denom2*(long double)PROBABILITY_OF_MOSTLY_SPAM_WORD; denom1 = denom1*(long double)(1- PROBABILITY_OF_MOSTLY_SPAM_WORD); } doc_spam_prob= (numerator/(denom1+denom2)); return 0; } I tried Float, double and even long double datatypes but still same problem. Hence, say in a 100K words document I am analyzing, if just 162 words are having 1% spam probability and remaining 99838 are conspicuously spam words, then still my app will say it as Not Spam doc because of Precision error (as numerator easily goes to ZERO)!!!. This is the first time I am hitting such issue. So how exactly should this problem be tackled?

    Read the article

  • SQL SERVER – SmallDateTime and Precision – A Continuous Confusion

    - by pinaldave
    Some kinds of confusion never go away. Here is one of the ancient confusing things in SQL. The precision of the SmallDateTime is one concept that confuses a lot of people, proven by the many messages I receive everyday relating to this subject. Let me start with the question: What is the precision of the SMALLDATETIME datatypes? What is your answer? Write it down on your notepad. Now if you do not want to continue reading the blog post, head to my previous blog post over here: SQL SERVER – Precision of SMALLDATETIME. A Social Media Question Since the increase of social media conversations, I noticed that the amount of the comments I receive on this blog is a bit staggering. I receive lots of questions on facebook, twitter or Google+. One of the very interesting questions yesterday was asked on Facebook by Raghavendra. I am re-organizing his script and asking all of the questions he has asked me. Let us see if we could help him with his question: CREATE TABLE #temp (name VARCHAR(100),registered smalldatetime) GO DECLARE @test smalldatetime SET @test=GETDATE() INSERT INTO #temp VALUES ('Value1',@test) INSERT INTO #temp VALUES ('Value2',@test) GO SELECT * FROM #temp ORDER BY registered DESC GO DROP TABLE #temp GO Now when the above script is ran, we will get the following result: Well, the expectation of the query was to have the following result. The row which was inserted last was expected to return as first row in result set as the ORDER BY descending. Side note: Because the requirement is to get the latest data, we can’t use any  column other than smalldatetime column in order by. If we use name column in the order by, we will get an incorrect result as it can be any name. My Initial Reaction My initial reaction was as follows: 1) DataType DateTime2: If file precision of the column is expected from the column which store date and time, it should not be smalldatetime. The precision of the column smalldatetime is One Minute (Read Here) for finer precision use DateTime or DateTime2 data type. Here is the code which includes above suggestion: CREATE TABLE #temp (name VARCHAR(100), registered datetime2) GO DECLARE @test datetime2 SET @test=GETDATE() INSERT INTO #temp VALUES ('Value1',@test) INSERT INTO #temp VALUES ('Value2',@test) GO SELECT * FROM #temp ORDER BY registered DESC GO DROP TABLE #temp GO 2) Tie Breaker Identity: There are always possibilities that two rows were inserted at the same time. In that case, you may need a tie breaker. If you have an increasing identity column, you can use that as a tie breaker as well. CREATE TABLE #temp (ID INT IDENTITY(1,1), name VARCHAR(100),registered datetime2) GO DECLARE @test datetime2 SET @test=GETDATE() INSERT INTO #temp VALUES ('Value1',@test) INSERT INTO #temp VALUES ('Value2',@test) GO SELECT * FROM #temp ORDER BY ID DESC GO DROP TABLE #temp GO Those two were the quick suggestions I provided. It is not necessary that you should use both advices. It is possible that one can use only DATETIME datatype or Identity column can have datatype of BIGINT or have another tie breaker. An Alternate NO Solution In the facebook thread this was also discussed as one of the solutions: CREATE TABLE #temp (name VARCHAR(100),registered smalldatetime) GO DECLARE @test smalldatetime SET @test=GETDATE() INSERT INTO #temp VALUES ('Value1',@test) INSERT INTO #temp VALUES ('Value2',@test) GO SELECT name, registered, ROW_NUMBER() OVER(ORDER BY registered DESC) AS "Row Number" FROM #temp ORDER BY 3 DESC GO DROP TABLE #temp GO However, I believe it is not the solution and can be further misleading if used in a production server. Here is the example of why it is not a good solution: CREATE TABLE #temp (name VARCHAR(100) NOT NULL,registered smalldatetime) GO DECLARE @test smalldatetime SET @test=GETDATE() INSERT INTO #temp VALUES ('Value1',@test) INSERT INTO #temp VALUES ('Value2',@test) GO -- Before Index SELECT name, registered, ROW_NUMBER() OVER(ORDER BY registered DESC) AS "Row Number" FROM #temp ORDER BY 3 DESC GO -- Create Index ALTER TABLE #temp ADD CONSTRAINT [PK_#temp] PRIMARY KEY CLUSTERED (name DESC) GO -- After Index SELECT name, registered, ROW_NUMBER() OVER(ORDER BY registered DESC) AS "Row Number" FROM #temp ORDER BY 3 DESC GO DROP TABLE #temp GO Now let us examine the resultset. You will notice that an index which is created on the base table which is (indeed) schema change the table but can affect the resultset. As you can see, an index can change the resultset, so this method is not yet perfect to get the latest inserted resultset. No Schema Change Requirement After giving these two suggestions, I was waiting for the feedback of the asker. However, the requirement of the asker was there can’t be any schema change because the application was used by many other applications. I validated again, and of course, the requirement is no schema change at all. No addition of the column of change of datatypes of any other columns. There is no further help as well. This is indeed an interesting question. I personally can’t think of any solution which I could provide him given the requirement of no schema change. Can you think of any other solution to this? Need of Database Designer This question once again brings up another ancient question:  “Do we need a database designer?” I often come across databases which are facing major performance problems or have redundant data. Normalization is often ignored when a database is built fast under a very tight deadline. Often I come across a database which has table with unnecessary columns and performance problems. While working as Developer Lead in my earlier jobs, I have seen developers adding columns to tables without anybody’s consent and retrieving them as SELECT *.  There is a lot to discuss on this subject in detail, but for now, let’s discuss the question first. Do you have any suggestions for the above question? Reference: Pinal Dave (http://blog.sqlauthority.com) Filed under: CodeProject, Developer Training, PostADay, SQL, SQL Authority, SQL DateTime, SQL Query, SQL Server, SQL Tips and Tricks, SQLServer, T SQL, Technology

    Read the article

  • Arbitrary precision arithmetic with Ruby

    - by macek
    How the heck does Ruby do this? Does Jörg or anyone else know what's happening behind the scenes? irb(main):001:0> 999**999 368063488259223267894700840060521865838338232037353204655959621437025609300472231530103873614505175218691345257589896391130393189447969771645832382192366076536631132001776175977932178658703660778465765811830827876982014124022948671975678131724958064427949902810498973271030787716781467419524180040734398996952930832508934116945966120176735120823151959779536852290090377452502236990839453416790640456116471139751546750048602189291028640970574762600185950226138244530187489211615864021135312077912018844630780307462205252807737757672094320692373101032517459518497524015120165166724189816766397247824175394802028228160027100623998873667435799073054618906855460488351426611310634023489044291860510352301912426608488807462312126590206830413782664554260411266378866626653755763627796569082931785645600816236891168141774993267488171702172191072731069216881668294625679492696148976999868715671440874206427212056717373099639711168901197440416590226524192782842896415414611688187391232048327738965820265934093108172054875188246591760877131657895633586576611857277011782497943522945011248430439201297015119468730712364007639373910811953430309476832453230123996750235710787086641070310288725389595138936784715274150426495416196669832679980253436807864187160054589045664027158817958549374490512399055448819148487049363674611664609890030088549591992466360050042566270348330911795487647045949301286614658650071299695652245266080672989921799342509291635330827874264789587306974472327718704306352445925996155619153783913237212716010410294999877569745287353422903443387562746452522860420416689019732913798073773281533570910205207767157128174184873357050830752777900041943256738499067821488421053870869022738698816059810579221002560882999884763252161747566893835178558961142349304466506402373556318707175710866983035313122068321102457824112014969387225476259342872866363550383840720010832906695360553556647545295849966279980830561242960013654529514995113584909050813015198928283202189194615501403435553060147713139766323195743324848047347575473228198492343231496580885057330510949058490527738662697480293583612233134502078182014347192522391449087738579081585795613547198599661273567662441490401862839817822686573112998663038868314974259766039340894024308383451039874674061160538242392803580758232755749310843694194787991556647907091849600704712003371103926967137408125713631396699343733288014254084819379380555174777020843568689927348949484201042595271932630685747613835385434424807024615161848223715989797178155169951121052285149157137697718850449708843330475301440373094611119631361702936342263219382793996895988331701890693689862459020775599439506870005130750427949747071390095256759203426671803377068109744629909769176319526837824364926844730545524646494321826241925107158040561607706364484910978348669388142016838792902926158979355432483611517588605967745393958061959024834251565197963477521095821435651996730128376734574843289089682710350244222290017891280419782767803785277960834729869249991658417000499998999

    Read the article

  • Precision loss when transforming from cartesian to isometric

    - by Justin Skiles
    My goal is to display a tile map in isometric projection. This tile map has 25 tiles across and 25 tiles down. Each tile is 32x32. See below for how I'm accomplishing this. World Space World Space to Screen Space Rotation (45 degrees) Using a 2D rotation matrix, I use the following: double rotation = Math.PI / 4; double rotatedX = ((tileWorldX * Math.Cos(rotation)) - ((tileWorldY * Math.Sin(rotation))); double rotatedY = ((tileWorldX * Math.Sin(rotation)) + (tileWorldY * Math.Cos(rotation))); World Space to Screen Space Scale (Y-axis reduced by 50%) Here I simply scale down the Y value by a factor of 0.5. Problem And it works, kind of. There are some tiny 1px-2px gaps between some of the tiles when rendering. I think there's some precision loss somewhere, or I'm not understanding how to get these tiles to fit together perfectly. I'm not truncating or converting my values to non-decimal types until I absolutely have to (when I pass to the render method, which only takes integers). I'm not sure how to guarantee pixel perfect rendering precision when I'm rotating and scaling on a level of higher precision. Any advice? Do I need to supply for information?

    Read the article

  • Game Clock Precision

    - by Philip
    I'm reading a fantastic article about game timer precision and here is a quote about 2/3 of the way into the article: If you start your game clock at about 4 billion (more precisely 2^32, or any large power of two) then your exponent, and hence your precision, will remain constant for the next ~4 billion seconds, or ~136 years. He doesn't give a concrete example of this though. Does this mean I would want to add 2^32 to the game clock value that I store at the beginning of each frame? Or is there a way to actually set the clock in Windows so that the numbers start at 2^32?

    Read the article

  • Essbase Data precision unraveled

    - by THE
    (guest reference added by Nancy) Anyone who has been working with Data import and exoport as well as the Essbase Excel Add In has probably come across a phenomenon that is called data precision: Lots of zeroes are added to any given number that has been calculated by Essbase, and this gets displayed as "10.0000000000001" or "9.99999999999999" instead of a simple "10" . This question is one of the recurring ones that Support get asked over and over again, and we therefore feel the need to give an explanation to it: I would like to point you to the note The Limits of Data Precision in Essbase (Doc ID 1311188.1) which explains in detail why these numbers are showing up and what to do about it.

    Read the article

  • SQL SERVER Precision of SMALLDATETIME A 1 MinutePrecision

    I am myself surprised that I am writing this post today. I am going to present one of the very known facts of SQL Server SMALLDATETIME datatype. Even though this is a very well-known datatype, many a time, I have seen developers getting confused with precision of the SMALLDATETIME datatype. The precision of the datatype [...]...Did you know that DotNetSlackers also publishes .net articles written by top known .net Authors? We already have over 80 articles in several categories including Silverlight. Take a look: here.

    Read the article

  • Temperature monitoring on Dell Precision M4500 under Linux: sensors-detect doesn't, what next?

    - by Tikitu
    I have a Dell Precision M4500, Intel Core i5 CPU, running Linux (Ubuntu Lucid), and would like to keep an eye on CPU temperature. I've tried lm-sensors: sensors-detect didn't find any sensors; following its hint ("This is relatively common on laptops, where thermal management is handled by ACPI rather than the OS.") I tried acpi -V but got nothing thermal. The Gnome panel applet "Hardware Sensors Monitor" reports on GPU temperature but nothing else. What should I be trying next?

    Read the article

  • How can I change the precision of printing with the stl?

    - by mmr
    This might be a repeat, but my google-fu failed to find it. I want to print numbers to a file using the stl with the number of decimal places, rather than overall precision. So, if I do this: int precision = 16; std::vector<double> thePoint(3); thePoint[0] = 86.3671436; thePoint[0] = -334.8866574; thePoint[0] = 24.2814; ofstream file1(tempFileName, ios::trunc); file1 << std::setprecision(precision) << thePoint[0] << "\\" << thePoint[1] << "\\" << thePoint[2] << "\\"; I'll get numbers like this: 86.36714359999999\-334.8866574\24.28140258789063 What I want is this: 86.37\-334.89\24.28 In other words, truncating at two decimal points. If I set precision to be 4, then I'll get 86.37\-334.9\24.28 ie, the second number is improperly truncated. I do not want to have to manipulate each number explicitly to get the truncation, especially because I seem to be getting the occasional 9 repeating or 0000000001 or something like that that's left behind. I'm sure there's something obvious, like using the printf(%.2f) or something like that, but I'm unsure how to mix that with the stl << and ofstream.

    Read the article

  • Rails: Cannot add :precision or :scale options with change_column in a migration?

    - by Josh Pinter
    This seems to have been asked before: http://stackoverflow.com/questions/1402547/rails-decimal-precision-and-scale But when running a change_column migration for :precision or :scale they don't actually affect the schema or database, but db:migrate runs without errors. My migration file looks like this: class ChangePrecisionAndScaleOfPaybackPeriodInTags < ActiveRecord::Migration def self.up change_column :tags, :payback_period, :decimal, { :scale => 3, :precision => 10 } end def self.down change_column :tags, :payback_period, :decimal end end But my schema (and the data) remains as: t.decimal "rate" # previous column t.decimal "payback_period" t.string "component_type" # next column Anybody else have this issue? Thanks, Josh

    Read the article

  • "possible loss of precision" is Java going crazy or I'm missing something?

    - by Lo'oris
    I'm getting a "loss of precision" error when there should be none, AFAIK. this is an instance variable: byte move=0; this happens in a method of this class: this.move=(this.move<<4)|(byte)(Guy.moven.indexOf("left")&0xF); move is a byte, move is still a byte, and the rest is being cast to a byte. I get this error: [javac] /Users/looris/Sviluppo/dumdedum/client/src/net/looris/android/toutry/Guy.java:245: possible loss of precision [javac] found : int [javac] required: byte [javac] this.move=(this.move<<4)|(byte)(Guy.moven.indexOf("left")&0xF); [javac] ^ I've tried many variations but I still get the same error. I'm now clueless.

    Read the article

  • SQL Server cast fails with arithmetic overflow

    - by Jamie Ide
    According to the entry for decimal and numeric data types in SQL Server 2008 Books Online, precision is: p (precision) The maximum total number of decimal digits that can be stored, both to the left and to the right of the decimal point. The precision must be a value from 1 through the maximum precision of 38. The default precision is 18. However, the second select below fails with "Arithmetic overflow error converting int to data type numeric." SELECT CAST(123456789 as decimal(9,0)) SELECT CAST(123456789 as decimal(9,1))

    Read the article

  • Java - How to avoid loss of precision during divide and cast to int?

    - by David Buckley
    I have a situation where I need to find out how many times an int goes into a decimal, but in certain cases, I'm losing precision. Here is the method: public int test(double decimalAmount, int divisor) { return (int) (decimalAmount/ (1d / divisor)); } The problem with this is if I pass in 1.2 as the decimal amount and 5 as the divisor, I get 5 instead of 6. How can I restrusture this so I know how many times 5 goes into the decimal amount as an int?

    Read the article

  • How precise is the internal clock of a modern PC?

    - by mafutrct
    I know that 10 years ago, typical clock precision equaled a system-tick, which was in the range of 10-30ms. Over the past years, precision was increased in multiple steps. Nowadays, there are ways to measure time intervals in actual nanoseconds. However, usual frameworks still return time with a precision of only around 15ms. My question is, which steps did increase the precision, how is it possible to measure in nanoseconds, and why are we still often getting less-than-microsecond precision (for instance in .NET). (Disclaimer: It strikes me as odd that this was not asked before, so I guess I missed this question when I searched. Please close and point me to the question in that case, thanks. I believe this belongs on SO and not on any other SOFU site. I understand the difference between precision and accuracy.)

    Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >