For questions related to numerical precision in programming.
This was a question I was asked at my recent interview and I want to know (I don't actually remember …
c++ floating-point precisionI need to calculate a list of very small numbers such as (0.1)^1000, 0.2^(1200), and then normalize them so they will sum …
r precision underflowI know the 0.1 decimal number cannot be represented exactly with a finite binary number (explanation), so double n = 0.1 will lose …
java floating-point double precisionImagine you have a large array of floating point numbers, of all kinds of sizes. What is the most correct …
algorithm language-agnostic floating-point precisionNo, this is not another "Why is (1/3.0)*3 != 1" question. I've been reading about floating-points a lot lately; specifically, how the same …
c# .net floating-point precision ieee-754I have medium size C99 program which uses long double type (80bit) for floating-point computation. I want to improve precision …
gcc floating-point precision 128-bitI wanted to know how I can generate pi to the nth digit. I have a couple of basic ideas. …
java precision bigdecimal piI am working with another codebase that I have no control over, and that I cannot see. I can send …
javascript decimal precision tofixedI'm currently using the charting within .NET using System.Windows.Forms.DataVisualization.Charting.Chart. Thus far it seems very powerful, …
c# precision data-visualization mschart intervalsI would like to obtain a number from stringstream and set it to 5 significant figures. How do I do this? …
c++ precision significant-digits