Top "Floating-point-precision" questions

For questions related to numerical precision in programming.

Printf width specifier to maintain precision of floating-point value

Is there a printf width specifier which can be applied to a floating point specifier that would automatically format the …

c floating-point printf c99 floating-point-precision
Set specific precision of a BigDecimal

I have an XSD that requires me to use a BigDecimal for a lat/lon. I currently have the lat/…

java bigdecimal floating-point-precision
Python: Find index of minimum item in list of floats

How can I find the index of the minimum item in a Python list of floats? If they were integers, …

python list floating-point minimum floating-point-precision
Does parseDouble exist in JavaScript?

In JavaScript, I have a number which is 21 digits, and I want to parse it. Does a parseDouble method exist …

javascript floating-point-precision
What's the C++ suffix for long double literals?

In C++ (and C), a floating point literal without suffix defaults to double, while the suffix f implies a float. …

c++ c literals floating-point-precision
How to safely floor or ceil a CGFloat to int?

I often need to floor or ceil a CGFloat to an int, for calculation of an array index. The problem …

ios math floating-point floating-accuracy floating-point-precision
How to avoid floating point errors?

I was trying to write a function to approximate square roots (I know there's the math module...I want to …

python python-3.x floating-point floating-point-precision
python convert a list of float to string

I have a list of floats in Python and when I convert it into a string, I get the follwoing [1883.95, 1878.3299999999999, 1869.4300000000001, 1863.4000000000001] …

python list formatting floating-point-precision floating-point-conversion
PostgreSQL: what is the difference between float(1) and float(24)?

I am having a hard time understanding the precision parameter p for float(p) in PostgreSQL. For example, float(1) and …

postgresql floating-point precision sqldatatypes floating-point-precision
In Java, when is a float equal to zero?

For what values of x does the test (x == 0) return true? Is there some kind of margin or does the …

java zero floating-point-precision