For questions related to numerical precision in programming.
A few interesting observations w.r.t equals operator on 0 and 0.0 new Double(0.0).equals(0) returns false, while new Double(0.0).equals(0.0) …
java double precision bigdecimal zeroI have an exam question I am revising for and the question is for 4 marks. "In java we can assign …
java floating-point double precisionTest case: NSLog(@"%f", M_PI); NSLog(@"%@", [NSString stringWithFormat:@"%f", M_PI]); NSLog(@"%@", [NSNumber numberWithDouble:M_PI]); Results: 3.141593 3.141593 3.141592653589793 Conclusions: 1) Printing …
iphone cocoa-touch double precision nslogWhy do some numbers lose accuracy when stored as floating point numbers? For example, the decimal number 9.2 can be expressed …
floating-point language-agnostic precisionI wrote a program to demonstrate floating point error in Go: func main() { a := float64(0.2) a += 0.1 a -= 0.3 var i …
go floating-point precisionDoes anyone know how to find out the precision of long double on a specific platform? I appear to be …
c++ precision long-doubleA single/double/extended-precision floating-point representation of Pi is accurate up to how many decimal places?
floating-point precision floating-accuracyI want to control the precision for a double during a comparison, and then come back to default precision, with …
c++ floating-point default precision rounding