For questions related to numerical precision in programming.
I'm trying to compare GPU to CPU performance. For the NVIDIA GPU I've been using the cudaEvent_t types to …
c linux timer precision high-resolutionI am using BigDecimal to get some price values. Requirement is something like this, what ever the value we fetch …
java precision bigdecimal decimal-pointI know a little bit about how floating-point numbers are represented, but not enough, I'm afraid. The general question is: …
floating-point precision numerical ieee-754Is it possible to set the display precision of a float in Ruby? Something like: z = 1/3 z.to_s #=> 0.33333333333333 …
ruby floating-point precisionI know you can't rely on equality between double or decimal type values normally, but I'm wondering if 0 is a …
c# .net floating-point precision zeroI'm using BigDecimal for my numbers in my application, for example, with JPA. I did a bit of researching about …
java jpa scale precision bigdecimalI was given this interview question recently: Given a 12-hour analog clock, compute in degree the smaller angle between the …
math floating-point precision angleThese lines in C# decimal a = 2m; decimal b = 2.0m; decimal c = 2.00000000m; decimal d = 2.000000000000000000000000000m; Console.WriteLine(a); Console.…
.net decimal precision