Top "Precision" questions

Rails: Cannot add :precision or :scale options with change_column in a migration?

This seems to have been asked before: rails decimal precision and scale But when running a change_column migration for :…

ruby-on-rails migration decimal precision scale

Haskell: Force floats to have two decimals

Using the following code snippet: (fromIntegral 100)/10.00 Using the Haskell '98 standard prelude, how do I represent the result with two …

haskell floating-point precision

Half-precision floating-point in Java

Is there a Java library anywhere that can perform computations on IEEE 754 half-precision numbers or convert them to and from …

java floating-point ieee-754 precision

Why do bean validation Min/Max constraints not support the double type?

Could somebody please explain to me why the JPA supports the double type as a field type, but the bean …

java constraints precision bean-validation

Ensuring C++ doubles are 64 bits

In my C++ program, I need to pull a 64 bit float from an external byte sequence. Is there some way …

c++ types precision compiler-construction ieee-754

More efficient and fast way of inverting matrices in c++ (big and small)

While googleing about matrix inversion algorithms I found that there are several ways (and opinions!) about how to do this …

c++ performance matrix precision matrix-inverse

Different std in pandas vs numpy

The standard deviation differs between pandas and numpy. Why and which one is the correct one? (the relative difference is 3.5% …

python numpy pandas precision

High precision floating point numbers in Haskell?

I know Haskell has native data types which allow you to have really big integers so things like >> …

haskell precision

Are all integer values perfectly represented as doubles?

My question is whether all integer values are guaranteed to have a perfect double representation. Consider the following code sample …

c++ double standards precision ieee-754

Getting the fractional part of a double value in integer without losing precision

i want to convert the fractional part of a double value with precision upto 4 digits into integer. but when i …

c double type-conversion precision floating