Top "Precision" questions

In MATLAB, are variables REALLY double-precision by default?

This question arose out of something strange that I noticed after investigating this question further... I always understood MATLAB variables …

variables matlab floating-point precision

What class to use for money representation?

What class should I use for representation of money to avoid most rounding errors? Should I use Decimal, or a …

python currency rounding-error arbitrary-precision precision

PostgreSQL: what is the difference between float(1) and float(24)?

I am having a hard time understanding the precision parameter p for float(p) in PostgreSQL. For example, float(1) and …

postgresql floating-point precision sqldatatypes floating-point-precision

C# Converting 20 digit precision double to string and back again

In C#. I have a double (which I've extracted from a database) that has 20 digit precision. In Visual Studio (using …

c# double precision tostring

How to Java String.format with a variable precision?

I'd like to vary the precision of a double representation in a string I'm formatting based on user input. Right …

java string rounding precision floating-accuracy

ValueError: pos_label=1 is not a valid label: array(['neg', 'pos'], dtype='<U3')

I recieve this error while trying to obtain the recall score. X_test = test_pos_vec + test_neg_vec Y_…

python machine-learning precision precision-recall

Preserving large numbers

I am trying to read a CSV file that has barcodes in the first column, but when R gets it …

r csv import precision numeric

Why is 24.0000 not equal to 24.0000 in MATLAB?

I am writing a program where I need to delete duplicate points stored in a matrix. The problem is that …

matlab floating-point precision

SQL Server decimal scale length - can be or has to be?

I have really simply question about DECIMAL (and maybe NUMERIC) type in SQL Server 2008 R2. MSDN said: (scale) The maximum …

sql-server decimal scale precision

Truncate a floating point number without rounding up

I have a floating point number that I want to truncate to 3 places but I don't want to round up. …

ruby math floating-point decimal precision