Top "Precision" questions

Why is the output of inv() and pinv() not equal in Matlab and Octave?

I have noticed that if A is a NxN matrix and it has the inverse matrix. But what the inv() …

matlab precision floating-accuracy matrix-inverse numerical-analysis

Determine the decimal precision of an input number

We have an interesting problem were we need to determine the decimal precision of a users input (textbox). Essentially we …

c# decimal precision decimal-point

How to perform unittest for floating point outputs? - python

Let's say I am writing a unit test for a function that returns a floating point number, I can do …

python unit-testing testing floating-point precision

Scikit: calculate precision and recall using cross_val_score function

I'm using scikit to perform a logistic regression on spam/ham data. X_train is my training data and y_…

python machine-learning scikit-learn precision logistic-regression

Is there a reliable way in JavaScript to obtain the number of decimal places of an arbitrary number?

It's important to note that I'm not looking for a rounding function. I am looking for a function that returns …

javascript floating-point decimal precision fractions

How to manually parse a floating point number from a string

Of course most languages have library functions for this, but suppose I want to do it myself. Suppose that the …

parsing floating-point precision

Arbitrary-Precision Decimals in C#

Possible Duplicates: Big integers in C# C# unlimited significant decimal digits (arbitrary precision) without java I read the question at …

c# math decimal precision j#

Why do I see a double variable initialized to some value like 21.4 as 21.399999618530273?

double r = 11.631; double theta = 21.4; In the debugger, these are shown as 11.631000000000000 and 21.399999618530273. How can I avoid this?

floating-point precision

Are doubles faster than floats in C#?

I'm writing an application which reads large arrays of floats and performs some simple operations with them. I'm using floats, …

c# performance floating-point precision

Why it is necessary to set precision for the fragment shader?

I learn WebGL. Next shaders work fine: // vertex.shader // precision mediump float; attribute vec4 a_Position; attribute float a_PointSize; …

opengl-es webgl shader precision