Numercially stable softmax

python numpy nan scientific-computing softmax
Abhishek Bhatia picture Abhishek Bhatia · Mar 4, 2017 · Viewed 27.3k times · Source

Is there a numerically stable way to compute softmax function below? I am getting values that becomes Nans in Neural network code.

np.exp(x)/np.sum(np.exp(y))

Answer

Paul Panzer picture Paul Panzer · Mar 5, 2017

The softmax exp(x)/sum(exp(x)) is actually numerically well-behaved. It has only positive terms, so we needn't worry about loss of significance, and the denominator is at least as large as the numerator, so the result is guaranteed to fall between 0 and 1.

The only accident that might happen is over- or under-flow in the exponentials. Overflow of a single or underflow of all elements of x will render the output more or less useless.

But it is easy to guard against that by using the identity softmax(x) = softmax(x + c) which holds for any scalar c: Subtracting max(x) from x leaves a vector that has only non-positive entries, ruling out overflow and at least one element that is zero ruling out a vanishing denominator (underflow in some but not all entries is harmless).

Footnote: theoretically, catastrophic accidents in the sum are possible, but you'd need a ridiculous number of terms. For example, even using 16 bit floats which can only resolve 3 decimals---compared to 15 decimals of a "normal" 64 bit float---we'd need between 2^1431 (~6 x 10^431) and 2^1432 to get a sum that is off by a factor of two.

Related questions

Removing nan values from an array

I want to figure out how to remove nan values from my array. My array looks something like this: x = [1400, 1500, 1600, nan, nan, nan ,1700] #Not in this exact configuration How can I remove the nan values from x?

Read More
convert nan value to zero

I have a 2D numpy array. Some of the values in this array are NaN. I want to perform certain operations using this array. For example consider the array: [[ 0. 43. 67. 0. 38.] [ 100. 86. 96. 100. 94.] [ 76. 79. 83. 89. 56.] [ 88. NaN 67. 89. 81.] [ 94. 79. 67. 89. 69.] [ 88. 79. 58. 72. 63.] [ 76. 79. 71. 67. 56.] [ 71. 71. NaN 56. 100.]] I am trying to take each row, one at …

Read More
Fast check for NaN in NumPy

I'm looking for the fastest way to check for the occurrence of NaN (np.nan) in a NumPy array X. np.isnan(X) is out of the question, since it builds a boolean array of shape X.shape, which is …

Read More