How would I implement the derivative of Leaky ReLU in Python without using Tensorflow?
Is there a better way than this? I want the function to return a numpy array
def dlrelu(x, alpha=.01):
# return alpha if x < 0 else 1
return np.array ([1 if i >= 0 else alpha for i in x])
Thanks in advance for the help
The method you use works, but strictly speaking you are computing the derivative with respect to the loss, or lower layer, so it might be wise to also pass the value from lower layer to compute the derivative (dl/dx).
Anyway, you can avoid using the loop which is more efficient for large x
. This is one way to do it:
def dlrelu(x, alpha=0.01):
dx = np.ones_like(x)
dx[x < 0] = alpha
return dx
If you passed the error from lower layer, it looks like this:
def dlrelu(dl, x, alpha=0.01):
""" dl and x have same shape. """
dx = np.ones_like(x)
dx[x < 0] = alpha
return dx*dl