What does "relu" stand for in tf.nn.relu?

tensorflow tensor
Aerin picture Aerin · Apr 19, 2017 · Viewed 15.3k times · Source

In its API documentation, it says "Computes rectified linear".

Is it Re(ctified) L(inear)... what is U then?

Answer

Phillip Bock picture Phillip Bock · Apr 19, 2017

Re(ctified) L(inear) (U)nit

Usually a layer in a neural network has some input, say a vector, and multiplies that by a weight matrix, resulting i.e. again in a vector.

Each value in the result (usually a float) is then considered an output. However, most layers in neural networks nowadays involve nonlinearities, hence an add-on function that, you might say, adds complexity to these output values. For long these have been sigmoids and tanhs.

But more recently people use a function that results in 0 if the input is negative, and the input itself if that input is 0 or positive. This specific add-on function (or better "activation function") is called a relu.

Related questions

How to print the value of a Tensor object in TensorFlow?

I have been using the introductory example of matrix multiplication in TensorFlow. matrix1 = tf.constant([[3., 3.]]) matrix2 = tf.constant([[2.],[2.]]) product = tf.matmul(matrix1, matrix2) When I print the product, it is displaying it as a Tensor object: <tensorflow.python.framework.…

Read More
How to get the dimensions of a tensor (in TensorFlow) at graph construction time?

I am trying an Op that is not behaving as expected. graph = tf.Graph() with graph.as_default(): train_dataset = tf.placeholder(tf.int32, shape=[128, 2]) embeddings = tf.Variable( tf.random_uniform([50000, 64], -1.0, 1.0)) embed = tf.nn.embedding_lookup(embeddings, train_dataset) …

Read More
tf.shape() get wrong shape in tensorflow

I define a tensor like this: x = tf.get_variable("x", [100]) But when I try to print shape of tensor : print( tf.shape(x) ) I get Tensor("Shape:0", shape=(1,), dtype=int32), why the result of output should not be shape=(100)

Read More