sum over a list of tensors in tensorflow

Question

I have a deep neural network where the weights between layers are stored in a list.

layers[j].weights I want to incluse the ridge penalty in my cost function. I need then to use something like tf.nn.l2_loss(layers[j].weights**2 for j in range(self.n_layers)) i.e. the squared sum of all the weights.

In particular the weights are defined as:

>>> avs.layers
[<neural_network.Layer object at 0x10a4b2a90>, <neural_network.Layer object at 0x10ac85080>, <neural_network.Layer object at 0x10b0f3278>, <neural_network.Layer object at 0x10b0eacf8>, <neural_network.Layer object at 0x10b145588>, <neural_network.Layer object at 0x10b165048>, <neural_network.Layer object at 0x10b155ba8>]
>>>
>>> avs.layers[0].weights
<tensorflow.python.ops.variables.Variable object at 0x10b026748>
>>> 

How can I do that in tensorflow ?

Answer 1

The standard way to sum a list of tensors is to use the tf.add_n() operation, which takes a list of tensors (each having the same size and shape) and produces a single tensor containing the sum.

For the particular problem that you have, I am assuming that each layers[j].weights could have a different size. Therefore you will need reduce each element down to a scalar before summing, e.g. using the tf.nn.l2_loss() function itself:

weights = [layers[j].weights for j in range(self.n_layers)]
losses = [tf.nn.l2_loss(w) for w in weights]
total_loss = tf.add_n(losses)

(Note however that when the values to be added are large, you may find it more efficient to calculate a sequence of tf.add() operations, since TensorFlow keeps the values of each of the add_n arguments in memory until all of them have been computed. A chain of add ops allows some of the computation to happen earlier.)