TensorFlow: How to use CudnnLSTM with variable input length (like dynamic_rnn)?

tensorflow lstm cudnn
melgor89 picture melgor89 · Oct 27, 2016 · Viewed 8.9k times · Source

I would like to speed up my LSTM network, but as I am using it for a OCR (where sequences have variable lenght), I can not use plain LSTM implementation. That is why I use "tf.nn.dynamic_rnn".

Based on benchmark of RNN in tensorflow (https://github.com/tensorflow/tensorflow/blob/754048a0453a04a761e112ae5d99c149eb9910dd/tensorflow/contrib/cudnn_rnn/python/kernel_tests/cudnn_rnn_ops_benchmark.py#L77), the CUDNN implementation is used for creating all model at once (it does not use "tf.nn.rnn" structure like others). I assume that it maybe impossible to use CUDNN with variable length, but maybe anybody success it?

Second this is using "tf.nn.bidirectional_dynamic_rnn", as I would like to use Bi-LSTM for OCR. But this should be resolved after implementing the first part.

Edit: It looks like "tf.contrib.cudnn_rnn.CudnnLSTM" have "bidirectional" implementation inside. So the only unknown this is that CUDNN can be used with variable input sequence.

Or maybe any working example which use 'CudnnLSTM' would be helpfull.

Answer

Aidas Liaudanskas picture Aidas Liaudanskas · Aug 14, 2017

Just found this:

tf.contrib.cudnn_rnn.CudnnLSTM currently does not support batches with sequences of different length, thus this is normally not an option to use.

Source: http://returnn.readthedocs.io/en/latest/tf_lstm_benchmark.html

Related questions

Tensorflow - ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)

Continuation from previous question: Tensorflow - TypeError: 'int' object is not iterable My training data is a list of lists each comprised of 1000 floats. For example, x_train[0] = [0.0, 0.0, 0.1, 0.25, 0.5, ...] Here is my model: model = Sequential() model.add(LSTM(128, activation='relu', input_…

Read More
How to deal with batches with variable-length sequences in TensorFlow?

I was trying to use an RNN (specifically, LSTM) for sequence prediction. However, I ran into an issue with variable sequence lengths. For example, sent_1 = "I am flying to Dubain" sent_2 = "I was traveling from US to Dubai" I am …

Read More
How to stack multiple lstm in keras?

I am using deep learning library keras and trying to stack multiple LSTM with no luck. Below is my code model = Sequential() model.add(LSTM(100,input_shape =(time_steps,vector_size))) model.add(LSTM(100)) The above code returns error in …

Read More