Shuffling training data with LSTM RNN

machine-learning keras lstm recurrent-neural-network
hellowill89 picture hellowill89 · Jun 27, 2017 · Viewed 10k times · Source

Since an LSTM RNN uses previous events to predict current sequences, why do we shuffle the training data? Don't we lose the temporal ordering of the training data? How is it still effective at making predictions after being trained on shuffled training data?

Answer

Brian Bartoldson picture Brian Bartoldson · Jun 27, 2017

In general, when you shuffle the training data (a set of sequences), you shuffle the order in which sequences are fed to the RNN, you don't shuffle the ordering within individual sequences. This is fine to do when your network is stateless:

Stateless Case:

The network's memory only persists for the duration of a sequence. Training on sequence B before sequence A doesn't matter because the network's memory state does not persist across sequences.

On the other hand:

Stateful Case:

The network's memory persists across sequences. Here, you cannot blindly shuffle your data and expect optimal results. Sequence A should be fed to the network before sequence B because A comes before B, and we want the network to evaluate sequence B with memory of what was in sequence A.

Related questions

What's the difference between a bidirectional LSTM and an LSTM?

Can someone please explain this? I know bidirectional LSTMs have a forward and backward pass but what is the advantage of this over a unidirectional LSTM? What is each of them better suited for?

Read More
How to calculate the number of parameters of an LSTM network?

Is there a way to calculate the total number of parameters in a LSTM network. I have found a example but I'm unsure of how correct this is or If I have understood it correctly. For eg consider the following …

Read More
Using pre-trained word2vec with LSTM for word generation

LSTM/RNN can be used for text generation. This shows way to use pre-trained GloVe word embeddings for Keras model. How to use pre-trained Word2Vec word embeddings with Keras LSTM model? This post did help. How to predict / generate …

Read More