Keras - Input a 3 channel image into LSTM

python keras lstm recurrent-neural-network
shubhamsingh picture shubhamsingh · Dec 6, 2017 · Viewed 9.8k times · Source

I have read a sequence of images into a numpy array with shape (7338, 225, 1024, 3) where 7338 is the sample size, 225 are the time steps and 1024 (32x32) are flattened image pixels, in 3 channels (RGB).

I have a sequential model with an LSTM layer:

model = Sequential()
model.add(LSTM(128, input_shape=(225, 1024, 3))

But this results in the error:

Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4

The documentation mentions that the input tensor for LSTM layer should be a 3D tensor with shape (batch_size, timesteps, input_dim), but in my case my input_dim is 2D.

What is the suggested way to input a 3 channel image into an LSTM layer in Keras?

Answer

Daniel Möller picture Daniel Möller · Dec 6, 2017

If you want the number of images to be a sequence (like a movie with frames), you need to put pixels AND channels as features:

input_shape = (225,3072)  #a 3D input where the batch size 7338 wasn't informed

If you want more processing before throwing 3072 features into an LSTM, you can combine or interleave 2D convolutions and LSTMs for a more refined model (not necessarily better, though, each application has its particular behavior).

You can also try to use the new ConvLSTM2D, which will take the five dimensional input:

input_shape=(225,32,32,3) #a 5D input where the batch size 7338 wasn't informed

I'd probably create a convolutional net with several TimeDistributed(Conv2D(...)) and TimeDistributed(MaxPooling2D(...)) before adding a TimeDistributed(Flatten()) and finally the LSTM(). This will very probably improve both your image understanding and the performance of the LSTM.

Related questions

Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (339732, 29)

My input is simply a csv file with 339732 rows and two columns : the first being 29 feature values, i.e. X the second being a binary label value, i.e. Y I am trying to train my data on a stacked …

Read More
ValueError: Input 0 is incompatible with layer lstm_13: expected ndim=3, found ndim=4

I am trying for multi-class classification and here are the details of my training input and output: train_input.shape= (1, 95000, 360) (95000 length input array with each element being an array of 360 length) train_output.shape = (1, 95000, 22) (22 Classes are there) model = Sequential() model.…

Read More
How to get the output shape of a layer in Keras?

I have the following code in Keras (Basically I am modifying this code for my use) and I get this error: 'ValueError: Error when checking target: expected conv3d_3 to have 5 dimensions, but got array with shape (10, 4096)' Code: from …

Read More