I'm trying to develop an Encoder model in keras for timeseries. The shape of data is (5039, 28, 1), meaning that my seq_len is 28 and I have one feature. For the first layer of the encoder, I'm using 112 hunits, second layer will have 56 and to be able to get back to the input shape for decoder, I had to add 3rd layer with 28 hunits (this autoencoder is supposed to reconstruct its input). But I don't know what is the correct approach to connect the LSTM layers together. AFAIK, I can either add RepeatVector
or return_seq=True
. You can see both of my models in the following code. I wonder what will be the difference and which approach is the correct one?
First model using return_sequence=True
:
inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112, return_sequences=True)(inputEncoder)
snd = LSTM(56, return_sequences=True)(firstEncLayer)
outEncoder = LSTM(28)(snd)
context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28,1))(context)
encoder_model = Model(inputEncoder, outEncoder)
firstDecoder = LSTM(112, return_sequences=True)(context_reshaped)
outDecoder = LSTM(1, return_sequences=True)(firstDecoder)
autoencoder = Model(inputEncoder, outDecoder)
Second model with RepeatVector
:
inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112)(inputEncoder)
firstEncLayer = RepeatVector(1)(firstEncLayer)
snd = LSTM(56)(firstEncLayer)
snd = RepeatVector(1)(snd)
outEncoder = LSTM(28)(snd)
encoder_model = Model(inputEncoder, outEncoder)
context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28, 1))(context)
firstDecoder = LSTM(112)(context_reshaped)
firstDecoder = RepeatVector(1)(firstDecoder)
sndDecoder = LSTM(28)(firstDecoder)
outDecoder = RepeatVector(1)(sndDecoder)
outDecoder = Reshape((28, 1))(outDecoder)
autoencoder = Model(inputEncoder, outDecoder)
You will probably have to see for yourself which one is better because it depends on the problem you're solving. However, I'm giving you the difference between the two approaches.
Essentially, return_sequences=True
returns all the outputs the encoder observed in the past, while RepeatVector
repeats the very last output of the encoder.