Predicting a multiple forward time step of a time series using LSTM
I want to predict certain values that are weekly predictable (low SNR). I need to predict the whole time series of a year formed by the weeks of the year (52 values - Figure 1)
My first idea was to develop a many-to-many LSTM model (Figure 2) using Keras over TensorFlow. I'm training the model with a 52 input layer (the given time series of previous year) and 52 predicted output layer (the time series of next year). The shape of train_X is (X_examples, 52, 1), in other words, X_examples to train, 52 timesteps of 1 feature each. I understand that Keras will consider the 52 inputs as a time series of the same domain. The shape of the train_Y are the same (y_examples, 52, 1). I added a TimeDistributed layer. My thought was that the algorithm will predict the values as a time series instead of isolated values (am I correct?)
The model's code in Keras is:
y = y.reshape(y.shape[0], 52, 1)
X = X.reshape(X.shape[0], 52, 1)
# design network
model = Sequential()
model.add(LSTM(n_neurons, input_shape=(X.shape[1], X.shape[2]), return_sequences=True))
model.add(TimeDistributed(Dense(1)))
model.compile(loss='mean_squared_error', optimizer='adam')
# fit network
model.fit(X, y, epochs=n_epochs, batch_size=n_batch, verbose=2)
The problem is that the algorithm is not learning the example. It is predicting values very similar to the attributes' values. Am I modeling the problem correctly?
Second question: Another idea is to train the algorithm with 1 input and 1 output, but then during the test how will I predict the whole 2015 time series without looking to the '1 input'? The test data will have a different shape than the training data.
Answer
Sharing the same concerns about having too little data, you can do that like this.
First, it's a good idea to keep your values between -1 and +1, so I'd normalize them first.
For the LSTM model, you must make sure you're using return_sequences=True.
There is nothing "wrong" with your model, but it may need more or less layers or units to achieve what you desire. (There is no clear answer to this, though).
Training the model to predict the next step:
All you need is to pass Y as a shifted X:
entireData = arrayWithShape((samples,52,1))
X = entireData[:,:-1,:]
y = entireData[:,1:,:]
Train the model using these.
Predicting the future:
Now, for predicting the future, since we need to use predicted elements as input for more predicted elements, we are going to use a loop and make the model stateful=True.
Create a model equal to the previous one, with these changes:
- All LSTM layers must have
stateful=True - The batch input shape must be
(batch_size,None, 1)- This allows variable lengths
Copy the weights of the previously trained model:
newModel.set_weights(oldModel.get_weights())
Predict only one sample at a time and never forget to call model.reset_states() before starting any sequence.
First predict with the sequence you already know (this will make sure the model prepares its states properly for predicting the future)
model.reset_states()
predictions = model.predict(entireData)
By the way we trained, the last step in predictions will be the first future element:
futureElement = predictions[:,-1:,:]
futureElements = []
futureElements.append(futureElement)
Now we make a loop where this element is the input. (Because of stateful, the model will understand it's a new input step of the previous sequence instead of a new sequence)
for i in range(howManyPredictions):
futureElement = model.predict(futureElement)
futureElements.append(futureElement)
This link contains a complete example predicting the future of two features: https://github.com/danmoller/TestRepo/blob/master/TestBookLSTM.ipynb