Request for example: Recurrent neural network for predicting next value in a sequence
Can anyone give me a practicale example of a recurrent neural network in (pybrain) python in order to predict the next value of a sequence ? (I've read the pybrain documentation and there is no clear example for it I think.) I also found this question. But I fail to see how it works in a more general case. So therefore I'm asking if anyone here could work out a clear example of how to predict the next value of a sequence in pybrain, with a recurrent neural network.
To give an example.
Say for example we have a sequence of numbers in the range [1,7].
First run (So first example): 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6
Second run (So second example): 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6
Third run (So third example): 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7
and so on.
Now given for example the start of a new sequence: 1 3 5 7 2 4 6 7 1 3
what is/are the next value(s)
This question might seem lazy, but I think there lacks a good and decent example of how to do this with pybrain.
Additionally: How can this be done if more than 1 feature is present:
Example:
Say for example we have several sequences (each sequence having 2 features) in the range [1,7].
First run (So first example): feature1: 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6
feature2: 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7
Second run (So second example): feature1: 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6
feature2: 1 2 3 7 2 3 4 6 2 3 5 6 7 2 4 7 1 3 3 5 6
Third run (So third example): feature1: 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7
feature2: 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6
and so on.
Now given for example the start of a new sequences:
feature 1: 1 3 5 7 2 4 6 7 1 3
feature 2: 1 2 3 7 2 3 4 6 2 4
what is/are the next value(s)
Feel free to use your own example as long it is similar to these examples and has some in depth explanation.
Answer
Issam Laradji's worked for me to predict sequence of sequences, except my version of pybrain required a tuple for the UnserpervisedDataSet object:
from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet
from pybrain.structure import LinearLayer
ds = SupervisedDataSet(21, 21)
ds.addSample(map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6'.split()),map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()))
ds.addSample(map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()),map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))
net = buildNetwork(21, 20, 21, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)
ts = UnsupervisedDataSet(21,)
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))
[ int(round(i)) for i in net.activateOnDataset(ts)[0]]
gives:
=> [1, 2, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6, 7, 1, 4, 6, 1, 2, 2, 3, 6]
To predict smaller sequences, just train it up as such, either as sub sequences or as overlapping sequences (overlapping shown here):
from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet
from pybrain.structure import LinearLayer
ds = SupervisedDataSet(10, 11)
z = map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split())
obsLen = 10
predLen = 11
for i in xrange(len(z)):
if i+(obsLen-1)+predLen < len(z):
ds.addSample([z[d] for d in range(i,i+obsLen)],[z[d] for d in range(i+1,i+1+predLen)])
net = buildNetwork(10, 20, 11, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)
ts = UnsupervisedDataSet(10,)
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3'.split()))
[ int(round(i)) for i in net.activateOnDataset(ts)[0]]
gives:
=> [3, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6]
Not too good...