I have a word2vec model in gensim trained over 98892 documents. For any given sentence that is not present in the sentences array (i.e. the set over which I trained the model), I need to update the model with that sentence so that querying it next time gives out some results. I am doing it like this:
new_sentence = ['moscow', 'weather', 'cold']
model.train(new_sentence)
and its printing this as logs:
2014-03-01 16:46:58,061 : INFO : training model with 1 workers on 98892 vocabulary and 100 features
2014-03-01 16:46:58,211 : INFO : reached the end of input; waiting to finish 1 outstanding jobs
2014-03-01 16:46:58,235 : INFO : training on 10 words took 0.1s, 174 words/s
Now, when I query with similar new_sentence for most positives (as model.most_similar(positive=new_sentence)
) it gives out error:
Traceback (most recent call last):
File "<pyshell#220>", line 1, in <module>
model.most_similar(positive=['moscow', 'weather', 'cold'])
File "/Library/Python/2.7/site-packages/gensim/models/word2vec.py", line 405, in most_similar
raise KeyError("word '%s' not in vocabulary" % word)
KeyError: "word 'cold' not in vocabulary"
Which indicates that the word 'cold' is not part of the vocabulary over which i trained the thing (am I right)?
So the question is: How to update the model so that it gives out all the possible similarities for the given new sentence?
train()
expects a sequence of sentences on input, not one sentence.
train()
only updates weights for existing feature vectors based on existing vocabulary. You cannot add new vocabulary (=new feature vectors) using train()
.