Tensorflow: Word2vec CBOW model

python tensorflow word2vec
gladys0313 picture gladys0313 · May 23, 2016 · Viewed 12.4k times · Source

I am new to tensorflow and to word2vec. I just studied the word2vec_basic.py which trains the model using Skip-Gram algorithm. Now I want to train using CBOW algorithm. Is it true that this can be achieved if I simply reverse the train_inputs and train_labels?

Answer

Zichen Wang picture Zichen Wang · Jun 21, 2016

I think CBOW model can not simply be achieved by flipping the train_inputs and the train_labels in Skip-gram because CBOW model architecture uses the sum of the vectors of surrounding words as one single instance for the classifier to predict. E.g., you should use [the, brown] together to predict quick rather than using the to predict quick.

enter image description here

To implement CBOW, you'll have to write a new generate_batch generator function and sum up the vectors of surrounding words before applying logistic regression. I wrote an example you can refer to: https://github.com/wangz10/tensorflow-playground/blob/master/word2vec.py#L105

Related questions

TensorFlow 'module' object has no attribute 'global_variables_initializer'

I'm new to Tensorflow I'm running a Deep learning Assignment from Udacity on iPython notebook. link And it has an error. AttributeError Traceback (most recent call last) `<ipython-input-18-3446420b5935>` in `<module>`() 2 3 with tf.Session(graph=…

Read More
Merging layers on Keras (dot product)

I've been following Towards Data Science's tutorial about word2vec and skip-gram models, but I stumbled upon a problem that I cannot solve, despite searching about it for hours and trying a lot of unsuccessful solutions. https://towardsdatascience.com/understanding-feature-engineering-part-4…

Read More
Use of fasttext Pre-trained word vector as embedding in tensorflow script

Can I use fasttext word vector like the ones here: https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md in a tensorflow script as an embedding vector instead of word2vec or glove without using the library fasttext

Read More