Why does word2Vec use cosine similarity?

nlp deep-learning word2vec
opus111 picture opus111 · Jul 17, 2016 · Viewed 7.4k times · Source

I have been reading the papers on Word2Vec (e.g. this one), and I think I understand training the vectors to maximize the probability of other words found in the same contexts.

However, I do not understand why cosine is the correct measure of word similarity. Cosine similarity says that two vectors point in the same direction, but they could have different magnitudes.

For example, cosine similarity makes sense comparing bag-of-words for documents. Two documents might be of different length, but have similar distributions of words.

Why not, say, Euclidean distance?

Can anyone one explain why cosine similarity works for word2Vec?

Answer

Aaron picture Aaron · Jul 17, 2016

Those two distance metrics are probably strongly correlated so it might not matter all that much which one you use. As you point out, cosine distance means we don't have to worry about the length of the vectors at all.

This paper indicates that there is a relationship between the frequency of the word and the length of the word2vec vector. http://arxiv.org/pdf/1508.02297v1.pdf

Related questions

CBOW v.s. skip-gram: why invert context and target words?

In this page, it is said that: [...] skip-gram inverts contexts and targets, and tries to predict each context word from its target word [...] However, looking at the training dataset it produces, the content of the X and Y pair seems …

Read More
Difference between Fasttext .vec and .bin file

I recently downloaded fasttext pretrained model for english. I got two files: wiki.en.vec wiki.en.bin I am not sure what is the difference between the two files?

Read More
Implement word2vec in Keras

I would like to implement word2vec algorithm in keras, Is this possible? How can I fit the model? Should I use custom loss function?

Read More