Python Gensim: how to calculate document similarity using the LDA model?

python nlp lda gensim
still_st picture still_st · Mar 16, 2014 · Viewed 23.6k times · Source

I've got a trained LDA model and I want to calculate the similarity score between two documents from the corpus I trained my model on. After studying all the Gensim tutorials and functions, I still can't get my head around it. Can somebody give me a hint? Thanks!

Answer

Radim picture Radim · Mar 31, 2014

Depends what similarity metric you want to use.

Cosine similarity is universally useful & built-in:

sim = gensim.matutils.cossim(vec_lda1, vec_lda2)

Hellinger distance is useful for similarity between probability distributions (such as LDA topics):

import numpy as np
dense1 = gensim.matutils.sparse2full(lda_vec1, lda.num_topics)
dense2 = gensim.matutils.sparse2full(lda_vec2, lda.num_topics)
sim = np.sqrt(0.5 * ((np.sqrt(dense1) - np.sqrt(dense2))**2).sum())

Related questions

How to print the LDA topics models from gensim? Python

Using gensim I was able to extract topics from a set of documents in LSA but how do I access the topics generated from the LDA models? When printing the lda.print_topics(10) the code gave the following error because …

Read More
LDA model generates different topics everytime i train on the same corpus

I am using python gensim to train an Latent Dirichlet Allocation (LDA) model from a small corpus of 231 sentences. However, each time i repeat the process, it generates different topics. Why does the same LDA parameters and corpus generate different …

Read More
How to predict the topic of a new query using a trained LDA model using gensim?

I have trained a corpus for LDA topic modelling using gensim. Going through the tutorial on the gensim website (this is not the whole code): question = 'Changelog generation from Github issues?'; temp = question.lower() for i in range(len(…

Read More