Unsupervised automatic tagging algorithms?

Question

I want to build a web application that lets users upload documents, videos, images, music, and then give them an ability to search them. Think of it as Dropbox + Semantic Search.

When user uploads a new file, e.g. Document1.docx, how could I automatically generate tags based on the content of the file? In other words no user input is needed to determine what the file is about. If suppose that Document1.docx is a research paper on data mining, then when user searches for data mining, or research paper, or document1, that file should be returned in search results, since data mining and research paper will most likely be potential auto-generated tags for that given document.

1. Which algorithms would you recommend for this problem?

2. Is there an natural language library that could do this for me?

3. Which machine learning techniques should I look into to improve tagging precision?

4. How could I extend this to video and image automatic tagging?

Thanks in advance!

Answer 1

The most common unsupervised machine learning model for this type of task is Latent Dirichlet Allocation (LDA). This model automatically infers a collection of topics over a corpus of documents based on the words in those documents. Running LDA on your set of documents would assign words with probability to certain topics when you search for them, and then you could retrieve the documents with the highest probabilities to be relevant to that word.

There have been some extensions to images and music as well, see http://cseweb.ucsd.edu/~dhu/docs/research_exam09.pdf.

LDA has several efficient implementations in several languages:

• many implementations from the original researchers
• http://mallet.cs.umass.edu/, written in Java and recommended by others on SO
• PLDA: a fast, parallelized C++ implementation