How to auto-tag content, algorithms and suggestions needed

Question

I am working with some really large databases of newspaper articles, I have them in a MySQL database, and I can query them all.

I am now searching for ways to help me tag these articles with somewhat descriptive tags.

All these articles is accessible from a URL that looks like this:

http://web.site/CATEGORY/this-is-the-title-slug

So at least I can use the category to figure what type of content that we are working with. However, I also want to tag based on the article-text.

My initial approach was doing this:

1. Get all articles
2. Get all words, remove all punctuation, split by space, and count them by occurrence
3. Analyze them, and filter common non-descriptive words out like "them", "I", "this", "these", "their" etc.
4. When all the common words was filtered out, the only thing left is words that is tag-worthy.

But this turned out to be a rather manual task, and not a very pretty or helpful approach.

This also suffered from the problem of words or names that are split by space, for example if 1.000 articles contains the name "John Doe", and 1.000 articles contains the name of "John Hanson", I would only get the word "John" out of it, not his first name, and last name.

Answer 1

Automatically tagging articles is really a research problem and you can spend a lot of time re-inventing the wheel when others have already done much of the work. I'd advise using one of the existing natural language processing toolkits like NLTK.

To get started, I would suggest looking at implementing a proper Tokeniser (much better than splitting by whitespace), and then take a look at Chunking and Stemming algorithms.

You might also want to count frequencies for n-grams, i.e. a sequences of words, instead of individual words. This would take care of "words split by a space". Toolkits like NLTK have functions in-built for this.

Finally, as you iteratively improve your algorithm, you might want to train on a random subset of the database and then try how the algorithm tags the remaining set of articles to see how well it works.