Does an algorithm exist to help detect the "primary topic" of an English sentence?

algorithm nlp semantics lexical-analysis
rockit picture rockit · Apr 4, 2011 · Viewed 17.3k times · Source

I'm trying to find out if there is a known algorithm that can detect the "key concept" of a sentence.

The use case is as follows:

  1. User enters a sentence as a query (Does chicken taste like turkey?)
  2. Our system identifies the concepts of the sentence (chicken, turkey)
  3. And it runs a search of our corpus content

The area that we're lacking in is identifying what the core "topic" of the sentence is really about. The sentence "Does chicken taste like turkey" has a primary topic of "chicken", because the user is asking about the taste of chicken. While "turkey" is a helper topic of less importance.

So... I'm trying to find out if there is an algorithm that will help me identify the primary topic of a sentence... Let me know if you are aware of any!!!

Answer

Michael picture Michael · May 3, 2011

I actually did a research project on this and won two competitions and am competing in nationals.

There are two steps to the method:

  1. Parse the sentence with a Context-Free Grammar
  2. In the resulting parse trees, find all nouns which are only subordinate to Noun-Phrase-like constituents

For example, "I ate pie" has 2 nouns: "I" and "pie". Looking at the parse tree, "pie" is inside of a Verb Phrase, so it cannot be a subject. "I", however, is only inside of NP-like constituents. being the only subject candidate, it is the subject. Find an early copy of this program on http://www.candlemind.com. Note that the vocabulary is limited to basic singular words, and there are no verb conjugations, so it has "man" but not "men", has "eat" but not "ate." Also, the CFG I used was hand-made an limited. I will be updating this program shortly.

Anyway, there are limitations to this program. My mentor pointed out in its currents state, it cannot recognize sentences with subjects that are "real" NPs (what grammar actually calls NPs). For example, "that the moon is flat is not a debate any longer." The subject is actually "that the moon is flat." However, the program would recognize "moon" as the subject. I will be fixing this shortly.

Anyway, this is good enough for most sentences...

My research paper can be found there too. Go to page 11 of it to read the methods.

Hope this helps.

Related questions

Is there an algorithm that tells the semantic similarity of two phrases

input: phrase 1, phrase 2 output: semantic similarity value (between 0 and 1), or the probability these two phrases are talking about the same thing

Read More
Measuring semantic similarity between two phrases

I want to measure semantic similarity between two phrases/sentences. Is there any framework that I can use directly and reliably? I have already checked out this question, but its pretty old and I couldn't find real helpful answer there. …

Read More
How does the Google "Did you mean?" Algorithm work?

I've been developing an internal website for a portfolio management tool. There is a lot of text data, company names etc. I've been really impressed with some search engines ability to very quickly respond to queries with "Did you mean: …

Read More