How to get a Token from a Lucene TokenStream?

java attributes lucene token tokenize
Eric Wilson picture Eric Wilson · Apr 14, 2010 · Viewed 43.1k times · Source

I'm trying to use Apache Lucene for tokenizing, and I am baffled at the process to obtain Tokens from a TokenStream.

The worst part is that I'm looking at the comments in the JavaDocs that address my question.

http://lucene.apache.org/java/3_0_1/api/core/org/apache/lucene/analysis/TokenStream.html#incrementToken%28%29

Somehow, an AttributeSource is supposed to be used, rather than Tokens. I'm totally at a loss.

Can anyone explain how to get token-like information from a TokenStream?

Answer

Adam Paynter picture Adam Paynter · Apr 14, 2010

Yeah, it's a little convoluted (compared to the good ol' way), but this should do it:

TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.getAttribute(OffsetAttribute.class);
TermAttribute termAttribute = tokenStream.getAttribute(TermAttribute.class);

while (tokenStream.incrementToken()) {
    int startOffset = offsetAttribute.startOffset();
    int endOffset = offsetAttribute.endOffset();
    String term = termAttribute.term();
}

Edit: The new way

According to Donotello, TermAttribute has been deprecated in favor of CharTermAttribute. According to jpountz (and Lucene's documentation), addAttribute is more desirable than getAttribute.

TokenStream tokenStream = analyzer.tokenStream(fieldName, reader);
OffsetAttribute offsetAttribute = tokenStream.addAttribute(OffsetAttribute.class);
CharTermAttribute charTermAttribute = tokenStream.addAttribute(CharTermAttribute.class);

tokenStream.reset();
while (tokenStream.incrementToken()) {
    int startOffset = offsetAttribute.startOffset();
    int endOffset = offsetAttribute.endOffset();
    String term = charTermAttribute.toString();
}

Related questions

private final static attribute vs private final attribute

In Java, what's the difference between: private final static int NUMBER = 10; and private final int NUMBER = 10; Both are private and final, the difference is the static attribute. What's better? And why?

Read More
How do you tell if a checkbox is selected in Selenium for Java?

I am using Selenium in Java to test the checking of a checkbox in a webapp. Here's the code: private boolean isChecked; private WebElement e; I declare e and assign it to the area where the checkbox is. isChecked = e.…

Read More
How to loop over a Class attributes in Java?

How can I loop over a class attributes in java dynamically. For eg : public class MyClass{ private type1 att1; private type2 att2; ... public void function(){ for(var in MyClass.Attributes){ System.out.println(var.class); } } } is this possible in Java?

Read More