How to represent text for classification in weka?

java machine-learning classification weka arff
Warren picture Warren · Nov 29, 2011 · Viewed 7.4k times · Source

Can you please let me know how to represent attribute or class for text classification in weka. By using what attribute can I do classification? word frequency or just word? What would be possible structure of ARFF format? Can you give me several lines of example of that structure?

Thank you very much in advance.

Answer

zdepablo picture zdepablo · Nov 30, 2011

One of the easiest alternatives is to start with an ARFF file for a two class problem like: 

@relation corpus 

@attribute text string
@attribute class {pos,neg}

@data
'long text with words ... ',pos

The text is represented as a String type and the class is a nominal with two values.

Then you could apply two filters:

  1. StringToWordVector that transforms the texts into a word vector representation. The filter uses an attribute for each word. You can tweak parameters to choose binary/frequency representation, stemming or stopwords. The best representation depends on the problem. If text are not long, usually binary representation is enough.
  2. Reorder to move the class atribute to the last position, Weka assumes it is there.

You may find more info and other approaches to transform your data in this Weka wiki page: http://weka.wikispaces.com/Text+categorization+with+WEKA

Related questions

Loding a weka model to a java code

I have saved the result of weka classification by right clicking on the model and selecting "save model". Now, I want to load it and work with in my "JAVA" application. How can I do that? Models could be Naive …

Read More
weka.core.UnassignedDatasetException when creating an unlabeled instance

I trained an IBK classifier with some training data that I created manually as following: ArrayList<Attribute> atts = new ArrayList<Attribute>(); ArrayList<String> classVal = new ArrayList<String>(); classVal.add("C1"); classVal.add("…

Read More
Best learning algorithm to make a decision tree in java?

I have a datasets with information like age, city, age of children, ... and a result (confirm, accept). To help modelisation of "workflow", I want to create automatically a decision tree based on previous datasets. I have take a look at …

Read More