How to use Bert for long text classification?

nlp text-classification bert-language-model
user1337896 picture user1337896 · Oct 31, 2019 · Viewed 15.7k times · Source

We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text How can BERT be used?

Answer

chefhose picture chefhose · Oct 31, 2019

You have basically three options:

  1. You cut the longer texts off and only use the first 512 Tokens. The original BERT implementation (and probably the others as well) truncates longer sequences automatically. For most cases, this option is sufficient.
  2. You can split your text in multiple subtexts, classifier each of them and combine the results back together ( choose the class which was predicted for most of the subtexts for example). This option is obviously more expensive.
  3. You can even feed the output token for each subtext (as in option 2) to another network (but you won't be able to fine-tune) as described in this discussion.

I would suggest to try option 1, and only if this is not good enough to consider the other options.

Related questions

Unable to train my keras model : (Data cardinality is ambiguous:)

I am using the bert-for-tf2 library to do a Multi-Class Classification problem. I created the model but training throws the following error: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-25-d9f382cba5d4> in <module>() …

Read More
How to rank features by their importance in a Weka classifier?

I use Weka to successfully build a classifier. I would now like to evaluate how effective or important my features are. Fot this I use AttributeSelection. But I don't know how to ouput the different features with their corresponding importance. …

Read More
Using Keras for text classification

I am struggling to approach the bag of words / vocabulary method for representing my input data as one hot vectors for my neural net model in keras. I would like to build a simple 3 layer network but I need help …

Read More