Python langdetect: choose between one language or the other only

python nlp language-detection
vandernath picture vandernath · May 15, 2016 · Viewed 8.8k times · Source

I'm using langdetect to determine the language of a set of strings which I know are either in English or French.

Sometimes, langdetect tells me the language is Romanian for a string I know is in French.

How can I make langdetect choose between English or French only, and not all other languages?

Thanks!

Answer

Philip Bergström picture Philip Bergström · Aug 24, 2018

Option 1

One option would be using the package langid instead. Then you can simply restrict the languages with a method call:

import langid
langid.set_languages(['fr', 'en'])  # ISO 639-1 codes
lang, score = langid.classify('This is a french or english text')
print(lang) # en

Option 2

If you really want to use the langdetect package, you can copy the package folder (if you're not sure where it is, use python -m site --user-site) and remove the profiles you don't need from the folder langdetect\profiles.

This is not a very dynamic solution though.

Related questions

How to get rid of punctuation using NLTK tokenizer?

I'm just starting to use NLTK and I don't quite understand how to get a list of words from text. If I use nltk.word_tokenize(), I get a list of words and punctuation. I need only the words instead. …

Read More
How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?

I'm working in a sentiment analysis problem the data looks like this: label instances 5 1190 4 838 3 239 1 204 2 127 So my data is unbalanced since 1190 instances are labeled with 5. For the classification Im using scikit's SVC. The problem is I do not know how to …

Read More
Stopword removal with NLTK

I am trying to process a user entered text by removing stopwords using nltk toolkit, but with stopword-removal the words like 'and', 'or', 'not' gets removed. I want these words to be present after stopword removal process as they are …

Read More