Difference between Fasttext .vec and .bin file

python nlp deep-learning word2vec fasttext
Bhushan Pant picture Bhushan Pant · Nov 5, 2017 · Viewed 9.2k times · Source

I recently downloaded fasttext pretrained model for english. I got two files:

  1. wiki.en.vec
  2. wiki.en.bin

I am not sure what is the difference between the two files?

Answer

fnl picture fnl · Mar 23, 2018

The .vec files contain only the aggregated word vectors, in plain-text. The .bin files in addition contain the model parameters, and crucially, the vectors for all the n-grams.

So if you want to encode words you did not train with using those n-grams (FastText's famous "subword information"), you need to find an API that can handle FastText .bin files (most only support the .vec files, however...).

Related questions

How to return history of validation loss in Keras

Using Anaconda Python 2.7 Windows 10. I am training a language model using the Keras exmaple: print('Build model...') model = Sequential() model.add(GRU(512, return_sequences=True, input_shape=(maxlen, len(chars)))) model.add(Dropout(0.2)) model.add(GRU(512, return_sequences=False)) …

Read More
How to get rid of punctuation using NLTK tokenizer?

I'm just starting to use NLTK and I don't quite understand how to get a list of words from text. If I use nltk.word_tokenize(), I get a list of words and punctuation. I need only the words instead. …

Read More
How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?

I'm working in a sentiment analysis problem the data looks like this: label instances 5 1190 4 838 3 239 1 204 2 127 So my data is unbalanced since 1190 instances are labeled with 5. For the classification Im using scikit's SVC. The problem is I do not know how to …

Read More